WO2023185835A1 - 制冷电器中的多相机视觉系统 - Google Patents

制冷电器中的多相机视觉系统 Download PDF

Info

Publication number
WO2023185835A1
WO2023185835A1 PCT/CN2023/084379 CN2023084379W WO2023185835A1 WO 2023185835 A1 WO2023185835 A1 WO 2023185835A1 CN 2023084379 W CN2023084379 W CN 2023084379W WO 2023185835 A1 WO2023185835 A1 WO 2023185835A1
Authority
WO
WIPO (PCT)
Prior art keywords
cameras
controller
refrigeration
data
different
Prior art date
Application number
PCT/CN2023/084379
Other languages
English (en)
French (fr)
Inventor
古德曼 施罗德迈克尔
Original Assignee
海尔智家股份有限公司
青岛海尔电冰箱有限公司
海尔美国电器解决方案有限公司
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by 海尔智家股份有限公司, 青岛海尔电冰箱有限公司, 海尔美国电器解决方案有限公司 filed Critical 海尔智家股份有限公司
Publication of WO2023185835A1 publication Critical patent/WO2023185835A1/zh

Links

Classifications

    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N7/00—Television systems
    • H04N7/18—Closed-circuit television [CCTV] systems, i.e. systems in which the video signal is not broadcast
    • H04N7/181—Closed-circuit television [CCTV] systems, i.e. systems in which the video signal is not broadcast for receiving images from a plurality of remote sources
    • F—MECHANICAL ENGINEERING; LIGHTING; HEATING; WEAPONS; BLASTING
    • F25—REFRIGERATION OR COOLING; COMBINED HEATING AND REFRIGERATION SYSTEMS; HEAT PUMP SYSTEMS; MANUFACTURE OR STORAGE OF ICE; LIQUEFACTION SOLIDIFICATION OF GASES
    • F25D—REFRIGERATORS; COLD ROOMS; ICE-BOXES; COOLING OR FREEZING APPARATUS NOT OTHERWISE PROVIDED FOR
    • F25D29/00—Arrangement or mounting of control or safety devices
    • F—MECHANICAL ENGINEERING; LIGHTING; HEATING; WEAPONS; BLASTING
    • F25—REFRIGERATION OR COOLING; COMBINED HEATING AND REFRIGERATION SYSTEMS; HEAT PUMP SYSTEMS; MANUFACTURE OR STORAGE OF ICE; LIQUEFACTION SOLIDIFICATION OF GASES
    • F25D—REFRIGERATORS; COLD ROOMS; ICE-BOXES; COOLING OR FREEZING APPARATUS NOT OTHERWISE PROVIDED FOR
    • F25D29/00—Arrangement or mounting of control or safety devices
    • F25D29/005—Mounting of control devices
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V20/00—Scenes; Scene-specific elements
    • G06V20/60—Type of objects
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
    • H04N23/90—Arrangement of cameras or camera modules, e.g. multiple cameras in TV studios or sports stadiums
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N5/00—Details of television systems
    • H04N5/222—Studio circuitry; Studio devices; Studio equipment
    • H04N5/262—Studio circuits, e.g. for mixing, switching-over, change of character of image, other special effects ; Cameras specially adapted for the electronic generation of special effects
    • H04N5/265—Mixing
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N7/00—Television systems
    • H04N7/18—Closed-circuit television [CCTV] systems, i.e. systems in which the video signal is not broadcast
    • H04N7/183—Closed-circuit television [CCTV] systems, i.e. systems in which the video signal is not broadcast for receiving images from a single remote source
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N7/00—Television systems
    • H04N7/18—Closed-circuit television [CCTV] systems, i.e. systems in which the video signal is not broadcast
    • H04N7/188—Capturing isolated or intermittent images triggered by the occurrence of a predetermined event, e.g. an object reaching a predetermined position
    • F—MECHANICAL ENGINEERING; LIGHTING; HEATING; WEAPONS; BLASTING
    • F25—REFRIGERATION OR COOLING; COMBINED HEATING AND REFRIGERATION SYSTEMS; HEAT PUMP SYSTEMS; MANUFACTURE OR STORAGE OF ICE; LIQUEFACTION SOLIDIFICATION OF GASES
    • F25D—REFRIGERATORS; COLD ROOMS; ICE-BOXES; COOLING OR FREEZING APPARATUS NOT OTHERWISE PROVIDED FOR
    • F25D2323/00—General constructional features not provided for in other groups of this subclass
    • F25D2323/02—Details of doors or covers not otherwise covered
    • F25D2323/021—French doors
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
    • G06V10/82—Arrangements for image or video recognition or understanding using pattern recognition or machine learning using neural networks
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V20/00—Scenes; Scene-specific elements
    • G06V20/60—Type of objects
    • G06V20/68—Food, e.g. fruit or vegetables

Definitions

  • the present invention relates generally to refrigeration appliances, and more particularly to a multi-camera vision system in a refrigeration appliance and a method of operating a multi-camera vision system.
  • Refrigeration appliances typically include a cabinet defining a refrigeration compartment for receiving food for storage. Additionally, the refrigeration appliance includes one or more doors that are rotatably hinged to the cabinet to allow selective access to food stored in the refrigeration compartment. Refrigeration appliances may also include various storage components installed within the refrigeration compartment and designed to facilitate storage of food items therein. Such storage components may include shelves, boxes, shelves, or drawers that receive food items within the refrigerated compartment and assist in organizing and arranging such food items.
  • a refrigeration appliance it is often desirable to have an updated inventory of items present within a refrigeration appliance, for example to facilitate reordering, ensure food freshness or avoid spoilage, etc.
  • Some conventional refrigeration appliances have systems for monitoring food in the refrigeration appliance. However, such systems typically require interaction with the user, for example via direct input via a control panel regarding added or removed food products.
  • some appliances include cameras for monitoring food as it is added to or removed from the refrigeration appliance.
  • traditional camera systems can have difficulty identifying specific objects, distinguishing between similar products, and accurately identifying the location of objects within a refrigeration chamber.
  • traditional camera systems including a single or limited number of cameras may have difficulty performing such tasks.
  • refrigeration appliance with a system for improved inventory management. More particularly, it would be particularly beneficial to include refrigeration appliances with an inventory management system having a multi-camera system capable of monitoring entry and exit of the inventory as well as the placement of objects within the refrigeration compartment.
  • a refrigeration appliance may include a box defining a refrigeration compartment.
  • the refrigeration appliance may also include a door rotatably hinged to the cabinet to provide selective access to the refrigeration compartment.
  • the refrigeration appliance may also include a camera assembly that can to be coupled to the cabinet and operable to monitor the refrigeration compartment.
  • a camera assembly may include multiple cameras that may be coupled to multiple cables. Multiple cameras are operable to simultaneously capture data associated with the refrigeration compartment. Each of the plurality of cameras may be coupled to a cable of the plurality of cables.
  • the camera assembly may also include a multiplexer device that may be coupled to multiple cables.
  • the multiplexer device is operable to multiplex different data signals simultaneously provided to the multiplexer device by a plurality of cameras via a plurality of cables, and output the multiplexed signal having the different data signals.
  • the different data signals may include data associated with the refrigeration compartment.
  • the camera assembly may also include a controller coupled to the multiplexer device. The controller may be configured to perform one or more operations based at least in part on receipt of the multiplexed signal.
  • a method of implementing inventory management within a refrigeration appliance may include a refrigeration compartment and a camera assembly having a plurality of cameras arranged to monitor the refrigeration compartment.
  • the method may include obtaining, by a controller operatively coupled to the camera assembly, a multiplexed signal from a multiplexer device coupled to the controller.
  • the multiplexed signal may include different data signals provided simultaneously to the multiplexer device by multiple cameras via multiple cables coupled to the multiplexer device and the multiple cameras.
  • the different data signals may include data associated with the refrigeration compartment.
  • the method may also include performing, by the controller, one or more operations based at least in part on receiving the multiplexed signal from the multiplexer device.
  • a refrigeration appliance in another exemplary embodiment, may include a box defining a refrigeration compartment.
  • the refrigeration appliance may also include a door rotatably hinged to the cabinet to provide selective access to the refrigeration compartment.
  • the refrigeration appliance may also include a camera assembly that may be coupled to the cabinet and operable to monitor the refrigeration compartment.
  • the camera assembly may include a first multiplexer device coupled to the first pair of cameras and the first cable.
  • the first multiplexer device is operable to output the first multiplexed signal onto the first cable.
  • the first multiplexed signal may comprise different first data signals provided simultaneously by the first pair of cameras to the first multiplexer device.
  • the camera assembly may also include a second multiplexer device coupled to the second pair of cameras and the second cable.
  • the second multiplexer device is operable to output the second multiplexed signal onto the second cable.
  • the second multiplexed signal may comprise a different second data signal provided simultaneously by the second pair of cameras to the second multiplexer device.
  • the camera assembly may also include a demultiplexer device coupled to the first cable and the second cable.
  • the demultiplexer means is operable to demultiplex the first multiplexed signal into a different first data signal and to demultiplex the second multiplexed signal into a different second data signal.
  • the camera assembly may also include a controller coupled to the demultiplexer device. The controller may be configured to perform one or more operations based at least in part on receiving at least one of the different first data signal or the different second data signal.
  • Figure 1 illustrates a perspective view of an exemplary non-limiting refrigeration appliance in accordance with one or more exemplary embodiments of the present invention.
  • FIG. 2 illustrates a perspective view of the exemplary refrigeration appliance of FIG. 1 with a door shown in an open position to reveal an exemplary non-limiting inventory management system in accordance with one or more exemplary embodiments of the present invention.
  • FIG. 3 illustrates a flowchart of an exemplary non-limiting method for operating the exemplary inventory management system of FIG. 2 in accordance with one or more exemplary embodiments of the present invention.
  • FIG. 4 illustrates a first image obtained using a camera of the exemplary inventory management system of FIG. 2 in accordance with one or more exemplary embodiments of the invention.
  • FIG. 5 illustrates a second image obtained using the camera of the exemplary inventory management system of FIG. 2 in accordance with one or more exemplary embodiments of the present invention.
  • FIG. 6 illustrates a diagram of an exemplary non-limiting image comparison and object recognition process using the exemplary inventory management system of FIG. 2 in accordance with one or more exemplary embodiments of the present invention.
  • FIG. 7 illustrates a diagram of an exemplary non-limiting object motion tracking process using the exemplary inventory management system of FIG. 2 in accordance with one or more exemplary embodiments of the present invention.
  • FIG. 8 illustrates a perspective view of the exemplary refrigeration appliance of FIG. 1 including an exemplary non-limiting inventory management system with multiple cameras in accordance with one or more exemplary embodiments of the present invention.
  • Figures 9, 10, and 11 each illustrate a block diagram of the exemplary inventory management system of Figures 2 and/or 8 in accordance with one or more exemplary embodiments of the present invention.
  • FIGS. 2, 8, and/or 9 illustrates a flowchart of an exemplary non-limiting method of operating the exemplary inventory management system of FIGS. 2, 8, and/or 9 in accordance with one or more exemplary embodiments of the present invention.
  • entity refers to a person, user, end-user, consumer, computing device and/or program (e.g., processor, computing hardware and/or software, application, etc.), agent, machine learning (ML) and/or artificial intelligence (AI) algorithms, models, systems and/or applications, and/or may implement and/or facilitate as described herein, exemplified in the drawings and/or in the appended claims
  • entity refers to a person, user, end-user, consumer, computing device and/or program (e.g., processor, computing hardware and/or software, application, etc.), agent, machine learning (ML) and/or artificial intelligence (AI) algorithms, models, systems and/or applications, and/or may implement and/or facilitate as described herein, exemplified in the drawings and/or in the appended claims
  • entity refers to a person, user, end-user, consumer, computing device and/or program (e.g., processor, computing hardware and/or software, application, etc.), agent, machine learning (ML) and/or artificial intelligence (AI
  • Coupled refers to chemical coupling (e.g., chemical bonding), communication coupling, electrical and/or electromagnetic coupling (e.g., capacitive coupling, inductive coupling, direct and/or connection coupling, etc.), mechanical coupling, Operable connection, optical connection and/or physical connection.
  • chemical coupling e.g., chemical bonding
  • communication coupling e.g., electrical and/or electromagnetic coupling
  • electrical and/or electromagnetic coupling e.g., capacitive coupling, inductive coupling, direct and/or connection coupling, etc.
  • mechanical coupling Operable connection, optical connection and/or physical connection.
  • upstream and downstream refer to relative directions with respect to fluid flow in a fluid pathway.
  • upstream refers to the direction the fluid flow is coming from
  • downstream refers to the direction the fluid flow is going.
  • the terms “includes” and “including” are intended to be inclusive in a manner similar to the term “comprising.”
  • the terms “or” and “and/or” are generally intended to be inclusive, that is, “A or B” or “A and/or B” are each intended to mean “A or B or both.”
  • first,” “second,” “third,” etc. may be used interchangeably to distinguish one component or entity from another component or entity, and these terms are not intended to mean that each The location, function, or importance of a part or entity.
  • Approximate language is intended to modify any quantitative representation that is susceptible to variation without resulting in a change in the underlying function to which it relates. Accordingly, values modified by terms such as “about,” “approximately,” and “approximately” are not limited to the precise values specified. In at least some cases, the approximate language may correspond to the precision of the instrument used to measure the value. For example, approximate language may refer to within a 10% margin of error.
  • Figure 1 illustrates a perspective view of an exemplary non-limiting refrigeration appliance 100 in accordance with one or more exemplary embodiments of the present invention.
  • the refrigeration appliance 100 generally defines a vertical direction V, a lateral direction L, and a lateral direction T, each of which are perpendicular to each other such that an orthogonal coordinate system is generally defined.
  • the refrigeration appliance 100 includes a case 102 that is generally used to house and/or support various components of the refrigeration appliance 100 and may also define one or more interior cavities of the refrigeration appliance 100 Room or room.
  • the terms "box,”"casing,” and the like are generally intended to refer to the outer frame or support structure for the refrigeration appliance 100, including, for example, any suitable number formed from any suitable material. , type and configuration of support structure, such as a system of elongated support members, multiple interconnected panels, or some combination thereof.
  • the box 102 does not necessarily need to be enclosed, but may simply include an open structure that supports the various components of the refrigeration appliance 100 . Rather, the case 102 may surround some or all of the interior of the case 102 . It should be understood that the box 102 may have any suitable size, shape, and configuration while remaining within the scope of the present invention.
  • the box 102 generally extends along the vertical direction V between the top 104 and the bottom 106, and along the lateral direction L between the first side 108 (for example, the left side when viewed from the front in FIG. 1) and the third side. It extends between two sides 110 (for example, the right side when viewed from the front in FIG. 1 ), and extends along the transverse direction T between the front side 112 and the rear side 114 .
  • terms such as “left,” “right,” “front,” “rear,” “top,” or “bottom” are used with reference to the user's perspective approaching the refrigeration appliance 100 .
  • Box 102 defines a refrigerated compartment for receiving food items for storage.
  • the box 102 defines a food preservation compartment 122 disposed at or adjacent the top 104 of the box 102 and a freezer compartment 124 disposed at or adjacent the bottom 106 of the box 102 .
  • the refrigeration appliance 100 is generally called a bottom-mounted refrigerator.
  • the benefits of the present invention are applicable to other types and styles of refrigeration appliances, such as overhead refrigeration appliances, side-by-side refrigeration appliances or single door refrigeration appliances.
  • aspects of the invention may be applied to other appliances as well. Accordingly, the descriptions set forth herein are for illustrative purposes only and are not intended to be limited in any respect to any particular appliance or configuration.
  • the refrigeration door 128 is rotatably hinged to the edge of the box 102 to selectively enter the food preservation compartment 122 .
  • a freezing door 130 is arranged below the refrigeration door 128 to selectively enter the freezing chamber 124 .
  • Freezer door 130 is coupled to a freezer drawer (not shown) slidably mounted within freezer compartment 124 .
  • the refrigeration door 128 forms a seal over the front opening 132 (Figs. 2 and 3) defined by the cabinet 102 (eg, extending in a plane defined by the vertical direction V and the lateral direction L).
  • a user may place items within the food preservation compartment 122 through the front opening 132 and may then close the refrigeration door 128 to facilitate climate control.
  • Refrigerator door 128 and freezer door 130 are shown in a closed configuration in FIG. 1 .
  • FIG. 1 Those skilled in the art will appreciate that other chamber and door configurations are possible and within the scope of the present invention.
  • FIG. 2 illustrates a perspective view of the refrigeration appliance 100 in accordance with one or more exemplary embodiments of the present invention, in which the refrigeration door 128 is shown in an open position to expose one or more components of the refrigeration appliance 100 and/or therein Object.
  • the storage component may include a box 134 and shelf 136. Each of these storage components is configured to receive one or more objects 182 (eg, food, beverages) and may assist in organizing such objects 182 .
  • the box 134 can be installed on the refrigeration door 128 or can be slid into the receiving space in the food preservation compartment 122 .
  • the storage components shown are for illustrative purposes only and that other storage components may be used and may have different sizes, shapes, and configurations.
  • Dispensing assembly 140 is typically used to dispense liquid water and/or ice. Although an exemplary dispensing assembly 140 is illustrated and described herein, it should be understood that various changes and modifications may be made to the dispensing assembly 140 while remaining within the scope of the invention.
  • the dispensing assembly 140 and its various components may be at least partially disposed within a dispenser recess 142 defined on one of the refrigeration doors 128 .
  • a dispenser recess 142 is defined on the front side 112 of the refrigeration appliance 100 so that a user can operate the dispensing assembly 140 without opening the refrigeration door 128 .
  • the dispenser recess 142 is provided at a predetermined height that is convenient for the user to take the ice and enables the user to take the ice without bending down.
  • dispenser recess 142 is disposed approximately at the level of the user's chest.
  • the dispensing assembly 140 includes an ice or water dispenser 144 that includes a drain 146 for draining ice from the dispensing assembly 140 .
  • An actuating mechanism 148 shown as a paddle, is mounted below the drain 146 to operate the ice or water dispenser 144 .
  • any suitable actuation mechanism may be used to operate ice dispenser 144 .
  • the ice or water dispenser 144 may include a sensor (eg, an ultrasonic sensor) or a button instead of a paddle.
  • the drain port 146 and the actuating mechanism 148 are external parts of the ice or water dispenser 144 and are mounted in the dispenser recess 142 .
  • refrigeration door 128 may define an ice bin compartment 150 (FIG. 2) that houses an ice maker and ice bin (not shown) configured to supply ice to the dispenser.
  • Container recess 142 may define an ice bin compartment 150 (FIG. 2) that houses an ice maker and ice bin (not shown) configured to supply ice to the dispenser.
  • Control panel 152 is provided to control operating modes.
  • the control panel 152 includes one or more selection inputs 154, such as knobs, buttons, touch screen interfaces, etc., such as a water dispensing button and an ice dispensing button, for selecting a desired operating mode, such as crushed ice or non-crushed ice.
  • input 154 may be used to specify a fill volume or a method of operating dispensing assembly 140 .
  • input 154 may be in communication with a processing device or controller 156 . Signals generated in controller 156 operate refrigeration appliance 100 and distribution assembly 140 in response to selector input 154 .
  • a display 158 such as an indicator light or screen, may be provided on the control panel 152. Display 158 may be in communication with controller 156 and may display information in response to signals from controller 156 .
  • processing device may refer to one or more microprocessors or semiconductor device and is not necessarily limited to a single component.
  • a processing device or controller eg, controller 156
  • a processing device or controller may be programmed to operate the refrigeration appliance 100 , the distribution assembly 140 , and one or more other components of the refrigeration appliance 100 .
  • a processing device or controller eg, controller 156) may include or be associated with one or more storage elements (eg, non-transitory storage media, non-transitory computer-readable storage media). In some embodiments, such storage elements include electrically erasable programmable read-only memory (EEPROM).
  • EEPROM electrically erasable programmable read-only memory
  • storage elements may store information that is accessible to a processing device or controller (eg, controller 156), including instructions that are executable by the processing device or controller.
  • the instructions may be software or any collection of instructions and/or data that, when executed by a processing device or controller (eg, controller 156 ), causes the processing device to perform operate.
  • external communication system 170 is used to allow interaction, data transfer, and other communications between refrigeration appliance 100 and one or more external devices.
  • the communication may be used to provide and receive various types of data, operating parameters, user information in various types of formats (e.g., data signals, media, images, video, audio, multiplexed or demultiplexed data signals). Instructions or notifications, performance characteristics, user preferences, or any other suitable information for improved performance of the refrigeration appliance 100 .
  • external communication system 170 may be used to communicate data or other information to enhance the performance of one or more external devices or appliances and/or to improve user interaction with such devices.
  • the external communication system 170 allows the controller 156 of the refrigeration appliance 100 to communicate with a separate device external to the refrigeration appliance 100, which is generally referred to herein as the external device 172. As described in greater detail below, these communications may be facilitated using wired or wireless connections, such as via network 174 .
  • external device 172 may be any suitable device separate from refrigeration appliance 100 that is configured to provide and/or receive communications, information, data, or commands to and/or from a user.
  • external device 172 may be, for example, a personal phone, smartphone, tablet, laptop or personal computer, wearable device, smart home system, or another mobile or remote device.
  • remote server 176 may communicate with refrigeration appliance 100 and/or external device 172 over network 174 .
  • remote server 176 may be a cloud-based server, thereby located at a remote location, such as in a separate state, country, or the like.
  • external device 172 may communicate with remote server 176 over network 174 (such as the Internet) to send and/or receive data or information, provide user input, receive user notifications or instructions, interact with or control refrigeration appliance 100 Refrigeration appliances, etc.
  • external device 172 and remote server 176 may communicate with refrigeration appliance 100 to communicate similar information.
  • remote server 176 may be configured to receive and analyze data generated by camera assembly 190 of refrigeration appliance 100 (Figs. 2 and Figure 3) Images, video, audio and/or other data obtained, for example to facilitate inventory analysis.
  • communications between refrigeration appliance 100, external device 172, remote server 176, and/or other user devices or appliances may be accomplished using any type of wired or wireless connection and using any suitable type of communication network, which is provided below.
  • external device 172 may communicate directly or indirectly with refrigeration appliance 100 through any suitable wired or wireless communication connection or interface (eg, network 174).
  • network 174 may include one or more of a local area network (LAN), a wide area network (WAN), a personal area network (PAN), the Internet, a cellular network, any other suitable short-range or long-range wireless network, and the like.
  • any suitable communication means or protocol may be used (such as via Radio, laser, infrared, Ethernet type devices and interfaces, etc.) to send communications. Additionally, such communications may use various communication protocols (e.g., Transmission Control Protocol/Internet Protocol (TCP/IP), Hypertext Transfer Protocol (HTTP), Simple Mail Transfer Protocol (SMTP), File Transfer Protocol (FTP), etc.) , encoding or format (e.g., Hypertext Markup Language (HTML), Extensible Markup Language (XML), etc.) and/or protection scheme (e.g., Virtual Private Network (VPN), Secure HTTP, Secure Shell (SSH), Secure Sockets layer (SSL, etc.).
  • TCP/IP Transmission Control Protocol/Internet Protocol
  • HTTP Hypertext Transfer Protocol
  • SMTP Simple Mail Transfer Protocol
  • FTP File Transfer Protocol
  • HTTP File Transfer Protocol
  • encoding or format e.g., Hypertext Markup Language (HTML), Extensible Markup Language (XML), etc.
  • protection scheme e.g., Virtual Private Network
  • an external communication system 170 in accordance with an exemplary embodiment of the present invention.
  • the exemplary functionality and configuration of the external communication system 170 provided herein are provided as examples only to facilitate describing aspects of the present invention.
  • System configurations may vary, other communication devices may be used to communicate directly or indirectly with one or more associated appliances, other communication protocols and procedures may be implemented, and the like. Such changes and modifications are considered to be within the scope of the invention.
  • the refrigeration appliance 100 may also include an inventory management system 180 that is typically configured to monitor one or more chambers of the refrigeration appliance 100 to monitor the addition and/or removal of inventory. More specifically, as described in greater detail below, the inventory management system 180 may include a plurality of sensors, cameras, or other detection devices for monitoring the food preservation compartment 122 and/or the freezer compartment 124 to detect items placed in the food preservation compartment. 122 and/or objects 182 (eg, food, beverages) in or removed from freezer compartment 124 . In this regard, the inventory management system 180 may use data from each of these devices to obtain a complete picture of the identity, location, and/or other qualitative or quantitative characteristics of the objects 182 within the food preservation compartment 122 and/or the freezer compartment 124 representation or knowledge. Although inventory management system 180 is described herein as monitoring food freshness compartment 122 to detect object 182, it should be understood that aspects of the invention may be used to monitor any other suitable appliance, chamber (eg, freezer 124) Objects or items in etc.
  • the inventory management system 180 may include a plurality of sensors
  • inventory management system 180 may include a camera assembly 190 coupled to refrigeration appliance 100 (eg, case 102 ), the camera assembly being generally configured and used to obtain images and/or images of refrigeration appliance 100 during operation. or video.
  • camera assembly 190 includes one or more Cameras 192, which are mounted to the box 102, the refrigeration door 128, or are otherwise configured to view the food freshness compartment 122.
  • the camera assembly 190 is described herein as being used to monitor the food preservation compartment 122 of the refrigeration appliance 100, it should be understood that aspects of the present invention may be used to monitor any other suitable area of any other suitable appliance, such as the freezer compartment 124. As best shown in FIG.
  • the camera 192 of the camera assembly 190 is mounted to the cabinet 102 at the front opening 132 of the food preservation compartment 122 and is oriented to have the camera 192 directed across the front opening 132 and/or into the food preservation chamber 122 .
  • the camera assembly 190 may include multiple cameras 192 disposed within the enclosure 102 and/or coupled (eg, mounted) to the enclosure, where the multiple cameras 192 Each of them has a designated monitoring area or monitoring range arranged around the food preservation chamber 122 .
  • the field of view of each camera 192 may be limited, directed, or focused on a specific monitoring zone, monitoring range, or specific area within the food preservation compartment 122 .
  • an inventory management system 180 having a plurality of cameras 192 is provided in accordance with one or more exemplary embodiments of the present invention. As shown, the cameras 192 can be mounted to the side walls of the food preservation chamber 122 and can be spaced along the vertical direction V to cover different monitoring areas.
  • camera assembly 190 may be used to facilitate an inventory management process of refrigeration appliance 100 . It can be seen that each camera 192 can be disposed at an opening of the food preservation chamber 122 to monitor objects 182 (eg, food, beverages) added to or taken out of the food preservation chamber 122 .
  • each camera 192 may be oriented in any other suitable manner for monitoring any other suitable area in or around the refrigeration appliance 100 .
  • the camera assembly 190 may include any suitable number, type, size and configuration of cameras 192 for obtaining images of any suitable zone or area within or around the refrigeration appliance 100, according to alternative embodiments.
  • each camera 192 may include features for adjusting its field of view and/or orientation.
  • the images and/or videos obtained by the camera assembly 190 may vary in quantity, frequency, angle, resolution, detail, etc., in order to enhance the clarity of specific areas around or within the refrigeration appliance 100 .
  • the controller 156 may be operable to illuminate the refrigeration chamber using one or more light sources prior to obtaining the image.
  • the controller 156 of the refrigeration appliance 100 (or any other suitable dedicated controller) may be communicatively coupled to the camera assembly 190 and may be programmed or used to analyze images obtained by the camera assembly 190, for example, to identify objects being Items added to or removed from the refrigeration appliance 100, as detailed below describe.
  • controller 156 may be coupled (e.g., electrically coupled, communicatively coupled, operatively coupled) to camera assembly 190 for analyzing one or more images and/or videos obtained by camera assembly 190 to extract information about food located Useful information about objects 182 in the freshness compartment 122.
  • images and/or videos obtained by camera assembly 190 may be used to extract barcodes, identify products, monitor product movement, or obtain other product information related to object 182 .
  • the analysis may be performed locally (eg, on controller 156) or may be sent to a remote server (eg, remote server 176 via external communications system 170) for analysis. This analysis is intended to facilitate inventory management, for example by identifying food items added to and/or removed from the refrigeration compartment.
  • Method 200 may be used to operate camera assembly 190, or any other suitable camera assembly for monitoring appliance operation or inventory.
  • controller 156 may be used to implement method 200. It should be understood, however, that the exemplary method 200 is discussed herein merely to describe exemplary aspects of the invention and is not intended to be limiting.
  • the method 200 includes: at step 210 , using a camera assembly to obtain a first image of a refrigeration compartment of a refrigeration appliance.
  • the camera assembly 190 of the refrigeration appliance 100 may obtain one or more images within the food preservation compartment 122, which images may include multiple objects 182 in its field of view.
  • the camera assembly 190 of the refrigeration appliance 100 may obtain one or more images of the food preservation compartment 122, the freezer compartment 124, or any other area or area in or around the refrigeration appliance 100 (e.g., in FIGS. 4 and 4 , respectively.
  • camera 192 is oriented downwardly from the top center of cabinet 102 and has a field of view covering the width of food preservation compartment 122 (eg, as shown in the photos of Figures 4 and 5). Furthermore, the field of view may be centered on the front opening 132 at the front of the box 102 , for example, where the refrigeration door 128 is positioned against the front of the box 102 . In this way, the field of view of the camera 192 and the resulting images may capture any motion or movement of objects entering and/or exiting the food preservation chamber 122 . Images obtained by camera assembly 190 may include one or more still images, one or more video clips, or any other suitable type and number of images suitable for identification of objects 182 (eg, food, beverages) or inventory analysis.
  • objects 182 eg, food, beverages
  • the camera assembly 190 may acquire images upon any suitable trigger, such as a time-based imaging schedule, in which the camera assembly 190 periodically images and monitors the food preservation chamber 122 .
  • the camera assembly 190 may periodically capture low-resolution images until motion is detected (eg, via image differentiation of the low-resolution images), at which time a or Multiple high resolution images.
  • the refrigeration appliance 100 may include one or more motion sensors (e.g., optical, acoustic, electromagnetic, etc.) that detect when an object 182 is added to or removed from the food preservation compartment 122 . A motion sensor is triggered, and the camera assembly 190 may be operatively coupled to such motion sensor to obtain images of the object 182 during such movement.
  • the refrigeration appliance 100 may include a door switch that detects when the refrigeration door 128 is opened, at which time the camera assembly 190 may begin to acquire one or more images.
  • images 300, 302 may be obtained continuously or periodically while refrigeration door 128 is open.
  • obtaining images 300, 302 may include determining that a door to the refrigeration appliance is open and capturing the image at a set frame rate while the door is open.
  • the movement of food between image frames may be used to determine whether object 182 is removed from or added to food preservation compartment 122 .
  • images obtained by camera assembly 190 may vary in quantity, frequency, angle, resolution, detail, etc., in order to enhance the clarity of object 182.
  • the controller 156 may be used to illuminate a refrigerator light (not shown) while the images 300, 302 are obtained. Other suitable triggers are possible and within the scope of the invention.
  • Step 220 may include analyzing the first image using a machine learning image recognition process to identify objects in the first image. It should be understood that this analysis may utilize any suitable image analysis technique, image decomposition, image segmentation, image processing, etc. This analysis may be performed entirely by controller 156, may be offloaded to a remote server for analysis, may be analyzed with user assistance (eg, via control panel 152), or may be analyzed in any other suitable manner. According to an exemplary embodiment of the present invention, the analysis performed at step 220 may include a machine learning image recognition process.
  • the image analysis may use any suitable image processing technique, image recognition process, or the like.
  • image analysis and the like may generally be used to refer to any suitable method of observation, analysis, image decomposition, feature extraction, image classification, etc. of one or more images, videos, or other visual representations of an object.
  • this image analysis may include the implementation of image processing techniques, image recognition techniques, or any suitable combination thereof.
  • image analysis may use any suitable image analysis software or algorithm to continuously or periodically monitor moving objects within the food preservation compartment 122 . It will be appreciated that this image analysis or processing may be performed locally (eg, by controller 156) or remotely (eg, by offloading the image data to a remote server or network, eg, remote server 176).
  • analysis of one or more images may include implementing image processing algorithms.
  • image processing and the like are generally intended to refer to any suitable method or algorithm for analyzing images that does not rely on artificial intelligence or machine learning techniques (e.g., in contrast to the machine learning image recognition process described below ).
  • image processing algorithms may rely on image differentiation, such as pixel-by-pixel comparison of two consecutive images. This comparison can help identify substantial differences between sequentially acquired images, for example, to identify movement, the presence of specific objects, the presence of specific conditions, etc.
  • image differentiation can be used to determine when pixel-level motion metrics pass a predetermined motion threshold.
  • the processing algorithm may also include measures for isolating or eliminating noise in the image comparison arising, for example, from image resolution, data transmission errors, inconsistent lighting, or other imaging errors. By removing this noise, image processing algorithms can improve accurate object detection, avoid false object detections, and isolate important objects, regions, or patterns within the image. Additionally or alternatively, the image processing algorithm may use other suitable techniques for identifying or identifying particular items or objects, such as edge matching, divide and conquer search, grayscale matching, histograms of receptive field responses, or another suitable example. process (e.g., executed at controller 156 based on one or more captured images from one or more cameras). Other image processing techniques are possible and within the scope of the invention.
  • image analysis may also include utilizing artificial intelligence (AI), such as machine learning image recognition processes, neural network classification modules, any other suitable artificial intelligence (AI) technology, and/or any other suitable image Analysis techniques, examples of which are described in more detail below.
  • AI artificial intelligence
  • each of the exemplary image analysis or evaluation processes described below may be used independently, jointly, or interchangeably to extract detailed information about the image being analyzed to facilitate the performance of one or more of the methods described herein or Improve appliance operation in other ways.
  • any suitable number of image processing, image recognition, or other image analysis techniques and combinations thereof may be used to obtain an accurate analysis of the acquired image.
  • the image recognition process may use any suitable artificial intelligence technique, for example, any suitable machine learning technique, or, for example, any suitable deep learning technique.
  • the image recognition process may include implementing a form of image recognition known as region-based convolutional neural network ("R-CNN") image recognition.
  • R-CNN may include taking an input image and extracting region proposals that include potential objects or regions of the image.
  • a "region suggestion" may be one or more regions in an image that may belong to a specific object, or may include adjacent regions that share common pixel characteristics.
  • a convolutional neural network is then used to compute features from the region proposals, and the extracted features are then used to determine the classification of each specific region.
  • the image segmentation process can be used with R-CNN image recognition.
  • image segmentation creates pixel-based masks for individual objects in an image and provides A more detailed or refined understanding of an object.
  • image segmentation can involve dividing the image into segments (e.g., into groups of pixels that contain similar properties) that Segments can be analyzed independently or in parallel to obtain a more detailed representation of one or more objects in the image. This may be referred to in this paper as "masked R-CNN" etc., as opposed to the regular R-CNN architecture.
  • Masked R-CNN can be based on Fast R-CNN which is slightly different from R-CNN.
  • R-CNN first applies a convolutional neural network (CNN) and then assigns it to region recommendations on the feature map instead of initially segmenting it into region recommendations.
  • CNN convolutional neural network
  • a standard CNN may be used to obtain, identify, or detect any other qualitative or quantitative data related to one or more objects or regions within one or more images.
  • a K-means algorithm may be used.
  • the image recognition process may use any other suitable neural network process while remaining within the scope of the present invention.
  • analyzing the one or more images may include using a deep belief network (DBN) image recognition process.
  • the DBN image recognition process can typically involve stacking many separate unsupervised networks that use the hidden layers of each network as input to the next layer.
  • the step of analyzing the one or more images may include implementing a deep neural network (DNN) image recognition process, which typically involves using a neural network with multiple layers between inputs and outputs (e.g., derived from biological neural networks). network-enabled and/or biological neural network-based computing systems).
  • DNN deep neural network
  • Other suitable image recognition processes, neural network processes, artificial intelligence analysis techniques, and combinations of the above or other known methods may be used while remaining within the scope of the present invention.
  • a neural network architecture such as VGG16, VGG19, or ResNet50 can be pretrained using public datasets, and then the final layer can be retrained using appliance-specific datasets.
  • the image recognition process may include detection of certain conditions based on comparison of initial conditions and/or may rely on image subtraction techniques, image stacking techniques, image stitching, etc. For example, subtracted images can be used to train neural networks with multiple classes for future comparisons and image classification.
  • the machine learning image recognition model can be actively trained by the appliance using new images, can be provided with training data from the manufacturer or from another remote source, or can be trained in any other suitable manner.
  • the image recognition process relies at least in part on a neural network trained with multiple images of appliances configured differently, experiencing different conditions, or interacting in different ways.
  • This training data can be stored locally or remotely, and can be transferred to a remote server for training other appliances and models.
  • image processing and machine learning image recognition processes can be used together to facilitate improved images Like analysis, object detection, or the extraction of other useful qualitative or quantitative data or information from one or more images that can be used to improve the operation or performance of the appliance. Indeed, the methods described herein may interchangeably use any or all of these techniques to improve the image analysis process and promote improved appliance performance and consumer satisfaction.
  • the image processing algorithms and machine learning image recognition processes described herein are exemplary only and are not intended to limit the scope of the invention in any way.
  • Step 230 may include obtaining a second image 302 using a camera assembly.
  • the second image 302 may be obtained immediately after the first image 300 is obtained in step 210 .
  • both the first image 300 and the second image 302 may be obtained while the object 182 is in the process of being inserted into or removed from the food preservation compartment 122 so that the trajectory of the object 182 may be determined, as described in greater detail below.
  • Step 240 may include analyzing the second image using a machine learning image recognition process to identify objects in the second image.
  • step 240 may include image analysis similar to that described above with respect to step 220.
  • the image analysis performed at step 220 may, for example, be based on training a machine learning model using similar objects 182 (eg, apples or oranges as exemplified herein) to identify elements within the first image 300 and the second image 302 . of multiple objects.
  • the machine learning image recognition process may provide a confidence score (eg, as generally identified by reference numeral 310 for each object 182 identified in Figures 4, 5, and 6).
  • confidence score 310 may generally represent the probability that an object has been appropriately recognized by the machine learning model.
  • the method 200 may further include obtaining a third image using the camera assembly 190 , wherein the third image also includes the object 182 from the first image 300 and the second image 302 .
  • Method 200 may also include analyzing the third image to identify the object in the third image; and increasing the confidence score to identify the object based at least in part on the analysis of the third image.
  • the confidence level may be increased, for example, as shown from the object recognition in Figures 4 and 5. Positive identification of the same orange in the third image further increases the confidence score. Conversely, negative identification of the same object 182 can be used to lower the confidence score.
  • the confidence score 310 may be an output from a machine learning model and may be based on any suitable characteristics of the object 182 being monitored or tracked.
  • each object 182 may have identifiable features, such as stems, discoloration, blemishes, or other features that may be identifiable and associated with that particular object 182 Characteristics (e.g., similar to the object's fingerprint).
  • Machine learning image recognition models can identify individual objects based on their specific fingerprints and can use identifiable features from other images to improve object recognition accuracy. Although this article describes this comparison of multiple images with respect to individual oranges or apples to improve the confidence score for object recognition, it should be understood that the model can be extrapolated to identify multiple objects using any suitable number of images. Either.
  • Step 250 may include determining a motion vector of the object based on the position of the object in the first image and the second image. Specifically, as best illustrated in Figure 7, a motion vector 320 of a first object 182 (eg, a first orange) is shown between the first image 300 and the second image 302. In this regard, if an object 182 (eg, an orange) is identified in both the first image 300 and the second image 302 , the method 200 may include determining a trajectory or motion vector 320 associated with movement of the object 182 . Furthermore, by positively identifying the motion vectors 320 of one or more objects 182 placed within the food preservation compartment 122, the confidence score 310 associated with the identification of a particular object 182 may be improved or increased.
  • a motion vector 320 of a first object 182 eg, a first orange
  • the method 200 may include determining a trajectory or motion vector 320 associated with movement of the object 182 .
  • the confidence score 310 associated with the identification of a particular object 182 may be improved or
  • method 200 may include analyzing first image 300 to identify a second object in first image 300 (eg, an apple positioned adjacent to an orange). Method 200 may also include determining a spatial relationship between first object 182 and second object 182 (eg, the relative placement of the two objects in three-dimensional space). Method 200 may also include determining a predicted motion vector for the second object based at least in part on motion vector 320 of first object 182 and a spatial relationship between the first object and the second object (e.g., as generally represented by reference numeral 322 logo).
  • the method 200 may include obtaining a plurality of images of the object 182 being added to or removed from the refrigeration chamber.
  • controller 156 or another suitable processing device may analyze these images to identify object 182 and/or its trajectory into or out of food preservation compartment 122 and/or freezer compartment 124 .
  • Controller 156 may monitor and track inventory within refrigeration appliance 100 by identifying whether objects 182 are added to or removed from food crisper 122 and/or freezer 124 .
  • the controller 156 may maintain a record of food items placed in or removed from the food preservation compartment 122 .
  • Figure 3 depicts an exemplary control method with steps performed in a specific order for purposes of example and discussion.
  • the steps of any of the methods described herein may be adapted, rearranged, expanded, omitted, or modified in various ways without departing from the scope of the invention.
  • aspects of these methods are illustrated using camera assembly 190 as an example, it should be understood that these methods may be applied to the operation of any suitable appliance and/or camera assembly.
  • the inventory management system 180 and method of operating a refrigeration appliance 200 as described above can generally help improve Inventory management within refrigeration appliances.
  • the system facilitates object recognition, where frame-by-frame object analysis methods can be used to support inventory management. This is advantageous when tracking multiple objects belonging to a single category (eg similar objects) stored in the refrigerator.
  • multiple images from a camera can be used to track items moving through its field of view, where objects are captured frame by frame. Consistency between frames of an object can be compared in a neural network. Neural networks can be designed to give the probability that two images belong to the same item. If multiple images of a single object are available, multiple comparisons can be made and the average confidence can be used.
  • Neural networks effectively generate feature vectors, or maps, for individual objects and compare them. High confidence vectors are given to objects that were positively identified between frames. The relative positions of unknown items can be used to identify them in the next step. If items move together, another item can be at a known location. If the item has not moved, it will be found in the same location. Either case can be used to identify items between contact frames.
  • the appliance-centric database may be built over the course of one or more interactions with the appliance (eg, many frames). Individual images of identical item identifications can be used for future comparisons, making tracking increasingly easier.
  • a method determines which item leaves the storage space upon retrieval, and also advises the user to remove the oldest item and show it in the image.
  • FIG. 9 illustrates a block diagram of an inventory management system 180 in accordance with one or more exemplary embodiments of the present invention.
  • the inventory management system 180 illustrated in FIG. 9 may include a camera assembly 190 .
  • inventory management system 180 and/or camera assembly 190 may include multiple cameras 192 , which may be coupled to multiplexer device 902 via multiple cables 904 .
  • each camera 192 may be coupled to the multiplexer device 902 via a single cable 904 , adapter 906 , and camera cable 908 .
  • multiplexer device 902 may be coupled to and/or integrated with a controller (eg, a microprocessor), such as controller 156 , which may constitute and/or include a single board computer ( SBC).
  • controller 156 may include, be coupled to, constitute and/or otherwise associated with an image signal processor (ISP), which may be operable and/or configured to One or more example implementations described herein process image data, video data, and/or audio data.
  • ISP image signal processor
  • individual cables 904 may constitute and/or include analog cables, digital cables, communication cables, communication cable, network cable, data cable, media cable, control cable, coaxial cable or another type of cable.
  • each cable 904 may be constructed and/or include a cable that may be used to communicate image data, video data, audio data, control data (eg, control signals), and/or other data between each camera 192 and the controller 156 .
  • Data cable e.g., coaxial cable).
  • each camera 192 may constitute and/or include a Mobile Industry Processor Interface (MIPI) camera (eg, MIPI camera module).
  • MIPI Mobile Industry Processor Interface
  • each camera cable 908 may constitute and/or include a MIPI camera cable.
  • each cable 904 may constitute and/or include a coaxial cable.
  • each adapter 906 may constitute and/or include a MIPI to coax adapter.
  • ISP image signal processor
  • SBC controller
  • ISP image signal processor
  • other types of cameras e.g., Universal Serial Bus (USB) cameras
  • cables For example, different combinations of USB cables
  • adapters For example, different numbers of Image Signal Processors (ISPs) and/or Single Board Computers (SBCs).
  • the camera 192 is operable to capture data associated with a refrigerated compartment (eg, the food preservation compartment 122 and/or the freezer compartment 124 ) simultaneously (eg, simultaneously, at about the same time) (eg, the food preservation compartment 122 and/or the freezer compartment 124 ). , image data, video data, audio data).
  • camera 192 may simultaneously capture images and/or video of one or more objects 182 placed within, added to, and/or removed from food crisper 122 and/or freezer 124 .
  • the inventory management system 180 and/or the camera Component 190 may use camera 192 to simultaneously capture images and/or videos of one or more objects 182 being added to or removed from food preservation compartment 122 .
  • the controller 156 may receive a signal indicating that the refrigeration door 128 and/or the freezer door 130 are open (eg, the controller 156 may receive such a signal from a motion sensor and/or a door sensor of the refrigeration appliance 100) .
  • controller 156 may operate (e.g., via inventory management system 180 , camera assembly 190 ) one or more of refrigerated door 128 and/or freezer door 130 when open. 192 cameras, to simultaneously capture such data associated with food preservation compartment 122 and/or freezer compartment 124, respectively (e.g., with one placed within, added to, and/or removed from food preservation compartment 122 and/or freezer compartment 124). or data associated with multiple objects 182).
  • multiplexer device 902 is operable to multiplex different data signals simultaneously provided to multiplexer device 902 by camera 192 via cable 904 .
  • the different data signals may include data associated with the food preservation compartment 122 and/or the freezer compartment 124 .
  • the different data signals may include images, such as images 300 , 302 and/or videos of one or more objects 182 being added to or removed from the food crisper 122 and/or the freezer 124 .
  • the multiplexer device 902 is further operable to output a multiplexed signal having different data signals associated with the food preservation compartment 122 and/or the freezer compartment 124 and the data described above.
  • multiplexer device 902 may be coupled to and/or integrated with a controller (eg, microprocessor, SBC) such as controller 156.
  • the controller 156 may receive a multiplexed signal that may be output by the multiplexer device 902 and may include a signal that may be simultaneously captured by the camera 192 as described above and provided to the multiplexer device 902 of different data signals.
  • the inventory management system 180 illustrated in the exemplary embodiment depicted in FIG. 9 may include a demultiplexer device (not shown in FIG. 9 ) that may be coupled to the multiplexer device 902 and the controller 156. Shows).
  • a demultiplexer device may be coupled to and/or integrated with the controller 156 and also coupled to the multiplexer device 902 such that the demultiplexer device may demultiplex Using the multiplexed signal output by the multiplexer device 902.
  • such a demultiplexer device may output and provide different data signals to the controller 156 when demultiplexing the multiplexed signal.
  • the controller 156 may perform one or more operations based at least in part on receiving the multiplexed signals and/or different data signals described above from the multiplexer device 902 .
  • controller 156 may locally analyze multiplexed signals, different data signals, and/or data therein (e.g., with food preservation chamber 122 and/or Freezer compartment 124 associated data, which may be simultaneously captured by camera 192).
  • the controller 156 may use one or more of the above-described machine learning and/or AI models, algorithms, and/or image recognition processes (e.g., CNN, R-CNN, DBN, DNN) to analyze the relationship between the food preservation chamber 122 and Image data (eg, data in images 300, 302) and/or video data is associated with freezer compartment 124 and is in the multiplexed signal.
  • the controller 156 may analyze such image and/or video data to monitor and/or maintain a or Records of multiple objects 182 (eg, food, beverages).
  • controller 156 may utilize external communication system 170 to communicate multiplexed signals, different data signals and/or data associated with food preservation compartment 122 and/or freezer compartment 124 to external devices 172 and/or via network 174 /or remote server 176.
  • the controller 156 may facilitate adjustments of the cameras 192 to adjust the monitoring range, monitoring area, or field of view of such cameras 192 .
  • multiplexer device 902 may be coupled to and/or integrated with controller 156 such that controller 156 may be coupled (e.g., electrically coupled, communicatively coupled, operatively coupled) to one or more Cable 904, which may be coupled to one or more cameras 192 (eg, via adapter 906 and camera cable 908).
  • controller 156 may send one or more control signals to cameras 192 via cable 904 to, for example, facilitate the above-described adjustments to such cameras 192 and/or another operation associated with such cameras 192 ( For example, powering on, off, adjusting camera settings).
  • MIPI cameras may be limited by the length of the MIPI camera cable coupled to and/or associated with each such MIPI camera (eg, a MIPI camera cable length of approximately 12 inches).
  • multiple cameras 192 may each be disposed on the refrigeration appliance 100 by using multiple cables 904, adapters 906, and camera cables 908 to couple the cameras 192 to the multiplexer device 902 and the controller 156 as illustrated in FIG. 9 At different positions on and/or in the box (for example, on and/or in the box 102, the food freshness chamber 122, the freezer 124, the refrigeration door 128, the freezer door 130).
  • a single controller such as controller 156, may be used to control such multiple cameras 192 disposed at such various locations on and/or within refrigeration appliance 100.
  • a single image signal processor may be used to process images captured (eg, captured simultaneously) by such multiple cameras 192 disposed at such various locations on and/or within the refrigeration appliance 100 (eg, multiple images 300, 302) and/or video, the image signal processor may be coupled to and/or integrated with the controller 156 (eg, the controller may include and/or constitute an SBC).
  • the inventory management system 180 illustrated in FIG. 9 may thereby reduce the need for use in the refrigeration appliance 100 to simultaneously obtain and/or process multiple items from various locations disposed on and/or within the refrigeration appliance 100.
  • Figure 10 illustrates a block diagram of an inventory management system 180 in accordance with one or more exemplary embodiments of the present invention.
  • the inventory management system 180 illustrated in FIG. 10 may constitute an exemplary non-limiting alternative implementation of the inventory management system 180 illustrated in FIG. 9 and described above.
  • inventory management system 180 and/or camera assembly 190 may include a device that may be coupled to first pair of cameras 192 (eg, via first pair of camera cables 908 , denoted as "first pair” in Figures 10 and 11) and first cable 904 (e.g., via first adapter 906)
  • such first multiplexer device 902 is operable to output a first multiplexed signal onto such first cable 904, wherein the first multiplexed signal may include a signal that may be transmitted by a first pair of cameras.
  • the different first data signals may include the above-described data associated with the food preservation compartment 122 and/or the freezer compartment 124 , which data may be simultaneously captured by the first pair of cameras 192 (e.g., via images, video, audio).
  • inventory management system 180 and/or camera assembly 190 may also include a second pair of cameras 192 that may be coupled to (e.g., via a second pair of camera cables).
  • 908 represented as a "second pair" in Figures 10 and 11
  • a second multiplexer device 902 of a second cable 904 e.g, via a second adapter 906.
  • such a second multiplexer device 902 is operable to output a second multiplexed signal onto such a second cable 904, wherein the second multiplexed signal may include a second multiplexed signal that may be transmitted by a second pair of cameras.
  • the different second data signals may include the above-described data associated with the food preservation compartment 122 and/or the freezer compartment 124 , which data may be simultaneously captured by the second pair of cameras 192 (e.g., via images, video, audio).
  • the first pair of cameras 192 and the second pair of cameras 192 may simultaneously (eg, simultaneously) capture the above-described image data associated with the food preservation compartment 122 and/or the freezer compartment 124 , video data and/or audio data.
  • the first pair of cameras 192 and the second pair of cameras 192 may simultaneously (eg, simultaneously) transmit signals to the first multiplexer via different first data signals and different second data signals, respectively.
  • the means 902 and the second multiplexer means 902 provide such image data, video data and/or audio data.
  • inventory management system 180 and/or camera assembly 190 may also include a demultiplexer that may be coupled to first cable 904 and second cable 904 Device 1002.
  • the demultiplexer device 1002 is operable to demultiplex a first multiplexed signal into a different first data signal and to demultiplex a second multiplexed signal into a different second data signal. Signal.
  • demultiplexer device 1002 may be coupled to and/or integrated with a controller (eg, controller 156).
  • the demultiplexer device 1002 may provide a different first data signal and/or a different data signal to the controller 156 when demultiplexing the first multiplexed signal and/or the second multiplexed signal respectively. second data signal.
  • the controller 156 may perform one or more operations. For example, controller 156 may perform one or more operations described above with reference to the exemplary embodiments described in FIGS. 1-9 .
  • controller 156 may: locally analyze (e.g., via CNN, R-CNN, DBN and/or DNN models, algorithms and/or processes) image data and/or video data of images 300, 302; monitor and/or maintain recording of object 182; transmitting signals and/or data (e.g., via external communication system 170 and network 174) to external device 172 and/or remote server 176; adjusting the monitoring range, monitoring area, or field of view of camera 192; and/ Or perform another operation based at least in part on receiving a different first data signal and/or a different second data signal from the demultiplexer device 1002 .
  • locally analyze e.g., via CNN, R-CNN, DBN and/or DNN models, algorithms and/or processes
  • image data e.g., via external communication system 170 and network 174
  • external device 172 and/or remote server 176 e.g., via external communication system 170 and network 174
  • Figure 11 illustrates a block diagram of an inventory management system 180 in accordance with one or more exemplary embodiments of the present invention.
  • the inventory management system 180 illustrated in FIG. 11 may constitute an exemplary, non-limiting alternative implementation of the inventory management system 180 illustrated in FIG. 10 and described above.
  • inventory management system 180 and/or camera assembly 190 may include a demultiplexer device 1002 that may be coupled to first cable 904 , second cable 904
  • the third multiplexer device 902 is operable to multiplex the first and second multiplexed signals that may be simultaneously provided by the first and second multiplexer devices 902 and 902 respectively. Signals are multiplexed.
  • the third multiplexer device 902 may output a third multiplexed signal, and the third multiplexed signal may include the above A different first data signal and a different second data signal.
  • the third multiplexer device 902 may output a third multiplexed signal that may include a different first data signal and a different data signal that may be simultaneously provided by the first pair of cameras 192 and the second pair of cameras 192 .
  • the second data signal is as described above with reference to Figure 10.
  • the third multiplexer device 902 and/or the demultiplexer device 1002 may be coupled to and/or integrated with a controller (eg, controller 156 as illustrated in Figure 11).
  • controller eg, controller 156 as illustrated in Figure 11
  • third multiplexer device 902 may provide a third multiplexed signal to demultiplexer device 1002 .
  • the demultiplexer device 1002 is operable to demultiplex the third multiplexed signal into: a first multiplexed signal and a second multiplexed signal; or the above-mentioned different first data signal and a different the second data signal.
  • the third multiplexer device 902 may be directly coupled to the controller 156 such that the third multiplexer device 902 may provide the third multiplexed signal to the controller 156 .
  • a first multiplexed signal, a second multiplexed signal are received from the demultiplexer device 1002 and/or the third multiplexer device 902 at least in part.
  • the controller 156 may perform one or more operations.
  • controller 156 may perform one or more operations described above with reference to the exemplary embodiments described in FIGS. 1-9 .
  • control The controller 156 may: locally analyze (e.g., via CNN, R-CNN, DBN and/or DNN models, algorithms and/or processes) the image data and/or video data of the images 300, 302; monitor and/or maintain the object 182 recording; transmitting signals and/or data (e.g., via external communication system 170 and network 174) to external device 172 and/or remote server 176; adjusting the monitoring range, monitoring area, or field of view of camera 192; and/or at least Another operation is performed based in part on receiving a different first data signal and/or a different second data signal from the demultiplexer device 1002 .
  • signals and/or data e.g., via external communication system 170 and network 174
  • external device 172 and/or remote server 176 adjusting the monitoring range, monitoring area, or field of view of camera 192
  • at least Another operation is performed based in part on receiving a different first data signal and/or a different second data signal from the demultiplexer device 1002 .
  • Figure 12 illustrates a flow of an exemplary non-limiting method 1200 of operating the inventory management system 180 described above and illustrated in Figures 2, 8 and/or 9 in accordance with one or more exemplary embodiments of the present invention.
  • Method 1200 may use, for example, controller 156, inventory management system 180 (e.g., an implementation of inventory management system 180 described above and exemplified in Figures 2, 8, and/or 9), and/or camera assembly 190 (e.g., , implementations of the camera assembly 190 described above and illustrated in FIG. 2, FIG. 8, and/or FIG. 9).
  • inventory management system 180 e.g., an implementation of inventory management system 180 described above and exemplified in Figures 2, 8, and/or 9
  • camera assembly 190 e.g., implementations of the camera assembly 190 described above and illustrated in FIG. 2, FIG. 8, and/or FIG. 9.
  • method 1200 may constitute an exemplary method of implementing inventory management within a refrigeration appliance (eg, refrigeration appliance 100 ), where the refrigeration appliance may include a refrigeration compartment (eg, food preservation compartment 122 , freezer compartment 124 ) and a camera assembly (eg, camera assembly 190) having a plurality of cameras (eg, camera 192) configured to monitor the refrigerated compartment.
  • a refrigeration appliance eg, refrigeration appliance 100
  • the refrigeration appliance may include a refrigeration compartment (eg, food preservation compartment 122 , freezer compartment 124 ) and a camera assembly (eg, camera assembly 190) having a plurality of cameras (eg, camera 192) configured to monitor the refrigerated compartment.
  • method 1200 may include, via a controller (eg, controller 156) operably coupled to a camera assembly (eg, camera assembly 190), from a multiplexer device (eg, multiplexer device) coupled to the controller 902) Obtain a multiplexed signal (e.g., the multiplexed signal described above with reference to FIG. 9) including a multiplexed signal from a plurality of cameras via a plurality of cables (eg, cable 904) coupled to a multiplexer device and the plurality of cameras. ) are simultaneously provided to different data signals (e.g., the different data signals described above with reference to FIG.
  • the different data signals including data associated with the refrigeration compartment (e.g., with the data provided in the food preservation compartment 122 and/or data associated with one or more objects 182 within, added to, and/or removed from the freezer 124).
  • method 1200 may include obtaining, by the controller, a signal indicating that a door associated with the refrigeration compartment is open (e.g., from a motion sensor of refrigeration appliance 100 and/or door switch signal). In these embodiments, method 1200 may also be performed by the controller operating multiple cameras to simultaneously capture data associated with the refrigeration compartment when the door is open based at least in part on receiving the signal (e.g., as described above with reference to FIG. 9 ).
  • method 1200 may include performing, by a controller, one or more operations based at least in part on receiving a multiplexed signal from a multiplexer device (e.g., controller 156 may perform the operations described above with reference to FIGS. 1-9 one or more operations described in the exemplary embodiments).
  • controller 156 may perform the operations described above with reference to FIGS. 1-9 one or more operations described in the exemplary embodiments.
  • method 1200 may include implementing, by a controller, a machine learning image recognition process based at least in part on receipt of multiplexed signals from a multiplexer device (e.g., CNN, R-CNN, DBN and/or DNN processes) to analyze the data associated with the refrigeration compartment.
  • a multiplexer device e.g., CNN, R-CNN, DBN and/or DNN processes
  • method 1200 may include analyzing, by the controller, the refrigerated compartment based at least in part on receipt of a multiplexed signal from a multiplexer device. associated data.
  • method 1200 may further include maintaining, by the controller, a record of food items placed in or removed from the refrigeration compartment based at least in part on analysis of data associated with the refrigeration compartment.
  • method 1200 may include: using, by the controller (eg, via network 174 ), an external communications system coupled to the controller (eg, external communications System 170) provides multiplexed signals, different data signals, and/or data associated with the refrigeration compartment to one or more remote computing devices external to the refrigeration appliance.
  • an external communications system coupled to the controller (eg, external communications System 170) provides multiplexed signals, different data signals, and/or data associated with the refrigeration compartment to one or more remote computing devices external to the refrigeration appliance.

Landscapes

  • Engineering & Computer Science (AREA)
  • Multimedia (AREA)
  • Signal Processing (AREA)
  • Physics & Mathematics (AREA)
  • Chemical & Material Sciences (AREA)
  • Combustion & Propulsion (AREA)
  • Mechanical Engineering (AREA)
  • Thermal Sciences (AREA)
  • General Engineering & Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Theoretical Computer Science (AREA)
  • Cold Air Circulating Systems And Constructional Details In Refrigerators (AREA)

Abstract

一种制冷电器包括：箱体，该箱体限定制冷间室；门体，该门体提供选择性到达制冷间室的途径；以及相机组件，该相机组件可操作为监测制冷间室。相机组件包括经由多个电缆联接到复用器装置的多个相机以及联接到复用器装置的控制器。多个相机可操作为同时捕获与制冷间室相关联的数据。多个相机中的每个相机联接到多个电缆中的电缆。复用器装置可操作为复用具有与制冷间室相关联的数据的不同数据信号，并且输出包括不同数据信号的复用信号，数据信号由多个相机经由多个电缆同时提供给复用器装置。控制器被配置为部分基于复用信号的接收而执行操作。

Description

制冷电器中的多相机视觉系统 技术领域
本发明总体涉及制冷电器，更具体地涉及制冷电器中的多相机视觉系统和操作多相机视觉系统的方法。
背景技术
制冷电器通常包括箱体，该箱体限定用于接收食品以便储存的制冷间室。另外，制冷电器包括一个或多个门体，这些门体可旋转地铰接到箱体，以允许选择性地接近制冷间室中储存的食品。制冷电器还可以包括安装在制冷间室内并且设计成便于在其中储存食品的各种储存部件。这种储存部件可以包括在制冷间室内接收食品并且辅助组织和布置这种食品的搁架、盒、层架或抽屉。
特别地，经常期望具有存在于制冷电器内的物品的更新库存，例如以便于重新排序、确保食物新鲜或避免变质等。由此，可能期望监测添加到制冷电器或从制冷电器取出的食品，并且获得与这些食品的存在、数量或质量有关的其他信息。某些传统的制冷电器具有用于监测制冷电器中的食品的系统。然而，这种系统通常需要与用户交互，例如，经由通过控制面板进行的关于添加或取出的食品的直接输入。作为对比，某些电器包括用于在食品被添加到制冷电器或从制冷电器取出时监测食品的相机。然而，传统的相机系统可能难以识别特定对象、区分类似产品以及精确地识别制冷间室内的对象的位置。特别地，包括单个或有限数量的相机的传统相机系统可能难以执行这样的任务。
因此，具有用于改进库存管理的系统的制冷电器将是有用的。更特别地，包括具有能够监测进入和离开库存以及对象在制冷间室内的放置的多相机系统的库存管理系统的制冷电器将是特别有益的。
发明内容
本发明的各个方面以及优点将会在下文的描述中进行阐述，或者是可以通过描述来获知，或者是可以通过实施实施方式而学到。
在一个示例性实施方式中，提供了一种制冷电器。该制冷电器可以包括箱体，该箱体限定制冷间室。制冷电器还可以包括门体，该门体可旋转地铰接到箱体，以提供选择性地到达制冷间室的途径。制冷电器还可以包括相机组件，该相机组件可 以联接到箱体并且可操作为监测制冷间室。相机组件可以包括可以联接到多个电缆的多个相机。多个相机可操作为同时捕获与制冷间室相关联的数据。多个相机中的每个相机可以联接到多个电缆中的电缆。相机组件还可以包括可以联接到多个电缆的复用器装置。复用器装置可操作为复用由多个相机经由多个电缆同时提供给复用器装置的不同数据信号，并且输出具有不同数据信号的复用信号。不同数据信号可以包括与制冷间室相关联的数据。相机组件还可以包括联接到复用器装置的控制器。控制器可被配置为至少部分基于复用信号的接收而执行一个或多个操作。
在另一示例性实施方式中，提供了一种在制冷电器内实施库存管理的方法。该制冷电器可以包括制冷间室和具有多个相机的相机组件，该多个相机被设置为监测制冷间室。该方法可以包括：通过可操作地联接到相机组件的控制器从联接到控制器的复用器装置获得复用信号。复用信号可以包括由多个相机经由联接到复用器装置和多个相机的多个电缆同时提供给复用器装置的不同数据信号。不同数据信号可以包括与制冷间室相关联的数据。方法还可包括：由控制器至少部分基于从复用器装置接收到复用信号而执行一个或多个操作。
在另一示例性实施方式中，提供了一种制冷电器。该制冷电器可以包括箱体，该箱体限定制冷间室。制冷电器还可以包括门体，该门体可旋转地铰接到箱体，以提供选择性地到达制冷间室的途径。制冷电器还可以包括相机组件，该相机组件可以联接到箱体并且可操作为监测制冷间室。相机组件可以包括可以联接到第一对相机和第一电缆的第一复用器装置。第一复用器装置可操作为将第一复用信号输出到第一电缆上。第一复用信号可以包括由第一对相机同时提供给第一复用器装置的不同的第一数据信号。相机组件还可以包括可以联接到第二对相机和第二电缆的第二复用器装置。第二复用器装置可操作为将第二复用信号输出到第二电缆上。第二复用信号可以包括由第二对相机同时提供给第二复用器装置的不同的第二数据信号。相机组件还可以包括可以联接到第一电缆和第二电缆的解复用器装置。解复用器装置可操作为将第一复用信号解复用为不同的第一数据信号，并且将第二复用信号解复用为不同的第二数据信号。相机组件还可以包括可以联接到解复用器装置的控制器。控制器可以被配置为至少部分地基于接收到不同的第一数据信号或不同的第二数据信号中的至少一个来执行一个或多个操作。
参照下文的描述以及所附权利要求，本发明的各个实施方式的这些和其它的特征、方面以及优点将变得更容易理解。结合在本说明书中并且构成本说明书一部分的附图显示了本发明的实施方式并且与描述一起用于对本发明的有关原理进行解 释。
附图说明
参照附图，说明书中阐述了面向本领域普通技术人员的本发明的完整公开，这种公开使得本领域普通技术人员能够实现本发明，包括本发明的最佳实施例。
图1示例了根据本发明的一个或多个示例性实施方式的示例性非限制性制冷电器的立体图。
图2示例了根据本发明的一个或多个示例性实施方式的图1的示例性制冷电器的立体图，其中门体被示出为处于打开位置以露出示例性非限制性库存管理系统。
图3示例了根据本发明的一个或多个示例性实施方式的用于操作图2的示例性库存管理系统的示例性非限制性方法的流程图。
图4示例了根据本发明的一个或多个示例性实施方式的使用图2的示例性库存管理系统的相机获得的第一图像。
图5示例了根据本发明的一个或多个示例性实施方式的使用图2的示例性库存管理系统的相机获得的第二图像。
图6示例了根据本发明的一个或多个示例性实施方式的使用图2的示例性库存管理系统的示例性非限制性图像比较和对象识别过程的图。
图7示例了根据本发明的一个或多个示例性实施方式的使用图2的示例性库存管理系统的示例性非限制性对象运动跟踪过程的图。
图8示例了根据本发明的一个或多个示例性实施方式的包括具有多个相机的示例性非限制性库存管理系统的图1的示例性制冷电器的立体图。
图9、图10和图11各自示例了根据本发明的一个或多个示例性实施方式的图2和/或图8的示例性库存管理系统的框图。
图12示例了根据本发明的一个或多个示例性实施方式的操作图2、图8和/或图9的示例性库存管理系统的示例性非限制性方法的流程图。
附图标记在本说明书和/或附图中的重复使用旨在表示本发明的相同或相似的特征、元件或操作。
具体实施方式
现在将详细地参照本发明的实施方式，其中的一个或多个示例示于附图中。每个示例都以对本发明进行解释的方式给出，并不对本发明构成限制。实际上，对于 本领域技术人员而言显而易见的是，能够在不偏离本发明的范围或者精神的前提下对本发明进行多种改型和变型。例如，作为一个实施方式的一部分示出或者进行描述的特征能够用于另一个实施方式，从而产生又一个实施方式。因此，期望的是，本发明覆盖落入所附权利要求及其等同形式的范围内的这些改型以及变型。
如本文所引用的，术语“实体”是指人、用户、最终用户、消费者、计算装置和/或程序(例如，处理器、计算硬件和/或软件、应用等)、代理、机器学习(ML)和/或人工智能(AI)算法、模型、系统和/或应用、和/或可以实施和/或促进如本文所述的、在附图中示例的和/或在所附权利要求中包括的本发明的一个或多个实施方式的实施的另一类型的实体。如本文所用的，术语“联接”是指化学联接(例如，化学键合)、通信联接、电和/或电磁联接(例如，电容联接、电感联接、直接和/或连接联接等)、机械联接、可操作联接、光学联接和/或物理联接。
如本文所用的，术语“上游”和“下游”是指相对于流体通路中的流体流动的相对方向。例如，“上游”是指流体流动的来向，而“下游”是指流体流动的去向。如本文所引用的，术语“包括(includes)”和“包括(including)”旨在以类似于术语“包括(comprising)”的方式为包括的。如本文所引用的，术语“或”以及“和/或”通常旨在是包括的，即“A或B”或“A和/或B”各自旨在意指“A或B或两者”。如本文所引用的，术语“第一”、“第二”、“第三”等可以互换使用以将一个部件或实体与另一个部件或实体区分开，并且这些术语并不旨在表示各个部件或实体的位置、功能或重要性。
如本文在整个说明书和权利要求书中使用的近似语言被应用于修饰任何定量表示，该定量表示可容许在不导致其相关的基本功能改变的情况下变化。因此，由诸如“大约”、“近似”以及“大致”的术语修饰的值不限于所指定的精确值。在至少一些情况下，近似语言可对应于用于测量值的仪器的精度。例如，近似语言可以指在10％的误差范围内。
现在参考附图。将描述根据本发明的一个或多个实施方式的示例性制冷电器、库存管理系统、相机组件和对应的操作方法。
图1示例了根据本发明的一个或多个示例性实施方式的示例性非限制性制冷电器100的立体图。如图示例，制冷电器100通常限定竖向V、侧向L和横向T，竖向V、侧向L和横向T中的每一个相互垂直，使得大体限定正交坐标系。
根据示例性实施方式，制冷电器100包括箱体102，该箱体102通常用于容纳和/或支撑制冷电器100的各种部件，并且还可限定制冷电器100的一个或多个内部腔 室或间室。在这点上，如本文所用的，术语“箱体”、“壳体”等通常旨在指用于制冷电器100的外框架或支撑结构，例如，包括由任何合适的材料形成的任何合适数量、类型和构造的支撑结构，诸如细长支撑构件、多个互连面板或其一些组合的系统。应当理解，箱体102不一定需要围合，而是可以简单地包括支撑制冷电器100的各种元件的开放结构。相反，箱体102可以包围箱体102内部的一些或所有部分。应当理解，箱体102可具有任何合适的尺寸、形状和构造，同时保持在本发明的范围内。
如图示例，箱体102通常沿着竖向V在顶部104与底部106之间延伸，沿着侧向L在第一侧108(例如，如图1中从前方观察时的左侧)与第二侧110(例如，如图1中从前方观察时的右侧)之间延伸，并且沿着横向T在前侧112与后侧114之间延伸。一般而言，诸如“左”、“右”、“前”、“后”、“顶部”或“底部”的术语是参考用户接近制冷电器100的视角来使用的。
箱体102限定用于接收食品以便储存的制冷间室。特别地，箱体102限定设置在箱体102的顶部104处或与其相邻设置的食物保鲜室122和布置在箱体102的底部106处或与其相邻布置的冷冻室124。由此可见，制冷电器100通常被称为底置式冰箱。然而，认识到，本发明的益处适用于其他类型和样式的制冷电器，例如，顶置式制冷电器、对开门式制冷电器或单门制冷电器。而且，本发明的方面也可以适用于其他电器。因此，本文阐述的描述仅出于说明目的，而无意于在任何方面限于任何特定的电器或配置。
冷藏门体128可旋转地铰接到箱体102的边缘，以便选择性地进入食物保鲜室122。另外，在冷藏门体128的下方布置冷冻门体130，以便选择性地进入冷冻室124。冷冻门体130联接至可滑动地安装在冷冻室124内的冷冻抽屉(未示出)。通常，冷藏门体128在由箱体102限定的前开口132(图2和图3)(例如，在由竖向V和侧向L限定的平面内延伸)上形成密封。在这点上，当冷藏门体128打开时，用户可以通过前开口132将物品放置在食物保鲜室122内，然后可以关闭冷藏门体128以便于气候控制。冷藏门体128和冷冻门体130在图1中被示出为处于关闭构造。本领域技术人员将理解，其它腔室和门体构造是可行的，并且在本发明的范围内。
图2示例了根据本发明的一个或多个示例性实施方式的制冷电器100的立体图，其中冷藏门体128被示出为处于打开位置以露出制冷电器100的一个或多个部件和/或其中的对象。如图2所示，如本领域技术人员将理解的，各种储存部件被安装在食物保鲜室122内，以促进食品在其中的储存。特别地，储存部件可以包括盒134 和层架136。这些储存部件中的每一个都被构造为接收一个或多个对象182(例如，食品、饮料)，并且可以辅助组织这样的对象182。如图示例，盒134可以安装在冷藏门体128上或者可以滑入食物保鲜室122中的容纳空间中。应当理解，所示的储存部件仅用于说明的目的，并且可以使用其它储存部件，并且其它储存部件可以具有不同的尺寸、形状以及构造。
再次参见图1，将描述根据本发明的示例性实施方式的分配组件140。虽然将示例并描述分配组件140的几个不同的示例性实施方式，但类似的附图标记可用于指代类似的部件和特征。分配组件140通常用于分配液态水和/或冰。虽然在本文中示例并描述了示例性分配组件140，但应当理解，可以在保持在本发明的范围内的同时对分配组件140进行各种变更和修改。
分配组件140及其各种部件可以至少部分地设置在限定于冷藏门体128中的一个上的分配器凹部142内。在这点上，分配器凹部142限定在制冷电器100的前侧112上，使得用户可以在不打开冷藏门体128的情况下操作分配组件140。另外，分配器凹部142设置在预定高度处，该预定高度方便用户取冰，并且使得用户能够在不需要弯腰的情况下取冰。在示例性实施方式中，分配器凹部142设置在接近用户的胸部水平的位置处。
分配组件140包括冰或水分配器144，该分配器包括用于从分配组件140排出冰的排放口146。被示出为拨片的致动机构148安装在排放口146下方，以便操作冰或水分配器144。在可选示例性实施方式中，可以使用任意合适的致动机构来操作冰分配器144。例如，冰或水分配器144可以包括传感器(例如超声传感器)或按钮，而不是拨片。排放口146和致动机构148是冰或水分配器144的外部零件，并且安装在分配器凹部142中。与之相比，冷藏门体128可以限定容纳制冰机和储冰盒(未示出)的冰盒室150(图2)，该制冰机和储冰盒被构造成将冰供应至分配器凹部142。
设置控制面板152，以便控制操作模式。例如，控制面板152包括一个或多个选择输入154，诸如旋钮、按钮、触摸屏界面等，诸如水分配按钮和冰分配按钮，用于选择期望的操作模式，诸如碎冰或非碎冰。另外，输入154可以用于指定填充容积或操作分配组件140的方法。在这点上，输入154可以与处理装置或控制器156通信。在控制器156中生成的信号响应于选择器输入154操作制冷电器100和分配组件140。另外，可以在控制面板152上设置显示器158，诸如指示灯或屏幕。显示器158可以与控制器156通信，并且可以响应于来自控制器156的信号而显示信息。
如本文所用的，“处理装置”或“控制器”可以指一个或多个微处理器或半导体 装置，并且不必限于单个元件。处理装置或控制器(例如，控制器156)可以被编程为操作制冷电器100、分配组件140以及制冷电器100的一个或多个其他部件。处理装置或控制器(例如，控制器156)可以包括一个或多个存储元件(例如，非暂时性存储介质、非暂时性计算机可读存储介质)或与其相关联。在一些实施方式中，这种存储元件包括电可擦可编程只读存储器(EEPROM)。通常，存储元件可以存储处理装置或控制器(例如，控制器156)可访问的信息，包括可以由处理装置或控制器执行的指令。可选地，指令可以是软件或指令和/或数据的任意集合，该软件或指令和/或数据的任意集合在由处理装置或控制器(例如，控制器156)执行时，使得处理装置执行操作。
仍然参见图1，将描述根据本发明的示例性实施方式的外部通信系统170的示意图。通常，外部通信系统170用于允许制冷电器100与一个或多个外部装置之间的交互、数据传送和其他通信。例如，该通信可以用于提供和接收各种类型的格式(例如，数据信号、媒体、图像、视频、音频、复用或解复用的数据信号)的各种类型的数据、操作参数、用户指令或通知、性能特性、用户偏好或用于制冷电器100的改进性能的任何其它合适的信息。另外，应当理解，外部通信系统170可用于传送数据或其它信息，以提高一个或多个外部装置或电器的性能和/或改进与这种装置的用户交互。
例如，外部通信系统170允许制冷电器100的控制器156与制冷电器100外部的单独装置通信，该单独装置在本文中通常被称为外部装置172。如以下更详细描述的，这些通信可以使用有线或无线连接(诸如经由网络174)来促进。通常，外部装置172可以是与制冷电器100分开的任何合适的装置，该装置被配置为向用户提供和/或从用户接收通信、信息、数据或命令。在这点上，外部装置172可以是例如个人电话、智能电话、平板电脑、膝上型或个人计算机、可穿戴装置、智能家庭系统或者另一移动或远程装置。
另外，远程服务器176可以通过网络174与制冷电器100和/或外部装置172通信。在这点上，例如，远程服务器176可以是基于云的服务器，由此位于远处位置，诸如在单独的州、国家等。根据示例性实施方式，外部装置172可通过网络174(诸如因特网)与远程服务器176通信，以发送和/或接收数据或信息、提供用户输入、接收用户通知或指令、与制冷电器100交互或控制制冷电器等。另外，外部装置172和远程服务器176可以与制冷电器100通信以传送类似的信息。根据示例性实施方式，远程服务器176可被配置为接收和分析由制冷电器100的相机组件190(图2和 图3)获得的图像、视频、音频和/或其他数据，例如以便于库存分析。
通常，可以使用任何类型的有线或无线连接并且使用任何合适类型的通信网络来进行制冷电器100、外部装置172、远程服务器176和/或其它用户装置或电器之间的通信，下面提供了通信网络的非限制性示例。例如，外部装置172可以通过任何合适的有线或无线通信连接或接口(例如网络174)与制冷电器100直接或间接通信。例如，网络174可以包括局域网(LAN)、广域网(WAN)、个域网(PAN)、因特网、蜂窝网络、任何其他合适的短程或远程无线网络等中的一个或多个。另外，可以使用任何合适的通信装置或协议(诸如经由无线电、激光、红外、以太网类型的装置和接口等)来发送通信。另外，这种通信可以使用各种通信协议(例如，传输控制协议/互联网协议(TCP/IP)、超文本传输协议(HTTP)、简单邮件传送协议(SMTP)、文件传送协议(FTP)等)、编码或格式(例如，超文本标记语言(HTML)、可扩展标记语言(XML)等)和/或保护方案(例如，虚拟专用网络(VPN)、安全HTTP、安全外壳(SSH)、安全套接层(SSL)等)。
本文描述了根据本发明的示例性实施方式的外部通信系统170。然而，应当理解，本文提供的外部通信系统170的示例性功能和配置仅用作示例，以便于描述本发明的各方面。系统配置可以变化，其他通信装置可以用于直接或间接地与一个或多个关联的电器通信，可以实施其他通信协议和步骤等。这些变化和修改被认为在本发明的范围内。
现在一般参见图2，制冷电器100还可以包括库存管理系统180，该库存管理系统通常配置为监测制冷电器100的一个或多个腔室，以监测库存的添加和/或取出。更具体地，如以下更详细地描述的，库存管理系统180可以包括多个传感器、相机或其他检测装置，其用于监测食物保鲜室122和/或冷冻室124，以检测放置在食物保鲜室122和/或冷冻室124中或从其取出的对象182(例如，食品、饮料)。在这点上，库存管理系统180可以使用来自这些装置中的每一个的数据来获得食物保鲜室122和/或冷冻室124内的对象182的身份、位置和/或其它定性或定量特性的完整表示或知识。尽管库存管理系统180在本文中被描述为监测食物保鲜室122，以便检测对象182，但是应当理解，本发明的各方面可以用于监测任何其他合适的电器、腔室(例如，冷冻室124)等中的对象或物品。
如图2示意性所示，库存管理系统180可以包括联接到制冷电器100(例如，箱体102)的相机组件190，该相机组件通常设置和用于在运行期间获得制冷电器100的图像和/或视频。具体地，根据所示例的实施方式，相机组件190包括一个或多个 相机192，这些相机安装到箱体102、冷藏门体128或以其他方式设置为看得见食物保鲜室122。尽管本文将相机组件190描述为用于监测制冷电器100的食物保鲜室122，但是应当理解，本发明的各方面可以用于监测任何其他合适的电器的任何其他合适的区域，例如冷冻室124。如图2中最佳示出的，相机组件190的相机192在食物保鲜室122的前开口132处安装到箱体102，并且被定向为具有被引导跨过前开口132和/或进入食物保鲜室122中的视场。
尽管在图2中示例了单个相机192，但是应当理解，相机组件190可以包括设置在箱体102内和/或联接(例如，安装)到箱体的多个相机192，其中，多个相机192中的每一个具有设置在食物保鲜室122周围的指定监测区或监测范围。在这点上，例如，各个相机192的视场可以被限制到、引导到或聚焦在食物保鲜室122内的特定监测区、监测范围或特定区域上。具体地，现在简要地参见图8，提供了根据本发明的一个或多个示例性实施方式的具有多个相机192的库存管理系统180。如图所示，相机192可以安装到食物保鲜室122的侧壁并且可以沿着竖向V隔开，以覆盖不同的监测区。
然而，特别地，可能期望将各个相机192设置为接近食物保鲜室122的前开口132并且定向各个相机192，使得各个相机192的视场被引导到食物保鲜室122中。这样，可以减轻或完全避免与获得制冷电器100的用户的图像有关的隐私问题。根据示例性实施方式，相机组件190可以用于促进制冷电器100的库存管理过程。由此可见，各个相机192可以设置在食物保鲜室122的开口处，以监测被添加到食物保鲜室122或从其中取出的对象182(例如，食品、饮料)。
根据另一些实施方式，各个相机192可以以任意其他合适的方式定向成用于监测制冷电器100内或周围的任意其他合适的区域。应当理解，根据可选实施方式，相机组件190可以包括任意合适数量、类型、尺寸和配置的相机192，用于获得制冷电器100内或周围的任意合适的区或区域的图像。另外，应当理解，各个相机192可以包括用于调节其视场和/或取向的特征。
应当理解，由相机组件190获得的图像和/或视频可以在数量、频率、角度、分辨率、细节等方面变化，以便提高制冷电器100周围或内的特定区域的清晰度。另外，根据示例性实施方式，控制器156可以用于在获得图像之前使用一个或多个光源照亮制冷间室。特别地，制冷电器100的控制器156(或任意其他合适的专用控制器)可以通信地联接到相机组件190，并且可以被编程或用于分析由相机组件190获得的图像，例如，以便识别被添加到制冷电器100或从其取出的物品，如以下详细 描述的。
通常，控制器156可以联接(例如，电联接、通信联接、可操作地联接)到相机组件190，用于分析由相机组件190获得的一个或多个图像和/或视频，以提取关于位于食物保鲜室122内的对象182的有用信息。在这点上，例如，由相机组件190获得的图像和/或视频可以用于提取条形码、识别产品、监测产品的运动、或获得与对象182有关的其他产品信息。特别地，该分析可以在本地(例如，在控制器156上)执行，或者可以被发送到远程服务器(例如，经由外部通信系统170的远程服务器176)以用于分析。这种分析旨在例如通过识别被添加到制冷间室和/或从制冷间室取出的食品来促进库存管理。
既然已经呈现了根据本发明的示例性实施方式的制冷电器100和相机组件190的结构和构造，则提供用于操作相机组件190的示例性方法200。方法200可用于操作相机组件190，或操作用于监测电器操作或库存的任意其它合适的相机组件。在这点上，例如，控制器156可以用于实施方法200。然而，应当理解，示例性方法200在本文仅讨论为描述本发明的示例性方面，而不旨在限制。
如图3所示，方法200包括：在步骤210，使用相机组件获得制冷电器的制冷间室的第一图像。例如，继续上述示例，制冷电器100的相机组件190可以获得食物保鲜室122内的一个或多个图像，这些图像可以在其视场中包括多个对象182。在这点上，制冷电器100的相机组件190可以获得食物保鲜室122、冷冻室124或者制冷电器100内或周围的任何其它区或区域的一个或多个图像(例如，分别在图4和图5中标识的第一图像300和第二图像302)。具体地，根据示例性实施方式，相机192从箱体102的顶部中心向下定向，并且具有覆盖食物保鲜室122的宽度的视场(例如，如图4和图5的照片所示)。而且，该视场可以以箱体102的前部处的前开口132为中心，例如，在该开口处，抵靠箱体102的前部安置冷藏门体128。这样，相机192的视场以及所获得的结果图像可以捕获对象进入和/或离开食物保鲜室122的任意运动或移动。通过相机组件190获得的图像可以包括一个或多个静止图像、一个或多个视频剪辑、或者适合于识别对象182(例如，食品、饮料)或库存分析的任意其他合适类型和数量的图像。
特别地，相机组件190可以在任何合适的触发(诸如基于时间的成像时间表)时获得图像，在成像时间表中，相机组件190周期性地对食物保鲜室122进行成像和监测。根据另一些实施方式，相机组件190可以周期性地拍摄低分辨率图像，直到检测到运动(例如，经由低分辨率图像的图像区分)为止，此时可以获得一个或 多个高分辨率图像。根据另一些实施方式，制冷电器100可以包括一个或多个运动传感器(例如，光学的、声学的、电磁的等)，当对象182被添加到食物保鲜室122或从中取出时，该一个或多个运动传感器被触发，并且相机组件190可以可操作地联接到这样的运动传感器，以在这样的移动期间获得对象182的图像。
根据另一些实施方式，制冷电器100可以包括门体开关，该门体开关检测冷藏门体128何时打开，在该时刻，相机组件190可以开始获得一个或多个图像。根据示例性实施方式，在冷藏门体128打开的同时，可以连续地或周期性地获得图像300、302。在这点上，获得图像300、302可以包括确定制冷电器的门体是打开的，并且在门体打开的同时以设定的帧率捕获图像。特别地，食品在图像帧之间的运动可以用于确定对象182是从食物保鲜室122中取出还是添加到其中。应当理解，由相机组件190获得的图像可以在数量、频率、角度、分辨率、细节等方面变化，以便提高对象182的清晰度。另外，根据示例性实施方式，控制器156可以用于在获得图像300、302的同时照亮冰箱灯(未示出)。其它合适的触发是可行的，并且在本发明的范围内。
步骤220可以包括：使用机器学习图像识别过程来分析第一图像，以识别第一图像中的对象。应当理解，该分析可以利用任意合适的图像分析技术、图像分解、图像分割、图像处理等。该分析可以完全由控制器156执行，可以卸载到远程服务器来分析，可以在用户辅助下分析(例如，经由控制面板152)，或者可以以任意其他合适的方式分析。根据本发明的示例性实施方式，在步骤220执行的分析可以包括机器学习图像识别过程。
根据示例性实施方式，该图像分析可以使用任何合适的图像处理技术、图像识别过程等。如本文所用的，术语“图像分析”等通常可以用于指代对象的一个或多个图像、视频或其他视觉表示的观察、分析、图像分解、特征提取、图像分类等的任何合适的方法。如以下更详细地解释的，该图像分析可以包括图像处理技术、图像识别技术或其任何适当组合的实施。在这点上，图像分析可以使用任何合适的图像分析软件或算法来持续地或周期性地监测食物保鲜室122内的移动对象。应当理解，该图像分析或处理可以在本地(例如，由控制器156)或远程(例如，通过将图像数据卸载到远程服务器或网络，例如，远程服务器176)执行。
具体地，对一个或多个图像的分析可以包括实施图像处理算法。如本文所用的，术语“图像处理”等通常旨在指代用于分析图像的不依赖于人工智能或机器学习技术的任何合适的方法或算法(例如，与以下描述的机器学习图像识别过程形成对比)。 例如，图像处理算法可以依赖于图像区分，例如两个连续图像的逐像素比较。该比较可以帮助识别顺序获得的图像之间的实质差异，例如，以识别移动、特定对象的存在、特定条件的存在等。例如，当特定条件存在时，可以获得一个或多个参考图像，并且这些参考图像可以被存储，以用于将来与在电器运行期间获得的图像进行比较。参考图像与获得的图像之间的相似性和/或差异可以用于提取用于提高电器性能的有用信息。例如，图像区分可以用于确定像素级运动度量何时通过预定运动阈值。
处理算法还可以包括用于隔离或消除例如由于图像分辨率、数据传输误差、不一致照明或其他成像误差而产生的图像比较中的噪声的措施。通过消除这种噪声，图像处理算法可以改善准确的对象检测，避免错误的对象检测，并且隔离图像内的重要对象、区域或图案。另外或可选地，图像处理算法可以使用用于识别或标识特定物品或对象的其他合适的技术，诸如边缘匹配、分治搜索、灰度匹配、感受野响应的直方图或另一合适的例程(例如，基于来自一个或多个相机的一个或多个捕获的图像在控制器156处执行)。其它图像处理技术也是可行的，并且在本发明的范围内。
除了上述图像处理技术之外，图像分析还可以包括利用人工智能(AI)，诸如机器学习图像识别过程、神经网络分类模块、任何其他合适的人工智能(AI)技术和/或任何其他合适的图像分析技术，其示例将在下面更详细地描述。而且，以下描述的各个示例性图像分析或评估过程可以独立地、共同地或可互换地使用，以提取关于被分析的图像的详细信息，从而促进本文描述的一个或多个方法的执行或以其他方式改进电器运行。根据示例性实施方式，可以使用任何合适数量的图像处理、图像识别或其他图像分析技术及其组合来获得对所获得的图像的准确分析。
在这点上，图像识别过程可以使用任意合适的人工智能技术，例如，任意合适的机器学习技术，或者例如，任意合适的深度学习技术。根据示例性实施方式，图像识别过程可以包括实施称为基于区域的卷积神经网络(“R-CNN”)图像识别的一种形式的图像识别。一般而言，R-CNN可包括取得输入图像并提取包括图像的潜在对象或区域的区域建议。在这点上，“区域建议”可以是图像中可能属于特定对象的一个或多个区域，或者可以包括共享共同像素特性的相邻区域。然后使用卷积神经网络来从区域建议计算特征，然后将使用所提取的特征来确定各个特定区域的分类。
根据另一些实施方式，可以将图像分割过程与R-CNN图像识别一起使用。通常，图像分割为图像中的各个对象创建基于像素的掩码，并且提供对给定图像内的各种 对象的更详细或更精细的理解。在这点上，代替处理整个图像(即，像素的大集合，其中许多像素可能不包含有用信息)，图像分割可以涉及将图像划分为片段(例如，划分为包含类似属性的像素组)，这些片段可以独立地或并行地分析，以获得图像中的一个或多个对象的更详细表示。这在本文中可以被称为“掩码R-CNN”等，与常规的R-CNN架构相反。例如，掩码R-CNN可以基于与R-CNN略微不同的快速R-CNN。例如，R-CNN首先应用卷积神经网络(CNN)，然后将其分配给特性图上的区域推荐，而不是初始地分割为区域推荐。另外，根据示例性实施方式，标准CNN可用于获得、识别或检测与一个或多个图像内的一个或多个对象或区域有关的任何其他定性或定量数据。在另外或可选实施方式中，可以使用K均值算法。
根据另一些实施方式，图像识别过程可以使用任意其他合适的神经网络过程，同时保持在本发明的范围内。例如，分析一个或多个图像的步骤可以包括使用深度信念网络(DBN)图像识别过程。DBN图像识别过程通常可以包括堆叠许多单独的无监督网络，这些网络使用各个网络的隐藏层作为下一层的输入。根据另一些实施方式，分析一个或多个图像的步骤可以包括实施深度神经网络(DNN)图像识别过程，其通常包括使用在输入与输出之间具有多个层的神经网络(例如，由生物神经网络启示和/或基于生物神经网络的计算系统)。可以使用其他合适的图像识别过程、神经网络过程、人工智能分析技术以及上述或其他已知方法的组合，同时保持在本发明的范围内。
另外，应当理解，可以使用各种传送技术，但是不需要使用这样的技术。如果使用传送技术学习，则可以利用公共数据集来预训练神经网络架构，诸如VGG16、VGG19或ResNet50，然后可以利用电器特定数据集来重新训练最后一层。另外或可选地，图像识别过程可包括基于初始条件的比较而检测某些条件和/或可依赖于图像减影技术、图像堆叠技术、图像拼接等。例如，减影图像可以用于训练具有多个类别的神经网络，以用于将来的比较和图像分类。
应当理解，机器学习图像识别模型可以由电器利用新图像主动训练，可以被提供有来自制造商或来自另一远程源的训练数据，或者可以以任何其它合适的方式训练。例如，根据示例性实施方式，该图像识别过程至少部分地依赖于神经网络，该神经网络利用不同配置的电器的多个图像训练、经历不同条件或以不同方式交互。该训练数据可以本地或远程地存储，并且可以被传送到远程服务器以用于训练其他电器和模型。
应当理解，图像处理和机器学习图像识别过程可以一起使用，以便于改进的图 像分析、对象检测，或者从一个或多个图像中提取可以用于改进电器的运行或性能的其他有用的定性或定量数据或信息。实际上，本文描述的方法可以可互换地使用这些技术中的任何或全部来改进图像分析过程并且促进改进的电器性能和消费者满意度。本文描述的图像处理算法和机器学习图像识别过程仅是示例性的，并且不旨在以任何方式限制本发明的范围。
步骤230可以包括：使用相机组件获得第二图像302。例如，第二图像302可以在步骤210获得第一图像300之后立即获得。通常，第一图像300和第二图像302都可以在对象182处于插入到食物保鲜室122中或从中取出的过程中的同时获得，使得可以确定对象182的轨迹，如下面更详细描述的。
步骤240可以包括：使用机器学习图像识别过程来分析第二图像，以识别第二图像中的对象。在这点上，步骤240可以包括与上面关于步骤220描述的图像分析类似的图像分析。
现在简要地参见图4、图5、图6和图7，示例了在方法200的实施期间由相机组件190获得的各种图像(例如，包括第一图像300和第二图像302)。例如如图4所示，在步骤220执行的图像分析可以例如基于使用类似对象182(例如，如本文示例为苹果或桔子)对机器学习模型的训练来识别第一图像300和第二图像302内的多个对象。除了对象识别之外，机器学习图像识别过程可以提供置信度分数(例如，如针对在图4、图5和图6中识别的各个对象182一般由附图标记310标识)。在这点上，例如，置信度分数310通常可以表示对象已经被机器学习模型适当识别的概率。
应当理解，通过在不同角度、不同时间、不同位置等获得同一对象182的更多图像，可以增加置信度分数310。因此，方法200还可包括以下步骤：使用相机组件190获得第三图像，其中，第三图像也包含来自第一图像300和第二图像302的对象182。方法200还可以包括：分析第三图像以识别第三图像中的对象；以及至少部分地基于对第三图像的分析来增加置信度分数，以识别对象。在这点上，如果机器学习模型将单个对象182识别为相同的桔子，则置信水平可以增加，例如，如从图4和图5中的对象识别所示。第三图像中相同桔子的肯定识别可进一步增加置信度分数。相反，相同对象182的否定识别可以用来降低置信度分数。
特别地，置信度分数310可以是来自机器学习模型的输出，并且可以基于被监测或跟踪的对象182的任何合适的特性。例如，各个对象182可以具有可识别的特征，诸如茎、变色、瑕疵、或其他可以是可识别的并且与该特定对象182相关联的 特征(例如，类似于该对象的指纹)。机器学习图像识别模型可以基于各个对象的特定指纹来识别各个对象，并且可以使用来自其他图像的可识别特征来提高对象识别的准确性。尽管本文关于单独的桔子或苹果描述了这种对多个图像的比较以提高对象识别的置信度分数，但是应当理解，模型可以被外推到使用任何合适数量的图像来识别多个对象中的任一个。
步骤250可以包括：基于对象在第一图像和第二图像中的位置来确定对象的运动向量。具体地，如图7中最佳示例的，在第一图像300与第二图像302之间示出了第一对象182(例如，第一桔子)的运动向量320。在这点上，如果在第一图像300和第二图像302中都识别出对象182(例如桔子)，则方法200可以包括：确定与该对象182的移动相关联的轨迹或运动向量320。而且，通过肯定地识别放置在食物保鲜室122内的一个或多个对象182的运动向量320，可以改进或增加与特定对象182的识别相关联的置信度分数310。
另外，多个对象中的相邻对象182及其相关联的运动向量320的识别可以提高对象识别及其相关联的运动向量320的置信度分数310。在这点上，例如，方法200可以包括：分析第一图像300，以识别第一图像300中的第二对象(例如与桔子相邻放置的苹果)。方法200还可以包括：确定第一对象182与第二对象182之间的空间关系(例如，两个对象在三维空间中的相对放置)。方法200还可包括：至少部分地基于第一对象182的运动向量320和第一对象与第二对象之间的空间关系来确定第二对象的预测运动向量(例如，如一般由附图标记322标识)。
由此，方法200可以包括：获得被添加到制冷间室或从制冷间室取出的对象182的多个图像。在这点上，继续上述示例，控制器156或另一合适的处理装置可以分析这些图像，以识别对象182和/或其进入或离开食物保鲜室122和/或冷冻室124的轨迹。通过识别对象182是被添加到食物保鲜室122和/或冷冻室124还是从其中取出，控制器156可以监测和跟踪制冷电器100内的库存。例如，控制器156可以保持放置在食物保鲜室122内或从其中取出的食品的记录。
图3描述了具有为了示例和讨论的目的而以特定顺序执行的步骤的示例性控制方法。使用本文所提供的发明内容，本领域普通技术人员将理解，本文所述的任意方法的步骤可以以各种方式改编、重新排列、扩展、省略或修改，而不脱离本发明的范围。而且，虽然使用相机组件190作为示例来说明了这些方法的各方面，但是应当理解，这些方法可以应用于任意合适的电器和/或相机组件的操作。
如上所述的库存管理系统180和操作制冷电器的方法200通常可以有助于改善 制冷电器内的库存管理。在这点上，该系统便于对象识别，其中，可以使用逐帧对象分析方法来支持库存管理。这在跟踪储存在冰箱中的属于单个类别的多个对象(例如，类似对象)时是有利的。在一些实施方式中，来自相机的多个图像可用于跟踪移动通过其视场的物品，其中逐帧地捕捉对象。可以在神经网络中比较对象的帧之间的一致性。神经网络可以被设计为给出两个图像属于同一物品的概率。如果单个对象的多个图像是可用的，则可以进行多次比较，然后可以使用平均置信度。
神经网络有效地为各个对象生成特征向量或映射并比较。高置信度向量被给予在帧之间肯定识别的对象。未知物品之间的相对位置可用于在下一步骤中识别它们。如果物品一起移动，则另一物品可位于已知的位置。如果物品没有移动，则将在相同的位置找到它。任一情况都可以用于联系帧之间的物品识别。可以在与电器的一个或多个交互(例如，许多帧)的过程中建立以电器为中心的数据库。相同物品识别的各个图像可用于将来的比较，从而使得跟踪变得越来越容易。
例如，如果基于一对帧存在给定桔子是相同桔子的50％置信度，则可以将相同桔子的较早图像运行相同比较，从而产生平均75％的高达90％置信度的匹配。由此，例如通过使用最大匹配置信度、使用平均匹配置信度、使用诸如四分位数、中值等的其它类似度量，有效地使用较早的图像可以带来匹配的置信度。因此，该方法对于跟踪进入电器中到达最终储存位置的物品和物品年龄(例如，即使物品被四处移动)是有用的。另外，方法(例如，方法200)确定在取出时哪个物品离开储存空间，并且还建议用户取出最早的物品并将其示出在图像中。
图9示例了根据本发明的一个或多个示例性实施方式的库存管理系统180的框图。尽管在图9中未描述，但是图9中示例的库存管理系统180可以包括相机组件190。在图9示例的示例性实施方式中，库存管理系统180和/或相机组件190可以包括多个相机192，这些相机可以经由多个电缆904联接到复用器装置902。例如，如图9所描述的实施方式示例，各个相机192可以经由单个电缆904、适配器906和相机电缆908联接到复用器装置902。在这个或另一个实施方式中，复用器装置902可以联接到控制器(例如微处理器)(诸如控制器156)和/或与其集成，该控制器可以构成和/或包括单板计算机(SBC)。在至少一个实施方式中，控制器156可以包括、联接到、构成和/或以其他方式关联于图像信号处理器(ISP)，该图像信号处理器可以是可操作的和/或被配置为根据本文所述的一个或多个示例性实施方式来处理图像数据、视频数据和/或音频数据。
在一些实施方式中，各个电缆904可以构成和/或包括模拟电缆、数字电缆、通 信电缆、网络电缆、数据电缆、媒体电缆、控制电缆、同轴电缆或另一类型的电缆。在一些实施方式中，各个电缆904可以构成和/或包括可以用于在各个相机192与控制器156之间传送图像数据、视频数据、音频数据、控制数据(例如，控制信号)和/或其他数据的电缆(例如，同轴电缆)。
在图9中描述的示例性实施方式中，各个相机192可以构成和/或包括移动行业处理器接口(MIPI)相机(例如，MIPI相机模块)。在这个或另一个实施方式中，各个相机电缆908可以构成和/或包括MIPI相机电缆。在这个或另一个实施方式中，各个电缆904可以构成和/或包括同轴电缆。在这个或另一个实施方式中，各个适配器906可构成和/或包括MIPI至同轴电缆适配器。
尽管本发明的一些示例性实施方式描述和/或描绘了同轴电缆、MIPI至同轴电缆适配器、MIPI相机电缆和MIPI相机的使用，但本发明并不限于此。例如，在不偏离本发明的意图和/或范围的情况下，可以根据本文所述的一个或多个实施方式来实施可操作为使用多个相机同时捕获图像数据、视频数据和/或音频数据、将多个相机的不同信号复用成单个信号和/或使用图像信号处理器(ISP)(例如，联接到控制器(例如，SBC)和/或与其集成的ISP)来处理单个信号的其他硬件的使用。例如，在不偏离本发明的意图和/或范围的情况下，可以根据本文所述的一个或多个实施方式来实施其他类型的相机(例如，通用串行总线(USB)相机)、电缆(例如，USB电缆)、适配器和/或不同数量的图像信号处理器(ISP)和/或单板计算机(SBC)的不同组合。
在至少一个实施方式中，相机192可操作为同时(例如，同时地、在大约相同的时间)捕获与制冷间室(例如，食物保鲜室122和/或冷冻室124)相关联的数据(例如，图像数据、视频数据、音频数据)。例如，相机192可以同时捕获放置在食物保鲜室122和/或冷冻室124内、添加到其中和/或从其中取出的一个或多个对象182的图像和/或视频。例如，当制冷电器100的运动传感器(例如，光学的、声学的、电磁的)和/或门体开关被触发和/或检测到冷藏门体128被打开时，库存管理系统180和/或相机组件190可以使用相机192来同时捕获被添加到食物保鲜室122或从其中取出的一个或多个对象182的图像和/或视频。在该示例中，控制器156可以接收指示冷藏门体128和/或冷冻门体130打开的信号(例如，控制器156可以从制冷电器100的运动传感器和/或门体传感器接收这样的信号)。在该示例中，至少部分地基于接收到这样的信号，控制器156可以在冷藏门体128和/或冷冻门体130打开时操作(例如，经由库存管理系统180、相机组件190)一个或多个相机192， 以同时捕获分别与食物保鲜室122和/或冷冻室124相关联的这样的数据(例如，与放置在食物保鲜室122和/或冷冻室124内、添加到其中和/或从其中取出的一个或多个对象182相关联的数据)。
在图9所描述的示例性实施方式中，复用器装置902可操作为复用由相机192经由电缆904同时提供给复用器装置902的不同数据信号。在这个或另一个实施方式中，不同的数据信号可以包括与食物保鲜室122和/或冷冻室124相关联的数据。例如，不同的数据信号可以包括图像，例如被添加到食物保鲜室122和/或冷冻室124或从其中取出的一个或多个对象182的图像300、302和/或视频。在至少一个实施方式中，复用器装置902还可操作为输出具有与食物保鲜室122和/或冷冻室124相关联的不同数据信号和上述数据的复用信号。
如图9所描述的示例性实施方式示例，复用器装置902可以联接到控制器(例如微处理器、SBC)(诸如控制器156)和/或与其集成。在至少一个实施方式中，控制器156可以接收复用信号，该复用信号可以由复用器装置902输出，并且可以包括可以由如上所述的相机192同时捕获并提供给复用器装置902的不同数据信号。
在可选或另外实施方式中，图9所描述的示例性实施方式中示例的库存管理系统180可以包括可以联接到复用器装置902和控制器156的解复用器装置(图9中未示出)。例如，在这些可选或另外实施方式中，这种解复用器装置可以联接到控制器156和/或与其集成，并且还联接到复用器装置902，使得解复用器装置可以解复用由复用器装置902输出的复用信号。在这些可选或另外示例性实施方式中，在将复用信号解复用时，这种解复用器装置可输出不同数据信号且将其提供给控制器156。
在一个或多个实施方式中，至少部分地基于从复用器装置902接收到上述复用信号和/或不同数据信号，控制器156可执行一个或多个操作。例如，如上文参见图1至图8所述，在一些实施方式中，控制器156可以在本地分析复用信号、不同数据信号和/或其中的数据(例如，与食物保鲜室122和/或冷冻室124相关联的数据，其可以由相机192同时捕获)。例如，控制器156可以使用上述机器学习和/或AI模型、算法和/或图像识别过程(例如，CNN、R-CNN、DBN、DNN)中的一个或多个来分析与食物保鲜室122和/或冷冻室124相关联并且处于复用信号中的图像数据(例如，图像300、302中的数据)和/或视频数据。在该示例中，控制器156可以分析这样的图像和/或视频数据以监测和/或保持放置在食物保鲜室122和/或冷冻室124内、添加到其中和/或从其中取出的一个或多个对象182(例如，食品、饮料)的记录。
在一些实施方式中，控制器156可以利用外部通信系统170经由网络174将与食物保鲜室122和/或冷冻室124相关联的复用信号、不同数据信号和/或数据传送到外部装置172和/或远程服务器176。在一些实施方式中，控制器156可以促进相机192的调节，以调节这样的相机192的监测范围、监测区或视场。例如，在这些实施方式中，复用器装置902可联接到控制器156和/或与其集成，使得控制器156可联接(例如，电联接、通信联接、可操作地联接)到一个或多个电缆904，该电缆可联接到一个或多个相机192(例如，经由适配器906和相机电缆908)。在这些实施方式中，控制器156可以经由电缆904向相机192发送一个或多个控制信号，以例如促进对这样的相机192的上述调节和/或与这样的相机192相关联的另一操作(例如，通电、断电、调节相机设置)。
应当理解，MIPI相机在制冷电器100上和/或内的设置可受到与各个这种MIPI相机联接和/或关联的MIPI相机电缆的长度限制(例如，MIPI相机电缆的长度约为12英寸)。然而，还应当理解，通过如图9示例使用多个电缆904、适配器906和相机电缆908来将相机192联接到复用器装置902和控制器156，多个相机192可以各自设置在制冷电器100上和/或内的不同位置处(例如，在箱体102、食物保鲜室122、冷冻室124、冷藏门体128、冷冻门体130上和/或内)。这样，在一些实施方式中，可以使用例如控制器156的单个控制器来控制设置在制冷电器100上和/或内的这种各个位置处的这种多个相机192。在这些或其它实施方式中，可以使用单个图像信号处理器来处理由设置在制冷电器100上和/或内的这种各个位置处的这种多个相机192捕获(例如，同时捕获)的图像(例如，多个图像300、302)和/或视频，该图像信号处理器可以联接到控制器156和/或与其集成(例如，控制器可以包括和/或构成SBC)。在这些实施方式中，图9中示例的库存管理系统180可以从而减少与在制冷电器100中使用以同时获得和/或处理来自设置在制冷电器100上和/或内的各个位置处的多个相机192的与食物保鲜室122和/或冷冻室124相关联的数据的硬件和/或软件部件相关联的数量、设计复杂度、设计占地面积和/或成本。
图10示例了根据本发明的一个或多个示例性实施方式的库存管理系统180的框图。例如，图10中示例的库存管理系统180可以构成图9中示例和上文所述的库存管理系统180的示例性非限制性可选实施方式。
在图10中描述的示例性实施方式中，库存管理系统180和/或相机组件190(图10中未示出)可以包括可以联接到第一对相机192(例如，经由第一对相机电缆908，在图10和图11中表示为“第一对”)和第一电缆904(例如，经由第一适配器906) 的第一复用器装置902。在该示例性实施方式中，这种第一复用器装置902可操作为将第一复用信号输出到这种第一电缆904上，其中，第一复用信号可包括可由第一对相机192(例如，经由第一对相机电缆908，在图10和图11中表示为“第一对”)同时提供给第一复用器装置902的不同的第一数据信号。在该示例性实施方式中，不同的第一数据信号可以包括与食物保鲜室122和/或冷冻室124相关联的上述数据，这些数据可以由第一对相机192同时捕获(例如，经由图像、视频、音频)。
在图10中描述的示例性实施方式中，库存管理系统180和/或相机组件190(图10中未示出)还可以包括可以联接到第二对相机192(例如，经由第二对相机电缆908，在图10和图11中表示为“第二对”)和第二电缆904(例如，经由第二适配器906)的第二复用器装置902。在该示例性实施方式中，这种第二复用器装置902可操作为将第二复用信号输出到这种第二电缆904上，其中，第二复用信号可包括可由第二对相机192(例如，经由第二对相机电缆908，在图10和图11中表示为“第二对”)同时提供给第二复用器装置902的不同的第二数据信号。在该示例性实施方式中，不同的第二数据信号可以包括与食物保鲜室122和/或冷冻室124相关联的上述数据，这些数据可以由第二对相机192同时捕获(例如，经由图像、视频、音频)。
在图10中描述的示例性实施方式中，第一对相机192和第二对相机192可以同时(例如，同时地)捕获与食物保鲜室122和/或冷冻室124相关联的上述图像数据、视频数据和/或音频数据。在该示例性实施方式中，第一对相机192和第二对相机192可以分别经由不同的第一数据信号和不同的第二数据信号来同时(例如，同时地)分别向第一复用器装置902和第二复用器装置902提供这样的图像数据、视频数据和/或音频数据。
在图10中描述的示例性实施方式中，库存管理系统180和/或相机组件190(图10中未示出)还可以包括可以联接到第一电缆904和第二电缆904的解复用器装置1002。在该示例性实施方式中，解复用器装置1002可操作为将第一复用信号解复用为不同的第一数据信号，并且将第二复用信号解复用为不同的第二数据信号。在该示例性实施方式中，解复用器装置1002可以联接到控制器(例如控制器156)和/或与其集成。在该示例性实施方式中，解复用器装置1002可以在分别解复用第一复用信号和/或第二复用信号时向控制器156提供不同的第一数据信号和/或不同的第二数据信号。
在图10中描述的示例性实施方式中，至少部分地基于从解复用器装置1002接收到不同的第一数据信号和/或不同的第二数据信号，控制器156可以执行一个或多 个操作。例如，控制器156可以执行上面参见图1至图9中描述的示例性实施方式描述的一个或多个操作。例如，控制器156可以：本地分析(例如，经由CNN、R-CNN、DBN和/或DNN模型、算法和/或过程)图像300、302的图像数据和/或视频数据；监测和/或保持对象182的记录；将信号和/或数据传送(例如，经由外部通信系统170和网络174)到外部装置172和/或远程服务器176；调节相机192的监测范围、监测区或视场；和/或至少部分地基于从解复用器装置1002接收到不同的第一数据信号和/或不同的第二数据信号来执行另一操作。
图11示例了根据本发明的一个或多个示例性实施方式的库存管理系统180的框图。例如，图11中示例的库存管理系统180可以构成图10中示例和上文所述的库存管理系统180的示例性非限制性可选实施方式。
在图11中描述的示例性实施方式中，库存管理系统180和/或相机组件190(图11中未示出)可以包括可以联接到第一电缆904、第二电缆904和解复用器装置1002的第三复用器装置902。在该示例性实施方式中，第三复用器装置902可操作为对可以分别由第一复用器装置902和第二复用器装置902同时提供的第一复用信号和第二复用信号进行复用。在该示例性实施方式中，基于对第一复用信号和第二复用信号进行复用，第三复用器装置902可以输出第三复用信号，该第三复用信号可以包括上述的不同的第一数据信号和不同的第二数据信号。例如，第三复用器装置902可以输出第三复用信号，该第三复用信号可以包括可以分别由第一对相机192和第二对相机192同时提供的不同的第一数据信号和不同的第二数据信号，如上面参见图10描述的。
在一些实施方式中，第三复用器装置902和/或解复用器装置1002可联接到控制器(例如如图11示例的控制器156)和/或与其集成。在图11中描述的示例性实施方式中，第三复用器装置902可将第三复用信号提供给解复用器装置1002。在该示例性实施方式中，解复用器装置1002可操作为将第三复用信号解复用为：第一复用信号和第二复用信号；或者上述不同的第一数据信号和不同的第二数据信号。在一些实施方式中，第三复用器装置902可直接联接到控制器156，使得第三复用器装置902可将第三复用信号提供给控制器156。在图10中描述的示例性实施方式或另一实施方式中，至少部分地基于从解复用器装置1002和/或第三复用器装置902接收到第一复用信号、第二复用信号、第三复用信号、不同的第一数据信号和/或不同的第二数据信号，控制器156可以执行一个或多个操作。例如，控制器156可以执行上面参见图1至图9中描述的示例性实施方式描述的一个或多个操作。例如，控 制器156可以：本地分析(例如，经由CNN、R-CNN、DBN和/或DNN模型、算法和/或过程)图像300、302的图像数据和/或视频数据；监测和/或保持对象182的记录；将信号和/或数据传送(例如，经由外部通信系统170和网络174)到外部装置172和/或远程服务器176；调节相机192的监测范围、监测区或视场；和/或至少部分地基于从解复用器装置1002接收到不同的第一数据信号和/或不同的第二数据信号来执行另一操作。
图12示例了根据本发明的一个或多个示例性实施方式的操作上文描述并且在图2、图8和/或图9中示例的库存管理系统180的示例性非限制性方法1200的流程图。方法1200可以使用例如控制器156、库存管理系统180(例如，上文描述并且在图2、图8和/或图9中示例的库存管理系统180的实施方式)和/或相机组件190(例如，上文描述并且在图2、图8和/或图9中示例的相机组件190的实施方式)来实施。在一些实施方式中，方法1200可以构成在制冷电器(例如，制冷电器100)内实施库存管理的示例性方法，其中，制冷电器可以包括制冷间室(例如，食物保鲜室122、冷冻室124)和具有被设置成监测制冷间室的多个相机(例如，相机192)的相机组件(例如，相机组件190)。
为了说明和讨论的目的，图12中示例的示例性实施方式描述了以特定顺序执行的操作。使用本文提供的公开内容，本领域普通技术人员将理解，在不偏离本发明的范围的情况下，方法1200或本文公开的任何其他方法的各种操作或步骤可以适配、修改、重新排列、同时执行，包括未示例的操作和/或以各种方式修改。
在1202，方法1200可以包括：通过可操作地联接到相机组件(例如，相机组件190)的控制器(例如，控制器156)从联接到控制器的复用器装置(例如，复用器装置902)获得复用信号(例如，上文参见图9描述的复用信号)，该复用信号包括由多个相机经由联接到复用器装置和多个相机的多个电缆(例如，电缆904)同时提供给复用器装置的不同数据信号(例如，上文参见图9描述的不同数据信号)，该不同数据信号包括与制冷间室相关联的数据(例如，与设置在食物保鲜室122和/或冷冻室124内、添加到其中和/或从其中取出的一个或多个对象182相关联的数据)。
虽然在图12中未描述，但是在一些实施方式中，方法1200可以包括：由控制器获得指示与制冷间室相关联的门体打开的信号(例如，来自制冷电器100的运动传感器和/或门体开关的信号)。在这些实施方式中，方法1200还可以由控制器至少部分地基于接收到信号在门体打开时操作多个相机以同时捕获与制冷间室相关联的数据(例如，如上文参见图9描述的)。
在1204，方法1200可包括：由控制器至少部分地基于从复用器装置接收到复用信号而执行一个或多个操作(例如，控制器156可执行上文参见图1到图9中描述的示例性实施方式描述的一个或多个操作)。例如，尽管在图12中未描述，但是在一些实施方式中，方法1200可以包括：由控制器至少部分地基于来自复用器装置的复用信号的接收来实施机器学习图像识别过程(例如，CNN、R-CNN、DBN和/或DNN过程)以分析与制冷间室相关联的数据。
在另一示例中，尽管在图12中未描述，但是在一些实施方式中，方法1200可以包括：由控制器至少部分地基于来自复用器装置的复用信号的接收来分析与制冷间室相关联的数据。在该示例中，方法1200还可以包括：由控制器至少部分地基于与制冷间室相关联的数据的分析来保持放置在制冷间室内或从其中取出的食品的记录。
在另一示例中，尽管在图12中未描述，但是在一些实施方式中，方法1200可以包括：由控制器(例如，经由网络174)使用联接到控制器的外部通信系统(例如，外部通信系统170)将复用信号、不同的数据信号和/或与制冷间室相关联的数据提供到在制冷电器外部的一个或多个远程计算装置。
本书面描述将示例用于公开本发明(其中包括最佳实施例)，并且还使本领域技术人员能够实施本发明(其中包括制造和使用任意装置或系统并且执行所包含的任意方法)。本发明的可专利范围通过权利要求进行限定，并且可以包括本领域技术人员能够想到的其它的示例。如果这种其它的示例包括与权利要求的字面语言没有区别的结构元件，或者如果这种其它的示例包括与权利要求的字面语言没有实质区别的等同结构元件，则期望这种其它的示例落入权利要求的范围中。

Claims (20)

  1. 一种制冷电器，其特征在于，包括：
    箱体，所述箱体限定制冷间室；
    门体，所述门体可旋转地铰接到所述箱体，以提供选择性地到达所述制冷间室的途径；
    相机组件，所述相机组件联接到所述箱体并且可操作为监测所述制冷间室，所述相机组件包括：
    多个相机，所述多个相机联接到多个电缆，所述多个相机可操作为同时捕获与所述制冷间室相关联的数据，所述多个相机中的每个相机联接到所述多个电缆中的电缆；
    复用器装置，所述复用器装置联接到所述多个电缆，所述复用器装置可操作为复用由所述多个相机经由所述多个电缆同时提供给所述复用器装置的不同数据信号，并且输出包括所述不同数据信号的复用信号，所述不同数据信号包括与所述制冷间室相关联的所述数据；以及
    控制器，所述控制器联接到所述复用器装置，所述控制器被配置为至少部分基于所述复用信号的接收而执行一个或多个操作。
  2. 根据权利要求1所述的制冷电器，其特征在于，所述制冷电器还包括联接到所述控制器和所述复用器装置的解复用器装置，所述解复用器装置可操作为解复用所述复用信号并输出所述不同的数据信号。
  3. 根据权利要求1所述的制冷电器，其特征在于，所述电缆和所述多个电缆中的每个电缆包括同轴电缆，并且其中，所述多个相机中的每个相机包括移动行业处理器接口相机。
  4. 根据权利要求1所述的制冷电器，其特征在于，所述多个相机中的每个相机经由移动行业处理器接口相机电缆联接到所述多个电缆中的所述电缆。
  5. 根据权利要求1所述的制冷电器，其特征在于，所述多个相机中的每个相机均联接到所述箱体的不同部分，其中，所述多个相机中的每个相机均可操作为监测指定的监测区或范围，并且其中，所述多个相机中的每个相机均具有指向针对所述相机指定的监测区或范围的视场。
  6. 根据权利要求1所述的制冷电器，其特征在于，与所述制冷间室相关联的所述数据包括图像数据、视频数据或音频数据中的至少一种。
  7. 根据权利要求1所述的制冷电器，其特征在于，所述控制器被配置为使用机器学习图像识别过程来分析与所述制冷间室相关联的所述数据。
  8. 根据权利要求7所述的制冷电器，其特征在于，所述机器学习图像识别过程包括卷积神经网络、基于区域的卷积神经网络、深度信念网络或深度神经网络图像识别过程中的至少一个。
  9. 根据权利要求1所述的制冷电器，其特征在于，所述控制器被配置为至少部分地基于对与所述制冷间室相关联的所述数据的分析来保持放置在所述制冷间室内或从所述制冷间室取出的食品的记录。
  10. 根据权利要求1所述的制冷电器，其特征在于，所述制冷电器还包括联接到所述控制器的外部通信系统，所述外部通信系统可操作为与在所述制冷电器外部的一个或多个远程计算装置通信，其中，所述控制器被配置为使用所述外部通信系统将与所述制冷间室相关联的所述复用信号、所述不同数据信号或所述数据中的至少一个提供给所述一个或多个远程计算装置。
  11. 根据权利要求1所述的制冷电器，其特征在于，所述控制器被配置为：
    获得指示所述制冷电器的所述门体打开的信号；以及
    至少部分地基于所述信号的接收在所述门体打开时操作所述多个相机以同时捕获与所述制冷间室相关联的所述数据。
  12. 一种在制冷电器内实施库存管理的方法，其特征在于，所述制冷电器包括制冷间室和包括设置为监测所述制冷间室的多个相机的相机组件，所述方法包括：
    通过可操作地联接到所述相机组件的控制器从联接到所述控制器的复用器装置获得复用信号，所述复用信号包括由所述多个相机经由联接到所述复用器装置和所述多个相机的多个电缆同时提供给所述复用器装置的不同数据信号，所述不同数据信号包括与所述制冷间室相关联的数据；以及
    由所述控制器至少部分地基于从所述复用器装置接收到所述复用信号而执行一个或多个操作。
  13. 根据权利要求12所述的方法，其特征在于，所述方法还包括：
    由所述控制器至少部分地基于从所述复用器装置接收到所述复用信号来实施机器学习图像识别过程，以分析与所述制冷间室相关联的所述数据。
  14. 根据权利要求12所述的方法，其特征在于，所述方法还包括：
    由所述控制器至少部分地基于从所述复用器装置接收到所述复用信号来分析与所述制冷间室相关联的所述数据；以及
    由所述控制器至少部分地基于对与所述制冷间室相关联的所述数据的分析来保持放置在所述制冷间室内或从所述制冷间室取出的食品的记录。
  15. 根据权利要求12所述的方法，其特征在于，所述方法还包括：
    由所述控制器使用联接到所述控制器的外部通信系统将与所述制冷间室相关联的所述复用信号、所述不同数据信号或所述数据中的至少一个提供给在所述制冷电器外部的一个或多个远程计算装置。
  16. 根据权利要求12所述的方法，其特征在于，所述方法还包括：
    由所述控制器获得指示与所述制冷间室相关联的门体打开的信号；以及
    由所述控制器至少部分地基于所述信号的接收在所述门体打开时操作所述多个相机以同时捕获与所述制冷间室相关联的所述数据。
  17. 一种制冷电器，其特征在于，包括：
    箱体，所述箱体限定制冷间室；
    门体，所述门体可旋转地铰接到所述箱体，以提供选择性地到达所述制冷间室的途径；
    相机组件，所述相机组件联接到所述箱体并且可操作为监测所述制冷间室，所述相机组件包括：
    第一复用器装置，所述第一复用器装置联接到第一对相机和第一电缆，所述第一复用器装置可操作为将第一复用信号输出到所述第一电缆上，所述第一复用信号包括由所述第一对相机同时提供给所述第一复用器装置的不同的第一数据信号；
    第二复用器装置，所述第二复用器装置联接到第二对相机和第二电缆，所述第二复用器装置可操作为将第二复用信号输出到所述第二电缆上，所述第二复用信号包括由所述第二对相机同时提供给所述第二复用器装置的不同的第二数据信号；
    解复用器装置，所述解复用器装置联接到所述第一电缆和所述第二电缆，所述解复用器装置可操作为将所述第一复用信号解复用为所述不同的第一数据信号，并且将所述第二复用信号解复用为所述不同的第二数据信号；以及
    控制器，所述控制器联接到所述解复用器装置，所述控制器被配置为至少部分地基于接收到所述不同的第一数据信号或所述不同的第二数据信号中的至少一个来执行一个或多个操作。
  18. 根据权利要求17所述的制冷电器，其特征在于，所述第一电缆和所述第二 电缆中的每个电缆包括同轴电缆，其中，所述第一对相机和所述第二对相机中的每个相机包括移动行业处理器接口相机，并且其中，所述不同的第一数据信号和所述不同的第二数据信号分别由所述第一对相机和所述第二对相机同时分别提供给所述第一复用器装置和所述第二复用器装置。
  19. 根据权利要求17所述的制冷电器，其特征在于，所述制冷电器还包括第三复用器装置，所述第三复用器装置联接到所述第一电缆、所述第二电缆和所述解复用器装置，所述第三复用器装置可操作为复用分别由所述第一复用器装置和所述第二复用器装置同时提供的所述第一复用信号和所述第二复用信号，以输出包括所述不同的第一数据信号和所述不同的第二数据信号的第三复用信号，其中，所述不同的第一数据信号和所述不同的第二数据信号分别由所述第一对相机和所述第二对相机同时提供。
  20. 根据权利要求19所述的制冷电器，其特征在于，所述解复用器装置可操作为将所述第三复用信号解复用为以下信号中的至少一种：所述第一复用信号和所述第二复用信号；或者所述不同的第一数据信号和所述不同的第二数据信号。
PCT/CN2023/084379 2022-03-28 2023-03-28 制冷电器中的多相机视觉系统 WO2023185835A1 (zh)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
US17/705,713 US20230308611A1 (en) 2022-03-28 2022-03-28 Multi-camera vision system in a refrigerator appliance
US17/705,713 2022-03-28

Publications (1)

Publication Number Publication Date
WO2023185835A1 true WO2023185835A1 (zh) 2023-10-05

Family

ID=88096826

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2023/084379 WO2023185835A1 (zh) 2022-03-28 2023-03-28 制冷电器中的多相机视觉系统

Country Status (2)

Country Link
US (1) US20230308611A1 (zh)
WO (1) WO2023185835A1 (zh)

Citations (10)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN101589619A (zh) * 2007-11-20 2009-11-25 索尼株式会社 信息处理装置、信息处理方法、显示控制装置、显示控制方法和程序
US20140252091A1 (en) * 2013-03-07 2014-09-11 Canopy Tours Inc. Apparatus and Method for Customized Product Data Management
US20170039511A1 (en) * 2015-08-05 2017-02-09 Whirlpool Corporation Object recognition system for an appliance and method for managing household inventory of consumables
US20170041520A1 (en) * 2015-08-05 2017-02-09 Whirlpool Corporation Refrigerators having internal content cameras, and methods of operating the same
CN107923693A (zh) * 2015-09-01 2018-04-17 三星电子株式会社 冰箱
CN109631456A (zh) * 2017-10-06 2019-04-16 东芝生活电器株式会社 冰箱
CN110308830A (zh) * 2018-03-27 2019-10-08 辛纳普蒂克斯公司 用于显示设备多路复用和多路解复用的方法和系统
CN111051801A (zh) * 2017-08-30 2020-04-21 三星电子株式会社 冰箱及其控制方法
CN111043826A (zh) * 2018-10-11 2020-04-21 青岛海尔股份有限公司 冷藏冷冻装置及其控制方法
US20210166266A1 (en) * 2019-12-02 2021-06-03 Lg Electronics Inc. Artificially intelligent computing device and refrigerator control method using the same

Family Cites Families (12)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US7023913B1 (en) * 2000-06-14 2006-04-04 Monroe David A Digital security multimedia sensor
US6999613B2 (en) * 2001-12-28 2006-02-14 Koninklijke Philips Electronics N.V. Video monitoring and surveillance systems capable of handling asynchronously multiplexed video
US20090201992A1 (en) * 2005-10-07 2009-08-13 Jeong-Il Seo Method and apparatus for encoding and decoding hopping default view for multiple cameras system
US9380260B2 (en) * 2009-01-21 2016-06-28 Texas Instruments Incorporated Multichannel video port interface using no external memory
JP6313014B2 (ja) * 2013-08-27 2018-04-18 東芝ライフスタイル株式会社 カメラシステム、冷蔵庫
US10956856B2 (en) * 2015-01-23 2021-03-23 Samsung Electronics Co., Ltd. Object recognition for a storage structure
US10341606B2 (en) * 2017-05-24 2019-07-02 SA Photonics, Inc. Systems and method of transmitting information from monochrome sensors
US11353259B2 (en) * 2018-09-18 2022-06-07 Samsung Electronics Co., Ltd. Augmented-reality refrigerator and method of controlling thereof
WO2020222335A1 (ko) * 2019-05-02 2020-11-05 엘지전자 주식회사 인공 지능형 냉장 장치와 그 제어 방법
KR102234691B1 (ko) * 2019-08-09 2021-04-02 엘지전자 주식회사 인공 지능을 이용하여, 물품을 관리하는 냉장고 및 그의 동작 방법
US20220147042A1 (en) * 2020-09-18 2022-05-12 Guident Ltd. Near Real-Time Data and Video Streaming System for a Vehicle, Robot or Drone
WO2022215771A1 (ko) * 2021-04-07 2022-10-13 엘지전자 주식회사 냉장고

Patent Citations (10)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN101589619A (zh) * 2007-11-20 2009-11-25 索尼株式会社 信息处理装置、信息处理方法、显示控制装置、显示控制方法和程序
US20140252091A1 (en) * 2013-03-07 2014-09-11 Canopy Tours Inc. Apparatus and Method for Customized Product Data Management
US20170039511A1 (en) * 2015-08-05 2017-02-09 Whirlpool Corporation Object recognition system for an appliance and method for managing household inventory of consumables
US20170041520A1 (en) * 2015-08-05 2017-02-09 Whirlpool Corporation Refrigerators having internal content cameras, and methods of operating the same
CN107923693A (zh) * 2015-09-01 2018-04-17 三星电子株式会社 冰箱
CN111051801A (zh) * 2017-08-30 2020-04-21 三星电子株式会社 冰箱及其控制方法
CN109631456A (zh) * 2017-10-06 2019-04-16 东芝生活电器株式会社 冰箱
CN110308830A (zh) * 2018-03-27 2019-10-08 辛纳普蒂克斯公司 用于显示设备多路复用和多路解复用的方法和系统
CN111043826A (zh) * 2018-10-11 2020-04-21 青岛海尔股份有限公司 冷藏冷冻装置及其控制方法
US20210166266A1 (en) * 2019-12-02 2021-06-03 Lg Electronics Inc. Artificially intelligent computing device and refrigerator control method using the same

Also Published As

Publication number Publication date
US20230308611A1 (en) 2023-09-28

Similar Documents

Publication Publication Date Title
US20170286760A1 (en) Method and system of temporal segmentation for gesture analysis
WO2014181323A1 (en) System and method of retail image analysis
US20220015572A1 (en) Method and system for foodstuff identification
WO2022105718A1 (zh) 制冷电器的库存管理系统
US20220325946A1 (en) Selective image capture using a plurality of cameras in a refrigerator appliance
CN107527363B (zh) 一种冷藏装置存储物管理系统和冷藏装置
WO2023185835A1 (zh) 制冷电器中的多相机视觉系统
WO2021091481A1 (en) System for object identification and content quantity estimation through use of thermal and visible spectrum images
CN107527060B (zh) 一种冷藏装置存储物管理系统和冷藏装置
US11796250B1 (en) Multi-camera vision system facilitating detection of door position using audio data
WO2023036099A1 (zh) 制冷电器中的库存管理系统
JP7113281B2 (ja) 冷蔵庫
CN107526991B (zh) 一种冷藏装置存储物管理系统和冷藏装置
US20220414391A1 (en) Inventory management system in a refrigerator appliance
US20240111897A1 (en) Multi-camera vision system facilitating authentication and secure data transfer
WO2023151694A1 (zh) 具有智能门体警报的制冷电器
WO2023046040A1 (zh) 制冷电器中的库存管理系统
WO2023131202A1 (zh) 带有智能抽屉的制冷电器
US20240056555A1 (en) Adjustable camera assembly in a refrigerator appliance
WO2024022401A1 (zh) 用于制冷电器的智能可调节层架
WO2022262683A1 (zh) 制冷电器中的库存管理系统
WO2023020499A1 (zh) 制冷设备
US20240068742A1 (en) Gasket leak detection in a refrigerator appliance
WO2023020498A1 (zh) 具有并列相机的制冷设备
CN113465268B (zh) 一种冰箱及食材识别方法

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 23778200

Country of ref document: EP

Kind code of ref document: A1