US20160138859A1

US20160138859A1 - Refrigeration appliance system

Info

Publication number: US20160138859A1
Application number: US14/898,187
Authority: US
Inventors: Peter Stimpfig; Gerhard Wetzl; Wolfgang Schramm; Jens Wichert
Original assignee: BSH Hausgeraete GmbH
Current assignee: BSH Hausgeraete GmbH
Priority date: 2013-06-14
Filing date: 2014-06-04
Publication date: 2016-05-19
Also published as: CN105358925A; DE102013211096A1; EP3008413A1; WO2014198610A1

Abstract

A refrigeration appliance system includes a refrigeration appliance having a camera module for acquiring or detecting image data of an item to be refrigerated in a cold or refrigeration compartment of the refrigeration appliance and a wireless camera-module operating device for operating the camera module.

Description

The present invention relates to a refrigeration appliance system which comprises a refrigeration appliance having a camera module.
When a refrigeration appliance has a camera module and a fixed display, recorded image data can be displayed only locally on the refrigeration appliance. The refrigeration appliance is controlled by means of separately provided switches or controllers.
The object of the invention is to define a refrigeration appliance system that is easier to operate.
This object is achieved by the subject matter having the features claimed in the independent claim. The subject matter of the figures, description and dependent claims contains advantageous embodiments of the invention.
The object is achieved according to one aspect of the invention by a refrigeration appliance system which comprises a refrigeration appliance having a camera module for acquiring image data about refrigerated goods in a cold compartment of the refrigeration appliance and comprises a wireless camera-module operating device for operating the camera module. This achieves the technical advantage, for example, that the camera module is easier to control.
A refrigeration appliance is understood to include in particular a domestic refrigeration appliance, i.e. a refrigeration appliance which is used for domestic tasks in households or in the catering sector, and in particular is used for storing food and/or beverages at specific temperatures, for instance an appliance such as a refrigerator, a freezer, a combined fridge-freezer, a chest freezer or a wine cooler.
In an advantageous embodiment of the refrigeration appliance system, the camera-module operating device comprises a touch-sensitive display for controlling the camera module and displaying the acquired image data. This achieves the technical advantage, for example, that the camera modules can be controlled easily without buttons.
In another advantageous embodiment of the refrigeration appliance system, the refrigeration appliance comprises a plurality of camera modules, and the camera-module operating device comprises means for selecting a camera module. This achieves the technical advantage, for example, that a specific camera module can be selected for control and display.
In another advantageous embodiment of the refrigeration appliance system, the camera modules are arranged for the purpose of generating an all-round view of the refrigerated goods, and the camera-module operating device is designed to control a viewing angle of the all-round view. This achieves the technical advantage, for example, that the refrigerated goods can be displayed on the display device from different sides.
In another advantageous embodiment of the refrigeration appliance system, the refrigeration appliance comprises a first camera module for acquiring first image data about the refrigerated goods from a first perspective, and a second camera module for acquiring second image data about the refrigerated goods from a second perspective. This achieves the technical advantage, for example, that the refrigerated goods can be recorded and displayed from different sides.
In another advantageous embodiment of the refrigeration appliance system, the first camera module is arranged in a door of the refrigeration appliance, and the second camera module is arranged in a compartment ceiling. This achieves the technical advantage, for example, that the perspectives obtained thereby complement each other well, and it is possible to obtain extensive information about the refrigerated goods.
In another embodiment of the refrigeration appliance system, the camera-module operating device is designed to switch between displaying the first perspective and the second perspective on the camera-module operating device being tilted. This achieves the technical advantage, for example, that it is easier to make a change in perspective.
In another advantageous embodiment of the refrigeration appliance system, the refrigeration appliance comprises a camera array comprising a multiplicity of camera modules for generating a view composed from individual images. This achieves the technical advantage, for example, that the camera modules can be generated on the basis of a panoramic view.
In another advantageous embodiment of the refrigeration appliance system, the camera array is arranged in a door of the refrigeration appliance. This achieves the technical advantage, for example, that the panoramic view is obtained from the same perspective from which a user would also perceive the contents of the refrigeration appliance with the door open.
In another advantageous embodiment of the refrigeration appliance system, the refrigeration appliance comprises a position-detection device for detecting a position of the camera-module operating device placed on an outer surface of the refrigeration appliance. This achieves the technical advantage, for example, that the refrigeration appliance can use the position to perform further control actions, for instance for a lighting device.
In another advantageous embodiment of the refrigeration appliance system, the camera-module operating device is designed to display an image region associated with the detected position of the camera-module operating device. This achieves the technical advantage, for example, that according to the detected position, those images that correspond to the position are displayed on the display device. This allows the contents of the refrigeration appliance to be displayed in an X-ray-like manner.
In another advantageous embodiment of the refrigeration appliance system, the refrigeration appliance comprises a lighting device for illuminating the refrigerated goods in the cold compartment. This achieves the technical advantage, for example, that the refrigerated goods can be recorded under defined light conditions.
In another advantageous embodiment of the refrigeration appliance system, the lighting device can be controlled by the camera-module operating device. This achieves the technical advantage, for example, of implementing straightforward and switch-free control of the lighting device.
In another advantageous embodiment of the refrigeration appliance system, the camera-module operating device is designed to compare the acquired image data about the refrigerated goods with stored image data about the refrigerated goods. This achieves the technical advantage, for example, of being able to show differences in the cold compartment.
In another advantageous embodiment of the refrigeration appliance system, the camera-module operating device is designed to identify the refrigerated goods from the acquired image data. This achieves the technical advantage, for example, that a list of contents of the refrigeration appliance can be generated or the type of the refrigerated goods can be determined.

Exemplary embodiments of the invention are described in more detail below and illustrated in the drawings, in which:

FIG. 1 is a schematic view of a refrigeration appliance;

FIG. 2A is a perspective view of a cold compartment having two camera modules;

FIG. 2B is a front view of the cold compartment;

FIG. 3A is a perspective view of a cold compartment having one camera module;

FIG. 3B is a plan view of the cold compartment;

FIG. 4 is a perspective view of a cold compartment having a plurality of camera modules;

FIG. 5 shows two views of different camera modules;

FIG. 6 is a plan view and a side view of the cold compartment in conjunction with an angle of view;

FIG. 7 is a plan view of the cold compartment in which a plurality of camera modules are used;

FIG. 8 is a perspective view of a cold compartment having a sensor mat;

FIG. 9 is a view of a refrigeration appliance having a data bus;

FIG. 10 is another view of a refrigeration appliance having a data bus;

FIG. 10A is a schematic view of an arrangement of the camera modules in a door;

FIG. 11 is a schematic view of a data bus having distribution nodes;

FIG. 12 is a view of a processor unit;

FIG. 13 shows images of refrigerated goods in the cold compartment and shows an image analysis;

FIG. 14 is a view of a camera-module operating device having a touch-sensitive display;

FIG. 15 is a schematic view of a refrigeration appliance having a camera module in a door; and

FIG. 16A-C are schematic views of different connections of camera modules.

FIG. 1 shows a refrigerator as an example of a refrigeration appliance 100 comprising a top and bottom refrigerator door 111. The refrigeration appliance 100 is a stand-alone appliance.
The refrigerator is used, for example, for chilling food and comprises a refrigerant circuit containing an evaporator, a compressor, a condenser and a flow regulator. The evaporator is a heat exchanger in which the liquid refrigerant, after expansion, is evaporated by absorbing heat from the medium to be cooled, i.e. from the air inside the refrigerator.
The compressor is a mechanically driven component which extracts the refrigerant vapor from the evaporator and discharges said vapor at a raised pressure to the condenser. The condenser is a heat exchanger in which the vaporized refrigerant, after compression, becomes a liquid by releasing heat to an external cooling medium, i.e. to the surrounding air. The flow regulator is a device designed to constantly reduce the pressure by narrowing the cross section.
The refrigerant is a fluid which is used for heat transfer in the refrigeration system and which absorbs heat when at a low temperature and low pressure, and releases heat when at a higher temperature and higher pressure, usually involving a change in state of the fluid.
The refrigeration appliance 100 comprises the top and bottom door 111, which closes a cold compartment inside the refrigeration appliance 100. The refrigeration appliance 100 also comprises a side wall 109. Inside the refrigeration appliance 100 is an electronic camera module, which is able to record the refrigerated goods inside the refrigeration appliance 100 using a single image, and to generate image data. The camera module 101 may be able to be directed, pivoted or adjusted by the customer, for example.
FIG. 2A shows a perspective view of the cold compartment 105 having two camera modules 101. The cold compartment 105 forms a receptacle for the refrigerated goods 103 inside the refrigeration appliance 100. A beaker and a food bag are inside the cold compartment 105 as examples of the refrigerated goods 103. The two camera modules 101 are arranged at the same height and spaced horizontally apart from one another on a front face 131 of the cold compartment 105, for example in the door 111 of the refrigeration appliance 100. The refrigerated goods 103 are recorded by the two camera modules 101 from the front face of the refrigeration appliance 100. The camera modules 101 on the front face 131 of the cold compartment 105 make it possible to emulate the view of a user of the contents of the cold compartment 105.
A lighting device 102 is provided for illuminating the cold compartment 105 during a recording. For example the lighting device 102 is an infrared light or illumination by visible light. The lighting device 102 is arranged in a compartment ceiling of the cold compartment 105 for instance. One and the same image can be recorded using different illumination for additional image information or higher resolution.
The design of the interior of the cold compartment 105 can be optimized with respect to the camera module 105, for example in terms of adjustability, a compartment floor arrangement, the position of the door racks relative to the compartment floors or adjustability of the door racks. The camera modules 101 are, for example, inclined towards the center of the refrigeration appliance. The camera modules 101 can be movable. A camera module 101 in a side wall has the advantage of gaining a view towards the door 111. In addition, reflective coatings, glass beads, mini retro-reflectors can be used.
A keep-fresh compartment does not have a dedicated camera, and has a transparent lid, transparent compartment floor and no opaque strip at the compartment floor edge.
The camera modules 101 comprise a light sensor array containing color sensors, for instance a CCD chip. The light sensor array has, for example, a minimum resolution of 640×480 pixels (VGA) and three color channels each with a color depth of at least 8 bits, for example RGB color channels. The camera module 101 can be designed, for example, such that it records not only image data in the visible region but additionally also image data in the infrared region.
CCD image sensors suitable for use in the light sensor array consist of an array of light-sensitive photodiodes. These may be rectangular, square or polygonal with edge lengths of 1.4 μm to more than 20 μm. The larger the surface area of the photodiodes, the higher the light sensitivity and the dynamic range of the CCD image sensor. For the same sensor size, however, the image resolution is lower in this case. The camera modules 101 have an interface for communicating with a processor unit. The sensor data obtained by the light sensor array can be processed by the processor unit. The power of the system is of the order of 3 W when active, which is only on demand, and 0.1 W in standby.
FIG. 2B shows a front view of the cold compartment 105 from the perspective of one of the camera modules 101. In the front view, the outlines of the refrigerated goods 103 can be seen in front of the background of a rear wall 129 of the cold compartment 105.
By using two camera modules 101 spaced horizontally apart, it is possible, for example, to record two images of the refrigerated goods 103, which in a stereoscopic view convey to the user a three-dimensional image of the refrigerated goods 103. In addition, using two camera modules 101 results in the advantage that even if one of the camera modules 101 should be obstructed by refrigerated goods 103, the interior of the cold compartment 105 can be recorded by the other of the camera modules 101.
FIG. 3A shows another perspective view of the cold compartment 105 having one camera module 101. The camera module 101 is arranged in the compartment ceiling 113 of the cold compartment 105. Arranging the camera module 101 in the compartment ceiling 113 allows a fundamental change in the direction of view and an overhead view onto the contents of the cold compartment 105.
FIG. 3B shows an overhead view onto the cold compartment 105 from the perspective of the camera module 103 arranged in the compartment ceiling 113. This overhead view can be used for visualizing the contents of the cold compartment 105.
As a general rule, the camera module can also be arranged on the rear wall 129 or on a side wall 109 of the cold compartment 105. It is hence possible to enlarge the field of vision for visualizing the contents and to record different perspectives inside the cold compartment 105 or an all-round view.
FIG. 4 shows a perspective view of a cold compartment 105 having a plurality of camera modules 101. The camera modules 101 are arranged such that the refrigerated goods 103 can be recorded from every side and under different viewing angles. By combining the images recorded by the individual camera modules 101, it is possible to generate an all-round view of the refrigerated goods 103 in the cold compartment 105.
In order for the camera module 101 to be able to record the refrigerated goods 103 with sufficient brightness, a lighting device 102 is implemented inside the cold compartment 105. The camera module 101 and the lighting device 102 can be arranged in a shared installation space, for example.
A flash light unit for emitting a flash of light during a recording is arranged inside the refrigeration appliance 100 as the lighting device 102 for instance, so that a recording can be obtained using flash light. The flash light unit is arranged on the camera module 101 for example. The flash light unit can be actuated by an additional controller, which outputs a control signal for each camera module 101. The light intensity of the flash can be measured using a special flash-light meter or a dual-function exposure meter.
In addition, the refrigeration appliance 100 can comprise an LED array as the lighting device 102, which is arranged such that the refrigerated goods 103 in the cold compartment 105 are illuminated as uniformly as possible from all sides. The LED array can be controlled centrally. In addition, the LED array provides a simple and energy-saving way of implementing uniform or adaptive illumination of the cold compartment 105 and of reducing cast shadows.
As a general rule, however, any other suitable lighting devices can also be used, such as LED strips, incandescent lamps or neon tubes for instance. When the refrigeration appliance door is open, ambient light can be used as the interior lighting.
FIG. 5 shows two different camera modules 101-1 and 101-2. Two types of camera modules 101-1 and 101-2, which are suitable for fitting in the refrigeration appliance 100, are available for image acquisition. The assembled camera modules 101-1 and 101-2 for instance comprise a light sensor array, a lens and a microcontroller 104.
Camera modules 101-1 and 101-2 having built-in microcontroller 104 can be integrated easily into the refrigerated appliance 100. Integration is also made simpler if there is a driver for the particular camera module 101-1 and 101-2. If the camera modules 101-1 and 101-2 comprise a USB interface, the camera modules 101-1 and 101-2 can be connected directly to a USB data bus or a PC. The camera modules 101-1 and 101-2 comprise an imager, which provides the raw image data for analysis.
The camera modules 101-1 and 101-2 are protected against fruit acid, dust and moisture. The camera modules 101-1 and 101-2 can be arranged in an encapsulated housing and/or provided with lens heating by an IR-LED and/or with a lens coating in order to prevent fogging.
The first camera module 101-1 comprises a pin-hole lens 135 and a circuit board 133, on which are arranged a chip comprising a light sensor array and a microcontroller 104. The camera module 101-1 is formed by a pin-hole camera 107, which can have a glass cover to protect against soiling. Although the light intensity of the pin-hole camera 107 is less than that of focusing cameras, the pin-hole camera 107 has a greater depth of field. The lower light intensity, however, can be compensated by a longer exposure time, because no movements occur inside the cold compartment 105.
The second camera module 101-2 comprises a lens 109 and a circuit board 133, on which are arranged a chip comprising a light sensor array and a microcontroller. The lens 109 may have a focal length of less than 10 mm (miniature lens), for example, so that the lens 109 also produces a large depth of field and a near-field region can be imaged. The lens 109 is a wide-angle lens, for instance, with an angle of view of greater than 60°. The angle of view defines the angle range that can be captured by the camera module 101-2. In addition, the camera modules 101-1 and 101-2 can comprise additional deflection mechanisms, for instance devices such as hinged or rotating prisms, mirrors, lenses or semiconductor mirrors.
The dimensions of the circuit board 133 are 32 mm×32 mm for example. On the circuit board 133 are formed, for example, electrical circuits for the microcontroller 104 and an interface for connecting the camera module 101 to a data bus so that image data can be provided as a file. The circuit board 133 is formed in the foam in the door 111, for example.
The height of the camera module 101-1 and 101-2 is defined by the lens 109 or the pin-hole lens 135 and lies between 10 mm and 30 mm for instance. Pin-hole lenses 135 allow a lower depth of the camera module 101 of less than 20 mm. Furthermore, a camera module 101-1 having a pin-hole lens 135 requires a smaller chamber or recess for fitting in a wall of the refrigeration appliance 100. For a camera module 101-1 having a pin-hole lens 135, the depth of the recess equals 2 mm to 3 mm, for example, whereas for a camera module 101-2 having a lens 109, the depth of the recess equals about 10 mm. The camera module 101-1 having the pin-hole lens 135 can also be implemented without circuit board 133 and can have dimensions of 10 mm×10 mm×7 mm, for example. A pin-hole camera 107 having a pin-hole lens 135 can permit an angle of view of 120°, for example.
The camera modules 101-1 and 101-2 can have different interfaces for transmitting image data, for instance interfaces such as an RS232 interface, an SPI interface, an IIC interface or a USB interface.
FIG. 6 shows a plan view and a side view of the cold compartment 105 in conjunction with different angles of view for a cold compartment 105 of size 20 cm×60 cm×50 cm. The angle of view defines the angle range that can be imaged by the camera module 101. Different viewable regions for the camera module 101 are obtained with different angles of view. There are individual areas 137 to the sides of the camera module 101 in the cold compartment 105 that are not visible and that become smaller as the angle of view gets bigger.
FIG. 7 shows a plan view of the cold compartment 105 in which a plurality of camera modules 105 are used. An arrangement having four camera modules 101 with an angle of view 40° is shown on the left. An arrangement having three camera modules 101 with an angle of view 60° is shown on the right by way of comparison. With a larger angle of view, fewer camera modules 101 can be used to cover an identical viewable region 139.
The angle of view determines the number of camera modules 101 to be able to generate a continuous image. A compromise can be found between the angle of view and the number of camera modules 101 depending on the size of the cold compartment 105. A larger angle of view, however, produces greater distortion. For instance a cushion-shaped distortion 141 or a barrel-shaped distortion 143 is produced. The nature and size of the distortion 141 and 143 can be identified and corrected for further image processing. A pattern can be used for this purpose.
FIG. 8 shows a cold compartment 105 having a sensor mat 115 for detecting a change in a load on a compartment floor 117. The membrane-like sensor mat 115 can be used for detecting weight changes. The sensor mat 115 is a few millimeters thick and is placed on the compartment floor 117. The sensor mat 115 measures the change in loads on the horizontal, flat and fixed compartment floor 117 and is suitable for monitoring free-standing and stacked refrigerated goods 103 on the compartment floor 117. This can be used, for example, for determining how full containers are by comparing the weight with an initial value. In addition, it is also possible to track changes in position of the refrigerated goods 103 and therefore to detect the position.
Moreover, the position of newly-inserted refrigerated goods 103-2 can be detected. If, for example, the refrigerated goods 103-2 are added to the refrigerated goods 103-1 on the compartment floor 117, the sensor mat 117 detects the surface region on which the refrigerated goods 103-2 have been added. Changes in the cold compartment 105 can thereby be detected in addition to a visual change in the image data acquired by the camera module 101.
In addition, the refrigeration appliance 100 can comprise a reader for radio tags (RFID tags), so that the refrigerated goods 103 in the cold compartment 105 can be registered by reading the tags. The reader for radio tags is an electronic circuit which by means of radio waves is able to read the information stored in electronic tags.
Both the radio-tag reader and the sensor mat 115 can be provided as additional sensors to complement optical image acquisition, for instance as an alternative or addition to the optical recognition of an EAN code. This increases reliability and opens up further applications.
In addition, the camera module 101 can comprise a controllable pivoting device for turning a recording direction of the camera module 101 and/or a controllable zooming device for magnifying a recording by the camera module 101. Furthermore, the camera module 101 can comprise a displacement device for controllable horizontal and/or vertical movement of the camera module 101 along a wall of the refrigeration appliance 100. The pivoting device, the displacement device and the zooming device can be implemented using a suitable mechanism driven by an electric motor and can be controlled by a processor unit via a data bus.
FIG. 9 shows a refrigeration appliance 100 having a data bus 121. The cold compartment 105 comprises four camera modules 101, which are arranged in various corners of the cold compartment 105. In addition, the refrigeration appliance 100 comprises a processor unit 119, to which the images are transmitted from the individual camera modules 101. The processor unit 119 comprises a processor and a memory and is used for processing the transmitted image data.
A data bus 121 is used for transmitting the image data from the camera module 101 to the processor unit 119. A data bus is a system for data transmission between a plurality of bus users over a shared transmission path, with the bus users not being involved in the data transmission between other bus users.
Each of the four camera modules 101 is connected to a separate data bus 121. Further devices can be connected to these data buses 121 and controlled. Part of the data bus 121 is integrated in the side wall 109 of the refrigeration appliance 100. The data bus 121 is formed by electrical conductors that transmit signals from the camera modules 101 to the processor unit 119.
The data bus 121 is a serial bus, for instance, which is used to transmit the individual bits or data packets in a serial data transmission sequentially (serially) over the conductor. The data bus 121 implements, for example, a USB standard, an I2C standard, an SPI standard or a CAN standard, or, for instance, a DBus2 standard or DBus3 standard. The data bus 121 is a high-speed bus (USB) for example. A serial data bus 121 has the advantage that it can be implemented using a small number of conductors.
The camera modules 101 and the processor unit 119 likewise have a serial interface for connecting the data bus 121. It is advantageous to use camera modules 101 and systems for which drivers already exist. For this purpose, the data bus 121 can be pre-fitted with connecting devices such as plug-fit connectors, for example, for connecting the camera modules 101. In this case, the camera module 101 can be changed and replaced easily if it has a fault. In additional the data bus 121 and connected components can be designed such that the components, for instance camera modules 101 or lighting devices 102, are supplied with electrical power or current via the data bus 121. This reduces the cabling costs inside the refrigeration appliance 100.
FIG. 10 shows another refrigeration appliance 100 having the data bus 121. The cold compartment 105 comprises four camera modules 101, which are arranged in the respective corners of the cold compartment 105. The four camera modules 101 are connected in pairs to the same data bus 121. Two of the camera modules 101 are arranged in a door 111 of the refrigeration appliance 100 and generate a front view of the cold compartment 105. The data bus 121 to these camera modules 101 is taken through a hinge 123 of the door 111 via a cable grommet.
A logical or physical address is assigned to the respective camera modules 101 and can be used to address the camera modules 101 on the data bus 121 individually. In addition, if required additional local distribution nodes can be provided. An image retrieval by the processor unit 119 via the data bus 121 should not take longer than a defined time period.
FIG. 10A shows a schematic view and a cross-sectional view of an arrangement of six camera modules 101 on the inside of a door 111. The door 111 has a door handle 169 on the outside. The camera modules 101 acquire the image data from an identical region inside a cold compartment 105 from different viewing angles. Two of the camera modules 101 at a time are grouped into pairs comprising a first and a second camera module 101 for stereoscopic image recording.
Two of the camera modules 101 at a time are arranged as a pair at the same height in the door 111. A first camera module 101 of the pair is arranged on a right-hand door pillar 167, and a second camera module 101 is arranged on a left-hand door pillar 167. The camera modules 101 are arranged in a corner of the door pillars 167 on the inside of the door 111. For example, the camera modules 101 are integrated in a frame of the door 111 so that they do not protrude out of the inside of the door 111.
Each camera module 101 is assigned a lighting device 102, for example in the form of a white LED in the door 111. Each lighting device 102 is arranged diagonally beside the associated camera module 101.
In the center of the door 111 is formed a door rack 165 into which refrigerated goods 103 such as bottles, for example, can be placed. The door rack 165 is not height-adjustable. Two of the camera modules 101 are located directly beneath the door rack 165, so that they cannot be obstructed by the refrigerated goods 103 inside the refrigeration appliance 100. Two further camera modules 101 are arranged in an upper region of the door 111 so that they too cannot be obstructed by refrigerated goods, and image recording can be performed from diagonally above.
The pairs of camera modules 101 are arranged at different heights on the inside of the door 111 so that the recording can be obtained at different viewing angles. This is done, for example, by each pair of camera modules 101 capturing an identical region inside a cold compartment 105 from different viewing angles.
The door 111 comprises an integral data bus 121 for connecting the camera modules 101 to a processor unit 119 so that the image data obtained by the camera modules 101 can be transmitted for further processing. All six camera modules 101 are connected to the same data bus 121, for example. This data bus 121 can be designed as a serial data bus so that the cabling costs for transmitting the image data can be low. The first and second camera module 101 of each pair can each comprise a zooming device for synchronous magnification of the stereoscopic image recording. In general, the camera modules 101 can also be arranged in different positions inside the refrigerated appliance, for instance on a rear wall of the refrigerated appliance 100.
FIG. 11 shows a schematic view of a data bus 121 in a refrigeration appliance 100 having a plurality of distribution nodes 125. The distribution nodes 125 each form an auxiliary module for connecting the camera modules 101 and comprise suitable interfaces. The distribution modules 125 are implemented, for example, by an embedded platform such as a BeagleBoard for instance.
The distribution nodes 125 are used for distributing image data and control signals between the processor unit 119 and the camera modules 101, and can form image conversion units. In addition, data compression can be performed in the distribution node 125. Furthermore, protocol conversion can also take place in the distribution nodes 125.
The camera modules 101 connected to the data bus 121 can comprise a microcontroller, which performs compression of the image data obtained. A compression facility for compressing the image data can be implemented in this way. This compression involves converting the image data obtained in bitmap format for example into image data based on the JPEG format and transmitting said image data in JPEG format via the data bus 121. This results in the advantage that the transmission is performed with less data and takes a shorter time period.
FIG. 12 shows a perspective view of the processor unit 119. The processor unit 119 comprises an interface 145 for transmitting the image data to an external network. For example, the interface 145 is a wireless WLAN interface, a Bluetooth interface, an Ethernet interface or a mobile communications interface such as GSM, UMTS or LTE. The image data can be supplied to an external network via the interface 145. Where required, a direct connection to a personal computer is possible via the interface 145. The personal computer can be used to process the sensor data, for example to remove distortions and noise, perform artificial sharpening or to increase contrast.
The processor unit 119 is used for processing the sensor data and image data from the camera modules 101. A program library, such as OpenVC for instance, containing algorithms for image processing and computer vision can be used for image analysis. Implementable image analysis methods are, for example, detecting a change, identifying a barcode or recognizing defined logos (pattern recognition). Linux can be used as the operating system for the processor unit 119.
The processor unit 119 can be used to retrieve the image data from the camera modules 101 and process the image data transmitted in this way. Image processing, for example suppressing noise or assembling the image data into larger units, can be used in this context. The image data can also be analyzed, for example difference images or difference objects can be detected. This analysis can be used to identify refrigerated goods 103 recorded by the camera modules 101 and to obtain additional information relating to these goods. The image data can be provided via a wireless network.
In addition, the processor unit 119 can control a lighting device 102 for illuminating the cold compartment 105 during a recording. For this purpose, the refrigeration appliance 100 comprises a lighting device 102, which is connected to the data bus 121 and can be controlled via this data bus.
FIG. 13 shows a first image recorded by the camera module 101 composed of first image data 147, and a second image recorded by the camera module 101 composed of second image data 149. The second image data 149 is recorded at a later point in time than the first image data 147. The processor unit 119 or an external camera-module operating device on which an application (app) is running can be used to implement a freshness-determination facility, which serves to determine a condition of freshness of the refrigerated goods on the basis of the first image data 147 and the second image data 149. The time interval between the first and the second image data 147 and 149 equals one day, for example.
In both cases, the lighting device 102 provides substantially identical lighting conditions in the cold compartment 105 in order to ensure comparability of the time-offset recordings. To prevent the image data 147 and 149 being falsified by ambient light entering the cold compartment 105 on opening the door 111 of the refrigeration appliance 100, the refrigeration appliance 100 comprises a sensor for detecting opening of the door 111. If the sensor detects that the door 111 of the refrigeration appliance 100 is open, no image data is acquired. The image data is only acquired once the door is closed 111. The image data is recorded after the door 111 is closed once uniform illumination of the interior of the cold compartment 105 has taken place.
It is also possible by comparing the first image data and the second image data to establish that refrigerated goods 103 have been added or removed. The image data 151 shows the subtraction of the second image data 149 from the first image data 147. The first image data 147 and the second image data 149 are, for example, three-color image data (RGB) with a color resolution of 24 bits per pixel.
The difference and the change between both sets of image data 147 and 149 can be calculated by the subtraction. In the example shown, the image region 153 displays the refrigerated goods 103 that have been removed from the cold compartment 105. Showing or hiding the contents of the cold compartment 105 and the calculated change can be performed on a display of the refrigeration appliance 100, for example on a tablet PC or a smartphone. In addition, an outline of the refrigerated goods 103 and the change brought about can be highlighted.
It is likewise possible to determine a change over time in the refrigerated goods 103 that are perishable. A freshness-determination facility formed by the processor unit 119 or the camera-module operating device by means of an application determines the condition of freshness of the refrigerated goods 103 on the basis of a color difference between the first image data 147 and the second image data 149. Then a suitable algorithm can be used to determine from a change in color that has occurred whether the refrigerated goods 103 are still fresh or usable. It is also possible to use the weight-change data acquired by the sensor mat 115 to determine freshness.
In addition, the freshness-determination facility may be designed to detect the type of the refrigerated goods 103 on the basis of the first image data or the second image data. The freshness-determination facility may, for example, automatically identify what sort of perishable goods the refrigerated goods 103 constitute. For instance the freshness-determination facility is able to identify by image analysis whether the refrigerated goods 103 in the cold compartment 105 consist of bananas or tomatoes. An image analysis algorithm compares the recording from the camera module 101 with pre-saved image data and thereby identifies the refrigerated goods 103.
Should automatic identification of the type of the refrigerated goods 103 fail, the refrigeration appliance 100 can comprise an input apparatus for manual input of the type of the refrigerated goods 103. This is formed, for example, by a touch-sensitive display or a display device on which different types of refrigerated goods 103 are displayed in menu form. A user selects a particular item of refrigerated goods, which the user has placed in the cold component 105, by touching the corresponding menu option. From the entered or detected type of the refrigerated goods 103, it is possible to determine more precisely the condition of freshness of the refrigerated goods 103 by means of a color difference.
A special camera module 101, which determines the image data not only in the visible spectrum region but simultaneously also in the infrared spectral region, can be used for detecting a condition of freshness. The camera module 101 can comprise a dichroic element, which separates a beam path of the infrared light from the beam path of the visible light.
The dichroic element is an optical prism, for example, which splits a light beam into two beams of different spectra or colors. The dichroic element is made of a glass, for instance, certain surfaces of which are provided with dichroic mirrors, which reflect or transmit the light depending on its wavelength.
The visible light is detected by a first light sensor array, and the separated infrared light by a second light sensor array. The required design generally determines the number, arrangement and type of the camera modules 101 and of the lighting.
In order to illuminate the refrigerated goods 103 during the image recording, a lighting device 102 is provided in the cold compartment 105, which lighting device provides identical illumination of the cold compartment 105 for every recording. In order to be able to perform a spectral analysis of the refrigerated goods 103 at a defined frequency, the lighting device 102 can emit monochromatic and narrowband light of a defined frequency, for example. For instance, the lighting device 102 comprises a tunable LED for emitting a monochromatic light of variable wavelength. The lighting device 102, however, can also be designed such that it emits polychromatic white light or infrared light.
The refrigeration appliance 100 can comprise a control and display device, which displays the determined condition of freshness of the refrigerated goods 103 to a user. If the detected color difference or a color change of, for instance, fruit or vegetables, exceeds a defined threshold value, a user can be informed via the display device. A tablet PC or smartphone can be used as the control and display device.
If the image data 147 and 149 is available in the processor unit 119 or the display device, further information can be extracted therefrom using image analysis algorithms, for instance using an application on the camera-module operating device. This information can then be displayed on the camera-module operating device. For instance, the recorded image data 147 and 149 can be used not only to detect changes but also to detect bar codes or QR codes of the refrigerated goods 103 or to perform character recognition. Further analysis of the codes can be performed via a product database on the Internet, in which database the relevant codes are stored. In addition, the ascertained product data can be associated with detected changes in the image data.
Image analysis employing pattern recognition can also be used to identify product logos. For this purpose, the product logos should be uniquely identifiable and not concealed by other refrigerated goods 103. Reliable optical recognition of the product can be guaranteed if the recording contains sufficient details of the product logo. Using suitable algorithms, the processor unit 119 can be trained for a multiplicity of products.
A list of contents of the refrigeration appliance 100 can be created in combination with product recognition. Product recognition is performed in this case using optical recognition of an EAN code on the product. Products such as fruit or vegetables, for instance, that have no EAN code can be selected manually from a selection list. Product registration can be performed in this context by identifying the EAN code. Once the product has been placed in the refrigeration appliance 100, the resultant change is then associated and highlighted in the image data (tracing).
The product database of the refrigeration appliance 100 can be extended to the entire kitchen area. This is done by initially selecting a storage location and registering the product by identifying the EAN code. Then the user puts the product away to store. The stock list can be updated when goods are removed. It is also possible to identify a product from the EAN code and to make an association with the image data.
Details providing additional information about a registered product, for instance information such as a best before date or a minimum stock level, can be used for monitoring purposes. In this case, the user can get the additional information (best before date) displayed when selecting the product on a display or in the database. If the product stock runs low or the product runs out, the user can be informed via the camera-module operating device.
In this context, for instance, the image data is merged with the product data by storing the outline information from an image analysis after the registration of a product. The product database can be used to implement other applications in addition to product monitoring. For instance these applications can include creating a recipe suggestion from the stock list or creating a shopping list for frequently purchased products. In this connection, an adaptive system can learn the habits of the user. An online interface to the stock list or to the image data allows convenient access on the move. The product registration and product database together with the display of additional information for the user provides extra added value in addition to pure image analysis and image display.
FIG. 14 shows a view of a portable camera-module operating device 300 having a touch-sensitive display 303 (touchscreen). The camera-module operating device 300 for example is a Tablet PC connected to the refrigeration appliance 100 via a WLAN, or a smartphone connected to the refrigeration appliance 100 via the mobile communications network. The camera-module operating device 300 can also be connected to the refrigeration appliance 100 via a USB interface, however. The USB interface, however, can also be used for charging the camera-module operating device 300.
An application such as a computer program, for instance, can be executed on the camera-module operating device 300. An installation kit can be provided for fitting the camera-module operating device 300. The camera-module operating device 300 is connected to a router via a wireless network. The camera-module operating device 300 can act as an image server or as a graphics processor and can back up the image data via a WLAN. The portable camera-module operating device 300 can be replaced or updated by a user.
Data can be transferred bi-directionally between the portable camera-module operating device 300 and the refrigeration appliance 100. The wireless camera-module operating device 300 is a component of a refrigeration appliance system and is used for displaying the image data acquired by the camera modules 101. The user can hence select the displayed cold compartment 105 in the application on the camera-module operating device 300. In addition, a view can be changed by swiping upwards or downwards on the touch-sensitive display 303 or by tilting the camera-module operating device 300 to right or left. An angle at which the camera-module operating device 300 is held can define, for instance, which cold compartment 105 is displayed.
The camera modules 101 inside the cold compartment 105 have a large angle of view. A plurality of camera modules 101 are used for a continuous representation of the front of view. The refrigeration appliance 100 can comprise a plurality of camera modules 101, which are arranged in the cold compartment 105 such that the refrigerated goods 103 are recorded from all sides, and an all-round view is generated by combining the individual images. The viewing angle in the all-round view onto the refrigerated goods 103 is selected by swiping left and right on the touch-sensitive display 303.
A camera module 101 having a large angle of view (fish eye) and arranged in a compartment ceiling 113 allows an overhead view onto the refrigerated goods 103 in the cold compartment 105. An additional camera module 101 is arranged in a door 111 of the refrigeration appliance 100. The cold compartment 105 additionally comprises a lighting device 102 for uniform illumination of the refrigerated goods 103. In this case and in all the other cases, the particular view can be selected by tilting the camera-module operating device 300.
In general, the refrigeration appliance 100 as a component of the refrigeration appliance system 200 can comprise a first camera module 101 for acquiring first image data about the refrigerated goods 103 from a first perspective, and a second camera module 101 for acquiring second image data about the refrigerated goods 103 from a second perspective. The perspective can generally be changed by touching the touch-sensitive display 303. The refrigeration appliance 100 can comprise any lighting device 102 that is suitable for illuminating the refrigerated goods 103 in the cold compartment 105 during image-data acquisition. The lighting device 102 can be controllable by the camera-module operating device 300.
On one side of the camera-module operating device 300 is displayed a menu 305 of options, which can be used to select the particular cold compartment 105 containing the relevant camera module 101. The contents of the cold compartment 105 are displayed when the cold compartment 105 is selected by touching the touch-sensitive display 303. The particular view can also be selected by swiping across the touch-sensitive display 103. It is possible to zoom-in or zoom-out the view, for instance, by a spreading movement with the fingers on the touch-sensitive display 103. It is also possible to display additional information such as, for example, a condition of freshness of the refrigerated goods 103 or changes in the image data compared with stored earlier image data.
The camera-module operating device 300 can compare the acquired image data about the refrigerated goods 103 with stored image data about the refrigerated goods 103. In addition, the camera-module operating device 300 can use an analysis algorithm to identify the refrigerated goods 103 from the acquired image data. The user can select and view on the camera-module operating device 300 the image data from the camera module 101 and the processed images, and get additional information displayed on said console, for instance information such as the allocation of a bar code to a specific product. For this purpose, the image data and the associated additional information are retrieved, processed and provided by the processor unit 119 of the refrigeration appliance 100. For this purpose, the camera data and additional information can be retrieved, processed and provided by an external central computer system. An association between the refrigeration appliance 100 and the camera-module operating device 300 can be made via a central processing unit, for example by means of Home Connect.
FIG. 15 shows a schematic view of a refrigeration appliance 100 having a camera array 127 in the door 111. The refrigeration appliance 100 comprises a camera array 127 having a multiplicity of camera modules 101 arranged in the form of an array on the inside of the door 111. Each of these camera modules 101 generates an individual front-view image from the inside of the refrigeration appliance 100. The camera modules 101 of the camera array 127 are arranged such that a combined view of the entire front view, such as when the door 111 is open, can be generated from the individual images. A panoramic image representing a continuous recording of the front side is composed from the camera array 127 in the refrigerator door 111. The camera modules 101 can be calibrated in order to improve an assembly of the individual images.
In addition, the refrigeration appliance 100 comprises a position-detection device for detecting a position of the camera-module operating device 300 placed on the outside of the refrigeration appliance 100. This position-detection device can be used, for example, to determine a coordinate pair specifying the position on the front face of the refrigeration appliance 100 in which the camera-module operating device 300 is placed.
Then an image region of the previously assembled view, which region is associated with the coordinate pair, can be transmitted to the camera-module operating device 300. This creates the impression that a user can use the wireless camera-module operating device 300 to look through the wall of the refrigeration appliance 100 in an X-ray-like manner. The user can also zoom in on regions of the assembled view. When the camera-module operating device 300 is moved in front of the front face of the refrigeration appliance 100, the corresponding image region moves on the camera-module operating device 300, and the cold compartment 105 that corresponds to the position of the camera-module operating device 300 is displayed. While the camera-module operating device 300 is moving over the refrigeration appliance 100 on the outside, the displayed image segment follows the position of the camera-module operating device 300. An image segment can be moved even when only one camera module is being used. All the regions are uniformly illuminated inside the refrigeration appliance 100 in this context.
The complexity of software development generally depends on the additional functions required such as, for instance, image acquisition, image analysis, product recognition (EAN code), detecting changes in an image, a product database and using recognition to allocate the products in the image data.
In the refrigeration appliance 100, it is possible to provide just the camera modules 101 and an interface to the camera-module operating device 300. The camera-module operating device 300 is used to control the camera modules 101 and to analyze the image data. Thus a minimum amount of control electronics can be implemented in the refrigeration appliance 100. A user is not tied to outdated hardware but can replace the camera-module operating device 300 as required. A door-opening and door-closing signal can optionally be transmitted for use by the camera-module operating device 300 to control the camera modules 101. The camera-module operating device 300 can transmit the image data by WLAN and display the contents of the refrigeration appliance 100.
For example, the image data is obtained every time the door is closed, then provided or transmitted to an external storage location and then transmitted immediately on request. Illumination remains available for a period of 300 to 500 ms immediately after a door closure. This achieves the advantage of recording the latest contents after any changes, adding a small amount of extra heat after a large amount of heat input, and allowing illumination solely by standard interior lighting.
FIG. 16A to 16C show different connections of camera modules 101. The camera modules 101 and the controller and be integrated in different ways.
In FIG. 16A, the camera module 101 comprises a control element 155 and a camera 157 as an integral unit and is connected to a network 161. In FIG. 16B, a plurality of camera modules 101 having the control elements 155 are connected to the network 161 via an integral image processing unit or a distribution node 125. In FIG. 16C, a multi-camera module is connected to the network 161. The multi-camera module can be connected to the network 161 using an integral or separate image processing unit 125. The multi-control element 155 is used to control a plurality of cameras 157 of the multi-camera module. An interface to a GSM network 163 can also be provided. The camera 157, the control elements 155, the multi-control element 155 and the image processing unit 125 can be physically identical units that perform different functions.
The camera modules 101 can be controlled by a processor in the refrigeration appliance or can be controlled separately from the refrigeration-appliance controller, for instance on the camera-module operating device 300 such as a smartphone and mobile Tablet PC. The camera-module operating device 300 having the screen can be attached to the door 111.
The camera-module operating device 300 can execute a program or an application (app), which is used to analyze the image data and to control the camera modules 101. For instance the application is used for determining freshness on the basis of the image data obtained. The freshness of the refrigerated goods 103 can be distinguished from a ripeness of the refrigerated goods 103.
In products which “feel the cold”, such as bananas, tomatoes, cucumbers, zucchini, egg plants and tropical fruits, the cell structure is affected by the cold. In general, weight, smell, taste, color, texture, for instance grooves, size, diameter or volume can be detected. The texture can be used to monitor whether a product is still crisp. Turgidity indicates the amount of water in the cells and is a measure of the freshness.
An application for determining the freshness acquires the information about products, analyzes this information and provides this information to other processing devices. For many products, the best before date is a reference date which is associated with the product and provided externally.
Unpackaged food does not have a best before date associated with the product. In addition, an alternative best before date can be used, which depends on the product, the time at which it was stored away, a refrigerator setting, a temperature graduation in the specific refrigeration appliance 100 and an air humidity. A traffic-light system is possible in this context:

- Red: best-before date cannot be identified
- Yellow: alternative best before date
- Green: best before date

Refrigerated goods such as meat may be packaged in opaque packaging, and therefore the condition of the meat cannot be monitored. The packaging can be labeled in a defined way, however, and may have a bar code and a best before date. Some products pose an increased health risk after expiry of the best before date, for instance products such as minced meat, poultry or fish products. The traffic-lights can be adapted for these products. To assess whether a product is still usable, the product specification contains a value that allows an assessment specifically according to the nature of the loss.


	Color

relative

absolute

Texture

	Food	e.g. red/brown	Weight	Size	Shape

Apple	+	+	+	+	+
Lettuce	+	−	+	(−)	+
Melon	+	+	−	+	−

The relative color can be assessed using a difference from the first image and a shade threshold value. Ripeness is identified from the shade of color.
The application can display guidance on storage in the refrigeration appliance 100, for instance

- “This food should not be kept in the refrigerator”
- “Minced meat is on top shelf. Note: store on a lower shelf/in a colder area”
- or
- “Note: adjust fridge to cooler setting or adjust automatic function; less food waste outweighs electricity costs.”

Some products have a Time Temperature Integrator (TTI), which indicates a product-specific actuation. The defined colors of the TTI can be read by sensors. Dietary advice can be generated by connecting to a storage guide. When stocking the refrigeration appliance 100, a stock list of all the items placed in the appliance can be created by passing these items across a camera module.
In addition, the application can use time-lapse to display the long-term change. In a virtual tour, an image of the contents of the refrigeration appliance can be displayed on the camera-module operating device 300 by the application. The functions may be a combination of language and bar code (EAN/QR), image analysis and image display. Products can be added. Synchronization with a photo can be carried out, and shopping lists can be generated.
In addition, the refrigeration appliance 100 can be designed such that a user can use a cell phone to request image data, making it possible to monitor freshness potentially even from remote locations. A GSM camera having MMS transmission can be used for this purpose. The camera and cell phone are connected via the GSM infrastructure. An SMS can be used to request an image, which is then transmitted as an MMS image.
Photos and videos can also be sent by email or MMS with a time stamp and date stamp. A photo or video can be sent on call at regular intervals. The photo resolution of the MMS for instance equals 160×120, 320×240, 640×480 pixels. If there is no GSM signal available, a recording is made in the internal memory. Power is supplied by battery, power pack or USB connection. Control can be performed directly at the refrigeration appliance 100, at a personal computer or by SMS from the mobile phone.
The mobile phone can be used to allow convenient remote control by Android app. The GSM camera can be actuated via GSM by a call or SMS and supports SMS commands. In addition, a power failure message can be issued. The viewing angle of the camera equals 60° or 90° with magnetic attachable wide-angle lens. A flash function can also be implemented.
In addition, a UMTS/HSDPA camera (WCDMA/TD-SCDMA) with a photo resolution of 1.3 Megapixels can be used. The UMTS/HSDPA camera has a tilt function and pivot function and a night-vision function. A UMTS video server is arranged separately from the camera.
In addition, an RJ45-10/100M Ethernet interface with or without WLAN can be provided. Network-compatible camera-module operating devices 300 are a laptop or television set for instance. It is thereby possible to separate multimedia components from refrigeration equipment. A multimedia interface may be a standard multimedia interface. In addition, optical fibers having a lens at one end can be used. The same color sensor can be used to read a plurality of optical fibers.
All the features illustrated and explained in association with the individual embodiments can be provided in various combinations in the subject matter according to the invention to achieve the advantageous effects of these embodiments simultaneously.
The scope of protection of the present invention is defined by the claims and is not restricted by the features explained in the description or illustrated in the figures.

LIST OF REFERENCES

100 refrigeration appliance
101 camera module
101-1 camera module
101-2 camera module
102 lighting device
103 refrigerated goods
104 microcontroller
105 cold compartment
107 pin-hole camera
109 side wall
111 door
113 compartment ceiling
115 sensor mat
117 compartment floor
119 processor unit
121 data bus
123 hinge
125 distribution node
127 camera array
129 rear wall
131 front face
133 circuit board
135 pin-hole lens
137 non-viewable region
139 viewable region
141 distortion
143 distortion
145 interface
147 image data
149 image data
151 subtraction
153 image region
155 control element
157 camera
161 network
163 GSM network
165 rack
167 door pillar
169 door handle
200 refrigeration appliance system
300 display device/camera-module operating device
303 touch-sensitive display
305 menu of options

Claims

1-15. (canceled)

16. A refrigeration appliance system, comprising:

a refrigeration appliance having a cold compartment;

a camera module associated with said refrigeration appliance for acquiring image data about refrigerated goods in said cold compartment; and

a wireless camera-module operating device associated with said refrigeration appliance for operating said camera module.

17. The refrigeration appliance system according to claim 16, wherein said camera-module operating device includes a touch-sensitive display for controlling said camera module and displaying the acquired image data.

18. The refrigeration appliance system according to claim 16, wherein said camera module is one of a plurality of camera modules associated with said refrigeration appliance, and said camera-module operating device includes a device for selecting one of said camera modules.

19. The refrigeration appliance system according to claim 18, wherein said camera modules are located for generating an all-round view of the refrigerated goods, and said camera-module operating device is constructed to control a viewing angle of the all-round view.

20. The refrigeration appliance system according to claim 16, wherein said camera module associated with said refrigeration appliance is one of a plurality of camera modules including a first camera module for acquiring first image data about the refrigerated goods from a first perspective and a second camera module for acquiring second image data about the refrigerated goods from a second perspective.

21. The refrigeration appliance system according to claim 20, wherein said refrigeration appliance has a door and a compartment ceiling, said first camera module is disposed in said door and said second camera module is disposed in said compartment ceiling.

22. The refrigeration appliance system according to claim 20, wherein said camera-module operating device is configured to switch between displaying the first perspective and the second perspective by tilting said camera-module operating device.

23. The refrigeration appliance system according to claim 16, wherein said camera module associated with said refrigeration appliance is one of a multiplicity of camera modules in a camera array for generating a view composed from individual images.

24. The refrigeration appliance system according to claim 23, wherein said refrigeration appliance has a door and said camera array is disposed in said door.

25. The refrigeration appliance system according to claim 23, wherein said refrigeration appliance has an outside, and said refrigeration appliance includes a position-detection device for detecting a position of said camera-module operating device placed on said outside of said refrigeration appliance.

26. The refrigeration appliance system according to claim 25, wherein said camera-module operating device is configured to display an image region associated with the detected position of said camera-module operating device.

27. The refrigeration appliance system according to claim 16, which further comprises a lighting device for illuminating the refrigerated goods in said cold compartment.

28. The refrigeration appliance system according to claim 27, wherein said lighting device is controllable by said camera-module operating device.

29. The refrigeration appliance system according to claim 16, wherein said camera-module operating device is configured to compare the acquired image data about the refrigerated goods with stored image data about the refrigerated goods.

30. The refrigeration appliance system according to claim 16, wherein said camera-module operating device is configured to identify the refrigerated goods from the acquired image data.