FR3143809A1 - System and method for acquiring a plurality of images and associated identification system and method - Google Patents

Abstract

The invention relates to a system (10) for acquiring images of a food product (5, 6) arranged in an acquisition zone (1), the acquisition system comprising: - a camera (12) configured to acquire a color image, the camera having a first optical axis, - a thermal detector (14) configured to acquire a thermal image, the thermal detector having a second optical axis, - a three-dimensional depth sensor (16) configured to acquire a image of the relief, the three-dimensional depth sensor having a third optical axis, - a processing unit (15) configured to receive each image and to generate three final images, the thermal detector being positioned between the camera and the three-dimensional depth sensor, the first optical axis, the second optical axis and the third optical axis being parallel to each other, the first optical axis and the third optical axis being spaced at a distance of between 5 and 20 centimeters. Figure for abstract: Fig. 2

Description

System and method for acquiring a plurality of images and associated identification system and method Technical field of the invention

The present invention generally relates to the field of product identification, particularly food products.

It finds a particularly advantageous application in the field of collective catering, to identify products positioned on a tray, in particular with a view to their automatic invoicing. It also finds a preferred application in the estimation of the nutritional and caloric characteristics of foods positioned on a tray. Finally, it finds an advantageous application in the context of quantifying food waste.

It relates more particularly to a system for acquiring images of a food product placed in an acquisition zone.

State of the art

In order to accelerate the flow of people in collective catering establishments, it is currently being sought to carry out faster collections.

It is known in particular from document FR3069685 an installation aimed at increasing the speed of collections to facilitate and streamline the movement of people in the product distribution area.

According to this document, this installation includes an acquisition station equipped with at least one camera oriented towards a surface for placing a tray comprising the different products selected by a user. According to this document, the acquisition station includes three cameras allowing images of the products present on the platform to be acquired from different angles. The acquired images are then processed to automatically identify the different products positioned on the tray and then issue the corresponding receipt.

Such an installation effectively makes it possible to automate the issuance of a ticket and therefore reduce human intervention to generate the collection ticket. However, the identification of products using this installation has limited reliability, in particular for example when a product is partially or totally hidden by a plate or a paper napkin or when it comes to identifying different products, with different prices and nutritional values, but with identical colors, containers or other visual aspects.

Presentation of the invention

The present invention proposes to improve the automatic identification of food products positioned on a tray, particularly when they are likely to be partially hidden by another product or a napkin or in the case of food products having different temperatures.

To this end, the invention relates to a system for acquiring a plurality of images of at least one food product placed in an acquisition zone. According to the invention, the acquisition system comprises:
- a camera configured to acquire at least one color image of the acquisition zone, the camera having a first optical axis,
- a thermal detector configured to acquire at least one thermal image of the acquisition zone, the thermal detector having a second optical axis,
- a three-dimensional depth sensor configured to acquire at least one image of the relief of the acquisition zone, the three-dimensional depth sensor having a third optical axis, and
- a processing unit configured to receive each image and to generate three final images of the acquisition zone,
the thermal detector being positioned between the camera and the three-dimensional depth sensor,
the first, second and third optical axes being parallel to each other,
the first optical axis and the third optical axis being spaced at a distance of between 5 and 20 centimeters.

The invention thus allows the acquisition of different types of images of the acquisition zone and therefore of the food product. In particular, the images acquired are not only limited to an acquisition of color images of the acquisition zone but also to an acquisition of thermal data and depth data. This then makes it possible to characterize the food product more precisely, by having visual characteristics of the food product but also other physical characteristics such as its temperature, dimensions and volume.

In addition, the relative arrangement of the different elements of the acquisition system makes it possible to ensure the quality of the images, in particular by preventing the representation of the acquisition zone on the different images from being truncated.

The images obtained are therefore particularly advantageous for subsequently allowing precise identification of the food product.

According to a variant of the invention, the first optical axis and the third optical axis are preferably spaced 6 centimeters apart.

This relative arrangement of the camera, the thermal detector and the three-dimensional depth sensor makes it possible to ensure the quality of the images, in particular by allowing a complete representation of the acquisition zone. All food products present in the acquisition zone are therefore visible by the camera, the thermal detector and the three-dimensional depth sensor.

According to one embodiment of the invention, the first optical axis, the second optical axis and the third optical axis are contained in the same plane.

This allows you to benefit from possible parallax effects which make it possible to enrich the visual information obtained from images acquired by several cameras making it possible to acquire color images, in terms of depth and perspective.

According to another embodiment of the invention, the first optical axis and the third optical axis being contained in a first plane, the second optical axis is contained in a second plane parallel to and distinct from the first plane. In other words, the projections of the three optical axes in a plane normal to them are not aligned.

According to the invention, the distance between the first plane and the second plane is less than or equal to 5 centimeters.

This allows a more compact arrangement of the camera, the thermal detector and the three-dimensional depth sensor while guaranteeing satisfactory quality of the images obtained.

According to one characteristic of the invention, the processing unit is configured to obtain two distinct color images of the acquisition zone according respectively to a first resolution and a second resolution and four final images are formed from one image in each of the two resolutions, of the thermal image and the relief image of the acquisition zone. The first resolution is higher than the second resolution. According to a variant of the invention, the first resolution corresponds to a high resolution and the second resolution corresponds to a medium resolution.

According to one characteristic of the invention, the processing unit is configured to obtain two distinct color images of the acquisition zone obtained each respectively according to a first angle of view and according to a second angle of view and the four final images are formed two color images, each according to an angle of view, the thermal image and the relief image of the acquisition zone.

The use of these four different types of images allows for more precise identification of food products present in the acquisition area. In addition, the acquisition of two color images with two different resolutions is particularly easy to implement.

According to another variant of the invention, the processing unit is configured to obtain three distinct color images of the acquisition zone each obtained respectively according to a first exposure, a second exposure and a third exposure, five final images are therefore formed a color image in each of the first exposure, the second exposure and the third exposure, the thermal image and the relief image of the acquisition zone.

The use of images with different exposures, an underexposed image and an overexposed image, makes it possible to limit the loss of information concerning dark parts (often poorly lit in practice) or very light parts (often subject to strong lighting) of the acquisition zone.

According to another characteristic of the invention, the color image being broken down into three primary colors, the five final images are formed of an image in each of the three primary colors, of the thermal image and of the image of the relief of the acquisition zone. Preferably, the decomposition is a decomposition according to the colors red green blue or even RGB for Red, Green, Blue in English.

The use of these five different types of images allows for more precise identification of food products present in the acquisition area.

According to another variant of the invention, the acquisition system also comprises a device for lighting the acquisition zone configured to emit continuous illumination or instantaneous illumination of the acquisition zone. Preferably, the lighting device emits white light whose color temperature is between 3500K and 6400K.

Such a lighting device makes it possible to correct the quality of the ambient lighting of the place where the device according to the invention is located. This lighting device is used in addition to the natural light present in the environment of the acquisition zone. This makes it possible to improve the quality of the images obtained by preventing certain parts of the acquisition zone from being visible or barely visible and/or by correcting the color rendering.

According to yet another variant of the invention, the camera comprises a polarizing filter.

This polarizing filter aims to attenuate, or even cancel, possible reflections on the specular parts of objects and products located in the acquisition zone.

According to a variant embodiment of the invention, the acquisition system also comprises communication means connected to the processing unit and configured to exchange data with a remote server.

The invention also relates to a system for identifying at least one food product, the identification system comprising:

- an acquisition zone adapted to receive the food product,

- a system for acquiring, according to the invention, a plurality of images of each food product, and

- a processing system configured to receive as input the three images generated by the processing unit and to provide as output identification data for each food product present in the acquisition zone.

This identification system makes it possible, from the acquired images, to identify the food products present in the acquisition zone. The identification data can then be used for billing or inventory applications.

According to a variant of the invention, the identification system also comprises a remote server configured to receive the three images generated by the processing unit of the acquisition system and the identification data determined by the system processing system identification and to transmit the identification data to an external entity.

Thus, the determined identification data can be transmitted to external entities, such as for example a collection device in the context of collective catering to enable billing for the food products chosen by the user concerned.

According to another variant of the invention, the acquisition system is positioned at a distance between 50 and 90 centimeters from the acquisition zone and, preferably, between 50 and 65 centimeters from the acquisition zone, preferably still, between 55 and 60 centimeters from the acquisition zone.

It has been noted that such ranges of distances make it possible to obtain an acquisition range with an optimal depth of field for good recognition of products while preserving ease of access compatible with the bulk of the platforms and products likely to be there.

The invention also relates to a method for acquiring a plurality of images of at least one food product placed in an acquisition zone, the method comprising steps of:
- acquisition of at least one color image of the acquisition zone,
- acquisition of a thermal image of the acquisition zone,
- acquisition of a relief image of the acquisition zone, and
- generation of three final images from each image, the three final images being formed of a color image, the thermal image and the relief image of the acquisition zone.

Within the meaning of the invention, a color image can be formed either from a monochrome image in a primary or component color or from an image formed from a variation of gray levels or from a polychrome image integrating three primary colors.

The invention finally relates to a method of identifying at least one food product placed in an acquisition zone, the identification method comprising steps of:
- acquisition of a plurality of images of each food product according to the acquisition method according to the invention, and
- identification of each food product by providing identification data for each food product present in the acquisition zone.

Of course, the different characteristics, variants and embodiments of the invention can be associated with each other in various combinations as long as they are not incompatible or exclusive of each other.

Detailed description of the invention

The description which follows with reference to the appended drawings, given as non-limiting examples, will make it clear what the invention consists of and how it can be carried out.

represents a schematic view from above of an acquisition zone according to the invention,

schematically represents an image acquisition system according to the invention,

represents a diagram illustrating the positioning of a camera, a thermal detector and a three-dimensional depth sensor in the image acquisition system of the ,

represents an example of a thermal image of the acquisition zone, obtained in the context of the present invention,

represents an example of an image of the relief of the acquisition zone, obtained in the context of the present invention,

schematically represents an identification system according to the invention,

represents, in flowchart form, an example of an image acquisition method according to the invention, and

represents, in flowchart form, an example of an identification method according to the invention.

The invention aims, according to a first application, to allow the automatic identification of products or foods in order to facilitate or even automate the process of invoicing the price of these products, particularly in the context of collective catering. For this purpose the invention proposes, as shown schematically in , to implement an identification system S which communicates via a communication network R with a payment or billing terminal T.

The identification system S then includes an acquisition zone 1 intended to receive products or food to be identified as will appear subsequently. Acquisition zone 1 is associated with an image acquisition system 10 which, according to the example illustrated, is placed above it. The identification system communicates via a communication network 2 with an SRV identification server in order in particular to allow the establishment of an invoice for the products 5, 6 positioned in the acquisition zone 1. According to the example illustrated, the products 5, 6 to be identified are placed on a serving tray of the type used in collective catering by customers to transport the items chosen for their meal. The tray is then placed in acquisition zone 1 by a customer wishing to have the receipt issued corresponding to their selection.

Thus, the invention relates in particular to an image acquisition system 10 as illustrated in and designated as a whole by the reference 10. According to this exemplary embodiment, the image acquisition system 10 comprises a camera 12, a thermal detector 14, a three-dimensional depth sensor 16, a processing unit 15 and means communication 18.

Advantageously according to the invention, the image acquisition system 10 is positioned at a distance h from the acquisition zone 1. This distance h is here between 50 and 90 cm. Preferably, this distance h is between 50 and 65 cm. More preferably, this distance is between 55 and 60 cm. This arrangement of the image acquisition system 10 relative to the acquisition zone 1 makes it possible to capture the entire acquisition zone 1 and to obtain images (whatever their type) of good quality for allow analysis by a processing system 24 of the identification system S.

As shown on the , the camera 12, the thermal detector 14 and the three-dimensional depth sensor 16 are positioned on a support 19. The support 19 has a generally rectangular shape. The front face 19A of the support 19 materializes a plane P.

As shown in the , the thermal detector 14 is here positioned between the camera 12 and the three-dimensional depth sensor 16.

Each of the camera 12, the thermal detector 14 and the three-dimensional depth sensor 16 are positioned on the support 19 so that their respective optical axes are perpendicular to the support 19. By definition, the optical axis of an optical system is defined as the axis of rotational symmetry of this optical system.

So, as shown in , the camera 12 has a first optical axis A1, orthogonal to the plane P of the support 19. The thermal detector 14 has a second optical axis A2, orthogonal to the plane P of the support 19. The three-dimensional depth sensor 16 has a third optical axis A3 , orthogonal to the plane P of the support 19. The first optical axis A1, the second optical axis A2 and the third optical axis A3 are parallel to each other.

Advantageously according to the invention, the first optical axis A1 and the third optical axis A3 are spaced by a distance d of between 5 and 20 centimeters (cm). This relative arrangement of the camera 12, the thermal detector 14 and the three-dimensional depth sensor 16 makes it possible to ensure the quality of the images, in particular by allowing a complete representation of the acquisition zone 1. All food products 5, 6 present in the acquisition zone 1 are therefore visible by the camera 12, the thermal detector 14 and the three-dimensional depth sensor 16.

Preferably, the first optical axis A1 and the third optical axis A3 are spaced by a distance d of the order of 6 cm.

On the example of the , the first optical axis A1 and the third optical axis A3 are contained in a first plane P1. The second optical axis A2 is, for its part, contained in a second plane P2, parallel and distinct from the first plane P1. In other words, this means that, for their positioning on the support 19, the camera 12, the thermal detector 14 and the three-dimensional depth sensor 16 are not aligned (here only the camera 12 and the three-dimensional depth sensor 16 are aligned according to the same plane P1).

Advantageously, the first plane P1 and the second plane P2 are spaced by a distance d ₁ of less than 5 cm. This allows a more compact arrangement of the camera 12, the thermal detector 14 and the three-dimensional depth sensor 16 while guaranteeing satisfactory quality of the images obtained.

The camera 12 here is an image acquisition system comprising a photographic sensor of the CMOS sensor type (for “ Complementary Metal – Oxide – Semiconductor ”). Such a sensor is characterized in particular by, on the one hand, the dimensions of its photosensitive surface of rectangular shape which can be summarized by its format and the length of the diagonal thereof, and, on the other hand, its resolution.

The sensor used has, for example, a 1/2.3' type format with a diagonal greater than 7 mm and a resolution between 2028x1520 and 8112x6080. In the present case, a sensor is used having an image format of 1/2.3, a diagonal of around 7.9 mm and a resolution of 4056 x 3040 such as the IMX 477 sensor marketed by Sony. Such a sensor has the advantage of having good sensitivity so that it is possible to obtain good quality images in ambient light without additional lighting. Thus, this makes it possible to avoid parasitic reflections induced by additional lighting and which harm the quality of the images acquired. Furthermore, such a sensor represents a good compromise between performance and the associated cost.

The photographic sensor is associated with optics making it possible to acquire an image encompassing the entire acquisition zone. This optic has, for example, a fixed focal length combined with the use of an equally fixed diaphragm.

Optionally, this optic, and therefore the camera, can be equipped with a polarizing filter. This polarizing filter aims to attenuate, or even cancel, the effects of possible reflections due to the nature of the products present in the acquisition zone 1. For example, plastic packaging around food products 5, 6 can be the cause of such thoughts.

Camera 12 makes it possible to acquire color images of acquisition zone 1 as will appear later.

The image acquisition system 10 also includes the thermal detector 14.

This thermal detector 14 is for example an infrared camera comprising an infrared sensor associated with optics making it possible to produce a thermal image encompassing at least the entire acquisition zone. This is mid-infrared, therefore wavelengths between 3 and 50 micrometers (µm). In the present case, the field of vision of this thermal detector 14 is for example 55 degrees by 35 degrees or 110 degrees by 75 degrees. The resolution of the thermal detector 14 is here at least 16x12 pixels. It is for example between 16x12 pixels and 64x48 pixels and, preferably, 32x24 pixels.

Here, the thermal detector 14 is for example the Melexis MLX90640 model.

The thermal detector 14 is configured to acquire at least one thermal image I _th of the acquisition zone 1 as will be described subsequently.

The image acquisition system 10 also includes the three-dimensional depth sensor 16.

This three-dimensional depth sensor 16 includes a stereoscopic sensor. The field of view of the three-dimensional depth sensor 16 is for example 65 degrees by 40 degrees. The resolution of this sensor is here between 640x360 pixels and 2560x1440, preferably 1280x720 pixels.

Here, the three-dimensional depth sensor 16 is for example an Intel® RealSense ^TM D415 type depth camera.

The three-dimensional depth sensor 16 is configured to acquire at least one image of the relief I _dpth of the acquisition zone 1, as described later.

The three-dimensional depth sensor 16 also includes a control unit and an image acquisition sensor. The three-dimensional depth sensor 16 is then also configured to acquire a color image of the acquisition zone 1.

As shown in the , the image acquisition system 10 also includes a lighting device 11. This lighting device 11 is used in addition to the ambient light present in the environment of the acquisition zone 1. This lighting device lighting can for example make it possible to obtain diffusing lighting in the environment of the acquisition zone 1.

This lighting device 11 is for example configured to emit continuous illumination of the acquisition zone 1. This is for example a lamp present in the environment of the acquisition zone.

As shown in the , the image acquisition system 10 also includes the processing unit 15. This processing unit 15 comprises for example a processor and a memory and ensures the control of the camera 12, the thermal detector 14 and the depth sensor three-dimensional 16 as well as the processing of images resulting from these.

As described below, the processing unit 15 is configured to trigger the acquisition of the different images by the camera 12, the thermal detector 14 and the three-dimensional depth sensor 16. The processing unit is also configured to receive each image acquired by the camera 12, the thermal detector 14 and the three-dimensional depth sensor 16 and to generate at least three final images of the acquisition zone 1 (an RGB color image, a thermal image I _th and an image of the relief I _dpth ). These three final images can then be analyzed so as, for example, to identify the food products 5, 6 located in the acquisition zone 1.

The processing unit 15 makes it possible in particular to ensure the correct alignment of the images, to adjust the perspectives associated with each acquired image and to resize them so that each image is centered on the acquisition zone. It also allows you to modify the format of the acquired images so that they can be analyzed subsequently. In particular, the processing unit 15 receives, for example in practice, a raw file from each camera and sensor associating data with each pixel. The processing unit 15 is then configured to transform these data so that they can be used for the identification of the food products 5, 6. The processing unit 15 can also compress the images so that those -these are remembered and transmitted more conveniently. The processing unit 15 can also implement a step of anonymization of the images received in order to ensure the confidentiality of the consumer's data, such as for example data related to bank cards, mobile phones or any other non-existent element. food possibly placed on the tray.

In a preferred embodiment, the processing unit is configured to obtain two distinct RGB color images of the acquisition zone 1 according to a first resolution and a second resolution respectively. Four final images are then formed here by a first RGB color image presenting a first resolution, a second RGB color image presenting a second resolution, a thermal image I _th and an image of the relief I _dpth of the acquisition zone 1. The first resolution is here higher than the second resolution. Here, the first resolution is called high and is for example 2560x1440 pixels or 2560x1600 pixels or even 3840x2160 pixels. The second resolution is here called medium and is for example 1920x1080 pixels or 1920x1200 pixels.

Finally, the image acquisition system 10 also includes the communication means 18. These communication means 18 are for example wireless communication means making it possible to establish wireless communication with a remote entity. These means of communication 18 are for example configured to exchange data with a remote server 25. Here, the means of communication 18 are for example configured to exchange the three (or more depending on the embodiment) final images generated with a remote server , allowing for example to store them externally to the image acquisition system 10 (for example for long-term backup of these images).

Certain alternative embodiments of the image acquisition system 10 can be implemented.

According to a variant embodiment, the first optical axis, the second optical axis and the third optical axis can be contained in the same plane

According to another alternative embodiment, the lighting device can be omitted from the image acquisition system. In fact, it is not strictly necessary, especially for environments in which ambient lighting is sufficient.

According to yet another alternative embodiment, the lighting device can be configured to emit instantaneous illumination of the acquisition zone.

According to yet another alternative embodiment, the image acquisition system can make it possible to obtain four final images formed from two distinct color images of the acquisition zone obtained each respectively from a first viewing angle and from a second angle. view, thermal image and relief image of the acquisition zone.

According to yet another alternative embodiment, the image acquisition system can make it possible to obtain five final images formed by three color images, each in a primary color, a thermal image and an image of the relief of the acquisition zone .

According to yet another alternative embodiment, the image acquisition system can make it possible to obtain five final images formed by an RGB color image, an underexposed image, an overexposed image, a thermal image and an image of the relief of the acquisition zone. In this variant, the overexposed image is obtained using a long exposure time and the underexposed image is obtained using a short exposure time. The RGB color image then corresponds to an average exposure compared to the other two images (underexposed and overexposed). The use of underexposed and overexposed images makes it possible to limit the loss of information concerning the dark parts (often poorly lit in practice) or the very light parts (often subject to strong lighting) of the acquisition zone.

The present invention aims to identify the food products 5, 6 present in the acquisition zone 1, from the images acquired by the acquisition system 10.

There represents, schematically, an identification system S of at least one food product 5, 6 present in the acquisition zone 1.

As shown in the , this identification system S comprises the image acquisition system 10 described above. This image acquisition system 10 therefore makes it possible to acquire images (for example three, four or five images according to the embodiments introduced previously) from the acquisition zone 1.

The identification system S also includes a processing system 24 configured to receive as input the final images generated and to provide as output identification data for each food product 5, 6.

In practice, the processing system 24 comprises for example an artificial neural network configured to receive these final generated images as input and to provide the identification data as output. This artificial neural network was previously trained using known images including identified food products. This artificial neural network is for example a multilayer neural network. Preferably, it is a deep multi-layer neural network. This is, for example, a convolutional neural network or a transformer.

The identification data therefore makes it possible here to identify the food product 5, 6 concerned. For example, this identification data specifies, for the food product 5, that it is a glass filled with cold fruit juice, or for the food product 6, that it is a plate filled with a dish of the day.

As shown in the , the identification system S finally includes a remote server SRV. This remote SRV server is for example configured to receive the final images generated (three or four or five depending on the implementation used) by the processing unit 15 of the image acquisition system 10 and the determined identification data. by the processing system 24 of the identification system S and transmit this identification data to an external entity 30. This external entity 30 is for example a collection device making it possible to generate an invoice for the food products concerned present in the acquisition zone 1, for example the payment or billing terminal T represented on the .

Alternatively, the processing system can be directly integrated into the acquisition system. The identification data is therefore directly determined within the acquisition system.

There is a flowchart representing an example of an image Acq acquisition method of at least one food product 5, 6 present in the acquisition zone 1 according to a first embodiment.

This Acq acquisition method is implemented when a user places an article (for example a tray supporting a food product 5, 6) in the acquisition zone 1. The presence of a product 5, 6 in the zone acquisition 1 triggers a command (step E0) to the image acquisition system 10 (and in particular the camera 12, the thermal detector 14 and the three-dimensional depth sensor 16) in order to trigger the acquisition of the different images.

In practice, the presence of a food product 5, 6 in the acquisition zone 1 is for example detected by a weighing system or by a tray presence detector.

In step E2, the camera 12 of the image acquisition system 10 receives the command to trigger image acquisition. It then acquires at least one color image of the acquisition zone 1. To do this, the processing unit 15 transmits, for example, a command concerning the exposure duration for the acquisition of the color image. The exposure duration is for example here between 12 and 32 milliseconds.

In practice, the color image is broken down on the basis of three primary colors: red, green and blue (according to the RGB or RGB color system for " red , green, blue " according to the acronym of Anglo-Saxon origin commonly used).

In this first embodiment, the camera 12 provides an RGB color image (taking into account the three primary colors). More specifically here, camera 12 provides an RGB color image with the first resolution. In other words, the camera 12 acquires an RGB color image with high resolution.

In step E4, the thermal detector 14 of the image acquisition system 10 receives the command to trigger image acquisition. It then acquires at least one thermal image I _th of acquisition zone 1.

In this description, the term “thermal image” means an image in which each pixel is associated with temperature data. A scale of colors or shades of colors makes it possible to transcribe the temperature differences in the acquisition zone 1 studied.

There represents an example of a thermal image obtained by the thermal detector 14. The temperature scale is here represented in the form of shades of gray (this representation allows in particular an optimization and a reduction of the size of the thermal image in order to improve the storage and transmission of this image). This scale extends here, on the representation of the , between 20 and 55 degrees Celsius (this scale being truncated at 20 degrees for temperatures below 20 degrees and limited to 55 degrees for temperatures above 55 degrees). In other words here, the temperature data obtained from the thermal detector 14 are processed, by the processing unit 15, according to band-pass filtering so as to obtain a relevant representation of the acquisition zone 1.

Preferably, the temperature range extends between 0 and 60 degrees Celsius. Alternatively, this temperature scale may be wider, depending on the model of the thermal detector used. For example, a thermal detector of the Melexis MLX90640 type can detect temperatures between -40 and 300 degrees Celsius.

The thermal detector 14 is particularly advantageous for allowing the distinction between food products which visually have the same appearance. For example, orange juice and an orange-colored soup have the same visual appearance (via, for example, the images acquired by camera 12) but different temperatures. The thermal image acquired by the thermal detector 14 therefore makes it possible to distinguish these two products.

The use of the thermal detector 14 is also advantageous in the case where a food product is masked by a concealment element. This concealment element is for example a paper towel or a lid placed on the food product. In this case, the images acquired by the camera 12 do not make it possible to distinguish the food product concerned. The thermal detector 14 makes it possible to highlight the presence of a product by characterizing it by its temperature.

In step E6, the three-dimensional depth sensor 16 of the image acquisition system 10 receives the command to trigger image acquisition. It then acquires at least one image of the relief I _dpth of acquisition zone 1. It also acquires another RGB color image of acquisition zone 1, this other RGB color image having a second resolution. The second resolution is lower than the first resolution (of the RGB color image acquired by the camera 12).

In this description, the term “relief image” (or image representing relief) is understood to mean an image representing depths or reliefs in the acquisition zone, relative to a reference plane. Here the reference plane is for example formed by the installation surface in acquisition zone 1 on which each serving tray is placed. In the case of a tray used to transport food products 5, 6, the reference plane is for example the plane of this tray.

There represents an example of a relief image obtained by the three-dimensional depth sensor. More particularly, this figure represents the height H (also called depth in this description) relative to the reference plane defined previously. The depth scale is represented here in the form of shades of gray (this representation allows in particular optimization and reduction of the size of the relief image in order to improve the storage and transmission of this image). This scale extends here between 19 and 105 millimeters (mm) (this scale being truncated at 19 mm for depths less than 19 mm and limited to 105 mm for depths greater than 105 mm). In other words here, the depth data obtained from the three-dimensional depth sensor 16 are processed, by the processing unit 15, according to band-pass filtering so as to obtain a relevant representation of the acquisition zone 1.

Alternatively, other depth scales, in particular wider, can be obtained depending on the three-dimensional depth sensor used.

Advantageously, the three-dimensional depth sensor 16 makes it possible, thanks to the detection of different depths, to improve the recognition of food products 5, 6. This sensor is particularly useful for enabling the detection of food products, such as bottles or cans, placed in a position preventing their identification (for example in a vertical position for bottles or cans). This then makes it possible to alert the user in order to allow them to position them in such a way as to make their identification possible (in a horizontal position for example). The three-dimensional depth sensor 16 also makes it possible to detect possible objects hidden in the other images based on a volumetric comparison between an expected volume and the volume observed for an object (for example in the case of a plate or a towel too high).

According to the example described previously, steps E2, E4 and E6 are implemented simultaneously.

As shown in the , the acquisition process Acq ends with step E8 during which the processing unit 15 of the image acquisition system 10 generates the final images from each image obtained in steps E2, E4, E6.

As described previously, in this first embodiment, the four final images are formed by a high resolution RGB color image, a medium resolution RGB color image, a thermal image I _th and a relief image I _dpth of the area d acquisition 1.

The final images generated can then be processed (for example, format processing of the “High Dynamic Range” type according to the commonly used Anglo-Saxon designation of origin or artificial intelligence algorithms allowing image processing) before be used to identify the food product 5, 6. They can also undergo encoding processing, for example in different gray levels.

The invention also relates to a second embodiment of the Acq image acquisition method of at least one food product 5, 6 present in the acquisition zone 1.

In this second embodiment, the Acq image acquisition method has the same steps and characteristics as the first embodiment described previously.

The only difference is in the images obtained at the output of camera 12. According to the second embodiment, at the output of camera 12, three images are obtained, one in each primary color.

Thus, during step E8 during which the processing unit 15 of the image acquisition system 10 generates five final images from each image obtained in steps E2, E4, E6, the five final images obtained are formed by three color images, each in a primary color, a thermal image I _th and a relief image I _dpth of acquisition zone 1.

The invention also relates to a third embodiment of the Acq image acquisition method of at least one food product 5, 6 present in the acquisition zone 1.

In this third embodiment, the Acq image acquisition method has the same steps and characteristics as the first embodiment described previously.

The only difference is in the output obtained from the image acquisition system. According to the third embodiment, five final images are obtained: an RGB color image, an underexposed image, an overexposed image, a thermal image and an image of the relief of the acquisition zone. In this third embodiment, the overexposed image is obtained using a long exposure time and the underexposed image is obtained using a short exposure time. The RGB color image then corresponds to an average exposure compared to the other two images (underexposed and overexposed).

Thus, during step E8 during which the processing unit 15 of the image acquisition system 10 generates five final images from each image obtained in steps E2, E4, E6, the five final images obtained are formed from an RGB color image, an underexposed image, an overexposed image, a thermal image and a relief image of the acquisition zone.

The invention also relates to a fourth embodiment of the Acq image acquisition method of at least one food product 5, 6 present in the acquisition zone 1.

In this fourth embodiment, the Acq image acquisition method has the same steps and characteristics as the first embodiment described previously.

The only difference is in the images obtained at the output of the camera 12. According to the fourth embodiment, during step E8 during which the processing unit 15 of the image acquisition system 10 generates the final images from each image obtained in steps E2, E4, E6, four final images obtained are formed by two RGB color images, a first according to a first viewing angle and a second according to a second viewing angle, a thermal image I _th and an image of the relief I _dpth of acquisition zone 1.

The invention also relates to a fifth embodiment of the Acq image acquisition method of at least one food product 5, 6 present in the acquisition zone 1.

In this fifth embodiment, the final images are three in number and made up of an RGB color image, a thermal image I _th and a relief image I _dpth of the acquisition zone 1.

Whatever the method of carrying out the image acquisition process, certain execution variants can be considered.

According to a variant embodiment, in step E2, the trigger command for the image acquisition system can be issued following a manual action, for example from the user who has just placed his tray in the zone d 'acquisition.

According to another alternative embodiment, steps E2, E4 and E6 can be implemented successively in a short time interval, for example less than 5 milliseconds.

The invention then also relates to a method of identifying Id of at least one food product 5, 6 placed in the acquisition zone 1. This identification method Id follows the image acquisition method Acq described previously . There represents, in flowchart form, an example of an Id identification method implemented according to the invention.

The identification method Id firstly comprises a step of acquisition Acq of a plurality of images of each food product 5, 6 present in the acquisition zone 1 according to the acquisition method Acq described previously.

The identification method Id then comprises a step E10. During this step, the processing system 24 of the identification system S identifies each food product 5, 6 by providing identification data associated with each food product 5, 6 from the final images generated (three or four or five depending on the implementation used) during the Acq acquisition step. As described previously, this step is for example implemented via an artificial neural network receiving, as input, the final images generated (three or four or five depending on the implementation used) and providing, in output, the identification data associated with each food product 5, 6.

Each identification data obtained is for example then transmitted to an external entity, such as a cash register device so as to invoice the user for the products he has chosen on his tray.

Advantageously according to the invention, the use of the thermal detector 14 also makes it possible to carry out quality control of the food products positioned in the acquisition zone, thanks to the known temperature data. The processing system 24 is then able to associate and transmit (to an external entity), for each food product, the associated identification data and the corresponding temperature data.

The invention also finds a particularly advantageous application in the estimation of the nutritional and caloric characteristics of foods. To do this, each identification data obtained is, for example, analyzed from a nutritional and caloric point of view, by comparison, for example, with data tables listing these characteristics for different foods.

The image acquisition system 10 also finds a particularly advantageous application for quantifying the portions of food remaining after the meal. For this, the acquisition zone is, for example, a space for placing a tray after the meal. The image acquisition system works in the same way as described previously to acquire the final images (three or four or five depending on the implementation used) of the acquisition zone after meal.

Then, a processing system identifies each product remaining on the tray by providing another identification data associated with each product remaining from the final images generated. Here too, this step is for example implemented via an artificial neural network receiving, as input, the final images generated and providing, as output, the other identification data associated with each remaining product

Each other data obtained is, for example, then transmitted to an external entity, such as an analysis device making it possible to evaluate the quantity of remaining food.

Thus, advantageously according to the invention, on the basis of the images acquired by the image acquisition system, it is possible to analyze the remaining quantities of food and therefore to quantify food waste, for example in a collective catering service.

Claims (17)

System (10) for acquiring a plurality of images of at least one food product (5, 6) arranged in an acquisition zone (1), the acquisition system (10) comprising:
- a camera (12) configured to acquire at least one color image of the acquisition zone (1), the camera (12) having a first optical axis (A1),
- a thermal detector (14) configured to acquire at least one thermal image (I _th ) of the acquisition zone (1), the thermal detector (14) having a second optical axis (A2),
- a three-dimensional depth sensor (16) configured to acquire at least one relief image (I _dpth ) of the acquisition zone (1), the three-dimensional depth sensor (16) having a third optical axis (A3), and
- a processing unit (15) configured to receive each image and to generate three final images of the acquisition zone (1),
the thermal detector (14) being positioned between the camera (12) and the three-dimensional depth sensor (16),
the first optical axis (A1), the second optical axis (A2) and the third optical axis (A3) being parallel to each other,
the first optical axis (A1) and the third optical axis (A3) being spaced by a distance ( d ) of between 5 and 20 centimeters. Acquisition system (10) according to claim 1, in which the first optical axis (A1) and the third optical axis (A3) are spaced 6 centimeters apart. Acquisition system (10) according to claim 1 or 2, in which the first optical axis (A1), the second optical axis (A2) and the third optical axis (A3) are contained in the same plane. Acquisition system (10) according to claim 1 or 2, in which, the first optical axis (A1) and the third optical axis (A3) being contained in a first plane (P1), the second optical axis (A2) is contained in a second plane (P2) parallel and distinct from the first plane (P1). Acquisition system (10) according to claim 4, in which the distance ( d ₁ ) between the first plane (P1) and the second plane (P2) is less than or equal to 5 centimeters. Acquisition system (10) according to any one of claims 1 to 5, in which the processing unit (15) is configured to obtain two distinct color images of the acquisition zone (1) respectively according to a first resolution and a second resolution, four final images being formed of a color image with a first resolution of a second color image with a second resolution, of the thermal image (I _th ) and of the relief image (I _dpth ) of the acquisition zone, the first resolution being greater than the second resolution. Acquisition system (10) according to any one of claims 1 to 5, in which the processing unit (15) is configured to obtain three distinct color images of the acquisition zone (1) each obtained respectively according to a first exposure, a second exposure and a third exposure, five final images being formed of a color image in each of the first exposure, the second exposure and the third exposure, the thermal image (I _th ) and the 'relief image (I _dpth ) of the acquisition zone. Acquisition system (10) according to any one of claims 1 to 5, in which, the color image being broken down into three primary colors, five final images are formed of an image in each of the three primary colors, of thermal image (I _th ) and the relief image (I _dpth ) of the acquisition zone. Acquisition system (10) according to any one of claims 1 to 8, also comprising a lighting device (11) of the acquisition zone (1) configured to emit continuous illumination or instantaneous illumination of the zone acquisition (1). Acquisition system (10) according to any one of claims 1 to 9, in which the camera 12) comprises a polarizing filter. Acquisition system (10) according to any one of claims 1 to 10, also comprising communication means (18) configured to exchange data with a remote server (25). Identification system (S) of at least one food product (5, 6), the identification system (S) comprising:
- an acquisition zone (1) adapted to receive the food product (5, 6),
- a system (10) for acquiring a plurality of images of each food product (5, 6) according to any one of claims 7 to 11, and
- a processing system (24) configured to receive as input the three color images generated by the processing unit (15) and to provide as output identification data for each food product (5, 6) present in the zone acquisition (1). Identification system (S) according to claim 12, also comprising a remote server (SRV) configured to receive the three images generated by the processing unit (15) of the acquisition system (10) and the identification data determined by the processing system (24) of the identification system (S) and to transmit the identification data to an external entity (30). Identification system (S) according to claim 12 or 13, in which the acquisition system (10) is positioned at a distance ( h ) of between 50 and 90 centimeters from the acquisition zone (1). Identification system (S) according to any one of claims 12 to 14, in which the acquisition system (10) is positioned at a distance ( h ) of between 55 and 60 centimeters from the acquisition zone (1 ). Acquisition method (Acq) of a plurality of images of at least one food product (5, 6) placed in an acquisition zone (1), the acquisition method (Acq) comprising steps of:
- acquisition of a color image of the acquisition zone (1),
- acquisition of a thermal image (I _th ) of the acquisition zone (1),
- acquisition of a relief image (I _dpth ) of the acquisition zone (1), and
- generation of three final images, each final image being formed of the color image, the thermal image (I _th ) and the relief image (I _dpth ) of the acquisition zone (1). Method for identifying (Id) of at least one food product (5, 6) placed in an acquisition zone (1), the identification method (Id) comprising steps of:
- acquisition of a plurality of images of each food product (5, 6) according to an acquisition method (Acq) according to claim 16, and
- identification of each food product (5, 6) by providing identification data for each food product (5, 6) present in the acquisition zone (1).