Classifications

• • G—PHYSICS
  • G06—COMPUTING; CALCULATING OR COUNTING
  • G06Q—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
  • G06Q10/00—Administration; Management
  • G06Q10/08—Logistics, e.g. warehousing, loading or distribution; Inventory or stock management
  • G06Q10/087—Inventory or stock management, e.g. order filling, procurement or balancing against orders
• • A—HUMAN NECESSITIES
  • A47—FURNITURE; DOMESTIC ARTICLES OR APPLIANCES; COFFEE MILLS; SPICE MILLS; SUCTION CLEANERS IN GENERAL
  • A47F—SPECIAL FURNITURE, FITTINGS, OR ACCESSORIES FOR SHOPS, STOREHOUSES, BARS, RESTAURANTS OR THE LIKE; PAYING COUNTERS
  • A47F3/00—Show cases or show cabinets
  • A47F3/04—Show cases or show cabinets air-conditioned, refrigerated
  • A47F3/0482—Details common to both closed and open types
  • A47F3/0495—Spraying, trickling or humidifying means
• • G—PHYSICS
  • G06—COMPUTING; CALCULATING OR COUNTING
  • G06F—ELECTRIC DIGITAL DATA PROCESSING
  • G06F18/00—Pattern recognition
  • G06F18/20—Analysing
  • G06F18/24—Classification techniques
  • G06F18/241—Classification techniques relating to the classification model, e.g. parametric or non-parametric approaches
  • G06F18/2413—Classification techniques relating to the classification model, e.g. parametric or non-parametric approaches based on distances to training or reference patterns
• • G—PHYSICS
  • G06—COMPUTING; CALCULATING OR COUNTING
  • G06Q—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
  • G06Q30/00—Commerce
  • G06Q30/02—Marketing; Price estimation or determination; Fundraising
• • G—PHYSICS
  • G06—COMPUTING; CALCULATING OR COUNTING
  • G06Q—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
  • G06Q30/00—Commerce
  • G06Q30/02—Marketing; Price estimation or determination; Fundraising
  • G06Q30/0281—Customer communication at a business location, e.g. providing product or service information, consulting
• • G—PHYSICS
  • G06—COMPUTING; CALCULATING OR COUNTING
  • G06Q—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
  • G06Q30/00—Commerce
  • G06Q30/06—Buying, selling or leasing transactions
• • G—PHYSICS
  • G06—COMPUTING; CALCULATING OR COUNTING
  • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
  • G06V10/00—Arrangements for image or video recognition or understanding
  • G06V10/40—Extraction of image or video features
  • G06V10/44—Local feature extraction by analysis of parts of the pattern, e.g. by detecting edges, contours, loops, corners, strokes or intersections; Connectivity analysis, e.g. of connected components
• • G—PHYSICS
  • G06—COMPUTING; CALCULATING OR COUNTING
  • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
  • G06V10/00—Arrangements for image or video recognition or understanding
  • G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
  • G06V10/82—Arrangements for image or video recognition or understanding using pattern recognition or machine learning using neural networks
• • G—PHYSICS
  • G06—COMPUTING; CALCULATING OR COUNTING
  • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
  • G06V20/00—Scenes; Scene-specific elements
  • G06V20/50—Context or environment of the image
  • G06V20/52—Surveillance or monitoring of activities, e.g. for recognising suspicious objects

Definitions

• the invention relates to the field of furniture for the presentation of products to retail sale, and more particularly to the field of displays or stalls. It also relates to the field of applications of artificial vision (also called digital vision or computer vision) and the acquisition and processing of information obtained by an artificial vision system. This information is used in processes for identifying products displayed on a stall, in particular plant products, and for monitoring their quantity.
• artificial vision also called digital vision or computer vision
• Modern sales furniture is no longer a simple horizontal plank on which products displayed for sale are stacked. Even if the principle has hardly changed, the horizontal boards of ancestral markets have become sophisticated stalls, incorporating specific functions, such as side protection strips, attractive lighting, sound system, scale, bag dispenser, a refrigeration device, a nebulization device, and devices for displaying regulatory information (for example concerning the name of the product or its price), advertising and / or utilities (for example concerning the use of the product).
• specific functions such as side protection strips, attractive lighting, sound system, scale, bag dispenser, a refrigeration device, a nebulization device, and devices for displaying regulatory information (for example concerning the name of the product or its price), advertising and / or utilities (for example concerning the use of the product).
• EP 2 175407 (Sanden Corp.) describes an information display system for a retail product display cabinet, which establishes communication between antennas integrated in the cabinet and beacons. RFID attached to the product, when the product is removed from the shelf.
• WO 2008/020132 (Supertec) describes another product management system which uses RFID tags affixed to products, and which can also detect loyalty cards provided with an RFID tag.
• the present invention aims to present a stall which exhibits at least some of these functions.
• the present invention relates to an installation for the exhibition of products, which comprises at least one product display surface, at least one means for capturing images of the exhibition surface, at least one information reproduction means, and a computer machine configured to perform certain functions which will be explained in greater detail below.
• It also relates to a system which comprises at least one of said installations and a computer machine external to the installation, configured to perform certain functions which will be explained in greater detail below, as well as radioelectric communication means allowing the installations to communicate. with said external computer machine.
• the invention applies in particular to a sales cabinet of the display or shelf type, which has an exhibition surface on which products intended to be picked up by a buyer are displayed. Said exposure surface may in particular be horizontal or inclined, over its entire surface or over part of its surface.
• the problem is solved by a combination of several means in an installation, and possibly also in a system, which comprises several elementary functionalities. All these functionalities are not obligatory within the framework of the present invention, and two or more of these elementary functionalities can be combined in certain embodiments of the invention.
• a first elementary functionality is the detection of the buyer's hand or arm near a given area of interest. It triggers the display of information specific to the product that is exposed in said area of interest. This functionality can be achieved by various means, known or new.
• a second basic functionality is the recognition of the product species on the display surface of a stall. It makes it possible to automatically map the display surface of the stall by product species. It also makes it possible to subdivide said exhibition area into areas of interest, each of which contains only one type of product. This recognition and subdivision can be done by store staff and entered manually into a human-machine interface (such as a computer tablet), but it is tedious.
• the system according to the invention performs this product species recognition and mapping automatically, using a computer machine specially configured to perform this task.
• An area of interest can have a rectangular or other shape.
• a third elementary functionality is the estimation of the quantity of a product of a given species in the area of interest.
• This estimate can relate to the relative quantity compared to an initial quantity. It allows the monitoring of the quantity, which may be the residual quantity of products of this species in the area of interest, and / or the quantity of these products sold. It facilitates the good management of stocks, and in particular makes it possible to trigger in good time (and preferably before the product area of interest is exhausted) the replenishment of the area of interest.
• the system is configured to generate alert messages, and to send these messages to a radio receiver.
• a fourth functionality is the adaptation of the nebulization to the nature of the products (ie to their species) and / or to the quantity of products.
• a fifth functionality is the adaptation of the refrigeration of the display surface of the area of interest to the nature of the products (ie their species) and / or the quantity of products.
• a first object of the present invention is an installation, such as a display, for the exhibition of products for sale, comprising:
• a product display area comprising at least one area known as the area of interest on which there are products of a given species
• a system comprising: a vision system comprising means for capturing images of said exposure surface, at least one information reproduction system, such as a display screen, means for detecting the penetration of an object within a predetermined perimeter above said product display surface, a computer machine configured to cooperate with said vision system, said information rendering system and said means for detecting the penetration of an object into said predetermined perimeter, characterized in that said computer machine is configured for:
• Said means for detecting the penetration of an object within a predetermined perimeter above said product display surface may form part of said vision system.
• - Said system is configured to recognize the products from their visual appearance, without using reference images for this recognition forming part of said system. This simplifies the use of the installation in the point of sale, because these reference images should be produced under the same conditions as on the installation, and preferably with products substantially identical with those with which the installation will be subsequently. used.
• - Said system is configured to allow the recognition of the species of products on the display surface of a stall, to establish a mapping of the display surface of the stall by species of product, and to subdivide said surface exhibition in areas of interest, each of which contains only one type of product. It also simplifies the use of the installation in the point of sale, as it is no longer necessary to reconfigure it each time the layout of the exhibition area is changed.
• Said computer machine can be configured to estimate the volume or quantity of said products of a given species, and / or to identify the products present in an area of interest according to their species.
• - Said system can be configured to trigger the restitution of information associated with the species of products located on said area of interest, by using said information restitution means, when the penetration of an object in said perimeter predetermined is detected by said detection means.
• Said information restitution means are selected from the group formed by: display screens (preferably arranged in front of, behind or above said product display area), printers, electronic shelf labels , man-machine interfaces provided with a data display means (and preferably with a keyboard, hardware or virtual).
• - Said system is configured to transmit or make available to a buyer information associated with the product he has previously selected by entering with an object or with his hand or with his arm in the predetermined perimeter above said zone of 'interest, when said buyer makes a request on said man-machine interface.
• - Said installation further comprises at least one device, called “device for modifying the local environment of the products", selected from the group formed by: a nebulization device, configured to spread a mist formed of drops of water. an average diameter of between 1 ⁇ m and 10 ⁇ m over at least one area of interest of said product display surface; a cooling device, configured to cool a surface of interest; a heating device, configured to heat a surface of interest, each of said devices preferably being configured to cooperate with a computer machine configured to control at least one of their operating parameters.
• - Said computer machine has been configured using an artificial neural system, so as to be able to identify the products present in an area of interest according to their species; said artificial neural system can be configured using the Mask R-CNN system.
• the installation is configured to display on said information rendering system: when said means for detecting the penetration of an object within the predetermined perimeter above any zone of interest have detected no penetration of '' an object in the perimeter above any of the areas of interest: information previously associated with a product present in any area of interest, and when the penetration of an object into the perimeter above of a zone of interest has been detected: information previously associated with the product which is found in said zone of interest.
• the invention makes it possible to identify information concerning the modification of the quantities of products of a certain species present on the stalls. This allows store staff to be alerted when this quantity drops below a predetermined value, in order to supply the stall with products of that species. This prevents the area dedicated to this type of product from being empty, which increases the turnover generated by this area.
• the system can be configured to generate, record and / or display to the store personnel the evolution of the stock and the sales of products of this species as a function of time, for example over a day, a week, a month, a quarter, or a year.
• the system can be configured so that the processing and display of data concerning the quantity of different products and its evolution can be done on the local computer machine and / or on the external computer machine.
• the installation for the exhibition of products according to the invention may comprise a transceiver which integrates it into a radio network.
• This radioelectric network comprises at least one installation according to the invention and an external computer machine, and may also include a remote information reproduction device (typically associated with said external computer machine), intended for the staff of the department or of the shop.
• a second object of the invention is a radio network, comprising at least one installation according to the invention, each of said installations being provided with a first receiver-transmitter, an external computer machine configured to communicate, via a second receiver-transmitter, with each of said installations, and configured to send information to at least one mobile receiver, said messages passing through another radio network.
• a third object of the present invention is a method of using an installation according to the invention, in which the quantity of a product present on an area of interest of the surface is determined at regular or irregular intervals. for displaying products, said process comprising the following successive steps:
• Another object of the invention is a method of using an installation according to the invention, in which the arrangement of the exhibition surface is carried out, by arranging on each of the different zones of said surface products d. 'a single species, said products being able to be different or identical from one zone to another, an image is taken of all or part of the exhibition surface, said image is analyzed to recognize the zones comprising products of a single species and the said product is identified for each zone and assigned an identifier, knowing that these three sub-steps can be carried out at the same time or successively, the mapping of the exhibition area is established by associating each zone with its identifier .
• the invention lends itself in particular to the sale of fruits and vegetables, but its field of application is not limited to these two types of products.
• FIGS 1 to 6 illustrate different aspects of the invention, but do not intend to limit the scope of the invention.
• FIG. 1 shows schematically an installation for the exhibition of products according to the invention.
• FIG. 2 shows another installation according to the invention.
• FIG. 3 schematically illustrates an inclined exhibition surface with two slopes, with geometric markings.
• FIG. 4 schematically shows a radio network according to the invention, which comprises two installations according to the invention and a central computer machine.
• FIG. 5 shows a logic diagram of a process for selecting the information displayed on a display screen of an installation according to the invention.
• FIG. 6 shows a logic diagram of a process for mapping a product display area of a facility according to the invention.
• Three-digit reference numbers refer to process steps (see Figures 5 and 6).
• buyer is meant here the customer of a self-service store who does his shopping and who approaches a stall to view and possibly apprehend products.
• An "display area” is the area of a stall on which products are arranged and displayed for buyers to see.
• An "area of interest” is an area of the display area of a stall where there are products displayed for sale, which are usually of one species.
• it can be bulk products (food products such as fruits or vegetables, non-food products such as pocket books on sale, small leather goods, lots of pairs of socks) or arranged in an orderly manner (in a crate for example).
• food products typically, we find on the different areas of a stall fruits and vegetables separated by species, because each species is offered at a specific price; on a given area of interest there is therefore typically a single species of fruit or vegetable (which may include, as this will be explained below, the case where these products are fruits or vegetables of a single plant variety).
• An “image capture area” is an area of the exhibition surface for which an image capture device captures images.
• An image capture area contains at least one area of interest, and may contain several. According to the invention, the image capture can be done continuously or discontinuously, at regular or irregular intervals.
• the installation according to the invention comprises a vision system, which comprises at least one device for capturing color images, which is typically disposed above the exhibition surface.
• Said image capture device may include a camera, configured to capture images. This capture can be done for example continuously, or at regular or irregular intervals.
• the vision system is linked to a computer machine with which it cooperates.
• This computer machine is advantageously a local computer machine, that is to say located in the installation (typically in its base), but alternatively it can also be an external computer machine, which communicates with the control system. vision via a wired or wireless link.
• the adjective “external” refers to the installation: the external computer machine is not integrated into said installation, it can be located on the sales area (typically in a designated space. as inaccessible to the public) or in an office located next to the sales area or above the sales area, or further away from the facility.
• Said external computer machine is called a “central computer machine” when it is configured to communicate and exchange data with several installations according to the invention.
• Said installation also comprises a device capable of detecting the entry of an object within a predetermined detection perimeter which is located above a zone of interest. This device can form part of said vision system.
• the installation according to the invention also comprises at least one information rendering system, which is advantageously a display system comprising at least one display screen and / or at least one electronic gondola label.
• the installation comprises a device capable of detecting the entry of an object into a detection perimeter.
• This object is typically the hand of a buyer who apprehends a product.
• Said computer machine is configured to cooperate with said device capable of detecting the entry of an object into a detection perimeter, to detect the arrival of a new object close to a given area of interest.
• a proximity detection device configured to detect the penetration of an object within a predetermined perimeter.
• Said perimeter is a space located above the products arranged on an area of interest.
• it may be a depth camera.
• These depth cameras are known as such; they include, for example, an infrared emitter and at least two receivers (preferably pixelated).
• the proximity camera is arranged above the exposure surface. It is configured to detect the arrival of a buyer's hand or arm over an area of interest. This makes it possible to identify the buyer's interest in products of a certain species.
• this identification of a buyer's interest in products of a certain species triggers the restitution of information associated with this product, by using the restitution means information.
• These means may include a display screen visible to the buyer placed in front, behind or above the area of interest.
• this information returned by the information recovery means may for example be specific to the product, and may include, for example, information on the price of the product, on its origin, on its nutritional value, or else specific advice for use (storage, preparation). They can also relate to complementary products which are often bought by buyers in general, or by that particular buyer (if it has been identified in advance individually), who buy that particular product.
• the vision system also includes an imaging device, which typically includes at least one camera, configured to capture images of at least one area of interest. Specifically, the camera captures shots of a capture area continuously or at regular intervals, such as every thirty seconds. These shots are processed by the vision system, as will be explained in more detail below.
• an imaging device typically includes at least one camera, configured to capture images of at least one area of interest. Specifically, the camera captures shots of a capture area continuously or at regular intervals, such as every thirty seconds. These shots are processed by the vision system, as will be explained in more detail below.
• an image capture device captures shots for several areas of interest (n areas of interest), each of which is loaded with products X n of a single species.
• the vision system is configured so that said image capture device can function as a device capable of detecting the entry of an object into a detection perimeter.
• the system according to the optical invention is configured to identify, by an appropriate image analysis method, the arrival of a hand or an arm in said detection perimeter.
• the system can be configured to count the number of such hand or arm penetration events within said detection area.
• the vision system can be configured to analyze the shape detected by the imaging device; if the shape of the object has a rectangular form factor (width: length) less than 1 and preferably between, for example, 1: 1 and 1:10, and if the angle formed by this rectangle and the edge of the furniture is between 35 ° and 145 °, the system identifies this object as an "arm" of a buyer. This analysis can be carried out by image analysis software of known type, suitably configured. If the vision system does not detect a new nearby object, it determines the volume of a specific product X n.
• This product X n is selected so that over a given period of time (which may be of the order of an hour or half an hour) the volume V n of each product X n is determined at least one times ; the selection of the product X n may or may not be a random selection.
• the value of the volume Vn can be stored in a file in one of the local or external computer machines (preferably in the local computer machine); this file can be a database.
• a preselected threshold which corresponds for example to 10% of the initial volume
• a message is sent to a predefined receiver, by a wired or wireless transmission channel. This message can be sent for example by SMS messaging ("short message service") or by email.
• the predefined receiver can be a fixed or mobile receiver, for example the cell phone of the department manager.
• the vision system detects a new nearby object near a specific product (referred to here as Xi), this new object is identified as a "hand", and the display system displays one or more messages corresponding to the product Xi .
• the hand appears as a different object from the background in one of the areas of interest.
• One or more depth cameras can be provided.
• the depth cameras can be integrated above the cabinet forming Lethal.
• said shooting device and said proximity detection device are integrated in the same housing.
• Such housings exist commercially, for example the product RealSense TM D435 from the company Intel®.
• the height of the cameras in relation to the sales area and their spacing depends on their optical characteristics, and in particular on their field of vision and their depth of vision.
• the height can typically be of the order of 1 to 2 meters.
• a height of approximately 1.0 meter and a distance between two neighboring cameras of approximately 1.2 meters are chosen.
• a height of approximately 2.0 meters and a distance between two neighboring cameras of approximately 2.4 meters are chosen. These values are suitable for example for the RealSense TM D435 camera, mentioned above.
• a camera and a display screen form a module called a recognition module. Said screen is located above the sales area, and preferably close to the area on the sales area where the products concerned by the display are located.
• the vision system and the computer machine with which it cooperates are configured to automatically detect the products to identify their species. These products are in particular fruits and vegetables. They can be placed in an area dedicated to them on the stall, in bulk or in crates.
• the system according to the invention makes it possible to identify, for example, loose carrots, bunched carrots, zucchini, green beans, cucumbers, lettuce, lemons.
• a first embodiment which is suitable in particular for products of a geometric shape which does not vary, or which does not vary much, from one product of the same species to another, such as manufactured parts , it is possible to use an image analysis system which detects the contours of the product. Thus, one can distinguish the products of two different species by their only geometric shape as defined by their contours. It is also possible to estimate the quantity of products visible on a surface or stacked in bulk.
• an image analysis system can be used which detects in addition to the contours of the product its color.
• the vision system and the computer machine with which it cooperates can be configured to automatically detect the products to identify their species when the products are bottles, hardware items (screws, brackets, dowels) , cosmetic items (tubes, flasks), hats, clothes, or more generally product packs.
• an advantageous embodiment which is suitable in particular for natural products such as fruits or vegetables, which may present within the same species a slightly different geometric shape, and / or a slightly different color and / or an inhomogeneous color, and / or an appearance presenting on the surface of the drawings (possibly of color) different, one uses a software known as artificial intelligence with an image recognition function.
• This software may be of a type which comprises an artificial neural network (also called a “neural network”). He must be configured prior to its use in a way specific to the products it must recognize. More precisely, a list of products to be recognized is defined, and in a first step, for each species of product, a library is provided comprising different images each comprising a product of this species.
• this library is used to train the neural network to recognize the products of a given species. This must be done for each of the product species.
• One procedure that can be used is described below.
• Mask R-CNN an artificial neural network known under the designation Mask R-CNN, which is described in a publication “Mask R-CNN” by K. He and al., available at https://arvix.org/abs/1703.06870.
• the Mask R-CNN system is a deep neural network designed for object detection and image segmentation. It generates bounding boxes and segmentation masks for each instance of an object in the image.
• the model is based on a network backbone (“backbone”) of the ResNet 101 type which was previously trained on a set of COCO (“Common Objects in Context”) type data, according to a method described in the publication “Deep Residual Learning for image Recognition ”by K. He et al, available at https://arxiv.org/abs/1512.03385, to which is added a network of characteristic pyramids (known as FPN,“ Featured Pyramid Network ”) the latter is described in the publication “Featured Pyramid Networks for Object Detection” by T. -Y. Lin et al., Available at https://arxiv.org/pdf/ 1612.03144.pdf.
• backbone network backbone
• COCO Common Objects in Context
• the vision system is configured to recognize the products in an image capture zone. To this end it cooperates with a computer machine. Said computer machine associated with the vision system is configured to subdivide said image capture zone into zones of interest, each of which only comprises products of the same species. This prevents department staff from having to configure or reconfigure the system each time a new type of product is placed on the exhibition area, and more generally, each time products are rearranged on the installation. . This subdivision uses the product identity information that has been acquired for each of the areas of interest within the image capture area.
• the vision system is configured to allow manual correction of the configuration of areas of interest, when a product of a certain species has been misrecognized or has not been recognized; this manual correction can be made by correcting the name of the product present in said area of interest.
• the system can be configured so that this correction can be carried out on a central computer machine which is part of the same radio system as the installation, and / or on a man / machine interface specific to the installation, or on a mobile terminal which communicates with a transceiver specific to said installation or with a transceiver specific to said central computer machine.
• a first step we identify for a given area of interest the product it contains.
• one or more information is associated with this product, so as to be able to restore at least one of this information by the information rendering system, in the event of activation of the system for this area of interest.
• This activation is done by the device capable of detecting the entry of an object into a detection perimeter, which includes said area of interest and extends above it. This allows this information to be displayed when a buyer is about to touch or apprehend a product in the area of interest.
• the vision system is configured to estimate the quantity of products.
• This quantity is expressed as a relative value with respect to an initial quantity; in certain cases (batch of products of regular shape and of little variable size) it can also be expressed fairly reliably in absolute quantity (mass or number).
• This estimation of the quantity of a product can be done in different ways.
• the average size of each product is calibrated (in units of length or in pixels). To this end we can divide the area of the area of interest by the size of a product. The number of products seen by the vision system is thus estimated. This approach works well for products of relatively uniform size and shape, such as apples or nuts.
• the volume (or a parameter representative of the volume) of the products in the area of interest is determined.
• a third step it is determined whether this volume (or this parameter which represents it) is below a predetermined threshold value. If so, an alert message is issued.
• the determined volume is recorded for the area of interest for which it was determined, along with the time at which it was determined.
• the image captured in the first step can include several areas of interest. This process is then carried out for one or more of these areas of interest.
• the capture of images for the purpose of recognizing the species of products and estimating the quantity of a product must be done at a time when no hand or other object is detected near the areas of interest, because the presence of a hand or another object can interfere with this recognition and estimation.
• the nebulization is stopped before the first step, and a certain period of time, which can be predetermined or determined by the system as a function of the gradual disappearance of the disturbance of the images captured by the nebulization, before proceeding to the first step; the nebulization can be restarted when the image which is used for determining the volume in the second step of said process has been recorded.
• Information on the quantity of products of a species present in the assigned area of interest can be used in different ways. In particular, this information will be brought to the attention of department or store management personnel.
• the installation for the exhibition of products according to the invention may comprise a transceiver which integrates it into a radio network.
• This radio network may in particular include a remote information retrieval device, intended for department or store management personnel. It can, for example, communicate messages to the personnel, referred to below as “alerts”, by any appropriate transmission channel, in particular by electronic mail or by SMS message.
• the system alerts store personnel to supply the stall with products of that species.
• the system analyzes at regular intervals (for example every hour) all the products recognized on the shot, and alerts the store staff if products of a certain species that were present during the previous recognition have disappeared in such a way as to create a disruption in the supply of products of this kind.
• the system determines the quantity of products at regular or irregular intervals, and determines the rate of decrease of this quantity; when this rate of decrease exceeds a predetermined threshold value, the system alerts the store staff to supply the stall with products of this species. This prevents the area dedicated to this type of product from being empty, knowing that an empty area does not generate sales.
• the computer system is configured to generate, record and / or display to the store personnel the evolution of the stock and the sales of products of this species as a function of time, for example over a day, a week, a month, a quarter, or a year.
• the system is configured to detect the rate of intruders (ie of products of a different species than that predominantly present in the area of interest) .
• the system alerts store staff to tidy up the stall.
• the vision system having identified the “hand” of a buyer which approaches an area of interest, and having defined the product X1 on the area of interest, triggers the display of information relating to said product X1 in which the buyer is interested and which is in the area of interest.
• This display is typically done on a display screen which is located near the area of interest, for example above it.
• Said information has been previously associated with the product X1 by an appropriate configuration of a computer machine which cooperates with said display screen.
• information can be displayed on an electronic shelf label (also called an electronic shelf label), for example on one or more segments of an electronic segment label.
• an electronic shelf label also called an electronic shelf label
• These gondola labels are known as such, exist commercially (for example offered by the company Pricer TM), and will not be explained in greater detail here.
• Said display can relate, for example, to information associated with the product, its price, its origin, its use (for example a cooking recipe), and / or it can relate to complementary products likely to be useful for the product.
• buyer of this product for example on the location of such a complementary product in the store.
• It can also relate simply to symbolic and / or graphic information (ie non-alphanumeric), such as a thank you message (possibly in the form of an emoticon), or can include such symbolic and / or graphic information.
• the system comprises a transmission means for transmitting information to the buyer.
• This transmission can be a wireless transmission, or a physical transmission or provision of information on a physical medium.
• This transmission or material provision typically follows a request made by the buyer.
• the installation can include a man-machine interface which allows the purchaser to form this request.
• the facility may include a printer for printing information onto a physical medium, such as paper.
• a printer for printing information onto a physical medium, such as paper.
• the man-machine interface can allow the buyer to enter his telephone number or an e-mail address, or to scan a code (such as a bar code or a QR code) with a terminal equipped with a camera, and the information is then sent to it by a wireless transmission means integrated into the installation, or by a transmission means external to the stall with which the installation communicates.
• Said man - machine interface can be a tablet provided with a screen, possibly touch screen, and / or a keyboard. It can be positioned on the shelf, for example on a side strip, or near the latter.
• Said information may be information related to the product, and the formation of the request by the purchaser to obtain its wireless transmission or its physical provision may follow the identification by the vision system of the " hand ”of the purchaser and of the product, as has been described in connection with the fourth aspect of the invention.
• the vision system having identified the "hand" of a buyer approaching an area of interest, and having defined the product X1 on the area of interest, as has been described. described above in relation to the fifth aspect of the invention, triggers the display of information which has been previously associated with said product X1 in which the buyer is interested and which is in the area of interest. This display can be done on a screen above the stall, and / or on said man-machine interface.
• the system can request a transmission request for the physical delivery of information that the buyer can enter into the human interface.
• the vision system is configured to recognize or characterize a state of a given product X1.
• This state is typically deduced from the modification of a visually perceptible characteristic of the product.
• This visually noticeable feature can be color.
• An optical characteristic related to wavelength is referred to here as "color”.
• the state of the product which can be recognized or characterized by a visually perceptible characteristic, such as color may be the state of freshness of the product. This particularly concerns food products, such as fruits and vegetables.
• Said color which makes it possible to assess the state of the product may be a color of the visible spectrum, and / or a color of the infrared spectrum and / or a color of the ultraviolet spectrum.
• said vision system may comprise one or more additional sensors which respond in the spectral range lying outside the visible spectrum, and / or an image capture device is used which responds both in the visible spectrum. and in a spectral range which lies outside the visible spectrum, typically in the near infrared.
• infrared light can allow the detection of defects invisible to the naked eye, which are located below the surface of the skin of fruits or vegetables, and which can indicate a certain aging of the product.
• the installation is used to adapt the temperature and / or the nebulization of the area of interest.
• said installation may comprise a nebulization system and / or a cooling or heating system, which are connected to the local computer machine, and certain operating parameters of which are controlled by said computer machine.
• Said nebulization system comprises a mist diffusion device connected to a generator of liquid microdroplets.
• Said diffusion device comprises at least one mist diffuser, which may for example be a mast extending above the exhibition surface, or a ramp mounted horizontally above the surface. exposure.
• Said microdroplet generator advantageously comprises a piezoelectric system capable of generating water droplets with an average diameter of between 1 ⁇ m and 10 ⁇ m, and preferably between 1 ⁇ m and 5 ⁇ m.
• said nebulization system is configured so as to allow individually controlled nebulization for each area of interest, or each group of contiguous areas of interest. This makes it possible to adapt with precision the intensity of the nebulization to each zone of interest (or each group of contiguous zones of interest), without spilling over into another neighboring zone which would not need nebulization, or which does not would not need a nebulization of this intensity.
• mist formed of droplets with an average diameter of between 1 ⁇ m and 10 ⁇ m, and preferably between 1 ⁇ m and 5 ⁇ m forms a sheet of mist which floats above the surface of exposure. Also because of this small droplet diameter, this fog sheet does not significantly wet the products, but cools them by evaporation on their surface or near their surface.
• Various means can be implemented to adapt the intensity of the nebulization from a mist diffusion device.
• ramps each comprising a plurality of nebulization orifices, the mist flow of which can be regulated using a flow regulation mechanism, for example by throttling; such a mechanism can be applied to an entire ramp or to individual orifices.
• This mechanism can be actuated by any known means, such as a stepping motor or a piezoelectric displacement device.
• Means can also be provided making it possible to regulate (i.e. reduce or increase) the flow of fog at the level of the fog generator.
• the electrical power absorbed by the piezoelectric element can be adjusted.
• a fog generator can serve one or more areas of interest.
• a generator can have multiple, individually adjustable piezoelectric elements, each of which serves a different area of interest.
• Said cooling or heating system advantageously cools or heats at least part of the exhibition surface.
• Such a cooling system advantageously comprises a cooled plate; the latter typically forms at least part of the display surface, so that the products rest on this cooled plate.
• Such cooled plates intended to equip stalls exist commercially and will not be described here in greater detail. The same is true of the heating plates which can be used in the context of the present invention.
• the system according to the invention is configured so that the intensity of nebulization and / or the setpoint temperature of the cooling or heating system can be modified according to a recognized product state. or characterized by the system. For example, it has been found that the color of peppers changes as they dry. In one embodiment of the method according to the invention, this evolution is recognized by the vision system, and the intensity of the nebulization is modified to be adapted to this state of the peppers recognized by the vision system.
• FIG. 1 shows an installation 1 for the exhibition of products according to the invention.
• the installation comprises a product display surface 2 mounted on a base 12.
• the installation comprises a superstructure 10, which comprises vertical poles 8, connected by crossbars 9.
• Said superstructure 10 supports lighting devices 7, and information restitution systems 4, in this case display screens. These screens 4 can be attached directly to a mast 8, or to a beam 11 supported by a mast 8.
• the mast also supports a vision system 3, in this case a box which comprises an image capture device, characterized by a capture perimeter 18, and a device capable of detecting the entry of an object into a perimeter. detection; this detection perimeter is typically located within said sensing perimeter 18 of the vision system 3.
• a vision system 3 in this case a box which comprises an image capture device, characterized by a capture perimeter 18, and a device capable of detecting the entry of an object into a perimeter. detection; this detection perimeter is typically located within said sensing perimeter 18 of the vision system 3.
• the vision system 3 communicates by a computer link (not shown in this figure), which can be of the wired or wireless type, with a local computer machine 29, which is typically disposed in the base of the installation 1.
• This local computer machine 29 is configured to process the data generated by the vision system 3, in particular the images captured by the image capture device and the data generated by the device capable of detecting the entry of an object into a detection perimeter. Said computer machine can also control the display of information on the screens 4.
• the installation 1 further comprises a man-machine interface 5, in this case a touch pad, placed near the display screen 4.
• This man-machine interface 5 also communicates with said local computer machine 29, which is configured to process the data that it generates and to send it data that it returns to the buyer.
• Display area 2 is occupied by three piles of products 6a, 6b, 6c, each of a given species, arranged in bulk or in an orderly manner, as appropriate, thus forming three areas of interest.
• a mist diffusion device 13 comprises in this example a horizontal ramp comprising two nebulization orifices 14 which spread mist 15 on the products 6c placed on an area of interest of product display area 2; other types of diffusers can be used.
• the fog is generated by a fog generator 30 which is typically disposed in the base 12 of the installation 1, and conveyed to the fog diffusion devices 13 by a system of ducts (not shown in the figures).
• a piezoelectric fog generator is used; these devices are known as such and will not be described here in greater detail.
• the installation 1 is part of a radio network, via a first transceiver 16, which is typically mounted on a mast 8.
• Said first transceiver is typically controlled by the computer machine. local 29. It forms part of a radio network and communicates with a second transceiver 17, which is connected, typically via a central computer machine 23 (shown in FIG. 4), to a remote screen 20 ( "Remote" from the facility 1) which displays information for store staff.
• Said central computer machine 23 is configured to communicate with several installations according to the invention.
• this information displayed is a curve which represents the quantity of one of the products of a given species 6a. This quantity decreases according to time, until it falls below a predetermined threshold value 21; this event then triggers an alert for the employees of the store who replenishes the product 6a concerned, which explains that at the next determination, the quantity of product 6a detected by the vision system 3 has increased.
• FIG. 2 shows another example of an installation 1, which does not include a nebulization system, but two zones each comprising four cooled plates 19a, 19b, 19c, 19d.
• Each human-machine interface 5, in this case a tablet, is associated with a printer 22 which allows the purchaser to print product-specific information. Said printer 22 can be controlled directly by the man-machine interface 5 or by the local computer machine 29.
• Figure 3 illustrates schematically an example of an estimation of the volume of products of a given species, for the case of an inclined exhibition surface with two slopes.
• markers are defined which are then detected by the system. Using these markers, one or more parameters representative of this inclination are determined. This could be, for example, the angle of inclination from the horizontal.
• six markers M1 M6 are used which will be detected by the vision system 3 connected to a local computer machine 29. The angle of inclination a and possibly also the width L and the height are thus calculated. h. These two distances can be expressed in units of length and / or in pixels. This surface will then be subtracted from the apparent volume of the products determined by the vision system.
• FIG. 4 schematically illustrates a radio network 26 according to the invention. It comprises a plurality of installations 1 according to the invention, each of which comprises its first transceiver 16, its information reproduction system 4 (such as a display screen) and its man-machine interface 5 intended to be used by a buyer 27. Said information rendering system 4 and said man-machine interface 5 communicate with the first transceiver by a computer link 25 which can be wired or wireless, and that may involve a microprocessor.
• Said first transceivers 16 are configured to communicate with a second transceiver 17.
• the latter communicates in this example with an external computer machine 23. This communication is bidirectional.
• Said external computer machine 23 is connected to a remote screen 20, which displays information intended for department or store management personnel.
• a buyer 27 can express on the man-machine interface 5 of one of the installations 1 his desire to receive information on his mobile phone 24 or on another receiver that he has designated; to this end, said external computer machine communicates with an external radio network, which has a so-called external transceiver 28, and which is configured to communicate with the mobile phone 24 of the purchaser 27.
• the external computer machine 23 (or the local computer machine 29, in particular if the data on the quantity are processed by it) is configured to communicate with a mobile receiver carried by a person responsible for the department to which the installation belongs; in this way said person can be alerted to the need to restock the facility with products of a given species.
• FIG. 5 shows a diagram which illustrates an embodiment for the method of display on the screens 3 of an installation 1 according to the invention.
• the screen 4 displays content relating to the various areas of interest of the installation; this content can be selected at random from all the content available for this installation.
• a request is sent to find out whether an area of interest has been selected by the system. This selection can in particular be triggered by the detection of an event by the device capable of detecting the penetration of an object into the detection perimeter which is located above an area of interest. If the system does not select an area of interest (step 120), the method starts over with step 100.
• step 130 the system displays on the screen 4 associated with the selected area of interest a content associated with the selected area of interest (step 140), depending on the product that had previously identified on this area of interest by the vision system.
• the duration of this display can be predetermined (for example 10 seconds).
• step 150 the method returns to the request step (step 110).
• the artificial neural network uses the Mask R-CNN system, which was mentioned above.
• three networks can then be trained in parallel, namely: a classifier to recognize the products (for example fruits and vegetables), a network to determine the limitation framework ("bounding box ”), and a network to identify the mask of the instance of the product detected.
• a metric called “Mean Average Precision” with a threshold of 0.5.
• the input data is generated with a library comprising actual photographs of products, on which the objects that correspond to these products are delimited.
• a library comprising actual photographs of products, on which the objects that correspond to these products are delimited.
• augmentation in a step called augmentation, we modify the initial shots by one or two random transformations selected from the following group: rotation (of 60 °, 120 ° or 360 ° ), mirror (vertical or horizontal), change in brightness, Gaussian blur, addition of Gaussian noise, affine transformations (translations), scaling (preferably by a factor chosen in the range [0.5, 1.2]).
• the learning by the Mask R-CNN network is then divided into three steps: in a first step, 15 learning periods (“epochs”) are used with a learning rate (“learning rate”). ”) T given, in a second step 15 other periods are used with a learning rate T / 10, in a third step 10 other periods are used with a learning rate T / 50.
• learning rate controls the convergence step
• This approach known in artificial intelligence, makes it possible to minimize the calculation time and prevents the mathematical model from getting stuck in a local minimum.
• FIG. 6 schematically illustrates a method used to determine the mapping of the exhibition surface 2 of an installation 1 according to the invention.
• a first step 210 called here "arrangement of the display surface"
• several crates are placed on the product display surface 2, each crate being filled with products of a single species; different crates may contain different products.
• products of a single species are stacked on different zones 6 of the display surface 2; different areas may contain different products. It is preferable that said zones are delimited between them; this can typically be done by using crates (where appropriate a space between two crates can be used as a zone), or by providing a delimitation between the zones, for example in the form of a separator delimiting two bordering zones, or again by positioning the products so that they form separate zones without mixing.
• Said products can be fruits or vegetables.
• the system takes a picture of the stall or part of the stall using the vision system 3; we thus obtain an image of all or part of the exhibition surface 2 of the installation 1.
• a third step 230 the system analyzes the image to recognize the different zones, each zone containing (in a visible manner on the shot) only products of the same species, knowing that certain zones may contain the same product. .
• This product is then identified (fourth step 240), preferably by its species, if not by its color and / or its shape. This identification can be done in different ways: for products which can be defined (and distinguished from products of different species present on the exhibition surface) by their geometric shape and / or by their color, a method of image analysis that searches for the contours and colors of surfaces.
• software is used comprising an artificial neural system, as will be described below; this system is configured to automatically recognize the species of a number of products which do not need to be identical, as long as they belong to the same species.
• the product can be recognized by its shape and / or color.
• the system assigns it an identifier, which can be any label (digital for example), or more precisely a typical image selected from an image library (for example an image showing a banana or an image showing an eggplant. ), or a name (for example "banana” or "aubergine”).
• the system also determines a confidence index for this identification or recognition.
• the product can be recognized by an artificial neural system, configured as described above.
• the steps of recognition 230 and identification 240 are inseparable and carried out at the same time.
• the system is configured to allow the user (department manager for example) to modify this identifier, in particular if the identification is incorrect, or to add an identifier. if no identifier has been proposed by the system for a given zone.
• This mapping operation can be done when the store opens, or it can be repeated whenever the arrangement of products on areas of the display area changes, for example when a product of a given species is exhausted and must be replaced by a product of another species.
• the installation according to the invention can be used with products of different species sold in bulk. It can be used with plant products of the same species but belonging to different varieties.
• different plant varieties belonging to the “pear” species can be treated as different “species”, that is to say that the installation according to the invention can be configured to distinguish between different varieties. pears, such as the varieties Abate, Celecommunicationna, Passe-Crassane, Comice, which are distinguished by their shape and / or color and / or other visually perceptible characteristics.
• the invention can be realized with numerous variants and / or additional functionalities, which can be combined with one another, and which can be combined with all the embodiments and all the variants described above with which they are compatible.
• the system can be configured to identify a crate, to determine their position on the zones, and / or to count them.
• the system is configured to count the number of penetration events of objects of the "hand" or "arm” type in the detection perimeter per unit of time (for example every hour). This information can be transmitted to the staff, and more particularly by means of a remote screen 20 which displays information for the employees of the store, or by means of the central computer machine 23.
• the visual recognition software was trained by taking pictures of fruits and vegetables taken under different point-of-sale conditions (dry atmosphere, humidification, lighting, etc.), with different levels of degradation or stages of freshness. Other images were also generated from these shots by adding noise, blurring, and / or modifying the magnification. This test was carried out with about sixty species and varieties of fruits and vegetables: The following species were used: lettuce, carrot, pepper, eggplant, tomato, leek, zucchini, cucumber, cauliflower, apple, pear, banana, pitaya, carambola, mangosteen, persimmon, green bean, plum. For apples, pears, tomatoes, zucchini, peppers, several varieties were used, namely:
• - Zucchini zucchini flowers, round zucchini, trumpet zucchini, white zucchini, yellow zucchini.
• Tomatoes marmande tomato, Yoom TM tomato, long tomato, green tomato, pineapple tomato, black tomato, cherry tomato, cocktail tomato, cluster cocktail tomato, beef tomato.
• the image with the recognized products was displayed on a screen, along with an estimate of their volume.
• Piloting information was sent to the nebulizers and to the chill plate, to adapt nebulization in at least one area of interest and / or to adapt the chill plate refrigeration.
• the system recognizes products in the different areas of interest of the exhibition area; it adapts the messages that are delivered on the display screen when a buyer's hand enters the perimeter above a given product, to that product.
• the buyer approaches an installation and enters with his hand into the sensing perimeter of the vision system, above an area of interest on which the product of his interest is located.
• the name of the product (which the buyer may not know, especially in the case of a so-called exotic fruit or in the case of a buyer who does not speak the language of the country) and its price are displayed on a screen placed in its field of vision.
• a recipe proposed by the system or selected by the purchaser from a list, is displayed on a screen of the system.
• the buyer then formulates a request to obtain a printout on paper or sending it wirelessly to his previously entered address (phone number, email address or other).
• Printing on paper is delivered to it, for example by a printer integrated into the installation.
• the system detects, for products of a first species, for example lettuce, that the quantity of products exposed in the zone of interest has fallen to- below a predefined threshold value.
• the radioelectric system which comprises said installation, a central computer machine and wireless transmission means, sends an electronic message, generated by the system automatically, to the portable terminal held by the department manager, warning it to restock. the lettuce.
• This sending of the electronic message can use a GSM network (for example an SMS type message) or other.
• the system determines the new quantity of lettuce available and saves the information in a memory space of the central computer machine.

Landscapes

• Engineering & Computer Science (AREA)
• Business, Economics & Management (AREA)
• Theoretical Computer Science (AREA)
• Physics & Mathematics (AREA)
• General Physics & Mathematics (AREA)
• Accounting & Taxation (AREA)
• Finance (AREA)
• Strategic Management (AREA)
• Development Economics (AREA)
• Economics (AREA)
• Multimedia (AREA)
• Evolutionary Computation (AREA)
• Computer Vision & Pattern Recognition (AREA)
• Marketing (AREA)
• General Business, Economics & Management (AREA)
• Entrepreneurship & Innovation (AREA)
• Artificial Intelligence (AREA)
• Operations Research (AREA)
• Software Systems (AREA)
• Quality & Reliability (AREA)
• Health & Medical Sciences (AREA)
• Human Resources & Organizations (AREA)
• Computing Systems (AREA)
• Databases & Information Systems (AREA)
• General Health & Medical Sciences (AREA)
• Medical Informatics (AREA)
• Tourism & Hospitality (AREA)
• Data Mining & Analysis (AREA)
• Game Theory and Decision Science (AREA)
• Bioinformatics & Cheminformatics (AREA)
• Bioinformatics & Computational Biology (AREA)
• Evolutionary Biology (AREA)
• General Engineering & Computer Science (AREA)
• Life Sciences & Earth Sciences (AREA)
• Thermal Sciences (AREA)
• Image Analysis (AREA)
• Management, Administration, Business Operations System, And Electronic Commerce (AREA)

Applications Claiming Priority (3)

Publications (1)

Family

ID=74595340

Family Applications (1)

Country Status (2)

Family Cites Families (8)

• 2021
  • 2021-02-11 WO PCT/IB2021/051119 patent/WO2021161207A1/fr unknown
  • 2021-02-11 EP EP21705276.0A patent/EP4104092A1/de active Pending

Also Published As

Similar Documents

Legal Events