Classifications

• • G—PHYSICS
  • G06—COMPUTING; CALCULATING OR COUNTING
  • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
  • G06V30/00—Character recognition; Recognising digital ink; Document-oriented image-based pattern recognition
  • G06V30/10—Character recognition
  • G06V30/14—Image acquisition
  • G06V30/1434—Special illumination such as grating, reflections or deflections, e.g. for characters with relief
• • G—PHYSICS
  • G06—COMPUTING; CALCULATING OR COUNTING
  • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
  • G06V10/00—Arrangements for image or video recognition or understanding
  • G06V10/10—Image acquisition
  • G06V10/12—Details of acquisition arrangements; Constructional details thereof
  • G06V10/14—Optical characteristics of the device performing the acquisition or on the illumination arrangements
  • G06V10/145—Illumination specially adapted for pattern recognition, e.g. using gratings
• • G—PHYSICS
  • G06—COMPUTING; CALCULATING OR COUNTING
  • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
  • G06V10/00—Arrangements for image or video recognition or understanding
  • G06V10/40—Extraction of image or video features
  • G06V10/60—Extraction of image or video features relating to illumination properties, e.g. using a reflectance or lighting model
• • G—PHYSICS
  • G06—COMPUTING; CALCULATING OR COUNTING
  • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
  • G06V30/00—Character recognition; Recognising digital ink; Document-oriented image-based pattern recognition
  • G06V30/10—Character recognition
  • G06V30/18—Extraction of features or characteristics of the image
  • G06V30/18124—Extraction of features or characteristics of the image related to illumination properties, e.g. according to a reflectance or lighting model

Definitions

• the present disclosure refers to a system and method for object recognition using 3D mapping and modeling of light.
• Computer vision is a field in rapid development due to abundant use of electronic devices capable of collecting information about their surroundings via sensors such as cameras, distance sensors such as LiDAR or radar, and depth camera systems based on structured light or stereo vision to name a few. These electronic devices provide raw image data to be processed by a computer processing unit and consequently develop an understanding of an environment or a scene using artificial intelligence and/or computer assistance algorithms. There are multiple ways how this understanding of the environment can be developed. In general, 2D or 3D images and/or maps are formed, and these images and/or maps are analyzed for developing an understanding of the scene and the objects in that scene. One prospect for improving computer vision is to measure the components of the chemical makeup of objects in the scene. While shape and appearance of objects in the environment acquired as 2D or 3D images can be used to develop an understanding of the environment, these techniques have some shortcomings.
• object recognition the capability of a computer vision system to identify an object in a scene is termed as "object recognition".
• object recognition a computer analyzing a picture and identifying/labelling a ball in that picture, sometimes with even further information such as the type of a ball (basketball, soccer ball, baseball), brand, the context, etc. fall under the term "object recognition”.
• Technique 1 Physical tags (image based): Barcodes, QR codes, serial numbers, text, patterns, holograms etc.
• Technique 2 Physical tags (scan/close contact based): Viewing angle dependent pigments, upconversion pigments, metachromics, colors (red/green), luminescent materials.
• Technique 3 Electronic tags (passive): RFID tags, etc. Devices attached to objects of interest without power, not necessarily visible but can operate at other frequencies (radio for example).
• Technique 4 Electronic tags (active): wireless communications, light, radio, vehicle to vehicle, vehicle to anything (X), etc. Powered devices on objects of interest that emit information in various forms.
• Technique 5 Feature detection (image based): Image analysis and identification, i.e. two wheels at certain distance for a car from side view; two eyes, a nose and mouth (in that order) for face recognition etc. This relies on known geometries/shapes.
• Technique 6 Deep learning/CNN based (image based): Training of a computer with many of pictures of labeled images of cars, faces etc. and the computer determining the features to detect and predicting if the objects of interest are present in new areas. Repeating of the training procedure for each class of object to be identified is required.
• Technique 7 Object tracking methods: Organizing items in a scene in a particular order and labeling the ordered objects at the beginning. Thereafter following the object in the scene with known color/geometry/3D coordinates. If the object leaves the scene and re-enters, the "recognition" is lost.
• Technique 1 When an object in the image is occluded or only a small portion of the object is in the view, the barcodes, logos etc. may not be readable. Furthermore, the barcodes etc. on flexible items may be distorted, limiting visibility. All sides of an object would have to carry large barcodes to be visible from a distance otherwise the object can only be recognized in close range and with the right orientation only. This could be a problem for example when a barcode on an object on the shelf at a store is to be scanned. When operating over a whole scene, technique 1 relies on ambient lighting that may vary.
• Upconversion pigments have limitations in viewing distances because of the low level of emitted light due to their small quantum yields. They require strong light probes. They are usually opaque and large particles limiting options for coatings. Further complicating their use is the fact that compared to fluorescence and light reflection, the upconversion response is slower. While some applications take advantage of this unique response time depending on the compound used, this is only possible when the time of flight distance for that sensor/object system is known in advance. This is rarely the case in computer vision applications. For these reasons, anti-counterfeiting sensors have covered/dark sections for reading, class 1 or 2 lasers as probes and a fixed and limited distance to the object of interest for accuracy.
• viewing angle dependent pigment systems only work in close range and require viewing at multiple angles. Also, the color is not uniform for visually pleasant effects. The spectrum of incident light must be managed to get correct measurements. Within a single image/scene, an object that has angle dependent color coating will have multiple colors visible to the camera along the sample dimensions.
• Luminescence based recognition under ambient lighting is a challenging task, as the reflective and luminescent components of the object are added together.
• luminescence based recognition will instead utilize a dark measurement condition and a priori knowledge of the excitation region of the luminescent material so the correct light probe/source can be used.
• Technique 3 Electronic tags such as RFID tags require the attachment of a circuit, power collector, and antenna to the item/object of interest, adding cost and complication to the design.
• RFID tags provide present or not type information but not precise location information unless many sensors over the scene are used.
• Technique 4 These active methods require the object of interest to be connected to a power source, which is cost-prohibitive for simple items like a soccer ball, a shirt, or a box of pasta and are therefore not practical.
• Technique 5 The prediction accuracy depends largely on the quality of the image and the position of the camera within the scene, as occlusions, different viewing angles, and the like can easily change the results.
• logo type images can be present in multiple places within the scene (i.e.
• a logo can be on a ball, a T-shirt, a hat, or a coffee mug) and the object recognition is by inference.
• the visual parameters of the object must be converted to mathematical parameters at great effort.
• Flexible objects that can change their shape are problematic as each possible shape must be included in the database. There is always inherent ambiguity as similarly shaped objects may be misidentified as the object of interest.
• Technique 6 The quality of the training data set determines the success of the method. For each object to be recognized/classified many training images are needed. The same occlusion and flexible object shape limitations as for Technique 5 apply. There is a need to train each class of material with thousands or more of images.
• edge or cloud computing For applications that require instant responses like autonomous driving or security, the latency is another important aspect.
• the amount of data that needs to be processed determines if edge or cloud computing is appropriate for the application, the latter being only possible if data loads are small.
• edge computing is used with heavy processing, the devices operating the systems get bulkier and limit ease of use and therefore implementation.
• a system for object recognition via a computer vision application comprising at least the following components:
• the object having an object specific reflectance spectral pattern and an object specific luminescence spectral pattern
• At least one light source which is configured to illuminate a scene which includes the at least one object under ambient light conditions, the at least one light source having light source specific radiance values, - a sensor which is configured to measure radiance data of the scene including the at least one object when the scene is illuminated by the light source,
• a scene mapping tool which is configured to map the scene rendering at least a partial 3D map of the scene
• a data processing unit which is configured to analyse data received from the scene mapping tool and to merge the analysed data with the light source specific radiance values, and, based thereon, to calculate radiance of light incident at points in the scene, particularly at points on the at least one object, and to combine the calculated radiance of light incident at the points in the scene with the measured radiance of light returned to the sensor from points in the scene, particularly from points on the at least one object, thus forming a model of light spectral distribution and intensity at the at least one object in the scene, and to extract/detect the object specific luminescence and/or reflectance spectral pattern of the at least one object to be recognized out of the model of light spectral distribution and intensity and to match the extracted/detected object specific luminescence and/or reflectance spectral pattern with the luminescence and/or reflectance spectral patterns stored in the data storage unit, and to identify a best matching luminescence and/or reflectance spectral pattern and, thus, its assigned object,
• the sensor, the scene mapping tool, the data storage unit and the data processing unit are in communicative connection with each other and linked together wirelessly and/or through wires and synchronized with the light source by default, thus forming an integrated system.
• a communicative connection between any of the components may be a wired or a wireless connection.
• Each suitable communication technology may be used.
• the respective components each may include one or more communications interface for communicating with each other. Such communication may be executed using a wired data transmission protocol, such as fiber distributed data interface (FDDI), digital subscriber line (DSL), Ethernet, asynchronous transfer mode (ATM), or any other wired transmission protocol.
• FDDI fiber distributed data interface
• DSL digital subscriber line
• Ethernet asynchronous transfer mode
• ATM asynchronous transfer mode
• the communication may be wirelessly via wireless communication networks using any of a variety of protocols, such as General Packet Radio Service (GPRS), Universal Mobile Telecommunications System (UMTS), Code Division Multiple Access (CDMA), Long Term Evolution (LTE), wireless Universal Serial Bus (USB), and/or any other wireless protocol.
• GPRS General Packet Radio Service
• UMTS Universal Mobile Telecommunications System
• CDMA Code Division Multiple Access
• LTE Long Term Evolution
• USB wireless Universal Serial Bus
• the respective communication may be a combination of a wireless and a wired communication.
• fluorescent and “luminescent” are used synonymously. The same applies to the terms “fluorescence” and "luminescence”.
• the points in the scene which are considered are in the field of view or line of sight of at least one of the light source, the sensor and the 3D mapping tool. If a point in the scene is not in line of sight of any of the three components, that point is not considered for forming the model. It is possible that the system comprises multiple sensors/cameras, light sources and/or mapping tools in the scene. Nevertheless, a partial coverage of the scene by any of those system components is sufficient, i.e. not all points in the scene need to be considered. It is to be stated that further calculation of radiance may be done inside, i. e. within the boundaries of, the at least partial 3D map obtained from the scene mapping tool.
• the 3D mapping tool i. e. the scene mapping tool is used to map part of the scene, then the 3D map is used to calculate radiance of light incident at points in the partially mapped scene.
• the light source can designed to connect automatically to at least one of the further components of the system such as the sensor, the scene mapping tool, the data storage unit and/or the data processing unit.
• the light source does not have to be linked to and/or networked with the other components of the system (if light source has predefined and known parameters, e. g. radiance values, pulse rates and timing, etc.), but need to be synchronized with the other components. This synchronization may be accomplished with measurements from the other components of the system, such as a spectral camera.
• the radiance of the light source is measured by at least one spectroradiometer, i.e. the system may be initialized with a spectroradiometer. However, generally this is only done for a setup of the system, but generally not in real time, i.e. not in operating mode of the system.
• the light source specific radiance values comprise spectral characteristics, power and/or an emission angle profile (light output profile) of the at least one light source in the scene.
• Radiance of the at least one light source at points of the at least one object in the scene is calculated by using the light source specific radiance values, particularly the spectral characteristics, the power and/or the emission angle profile of the at least one light source in the scene and mapping a distance from the at least one light source to the at least one object in the scene.
• the senor is a multispectral or hyperspectral camera.
• the sensor is generally an optical sensor with photon counting capabilities. More specifically, it may be a monochrome camera, or an RGB camera, or a multispectral camera, or a hyperspectral camera.
• the sensor may be a combination of any of the above, or the combination of any of the above with a tuneable or selectable filter set, such as, for example, a monochrome sensor with specific filters.
• the sensor may measure a single pixel of the scene, or measure many pixels at once.
• the optical sensor may be configured to count photons in a specific range of spectrum, particularly in more than three bands.
• a multispectral camera captures image data within specific wavelength ranges across the electromagnetic spectrum.
• the wavelengths may be separated by filters or by the use of instruments that are sensitive to particular wavelengths, including light from frequencies beyond the visible light range, i.e. infrared and ultra-violet.
• Spectral imaging can allow extraction of additional information the human eye fails to capture with its receptors for red, green and blue.
• a multispectral camera measures light in a small number (typically 3 to 15) of spectral bands.
• a hyperspectral camera is a special case of spectral camera where often hundreds of contiguous spectral bands are available.
• the scene mapping tool is configured to perform a scene mapping by using a technique based on time of flight (TOF), stereovision and/or structured light.
• the scene mapping tool may comprise at least one of a time of flight system, such as TOF-cameras, a stereovision-based system, a light probe which emits structured light or any combination thereof.
• the structured light may be, for example, infrared light. Time of flight measurements can use infrared light, visible light or radar.
• Alternative scene mapping tools are (ultra)sound-based systems.
• the system is configured to use physical location (received via GPS), compass orientation, time of day, and/or weather conditions to model an effect of solar radiation on the illumination of the at least one object in the scene. Those influencing factors are considered in the model, i.e. incorporated into the model.
• the system is configured to use information of reflective and fluorescence properties of not only the at at least one object but also of other items in the scene to improve radiance mapping of the scene by means of bidirectional reflectance distribution functions (BRDFs) and bidirectional fluorescence distribution functions (BFDFs) to account for interreflections of reflected and fluoresced light throughout the scene.
• BRDFs bidirectional reflectance distribution functions
• BFDFs bidirectional fluorescence distribution functions
• the system comprises at least one white tile located at least one point in the scene, the white tile being configured to be used to measure radiance of the light source at the at least one point in the scene, wherein the measured radiance of the light source at the at least one point in the scene is used in conjunction with the 3D map and the light output profile of the light source to estimate radiance at other points in the scene.
• Highly reflective white tile(s) in the scene can be used to measure radiance from the light source at that point in the scene. This will also give the spectral characteristics of the light source.
• estimates of the radiance at other points in the scene can then be made.
• the white tile(s) could also be used for "smart" systems that are networked with information about the light source to validate the calculations in addition to determining contributions from light sources outside of the system described.
• the present disclosure also refers to a method for object recognition via a computer vision application, the method comprising at least the following steps:
• a data processing unit which is programmed to analyze data received from the scene mapping tool and merge the analysed data with the light source specific radiance values to calculate radiance of light incident at points in the scene, particularly at points of the at least one object, and to combine the calculated radiance of light incident at the points in the scene with the measured radiance of light returned to the sensor from points in the scene, particularly from the at least one object, thus forming a model of light spectral distribution and intensity at the at least one object in the scene, and to extract/detect the object specific luminescence and/or reflectance spectral pattern of the at least one object to be recognized out of the model of light spectral distribution and intensity and to match the extracted/detected object specific luminescence and/or reflectance spectral pattern with the luminescence and/or reflectance spectral patterns stored in the data storage unit, and to identify a best matching luminescence and/or reflectance spectral pattern and, thus, its assigned object,
• a scene mapping is performed by using a technique based on time of flight (TOF) and/or structured light and/or stereocameras wherein at least one of a time of flight system, a sound-based system, a stereovision-based system or any combination thereof is used.
• TOF time of flight
• a sound-based system a stereovision-based system or any combination thereof is used.
• Infrared, visible, UV light can be used.
• radar, stereovision and/or ultrasound can be used here.
• radiance of the at least one light source at the at least one object in the scene is calculated using the light source specific radiance values, such as spectral characteristics, power and/or an emission angle profile of the at least one light source in the scene, and mapping a distance from the at least one light source to the at least one object in the scene.
• physical location (determined via GPS), compass orientation, time of day, and/or weather conditions may be used to model an effect of solar radiation on the illumination of the scene, thus adapting the model accordingly.
• information of the reflective and fluorescence properties of items (not only of the at least one object) in the scene is used to improve radiance mapping of the scene by means of bidirectional reflectance distribution functions (BRDFs) and bidirectional fluorescence distribution functions (BFDFs) to account for interreflections of reflected and fluoresced light throughout the scene.
• BRDFs bidirectional reflectance distribution functions
• BFDFs bidirectional fluorescence distribution functions
• the model of light spectral distribution and intensity can be analyzed and displayed on a 2D map or as a 3D view via a respective output device, such as a display or a screen configured to issue a 3D map/view.
• Embodiments of the invention may be used with or incorporated in a computer system that may be a standalone unit or include one or more remote terminals or devices in communication with a central computer, located, for example, in a cloud, via a network such as, for example, the Internet or an intranet.
• the data processing unit described herein and related components may be a portion of a local computer system or a remote computer or an online system or a combination thereof.
• the database i.e. the data storage unit and software described herein may be stored in computer internal memory or in a non- transitory computer-readable medium. Within the scope of the present disclosure the database may be part of the data storage unit or may represent the data storage unit itself.
• the terms "database” and "data storage unit” are used synonymously.
• the present disclosure further referes to a computer program product having instructions that are executable by a data processing unit as provided as component/part of the proposed system, the instructions cause the system to: analyse data received from the scene mapping tool,
• the present disclosure also refers to a non-transitory computer-readable medium storing instructions that, when executed by one or more data processing units as component(s) of the proposed system, cause the system to:
• the present disclosure describes a method for object recognition and a chemistry-based object recognition system comprising a light source(s), a sensor, particulary a camera, a database of luminescence and/or reflectance spectral patterns of different objects and a computer/data processing unit that is configured to compute a spectral match of such luminescent and/or reflective objects of the database using various algorithms, a 3D map of scenes and a model of light spectral distribution and intensity (illuminance) at target objects in the field of view of the sensor.
• luminescent/chemistry-based object recognition techniques are simplified and improved.
• the invention is further defined in the following examples.
• Fig. 1 shows schematically an arrangement of an embodiment of the system according to the invention.
• Figure 1 shows an embodiment of the system 100 for object recognition via a computer vision application.
• the system 100 comprises at least one object 1 10 which is to be recognized.
• the object 1 10 has an object-specific reflectance spectral pattern and an object-specific luminescence spectral pattern.
• the object 1 10 is further located in a scene 130.
• the system 100 further comprises a first light source 121 and a second light source 122. Both light sources are configured to illuminate the scene 130 including the at least one object 1 10, preferably under ambient light conditions.
• the system 100 further comprises a sensor 140 which is configured to measure radiance data of the scene 130 including the at least one object 1 10 when the scene 130 is illuminated by at least one of the light sources 121 and 122.
• the senor 140 is a multispectral or a hyperspectral camera.
• the system 100 further comprises a scene mapping tool 150 which is configured to map the scene 130 rendering at least a partial 3D map of the scene 130. Further shown is a data storage unit 160 which comprises luminescence and/or reflectance spectral patterns together with appropriately assigned respective objects.
• the system 100 further comprises a data processing unit 170 which is configured to analyze data received from the scene mapping tool 150, merge the analyzed data with light source specific radiance parameters/values and calculate radiance of light incident at points in the scene 130, particularly at points of the object 1 10.
• the radiance of light incident at a specific point in the scene 130 can be formulated via a function of light intensity l(x, y, z) with (x, y, z) designating the space coordinates of the specific point wthin the scene 130.
• the calculated radiance of light incident at the points in the scene 130 is combined with a measured radiance of light returned to the camera 140 from points in the scene, particularly from points of the object 1 10. Based on such combination of calculated radiance and measured radiance, a model of light spectral distribution and intensity at the object 1 10 in the scene is formed.
• the data processing unit 170 is further configured to calculate out of the model of light spectral distribution and intensity the object-specific luminescence and/or reflectance spectral pattern of the object 1 10 and to match the object-specific luminescence and/or reflectance spectral pattern of the object 1 10 with the luminescence and/or reflectance spectral patterns stored in the data storage unit 160. Thereby, a best matching luminescence and/or reflectance spectral pattern can be identified and the object 1 10 is identified as the object which is assigned within the database to this best matching luminescence and/or reflectance spectral pattern.
• the camera 140, the scene mapping tool 150, the database 160 and the data processing unit 170 are in communicative connection with each other and linked together wirelessly and/or through wires, thus forming an integrated system.
• the light sources 121 and 122 may be linked to, but must not be linked to, the other components of the system. However, the light sources have to be synchronized with the other components.
• the light sources 121 , 122 may be controlled by, for example, the data processing unit 170 or any other controller.
• a further sensor, such as a spectroradiometer, which is configured to measure radiance data of the light sources 121 , 122 may be useful but not necessary. Generally, a factory production specification will be available for the radiance of each light source 121 , 122.
• Information about the light sources 121 , 122 may be combined with the partial 3D map of the scene 130 which is provided by the scene mapping tool 150, in order to calculate radiance at different points in the scene 130. That means that light radiance at points of interest in the scene 130, particularly at points of the object 1 10 is calculated based on the properties of the light sources 121 and 122 and the 3D map of the scene outputted by the scene mapping tool 150 (3D mapping tool). Further information, such as information about a physical location, a compass orientation, a time of day, and weather conditions may be used to model an effect of solar radiation on the illumination of the scene 130.
• the scene mapping tool 150 may perform scene mapping using a technique based on time of flight and/or structured light using, for example, infrared light. However, visible light, radar, stereovision, and/or ultrasound may be possible alternatives.
• the scene mapping tool 150 may comprise at least one of a time of flight system (e. g. a LiDAR system), a sound-based system, a stereovision-based system or any combination thereof.
• Knowledge of reflective and fluorescent properties of objects/items in the scene 130 may be used to improve the scene mapping with techniques such as bidirectional reflectance distribution functions and bidirectional fluorescence distribution functions to account for interreflections of reflected and fluoresced light throughout the scene 130.
• the bidirectional reflectance distribution function indicates how light is reflected at an opaque surface within the scene 130.
• the 3D mapping performed by the scene mapping tool can be improved as further effects due to reflected and fluoresced light emitted by further objects in the scene can be considered.
• the 3D mapping is more realistic as there are generally more than only the at least one object to be recognized within the scene. Due to the knowledge or the measuring of spectral characteristics and power of the illuminants, i.e. the light sources 121 and 122 in the scene 130, and by mapping distances from the light sources 121 , 122 to a plurality of objects in the scene 130, such as the desk 131 and the chair 132 which are previously known, accurate radiances can be derived and calculated at any point in the scene 130.
• the scene mapping can be performed by the scene mapping tool 150 using a variety of different techniques.
• a most common technique is based on time of flight measurements.
• a further possibility is the usage of structured light.
• a 3D map of the scene can be formed, thus giving information about specific coordinates of the respective objects within the scene.
• the object-specific fluorescence spectral pattern can be filtered out of the calculated radiance model of the scene.
• the radiance mapping of the scene can be improved by using bidirectional reflectance distribution functions and bidirectional fluorescence distribution functions to account for interreflections of reflected and fluoresced light throughout the scene.

Landscapes

• Engineering & Computer Science (AREA)
• Physics & Mathematics (AREA)
• General Physics & Mathematics (AREA)
• Multimedia (AREA)
• Theoretical Computer Science (AREA)
• Computer Vision & Pattern Recognition (AREA)
• Software Systems (AREA)
• Artificial Intelligence (AREA)
• Image Processing (AREA)
• Image Analysis (AREA)
• Length Measuring Devices By Optical Means (AREA)
• Image Generation (AREA)
• Optical Radar Systems And Details Thereof (AREA)

Applications Claiming Priority (3)

Publications (1)

Family

ID=70977985

Family Applications (1)

Country Status (11)

Families Citing this family (2)

Family Cites Families (6)

• 2020
  • 2020-06-05 BR BR112021019027A patent/BR112021019027A2/pt unknown
  • 2020-06-05 JP JP2021572405A patent/JP7277615B2/ja active Active
  • 2020-06-05 EP EP20730650.7A patent/EP3980925A1/en active Pending
  • 2020-06-05 KR KR1020217039557A patent/KR20220004738A/ko unknown
  • 2020-06-05 CN CN202080034862.XA patent/CN113811888A/zh active Pending
  • 2020-06-05 CA CA3140449A patent/CA3140449A1/en active Pending
  • 2020-06-05 WO PCT/EP2020/065751 patent/WO2020245444A1/en active Application Filing
  • 2020-06-05 TW TW109119102A patent/TW202105330A/zh unknown
  • 2020-06-05 AU AU2020288708A patent/AU2020288708A1/en not_active Abandoned
  • 2020-06-05 SG SG11202113354UA patent/SG11202113354UA/en unknown
  • 2020-06-05 MX MX2021014833A patent/MX2021014833A/es unknown

Also Published As

Similar Documents

Legal Events