WO2022081057A1 - Material imaging analyzer and method for its use - Google Patents
Material imaging analyzer and method for its use Download PDFInfo
- Publication number
- WO2022081057A1 WO2022081057A1 PCT/SE2021/000008 SE2021000008W WO2022081057A1 WO 2022081057 A1 WO2022081057 A1 WO 2022081057A1 SE 2021000008 W SE2021000008 W SE 2021000008W WO 2022081057 A1 WO2022081057 A1 WO 2022081057A1
- Authority
- WO
- WIPO (PCT)
- Prior art keywords
- radiation emitting
- imaging analyzer
- accordance
- material imaging
- spectral
- Prior art date
Links
- 239000000463 material Substances 0.000 title claims abstract description 119
- 238000003384 imaging method Methods 0.000 title claims abstract description 64
- 238000000034 method Methods 0.000 title claims abstract description 53
- 230000005855 radiation Effects 0.000 claims abstract description 71
- 230000003595 spectral effect Effects 0.000 claims abstract description 46
- 238000012512 characterization method Methods 0.000 claims abstract description 19
- 230000005670 electromagnetic radiation Effects 0.000 claims description 44
- 230000003287 optical effect Effects 0.000 claims description 10
- 238000004364 calculation method Methods 0.000 claims description 8
- 238000000611 regression analysis Methods 0.000 claims description 5
- 238000004422 calculation algorithm Methods 0.000 claims description 4
- 238000001914 filtration Methods 0.000 claims description 3
- 238000012986 modification Methods 0.000 claims description 3
- 230000004048 modification Effects 0.000 claims description 3
- 238000000513 principal component analysis Methods 0.000 claims description 3
- 238000004590 computer program Methods 0.000 claims description 2
- 238000004458 analytical method Methods 0.000 description 14
- 238000005516 engineering process Methods 0.000 description 9
- 238000013473 artificial intelligence Methods 0.000 description 8
- 238000009826 distribution Methods 0.000 description 8
- 230000002123 temporal effect Effects 0.000 description 5
- 238000000701 chemical imaging Methods 0.000 description 4
- 238000005259 measurement Methods 0.000 description 4
- 230000001360 synchronised effect Effects 0.000 description 4
- 238000012549 training Methods 0.000 description 4
- 238000001514 detection method Methods 0.000 description 3
- 239000006185 dispersion Substances 0.000 description 3
- 230000003252 repetitive effect Effects 0.000 description 3
- 230000035945 sensitivity Effects 0.000 description 3
- 238000001228 spectrum Methods 0.000 description 3
- 230000003068 static effect Effects 0.000 description 3
- 208000034656 Contusions Diseases 0.000 description 2
- 239000010426 asphalt Substances 0.000 description 2
- 238000004891 communication Methods 0.000 description 2
- 238000013480 data collection Methods 0.000 description 2
- 230000006870 function Effects 0.000 description 2
- 238000007689 inspection Methods 0.000 description 2
- 238000007619 statistical method Methods 0.000 description 2
- XLYOFNOQVPJJNP-UHFFFAOYSA-N water Substances O XLYOFNOQVPJJNP-UHFFFAOYSA-N 0.000 description 2
- 239000002028 Biomass Substances 0.000 description 1
- 238000004566 IR spectroscopy Methods 0.000 description 1
- 241000220324 Pyrus Species 0.000 description 1
- 238000010521 absorption reaction Methods 0.000 description 1
- 238000013528 artificial neural network Methods 0.000 description 1
- 230000001427 coherent effect Effects 0.000 description 1
- 238000005094 computer simulation Methods 0.000 description 1
- 239000012141 concentrate Substances 0.000 description 1
- 238000013461 design Methods 0.000 description 1
- 238000011161 development Methods 0.000 description 1
- 235000013399 edible fruits Nutrition 0.000 description 1
- 229910052736 halogen Inorganic materials 0.000 description 1
- 150000002367 halogens Chemical class 0.000 description 1
- 238000005286 illumination Methods 0.000 description 1
- 230000001678 irradiating effect Effects 0.000 description 1
- 238000012423 maintenance Methods 0.000 description 1
- 238000000691 measurement method Methods 0.000 description 1
- 230000010355 oscillation Effects 0.000 description 1
- 235000021017 pears Nutrition 0.000 description 1
- 230000010363 phase shift Effects 0.000 description 1
- 238000012545 processing Methods 0.000 description 1
- 239000002994 raw material Substances 0.000 description 1
- 238000004064 recycling Methods 0.000 description 1
- 239000000126 substance Substances 0.000 description 1
- 238000012706 support-vector machine Methods 0.000 description 1
- 230000001629 suppression Effects 0.000 description 1
- 229910052724 xenon Inorganic materials 0.000 description 1
- FHNFHKCVQCLJFQ-UHFFFAOYSA-N xenon atom Chemical compound [Xe] FHNFHKCVQCLJFQ-UHFFFAOYSA-N 0.000 description 1
Classifications
-
- G—PHYSICS
- G01—MEASURING; TESTING
- G01N—INVESTIGATING OR ANALYSING MATERIALS BY DETERMINING THEIR CHEMICAL OR PHYSICAL PROPERTIES
- G01N21/00—Investigating or analysing materials by the use of optical means, i.e. using sub-millimetre waves, infrared, visible or ultraviolet light
- G01N21/17—Systems in which incident light is modified in accordance with the properties of the material investigated
- G01N21/25—Colour; Spectral properties, i.e. comparison of effect of material on the light at two or more different wavelengths or wavelength bands
-
- G—PHYSICS
- G01—MEASURING; TESTING
- G01N—INVESTIGATING OR ANALYSING MATERIALS BY DETERMINING THEIR CHEMICAL OR PHYSICAL PROPERTIES
- G01N21/00—Investigating or analysing materials by the use of optical means, i.e. using sub-millimetre waves, infrared, visible or ultraviolet light
- G01N21/17—Systems in which incident light is modified in accordance with the properties of the material investigated
- G01N21/25—Colour; Spectral properties, i.e. comparison of effect of material on the light at two or more different wavelengths or wavelength bands
- G01N21/255—Details, e.g. use of specially adapted sources, lighting or optical systems
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
- H04N23/60—Control of cameras or camera modules
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
- H04N23/60—Control of cameras or camera modules
- H04N23/665—Control of cameras or camera modules involving internal camera communication with the image sensor, e.g. synchronising or multiplexing SSIS control signals
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
- H04N23/70—Circuitry for compensating brightness variation in the scene
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
- H04N23/70—Circuitry for compensating brightness variation in the scene
- H04N23/73—Circuitry for compensating brightness variation in the scene by influencing the exposure time
-
- G—PHYSICS
- G01—MEASURING; TESTING
- G01J—MEASUREMENT OF INTENSITY, VELOCITY, SPECTRAL CONTENT, POLARISATION, PHASE OR PULSE CHARACTERISTICS OF INFRARED, VISIBLE OR ULTRAVIOLET LIGHT; COLORIMETRY; RADIATION PYROMETRY
- G01J3/00—Spectrometry; Spectrophotometry; Monochromators; Measuring colours
- G01J3/28—Investigating the spectrum
-
- G—PHYSICS
- G01—MEASURING; TESTING
- G01J—MEASUREMENT OF INTENSITY, VELOCITY, SPECTRAL CONTENT, POLARISATION, PHASE OR PULSE CHARACTERISTICS OF INFRARED, VISIBLE OR ULTRAVIOLET LIGHT; COLORIMETRY; RADIATION PYROMETRY
- G01J3/00—Spectrometry; Spectrophotometry; Monochromators; Measuring colours
- G01J3/28—Investigating the spectrum
- G01J3/2823—Imaging spectrometer
-
- G—PHYSICS
- G01—MEASURING; TESTING
- G01N—INVESTIGATING OR ANALYSING MATERIALS BY DETERMINING THEIR CHEMICAL OR PHYSICAL PROPERTIES
- G01N22/00—Investigating or analysing materials by the use of microwaves or radio waves, i.e. electromagnetic waves with a wavelength of one millimetre or more
-
- G—PHYSICS
- G01—MEASURING; TESTING
- G01N—INVESTIGATING OR ANALYSING MATERIALS BY DETERMINING THEIR CHEMICAL OR PHYSICAL PROPERTIES
- G01N22/00—Investigating or analysing materials by the use of microwaves or radio waves, i.e. electromagnetic waves with a wavelength of one millimetre or more
- G01N22/04—Investigating moisture content
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
- H04N23/10—Cameras or camera modules comprising electronic image sensors; Control thereof for generating image signals from different wavelengths
Definitions
- the present invention concerns a material imaging analyzer, which is suitable to be used for the classification and characterization of surface materials in accordance with the claims, as well as a method for its use in accordance with the claims.
- a scientific article by H. Jiang et.al [2] states that a few characteristic wavelengths can be selected from hyperspectral data using, inter alia, principal component analysis.
- the aim is to be able to classify materials based on the reflection of light with only a few different wavelengths.
- the method can be used to select the wavelengths of light emitters in an application-specific material analyzer.
- the application in this particular case concerns the detection of bruising on fruit.
- Patrik Jonsson describes in a doctoral thesis from Mid Sweden University [3] how a camera sensitive to light within the near infrared wavelength area can be used to classify road conditions: ice, snow, wet, dry asphalt.
- the system can provide guidance for better maintenance of winter roads.
- the technique is based on exposing a sequence of four images. For each of the four images, carefully selected optical band-pass filters are used that allow the use of Al where a subsequent classifier (KNN) identifies which areas within the camera's inspection area are coated with snow, water, ice or dry asphalt.
- KNN classifier
- the technique works well for the inspection of static areas, i.e. no movement in the image, this is due to the long time it takes to change filters and expose four images.
- the mechanical filter change is an expensive solution, has a questionable function in cold and is slow.
- the described technology is an application-specific material analyzer but works only for static surfaces for the reasons described.
- Guterman et.al describes in a patent [4] an imaging technique using out-of-sync amplitude modulated light.
- the fact that the modulated light has not been synchronized with the camera means that the radiation of the light cannot be timed to concentrate on the periods when the image sensor exposes images. This in turn leads to low signal-noise ratio (SNR), during, for example, the impact from unwanted bright sunlight.
- SNR signal-noise ratio
- Sten Lofving describes in a patent [5] a sensor for the detection of road conditions.
- This sensor uses modulated laser light but is not imaging, i.e. measures at a limited point.
- the measurement takes place at several points along the line following the direction of travel of the vehicle.
- the described equipment is an application-specific material analyzer but would need a scanning motion in two dimensions to become imaging.
- this camera can be used as a material analyzer.
- Johan Casselgren et.al describes in a scientific article [7] a method for spectral imaging and classification of road conditions.
- This article therefore describes a method for separating the contribution of active lighting from the background.
- Laser light is binarily modulated, on or off, synchronously with imaging in such a way that a sequence of six images is exposed: background light, lighting with laser 1, background light, laser 2, background light and laser 3.
- Lasers 1, 2 and 3 emit light at different wavelengths.
- the method thus allows spectral encoding of light, and that background light can be suppressed for static scenes, but the disadvantage is that the slightest movement in the image will generate an incorrect result due to the time it takes to expose six times.
- the purpose of the present invention is to eliminate or substantially reduce at least one of the aforementioned, or in the following description mentioned, problems.
- the purpose is achieved by a device and a method in accordance with the present invention.
- Fig. 1 shows how general components have been connected to a material imaging analyzer.
- Fig. 2 shows a sinusoidal pulse width modulated signal used to control the output of electromagnetic radiation. The drawing also shows how the pulses have been synchronized with the image capture.
- Fig. 3 describes the main components and the calculations performed in the spectral demodulator.
- Fig. 4 shows a preferred exemplifying embodiment of the material analyzer.
- Figs. 5 and 6 show alternative embodiments of the material imaging analyzer.
- Fig. 7 shows a method for selecting wavelengths Xi, equipping the radiation emitting device with radiation emitting devices and configuring the modulator/demodulator.
- Fig. 8 shows examples of three different situations for how the material imaging analyzer is used.
- Fig. 9 shows a schematic exemplification of the temporal signal spectrum of the video signal.
- a device referred to as a material imaging analyzer 114 and a method for its use and adapted applications of the material imaging analyzer 114 are shown.
- One or more surfaces 101 are assumed to consist of at least two different materials and/or at least one material with varying properties.
- the material imaging analyzer allows the materials of which the surfaces consist of to be classified and characterized by contactless measurement technology and where the image provides information about the distribution of materials across the surfaces.
- the contactless measurement technique means that the surfaces are irradiated with electromagnetic radiation and that the spectral distribution of reflected radiation forms the basis for classification and characterization of the materials.
- Typical application areas for the present invention are, for example: sorting materials for recycling, characterizing raw materials such as moisture content in biomass or classification and characterization of agricultural crops over large areas.
- the distinctive feature of the material imaging analyzer is that it needs to be adapted and trained for the set of different materials it can recognize and/or characterize.
- the material imaging analyzer 114 consists of at least two main components: at least one radiation emitting device 104, and at least one receiver 115.
- Fig. 1 shows the two-dimensional surface 101 analyzed by a radiation emitting device 104, consisting of at least two or more radiation emitting units 113.1 to 113. N.
- the radiation emitting units 113.1-113. N, which illuminate surface 101 are shown in the figures.
- the N number of radiation emitting units emits electromagnetic radiation with wavelengths from a minimum of 100 nm to a maximum of 3pm but where each individual radiation emitting unit is characterized by maximum intensity at wavelengths Xi to AN.
- the word light may be used synonymously with electromagnetic radiation within a given wavelength range. Wavelength intervals other than 100 nm to 3pm can occur in alternative embodiments.
- N and Ai to AN are parameters whose values are assigned in accordance with the method described in the present patent application.
- the value of parameter N indicates the number of radiation emitting units and at the same time is the control for the dimensioning of modulator 106, spectral demodulator 109 and Al unit 111.
- the radiation emitting units consist of LED lasers and/or LED diodes.
- An LED laser emits a near monochromatic, coherent light whilst a simpler LED diode emits light with a wider wavelength range than for the LED laser.
- flashing xenon lamps in combination with optical band-pass filters can be used.
- a camera 102 images a surface 101 by recording the electromagnetic radiation reflected in the surface during repetitive time intervals called exposure time.
- the camera includes at least one image detector.
- These image detectors can be of the one-dimensional type for line imaging or of a two-dimensional type for area imaging, the pixels of which are sensitive to all existing wavelengths i to AN.
- Camera 102 generates at least one digital video signal 103 and at least one trigger signal 107 whose pulses signal the beginning of repetitive image exposures. Trigger signals are commonly used in digital designs where they are used to simultaneously "fire" events into more than one module, commonly referred to as synchronization.
- a modulator 106 generates N sinusoidal pulse width modulated signals 105 whose pulses determine when the N number of radiation emitting units 113.1 to 113. N shall emit radiation.
- Pulses 22 start synchronously with the trigger signal 107, 21 coming from the camera 102.
- the trigger signal therefore allows pulses 22 to start at the same time as the camera's 102 image exposure but where the width 24 of the pulses is determined by modulator 106.
- pulses 22 in the exemplifying embodiment have a signal level higher than 80 % in relation to the maximum signal level, the corresponding radiation emitting units 113.1-113.
- N emit electromagnetic radiation, which means that electromagnetic radiation radiates from the radiation emitting units 113.1-113. N only during the times when the camera 102 exposes images. Limits other than 80 % for high signal levels can occur in alternative embodiments.
- the repetition frequency for camera image exposures 102 is indicated by Fc 27.
- the exposure time of camera 102 should ideally be selected to the maximum time during which any of the radiation emitting units 113.1-113.
- N emit electromagnetic radiation, i.e. maximum pulse width 24.
- the widths 24 of the pulses are modulated so that its width is directly proportional to a time and amplitude discrete sinusoidal signal 23, commonly known as SPWM.
- the time and amplitude discrete values 25 therefore constitute a digital representation of the sinusoidal signal 23.
- N number of sinusoidal pulse width modulated signals is thus generated in modulator 106 at different frequencies fi to fu 26 which can control the radiation emitting units 113.1-113.
- N characterized by wavelengths i to AN.
- Frequencies fi to fu are parameters whose values are assigned in accordance with the method described in the present patent application. Where applicable, the remaining technical description and claims can refer to these parameters.
- the digital signals 108 are proportional to pulse widths 24 and thus represent the time and amplitude discrete sinusoidal signals Si to SN and the corresponding cosine terms Ci to CN.
- the cosine terms have a phase shift of 90 degrees relative to the sine wave 23 that controls pulse widths 24.
- the digital signals 108 are used in conjunction with the video signal 103 in the spectral demodulator 109 to generate a series of digital signals Ri to RN, proportional to the intensity of the electromagnetic radiation characterized by wavelengths i to AN.
- the signals Ri to RN together constitute signatures for the materials that the surface 101 consists of.
- the spectral demodulator consists of a digital implementation of a superheterodyne lock-in amplifier and is schematically shown in Fig. 3.
- Sine and cosines terms, 31 and 32 are multiplied by the video signal V 33 in digital multipliers 39.
- the resulting signals, 34 and 35 are then filtered in a bank of linear low-pass digital filters 36. These filters operate temporally, i.e., filtering only in the time domain of each pixel.
- the time domain of a pixel describes how the pixel's value changes over time for the sequence of images that the digital video signal consists of.
- an arithmetic calculation 37 of the intensity of the reflected electromagnetic radiation Ri to RN, 38 for each respective wavelength i to N is done.
- artificial intelligence consisting of at least one computer program in the Al unit 111 is used.
- the algorithm "K Nearest Neighbor' (KNN) is used for the classification and characterization of the surface's 101 constituting materials.
- KNN K Nearest Neighbor'
- Examples of alternative artificial intelligence algorithms that can occur in alternative embodiments are neural networks or "Support Vector Machine” (SVM).
- Fig. 8 shows examples of three different situations where: material analyzer 81 is mounted on a vehicle or autonomous craft 82 for analysis of surfaces 83 over large areas, material analyzer 84 is mounted stationary over a conveyor belt 85 for continuous analysis of objects 86 passing by, material analyzer 87 is used manually by a user 88 for analysis of a selected object 89.
- a material imaging analyzer 114 needs to be adapted to the selected set of different materials that the user wishes to analyze.
- a prerequisite for the material analysis to be possible is that reflected electromagnetic radiation within the spectral wavelength range within which the receiver 115 is sensitive is also the carrier of information about the selected materials. This information is usually concentrated in a few single wavelengths/wavelength ranges within the sensitivity range of the receiver. Therefore, the following paragraphs describe a method for selecting these single wavelengths/wavelength areas.
- the radiation emitting device is then equipped in accordance with selected wavelengths/wavelength ranges and that the modulator and spectral demodulator are configured for the corresponding signal processing. Thereafter a method takes place in which the material analyzer learns to classify selected materials and learns to characterize the properties of the materials. This learning is obtained through the training of an artificial intelligence unit 111.
- Fig. 7 is a flow graph of operations that schematically compiles a method where the radiation emitting device 104 is equipped with a set of radiation emitting units 113.1-113. N and that modulator 106 and spectral demodulator 109 are configured.
- the method involves adapting the material imaging analyzer 114 to analyze the limited set of different materials that the material imaging analyzer 114 will be able to analyze.
- a spectrograph, hyperspectral camera or similar instrument is needed to collect data 71 for the spectral distribution of the electromagnetic radiation reflected in surfaces of a limited set of different materials and where some of the materials may have varying properties.
- principal component analysis 72 (a well-known statistical analysis method) is performed for the collected data, whereby projected data and projection coefficients are calculated.
- the coefficients then provide good guidance on which wavelengths/wavelength ranges, characterized by Xi 73, within the total analyzed spectral range that is most important for classifying and/or characterizing the different materials included in the analysis.
- Data collected at 71 is then filtered with simulated optical filters 74 according to the selections made at 73.
- the result after filtration is a spectral description of the materials with significantly lower spectral resolution than those obtained with spectrograph 71.
- Half of the filtered data is then used to train a classifier and/or for regression analysis of its properties, while the other half of the data is used for a simulated classification of material and/or simulated characterization of the properties 75 of the materials.
- the next step 76 will be to equip the radiation emitting device 104 with a set of radiation emitting units 113.1-113. N characterized by the Xi previously selected at 73. Frequencies 26, fi for the N number of sinusoidal signals 108 can then be selected and the modulator configured correspondingly.
- the spectral demodulator 109 is configured for the 2N number of low-pass filters 36 and 2N signal paths for other calculations 39, 37.
- the modulator and spectral demodulator configuration involves modifying the source code of software and/or firmware, after which the material imaging analyzer 114 is updated with new machine code and/or new configuration files for programmable logic.
- Fig. 9 provides a schematic drawing of the temporal amplitude spectrum 91 of the video signal 103.
- Temporal means an amplitude spectrum after analysis in the time domain of a single image element (pixel).
- the triangular areas 92 represent double bandwidth B for the video signal generated by a single radiation emitting unit 113.1-113.
- N. B is a parameter whose value is assigned in accordance with the method described here. Where applicable, the remaining technical description and claims can refer to parameter B.
- B can be calculated according to formula 93, where N is the number of radiation emitting units 113.1-113.
- N. Fc 1 , 97 is the repeating frequency for image exposures in the camera 102. The N frequencies fi that modulate the respective radiation emitting units 113.1-113.
- N can be selected in the exemplifying embodiment according to formulas 95 and 96. Other selections of frequencies fi can occur in alternative embodiments.
- Equation 93 indicates a correlation between the repetition frequency of image exposures Fc 97, the number of radiation emitting units N 113.1-113. N and bandwidth B 92. A higher repetition frequency for image exposures Fc 97 or fewer radiation emitting units N 113.1-113. N allows greater bandwidth B 92. What determines the need for bandwidth B is the presence of motion in the image, faster movements require greater bandwidth B for proper reproduction.
- the bandwidth B is the same bandwidth as in the exemplifying embodiment appropriately selected for the spectral demodulator's 2N number low-pass filter 36. Alternate embodiments can have low- pass filters 36 with bandwidth other than B.
- the present patent application describes in detail a device and a method.
- the device is a material imaging analyzer 114 for the classification and/or characterization of the materials of which a surface consists of.
- Classification refers to decisions about which materials, based on a limited set of different materials, a surface consists of.
- the ability to classify materials based on spectral data is created through a previous training of artificial intelligence.
- Characterization means the calculation of one or more metrics that quantify the properties of a material according to a mathematical relationship determined by regression analysis.
- Regression analysis refers to a statistical method for determining, based on training data, a mathematical relationship between spectral information and the properties of a surface.
- Both classification and characterization are the result of calculations performed with an artificial intelligence unit called Al unit 111. Training and regression analysis are also the result of calculations performed with artificial intelligence but by executing software in a separate computer.
- Analysis of materials is carried out by the device 114 irradiating a surface with spectrally modulated electromagnetic radiation while simultaneously quantifying reflected radiation.
- the device 114 demodulates quantified reflection, providing a spectral distribution of reflected radiation for each image element (pixel).
- a method schematically shown in Fig. 7, is used to adapt the device 114 to the limited set of different materials it is able to classify and/or characterize, where the set of different materials depends on the application.
- Fig. 4 shows a material imaging analyzer consisting of two main components: (1) a radiation emitting device 402, consisting of a group of radiation emitting units 413 and (2) a receiver 403. These two units are electrically connected via a cable 404, equipped with a connector and which mainly transmits signals Ti to TN, see 104 in Fig. 1.
- This allows the analyzer to be equipped with a replaceable radiation emitting device 402, consisting of alternative groups of radiation emitting units 413.
- the radiation emitting units are characterized by wavelengths Xi to AN selected based on the materials that need to be classified. Mechanically, components 402 and 403 are joined but detachable.
- a pixelated image detector 406 with one or two dimensions depicts the surface 401.
- the imaging is taken by focusing light with a detachable and replaceable lens 405 so that the image is projected onto the image detector 406.
- Algorithms and logic for 407 imaging, 408 spectral demodulator and modulator, 409 artificial intelligence, and 410 communication are in the exemplifying embodiment implemented on a field programmable logic array (FPGA) in combination with microprocessors on one and the same single integrated circuit.
- Wired or wireless communication 411 transmits registered materials and properties to a personal computer, industrial computer or similar platform for measurement data collection 412.
- Fig. 5 shows an alternative embodiment in which artificial intelligence 52 is moved from the receiver unit 51 and instead implemented as software in the computer for measurement data collection 53.
- Fig. 6 shows yet another alternative embodiment where the material imaging analyzer has its own built-in graphic display 62 and can thus operate independently without connection to a computer 412. Graphics are generated in 61 to visualize analysis results for a user 63.
- the present invention modulates the active electromagnetic radiation with a sinusoidal, pulse width modulated waveform (SPWM) in which the pulses are synchronized with image exposure.
- SPWM pulse width modulated waveform
- high light intensity can be created during the time of very short pulses, which in turn allows extremely short exposure times and thus minimizes contribution from surrounding electromagnetic radiation.
- the sinusoidal variation of pulse widths allows for a variable light energy that appears as a sinusoidal intensity in a temporal sequence of images.
- the sine modulated intensity can then be effectively separated from the unmodulated ambient electromagnetic radiation using superheterodyne lock-in where the different wavelengths of electromagnetic radiation have been modulated at different frequencies 26.
- Suppression of ambient electromagnetic radiation therefore takes place in two steps: (1) through synchronized light pulses in combination with short image exposures, (2) by demodulation with superheterodyne lock-in.
- the first step protects the image detector from saturation, for example, in the case of strong electromagnetic radiation from the sun. It is hugely important to include a way to protect the image detector from saturation, because no known demodulation technology is otherwise capable of suppressing the surrounding electromagnetic radiation.
- the intensity of the reflected electromagnetic radiation for each radiation emitting unit is calculated.
- This demodulation occurs on the basis of the same temporal amount of image data for all spectral channels and not different sets as in the case of time- multiplexed modulation described by Johan Casselgren et.al. Therefore, the present invention has excellent real-time properties and gives good results even where movement occurs in the image while suppressing the surrounding passive electromagnetic radiation in two steps in a particularly effective way. No other known technique of material imaging analysis has been able to combine these important properties.
- the present invention modulates electromagnetic radiation with different wavelengths instead of using optical components for dispersion or mechanical solutions for filter replacement. The material analyzer thus becomes cheaper, smaller and lighter than competing technologies.
- details may be omitted which are obvious to a professional in the field of the device and method. Such obvious details are included to the extent necessary for the satisfactory performance of the present device and method to be obtained.
Landscapes
- Engineering & Computer Science (AREA)
- Multimedia (AREA)
- Signal Processing (AREA)
- Physics & Mathematics (AREA)
- Spectroscopy & Molecular Physics (AREA)
- Health & Medical Sciences (AREA)
- Life Sciences & Earth Sciences (AREA)
- Chemical & Material Sciences (AREA)
- Analytical Chemistry (AREA)
- Biochemistry (AREA)
- General Health & Medical Sciences (AREA)
- General Physics & Mathematics (AREA)
- Immunology (AREA)
- Pathology (AREA)
- Investigating Or Analysing Materials By Optical Means (AREA)
- Analysing Materials By The Use Of Radiation (AREA)
Abstract
A material imaging analyzer (114), suitable for classification and characterization of the material of surfaces (101), comprising at least one radiation emitting device (104, 402) and at least one receiver (115). The radiation emitting device (104) emits light pulses modulated by at least one modulator (106) toward at least one surface (101) and that the reflected light from the surface (101) is recorded by at least one camera (102) in the receiver (115). The camera (102) generates video signals (103) which are transmitted to at least one spectral demodulator (109) that demodulates the generated video signals (103) and emits digital signals (Ri) to (RN) which are proportional to the intensity of the reflected light from the surface (101). The light emitted by the radiation emitting units (113) is controlled by sinusoidal pulse width modulated signals (105) modulated by the modulator (106), and that the modulator (106) comprises a synchronization function that synchronizes the start time of the emitted pulses (105) and light pulses from the radiation emitting units with the start times of the camera (102) exposures. The present invention also includes a method for the use of the material analyzer.
Description
Material Imaging Analyzer and Method for Its Use
Field of the Invention
The present invention concerns a material imaging analyzer, which is suitable to be used for the classification and characterization of surface materials in accordance with the claims, as well as a method for its use in accordance with the claims.
Background of the Invention
Techniques for the classification and characterization of materials by spectral imaging analysis of reflected electromagnetic radiation are already known. For example, it is known that when a material is irradiated with electromagnetic radiation within the wavelength range to which this invention refers, the molecules of the material and its clouds of electrons will oscillate due to the chemical bonding energy between the atoms of the molecules. This phenomenon can be approximated with a mechanical harmonic oscillation that in turn emits electromagnetic radiation, resulting in a scattered reflection of incoming radiation. Instead, for photon energies that match the corresponding resonance frequency of molecules or their harmonics, the molecule assumes a quantified higher energy level. This phenomenon thus results in a strong absorption of electromagnetic radiation into the material for the wavelengths that the molecules exhibit resonance for. In this way, the spectral distribution of the reflected radiation becomes a fingerprint of the analyzed material [1].
Furthermore, a scientific article by H. Jiang et.al [2] states that a few characteristic wavelengths can be selected from hyperspectral data using, inter alia, principal component analysis. The aim is to be able to classify materials based on the reflection of light with only a few different wavelengths. The method can be used to select the wavelengths of light emitters in an application-specific material analyzer. The application in this particular case concerns the detection of bruising on fruit.
Similarly, Patrik Jonsson describes in a doctoral thesis from Mid Sweden University [3] how a camera sensitive to light within the near infrared wavelength area can be used to classify road conditions: ice, snow, wet, dry asphalt. The system can provide guidance for better maintenance of winter roads. The technique is based on exposing a sequence of four images. For each of the four images, carefully selected optical band-pass filters are used that allow the use of Al where a subsequent classifier (KNN) identifies which areas within the
camera's inspection area are coated with snow, water, ice or dry asphalt. The technique works well for the inspection of static areas, i.e. no movement in the image, this is due to the long time it takes to change filters and expose four images. The mechanical filter change is an expensive solution, has a questionable function in cold and is slow. The described technology is an application-specific material analyzer but works only for static surfaces for the reasons described.
Guterman et.al describes in a patent [4] an imaging technique using out-of-sync amplitude modulated light. The fact that the modulated light has not been synchronized with the camera means that the radiation of the light cannot be timed to concentrate on the periods when the image sensor exposes images. This in turn leads to low signal-noise ratio (SNR), during, for example, the impact from unwanted bright sunlight. The technology is exemplified for video applications although it could theoretically be applied for spectral material analysis.
Sten Lofving describes in a patent [5] a sensor for the detection of road conditions. This sensor uses modulated laser light but is not imaging, i.e. measures at a limited point. When the sensor is mounted on a vehicle, the measurement takes place at several points along the line following the direction of travel of the vehicle. The described equipment is an application-specific material analyzer but would need a scanning motion in two dimensions to become imaging.
T. Hyvarinen et.al of the Finnish company Specim describes in a scientific article [6] how an spectral imaging camera with high spectral resolution has been constructed. The basic technique is commonly known as the "push-broom" which means that optical components such as lattice or prism are used to scatter the light that has passed through a narrow gap opening. Along one of the dimensions of the two-dimensional detector, a line image is obtained, while the other dimension provides a spectral distribution of light, dispersion. The technique only allows the imaging of one line and not imaging in two dimensions, like the technique Patrik Jonsson describes. However, if the object to be depicted has a linear motion relative to the camera, two dimensions can be depicted as a sequence of lines. There is also a problem with the sensitivity to light, which becomes very low for this technique as the captured light must pass through a narrow slit-shaped opening. The cost of such a camera is usually great as the optical components needed to achieve the dispersion of light
are expensive. The equipment is also often large and heavy due to the optical components.
In combination with, for example, halogen lighting, this camera can be used as a material analyzer.
Johan Casselgren et.al describes in a scientific article [7] a method for spectral imaging and classification of road conditions. There are problems in creating a controlled lighting environment outdoors with the help of active lighting. During the day there is a strong contribution of background light from the sun with very large variations. This article therefore describes a method for separating the contribution of active lighting from the background. Laser light is binarily modulated, on or off, synchronously with imaging in such a way that a sequence of six images is exposed: background light, lighting with laser 1, background light, laser 2, background light and laser 3. Lasers 1, 2 and 3 emit light at different wavelengths. The method thus allows spectral encoding of light, and that background light can be suppressed for static scenes, but the disadvantage is that the slightest movement in the image will generate an incorrect result due to the time it takes to expose six times.
S. Bhattacharyya et.al describes in a scientific article [8] how a superheterodyne lock-in amplifier can be implemented digitally. The method is very effective in separating weak sinusoidal signals from other unwanted noise. By selecting different frequencies on the reference signal, signals with different frequencies can also be detected. This technique can therefore be usefully used for spectral decoding of a sequence of images in which active lighting with different wavelengths has been modulated at different frequencies.
M. Lakka et.al describes in a scientific article [9] how a sinusoidal pulse width modulated signal generator (SPWM) is used as a method for converting electrical energy from DC to AC voltage. Furthermore, it describes how the generator was implemented digitally on a circuit with programmable logic, FPGA.
There are problems with existing material analysis techniques. Spectral cameras according to the technology "push-broom" can only image one line, they are large, heavy and expensive, have a low sensitivity to light and lack function to suppress the ambient light. The need to suppress the impact of the ambient passive light becomes particularly evident for those applications when material analysis is done in an outdoor environment under the influence
of strong and variable sunlight. Without suppressing the passive ambient light, the spectral distribution of the total light becomes variable over time, which in turn requires repetitive calibrations. In order to avoid these repeated calibrations, the overall (total) lighting environment should be dominated by the active lighting, which is significantly more stable over time. The existing technologies that use modulated light may indeed partially suppress ambient passive light, but in some cases are poor at managing real-time properties, i.e. movement in the imaging or they have built-in limitations in the ability to utilize radiated active lighting to maximize SNR. Techniques where multiple exposures are used in combination with mechanical filter changers become expensive, slow and have questionable mechanical reliability.
The purpose of the present invention is to eliminate or substantially reduce at least one of the aforementioned, or in the following description mentioned, problems. The purpose is achieved by a device and a method in accordance with the present invention.
Brief Description of the Drawings
In the following detailed description of the present invention, reference and references to the figures will take place. Each figure is briefly described in the following table of figures. The drawings are schematic and details may be omitted. The exemplifying embodiments of the material imaging analyzer shown in the figures do therefore not limit the scope of the present invention.
Fig. 1 shows how general components have been connected to a material imaging analyzer.
Fig. 2 shows a sinusoidal pulse width modulated signal used to control the output of electromagnetic radiation. The drawing also shows how the pulses have been synchronized with the image capture.
Fig. 3 describes the main components and the calculations performed in the spectral demodulator.
Fig. 4 shows a preferred exemplifying embodiment of the material analyzer.
Figs. 5 and 6 show alternative embodiments of the material imaging analyzer.
Fig. 7 shows a method for selecting wavelengths Xi, equipping the radiation emitting device with radiation emitting devices and configuring the modulator/demodulator.
Fig. 8 shows examples of three different situations for how the material imaging analyzer is used.
Fig. 9 shows a schematic exemplification of the temporal signal spectrum of the video signal.
Detailed Description of the Invention
With reference to the figures and in accordance with the present invention, a device referred to as a material imaging analyzer 114 and a method for its use and adapted applications of the material imaging analyzer 114 are shown. One or more surfaces 101 are assumed to consist of at least two different materials and/or at least one material with varying properties. The material imaging analyzer allows the materials of which the surfaces consist of to be classified and characterized by contactless measurement technology and where the image provides information about the distribution of materials across the surfaces. The contactless measurement technique means that the surfaces are irradiated with electromagnetic radiation and that the spectral distribution of reflected radiation forms the basis for classification and characterization of the materials. Typical application areas for the present invention are, for example: sorting materials for recycling, characterizing raw materials such as moisture content in biomass or classification and characterization of agricultural crops over large areas. The distinctive feature of the material imaging analyzer is that it needs to be adapted and trained for the set of different materials it can recognize and/or characterize.
The most significant number position in the numbered references always indicates the numbers 1 to 9 of the figures. The material imaging analyzer 114 consists of at least two main components: at least one radiation emitting device 104, and at least one receiver 115.
Fig. 1 shows the two-dimensional surface 101 analyzed by a radiation emitting device 104, consisting of at least two or more radiation emitting units 113.1 to 113. N. The radiation emitting units 113.1-113. N, which illuminate surface 101 are shown in the figures. The N number of radiation emitting units emits electromagnetic radiation with wavelengths from a minimum of 100 nm to a maximum of 3pm but where each individual radiation emitting unit is characterized by maximum intensity at wavelengths Xi to AN. In the present patent application, the word light may be used synonymously with electromagnetic radiation within a given wavelength range. Wavelength intervals other than 100 nm to 3pm can occur in
alternative embodiments. N and Ai to AN are parameters whose values are assigned in accordance with the method described in the present patent application. Where applicable, the remaining technical description and claims can refer to these parameters. The value of parameter N indicates the number of radiation emitting units and at the same time is the control for the dimensioning of modulator 106, spectral demodulator 109 and Al unit 111. In the exemplifying embodiment, the radiation emitting units consist of LED lasers and/or LED diodes. An LED laser emits a near monochromatic, coherent light whilst a simpler LED diode emits light with a wider wavelength range than for the LED laser. In alternative embodiments, flashing xenon lamps in combination with optical band-pass filters can be used. A camera 102 images a surface 101 by recording the electromagnetic radiation reflected in the surface during repetitive time intervals called exposure time. The camera includes at least one image detector. These image detectors can be of the one-dimensional type for line imaging or of a two-dimensional type for area imaging, the pixels of which are sensitive to all existing wavelengths i to AN. Camera 102 generates at least one digital video signal 103 and at least one trigger signal 107 whose pulses signal the beginning of repetitive image exposures. Trigger signals are commonly used in digital designs where they are used to simultaneously "fire" events into more than one module, commonly referred to as synchronization. A modulator 106 generates N sinusoidal pulse width modulated signals 105 whose pulses determine when the N number of radiation emitting units 113.1 to 113. N shall emit radiation.
With reference to Fig. 2, an exemplifying sinusoidal pulse width modulated signal is shown. Pulses 22 start synchronously with the trigger signal 107, 21 coming from the camera 102. The trigger signal therefore allows pulses 22 to start at the same time as the camera's 102 image exposure but where the width 24 of the pulses is determined by modulator 106. When pulses 22 in the exemplifying embodiment have a signal level higher than 80 % in relation to the maximum signal level, the corresponding radiation emitting units 113.1-113. N emit electromagnetic radiation, which means that electromagnetic radiation radiates from the radiation emitting units 113.1-113. N only during the times when the camera 102 exposes images. Limits other than 80 % for high signal levels can occur in alternative embodiments. The repetition frequency for camera image exposures 102 is indicated by Fc 27. The exposure time of camera 102 should ideally be selected to the maximum time during
which any of the radiation emitting units 113.1-113. N emit electromagnetic radiation, i.e. maximum pulse width 24. The widths 24 of the pulses are modulated so that its width is directly proportional to a time and amplitude discrete sinusoidal signal 23, commonly known as SPWM. The time and amplitude discrete values 25 therefore constitute a digital representation of the sinusoidal signal 23. N number of sinusoidal pulse width modulated signals is thus generated in modulator 106 at different frequencies fi to fu 26 which can control the radiation emitting units 113.1-113. N, characterized by wavelengths i to AN. Frequencies fi to fu are parameters whose values are assigned in accordance with the method described in the present patent application. Where applicable, the remaining technical description and claims can refer to these parameters. The digital signals 108 are proportional to pulse widths 24 and thus represent the time and amplitude discrete sinusoidal signals Si to SN and the corresponding cosine terms Ci to CN. The cosine terms have a phase shift of 90 degrees relative to the sine wave 23 that controls pulse widths 24. The digital signals 108 are used in conjunction with the video signal 103 in the spectral demodulator 109 to generate a series of digital signals Ri to RN, proportional to the intensity of the electromagnetic radiation characterized by wavelengths i to AN. The signals Ri to RN together constitute signatures for the materials that the surface 101 consists of.
The spectral demodulator consists of a digital implementation of a superheterodyne lock-in amplifier and is schematically shown in Fig. 3. Sine and cosines terms, 31 and 32, are multiplied by the video signal V 33 in digital multipliers 39. The resulting signals, 34 and 35, are then filtered in a bank of linear low-pass digital filters 36. These filters operate temporally, i.e., filtering only in the time domain of each pixel. The time domain of a pixel describes how the pixel's value changes over time for the sequence of images that the digital video signal consists of. Finally, an arithmetic calculation 37 of the intensity of the reflected electromagnetic radiation Ri to RN, 38 for each respective wavelength i to N is done. The fact that the sine and cosine terms, 31 and 32, have the same frequency fi 26 as for the corresponding emitted sinusoidal pulse width modulated electromagnetic radiation 22 with wavelength Ai means that the result of the arithmetic calculation 37 corresponds to the intensity of reflected electromagnetic radiation with the corresponding wavelength i.
In the exemplifying embodiment, artificial intelligence consisting of at least one computer program in the Al unit 111 is used. In its simplest form, the algorithm "K Nearest Neighbor'
(KNN) is used for the classification and characterization of the surface's 101 constituting materials. Examples of alternative artificial intelligence algorithms that can occur in alternative embodiments are neural networks or "Support Vector Machine" (SVM).
Fig. 8 shows examples of three different situations where: material analyzer 81 is mounted on a vehicle or autonomous craft 82 for analysis of surfaces 83 over large areas, material analyzer 84 is mounted stationary over a conveyor belt 85 for continuous analysis of objects 86 passing by, material analyzer 87 is used manually by a user 88 for analysis of a selected object 89.
The present invention, a material imaging analyzer 114 needs to be adapted to the selected set of different materials that the user wishes to analyze. A prerequisite for the material analysis to be possible is that reflected electromagnetic radiation within the spectral wavelength range within which the receiver 115 is sensitive is also the carrier of information about the selected materials. This information is usually concentrated in a few single wavelengths/wavelength ranges within the sensitivity range of the receiver. Therefore, the following paragraphs describe a method for selecting these single wavelengths/wavelength areas. The radiation emitting device is then equipped in accordance with selected wavelengths/wavelength ranges and that the modulator and spectral demodulator are configured for the corresponding signal processing. Thereafter a method takes place in which the material analyzer learns to classify selected materials and learns to characterize the properties of the materials. This learning is obtained through the training of an artificial intelligence unit 111.
Fig. 7 is a flow graph of operations that schematically compiles a method where the radiation emitting device 104 is equipped with a set of radiation emitting units 113.1-113. N and that modulator 106 and spectral demodulator 109 are configured. The method involves adapting the material imaging analyzer 114 to analyze the limited set of different materials that the material imaging analyzer 114 will be able to analyze. Initially, a spectrograph, hyperspectral camera or similar instrument is needed to collect data 71 for the spectral distribution of the electromagnetic radiation reflected in surfaces of a limited set of different materials and where some of the materials may have varying properties. After that, principal component analysis 72 (a well-known statistical analysis method) is performed for the collected data, whereby projected data and projection coefficients are calculated. The
coefficients then provide good guidance on which wavelengths/wavelength ranges, characterized by Xi 73, within the total analyzed spectral range that is most important for classifying and/or characterizing the different materials included in the analysis. Data collected at 71 is then filtered with simulated optical filters 74 according to the selections made at 73. The result after filtration is a spectral description of the materials with significantly lower spectral resolution than those obtained with spectrograph 71. Half of the filtered data is then used to train a classifier and/or for regression analysis of its properties, while the other half of the data is used for a simulated classification of material and/or simulated characterization of the properties 75 of the materials. There can also be other alternative distributions for the division of filtered data than half-half. If substandard classification or characterization occurs, operations 73, 74 and 75 must be repeated, whereby other wavelengths Xi and possibly more radiation emitting units N can be chosen. However, if the result of the classification and/or characterization was acceptable, the next step 76 will be to equip the radiation emitting device 104 with a set of radiation emitting units 113.1-113. N characterized by the Xi previously selected at 73. Frequencies 26, fi for the N number of sinusoidal signals 108 can then be selected and the modulator configured correspondingly. The spectral demodulator 109 is configured for the 2N number of low-pass filters 36 and 2N signal paths for other calculations 39, 37. The modulator and spectral demodulator configuration involves modifying the source code of software and/or firmware, after which the material imaging analyzer 114 is updated with new machine code and/or new configuration files for programmable logic.
Fig. 9 provides a schematic drawing of the temporal amplitude spectrum 91 of the video signal 103. Temporal means an amplitude spectrum after analysis in the time domain of a single image element (pixel). The triangular areas 92 represent double bandwidth B for the video signal generated by a single radiation emitting unit 113.1-113. N. B is a parameter whose value is assigned in accordance with the method described here. Where applicable, the remaining technical description and claims can refer to parameter B. B can be calculated according to formula 93, where N is the number of radiation emitting units 113.1-113. N. Fc 1 , 97 is the repeating frequency for image exposures in the camera 102. The N frequencies fi that modulate the respective radiation emitting units 113.1-113. N can be selected in the exemplifying embodiment according to formulas 95 and 96. Other selections
of frequencies fi can occur in alternative embodiments. Equation 93 indicates a correlation between the repetition frequency of image exposures Fc 97, the number of radiation emitting units N 113.1-113. N and bandwidth B 92. A higher repetition frequency for image exposures Fc 97 or fewer radiation emitting units N 113.1-113. N allows greater bandwidth B 92. What determines the need for bandwidth B is the presence of motion in the image, faster movements require greater bandwidth B for proper reproduction. The bandwidth B is the same bandwidth as in the exemplifying embodiment appropriately selected for the spectral demodulator's 2N number low-pass filter 36. Alternate embodiments can have low- pass filters 36 with bandwidth other than B.
The present patent application describes in detail a device and a method. The device is a material imaging analyzer 114 for the classification and/or characterization of the materials of which a surface consists of. Classification refers to decisions about which materials, based on a limited set of different materials, a surface consists of. The ability to classify materials based on spectral data is created through a previous training of artificial intelligence. Characterization means the calculation of one or more metrics that quantify the properties of a material according to a mathematical relationship determined by regression analysis. Regression analysis refers to a statistical method for determining, based on training data, a mathematical relationship between spectral information and the properties of a surface. Both classification and characterization are the result of calculations performed with an artificial intelligence unit called Al unit 111. Training and regression analysis are also the result of calculations performed with artificial intelligence but by executing software in a separate computer.
Analysis of materials is carried out by the device 114 irradiating a surface with spectrally modulated electromagnetic radiation while simultaneously quantifying reflected radiation. The device 114 demodulates quantified reflection, providing a spectral distribution of reflected radiation for each image element (pixel). A method, schematically shown in Fig. 7, is used to adapt the device 114 to the limited set of different materials it is able to classify and/or characterize, where the set of different materials depends on the application.
Exemplifying Embodiment
The following describes a preferred embodiment of the present material analyzer where it is used to classify the different materials of which a two-dimensional surface 401 consists of. Fig. 4 shows a material imaging analyzer consisting of two main components: (1) a radiation emitting device 402, consisting of a group of radiation emitting units 413 and (2) a receiver 403. These two units are electrically connected via a cable 404, equipped with a connector and which mainly transmits signals Ti to TN, see 104 in Fig. 1. This allows the analyzer to be equipped with a replaceable radiation emitting device 402, consisting of alternative groups of radiation emitting units 413. The radiation emitting units are characterized by wavelengths Xi to AN selected based on the materials that need to be classified. Mechanically, components 402 and 403 are joined but detachable.
A pixelated image detector 406 with one or two dimensions depicts the surface 401. The imaging is taken by focusing light with a detachable and replaceable lens 405 so that the image is projected onto the image detector 406. Algorithms and logic for 407 imaging, 408 spectral demodulator and modulator, 409 artificial intelligence, and 410 communication are in the exemplifying embodiment implemented on a field programmable logic array (FPGA) in combination with microprocessors on one and the same single integrated circuit. Wired or wireless communication 411 transmits registered materials and properties to a personal computer, industrial computer or similar platform for measurement data collection 412.
Fig. 5 shows an alternative embodiment in which artificial intelligence 52 is moved from the receiver unit 51 and instead implemented as software in the computer for measurement data collection 53.
Fig. 6 shows yet another alternative embodiment where the material imaging analyzer has its own built-in graphic display 62 and can thus operate independently without connection to a computer 412. Graphics are generated in 61 to visualize analysis results for a user 63.
Advantages of the Invention
Several advantages are achieved by the present invention. The most important is that at least one of the listed problems in the background or description with known techniques has been eliminated or reduced.
The present invention modulates the active electromagnetic radiation with a sinusoidal, pulse width modulated waveform (SPWM) in which the pulses are synchronized with image exposure. This means that the energy of the active electromagnetic radiation can be controlled to only the time intervals at which the image exposure occurs. With modern LEDs or LED laser diodes, high light intensity can be created during the time of very short pulses, which in turn allows extremely short exposure times and thus minimizes contribution from surrounding electromagnetic radiation. The sinusoidal variation of pulse widths allows for a variable light energy that appears as a sinusoidal intensity in a temporal sequence of images. The sine modulated intensity can then be effectively separated from the unmodulated ambient electromagnetic radiation using superheterodyne lock-in where the different wavelengths of electromagnetic radiation have been modulated at different frequencies 26. Suppression of ambient electromagnetic radiation therefore takes place in two steps: (1) through synchronized light pulses in combination with short image exposures, (2) by demodulation with superheterodyne lock-in. The first step protects the image detector from saturation, for example, in the case of strong electromagnetic radiation from the sun. It is hugely important to include a way to protect the image detector from saturation, because no known demodulation technology is otherwise capable of suppressing the surrounding electromagnetic radiation.
At demodulation, the intensity of the reflected electromagnetic radiation for each radiation emitting unit is calculated. This demodulation occurs on the basis of the same temporal amount of image data for all spectral channels and not different sets as in the case of time- multiplexed modulation described by Johan Casselgren et.al. Therefore, the present invention has excellent real-time properties and gives good results even where movement occurs in the image while suppressing the surrounding passive electromagnetic radiation in two steps in a particularly effective way. No other known technique of material imaging analysis has been able to combine these important properties.
The present invention modulates electromagnetic radiation with different wavelengths instead of using optical components for dispersion or mechanical solutions for filter replacement. The material analyzer thus becomes cheaper, smaller and lighter than competing technologies. In the detailed description of the present device and method, details may be omitted which are obvious to a professional in the field of the device and method. Such obvious details are included to the extent necessary for the satisfactory performance of the present device and method to be obtained.
Although certain preferred embodiments of the device and the method have been described in more detail, variations and modifications to the device and the method may become apparent to the professionals in the field to which the invention relates. All such modifications and variations are considered to fall within the scope of the subsequent claims.
References
[1] B. Stuart, Biological applications of infrared spectroscopy, Wiley, 1997
[2] H. Jiang et.al, "Wavelength Selection for Detection of Slight Bruises on Pears Based on Hyperspectral Imaging," Applied Sciences, v. 6, No. 12, pp. 450, MDPI 2016
[3] P. Jonsson, Surface Status Classification, Utilizing Image Sensor Technology and Computer Models, Doctoral thesis no. 219, Mid Sweden University, 2015.
[4] Guterman et.al, "Methods of producing video images that are independent of the background lighting", US Patent No. US 10,630,907. B2, 21st of April 2020.
[5] Sten Lofving, "Optical method of estimating properties of an ice or water coated measuring object", Swedish patent, number SE 531 949 C2, September 15, 2009.
[6] T. Hyvarinen et.al, "Compact high-resolution VIS/NIR hyperspectral sensor", Proceedings oftheSPIE - The International Society for Optical Engineering, v. 8032, Conference: Next- Generation Spectroscopic Technologies IV, 25-26 April 2011, Orlando, FL, USA.
[7] J. Casselgren et.al, "Road condition analysis using NIR illumination and compensating for surrounding light", Optics and Lasers in Engineering, v. 77, pp. 175-82, Feb. 2016
[8] S. Bhattacharyya et.al, "Implementation of Digital Lock-in Amplifier," Journal of Physics: Conference Series, v. 759, 2016
[9] M.Lakka et.al, "Development of an FPGA-Based SPWM Generator for High Switching Frequency DC/AC Inverters", IEEE Transactions on Power Electronics, v.29, No. 1, January
Claims
Claims
1. A material imaging analyzer (114), suitable for use in the classification and/or characterization of materials in at least one surface (101), which comprises at least one radiation emitting device (104) and at least one receiver (115), said radiation emitting device (104) arranged to emit pulses of electromagnetic radiation, modulated by at least one modulator (106), with a maximum intensity at wavelengths (Xi) to (XN) toward at least one surface (101) and that the receiver (115 ) comprises at least one camera (102) which is arranged to register the reflected electromagnetic radiation from the surface (101) and that the camera (102) is arranged to generates video signals (103) which are transmitted to at least one spectral demodulator (109) which is arranged to demodulate the generated video signals (103) and emits digital signals (Ri) to (RN) and that the radiation emitting device (104) comprises at least one first radiation emitting unit (113.1) arranged to emit modulated electromagnetic radiation with a maximum intensity at a first wavelength (Xi) and at least one second radiation emitting unit (113.2) arranged to emit modulated electromagnetic radiation with a maximum intensity at a second wavelength ( 2) characterized by that the modulator (106) is arranged to emit pulses (105, 22) modulated to sinusoidal, pulse width modulated waveforms that control the radiation emitting units' (113.1-113. N) emitted electromagnetic radiation and that the modulator (106) is arranged to synchronize the start time of the emitted pulses (105, 22) and the pulses of electromagnetic radiation from the radiation emitting units with the starting times of the camera (102) exposures, and that the digital signals (Ri) to (RN) emitted by the spectral demodulator (109) are in proportion to the intensity of the light reflected in the surface (101) and emitted from the respective radiation emitting units (113.1-113. N) with maximum radiated intensity at the corresponding wavelengths (Xi) to (XN).
2. The material imaging analyzer (114) in accordance with claim 1 characterized by that the radiation emitting units (113.1-113. N) are arranged to emit electromagnetic radiation with wavelengths in the range of 100 nm to 3pm.
3. The material imaging analyzer (114) in accordance with one of the claims 1 or 2 characterized by that the radiation emitting units (113.1-113. N) consist of LED lasers.
4. The material imaging analyzer (114) in accordance with one of the claims 1 to 2 characterized by that the radiation emitting units (113.1-113. N) consist of LED diodes.
5. The material imaging analyzer (114) in accordance with one of the claims 1 to 2 characterized by that the radiation emitting devices (113.1-113. N) consist of a combination of LED lasers and LED diodes.
6. The material imaging analyzer (114) in accordance with one of the previous claims 1 to 4 characterized by that the spectral demodulator (109) consists of a digital implementation of a superheterodyne lock-in amplifier.
7. The material imaging analyzer (114) in accordance with at least one of the previous claims characterized by that the material imaging analyzer comprises at least one Al unit (111)
8. The material imaging analyzer (114) in accordance with claim 7 characterized by that the Al unit (111) comprises at least one computer program comprising at least one algorithm, said Al unit (111) being arranged to classify the material constituting the surface (101) and/or characterize one or more metrics regarding the properties of the materials constituting the surface (101).
9. The material imaging analyzer (114) in accordance with claim 1 characterized by the camera (102) comprising at least one one-dimensional and/or at least one two- dimensional image detector whose pixels are sensitive to all existing wavelengths (Xi) to (AN).
10. The material imaging analyzer (114) in accordance with claim 1 characterized by that the radiation emitting units (113.1-113. N) are interchangeable and/or that the radiation emitting device (104) is interchangeable.
11. The material imaging analyzer (114) in accordance with claim 1 characterized by that the radiation emitting device (104) and the receiver (115), consist of separate units.
12. The material imaging analyzer (114) in accordance with claim 1 characterized by that the radiation emitting device (104) and the receiver (115) are integrated into one unit.
13. The material imaging analyzer (114) in accordance with at least one of the previous claims characterized by that the imaging analyzer comprises a bank of linear low- pass digital filters (36) that work temporally, that is filtering the time domain for each pixel data.
14. The material imaging analyzer (114) in accordance with at least one of the previous claims characterized by that it comprises at least one graphics generator.
15. A method for the use of the material imaging analyzer (114) in accordance with at least one of the claims 1 to 14 characterized by that the radiation emitting device (104) is equipped with a necessary N number of radiation emitting units (113.1 to 113. N), with maximum intensity at wavelengths (Xi) to (XN.), selected to enable the classification of materials in the surface (101) and/or to allow characterization of the properties of the materials in the surface (101), after which the modulator (106) and the spectral demodulator (109) are configured after the selected set of radiation emitting units (113.1-113. N), that the modulator (106) emits N number of sinusoidal pulse width modulated signals (105, 22) whose pulses (22) control the electromagnetic radiation emitted by the radiation emitting units (113.1-113. N) toward the surface (101), that the camera (102) emits at each exposure a trigger signal (107, 21) to the modulator (106) that synchronizes the start time of each pulse (22) with the start time of the camera's (102) exposure,
that the camera (102) records the reflected electromagnetic radiation and generates at least one digital video signal (103) which digital video signal (103) is demodulated in the spectral demodulator (109) and that the spectral demodulator (109) emits N number of digital signals (Ri) to (RN) which are proportional to the intensity at the corresponding N number of wavelengths (Xi) to (AN) of the reflected electromagnetic radiation from the surface (101), and that the digital signals (Ri) to (RN) form the basis for a classification of the materials constituting the surface and/or a characterization of the properties of the materials. The method for the use of the material imaging analyzer (114) in accordance with claim 15 characterized by that the sinusoidal pulse width modulated signals (105, 22) generated by the modulator (106) are generated with N number of different frequencies fi to fu. The method for the use of the material imaging analyzer (114) in accordance with claim 16 characterized by that the N number of different frequencies fi to fu are calculated according to formula (94) as follows fi= 2iB where the variable i is defined according to formula (95) as follows i = 1 ... N and that bandwidth B for the low-pass filters (36) is defined according to formula (93) as follows
B=Fc/4(N+l) where the variable Fc corresponds to the repetition frequency for the image exposures of the camera (102). The method for the use of the material imaging analyzer (114) in accordance with one of the claims 15 to 17 characterized by that when pulses (105, 22) emitted by the modulator (106) have a signal level higher than 80 % relative to the maximum signal level the modulator (106) can emit, the corresponding radiation emitting units (113.1-113. N) emit electromagnetic radiation, which means that electromagnetic
radiation is emitted from the radiation emitting units (113.1-113. N), only during the times when the camera (102) exposes images. The method for the use of the material imaging analyzer (114) in accordance with one of the claims 15 to 18 characterized by that the exposure time for the camera (102) is chosen to the time corresponding to the maximum pulse width (24), during which any of the radiation emitting units (113.1-113. N) emit electromagnetic radiation. The method for the use of the material imaging analyzer (114) in accordance with at least one of the claims 15 to 19 characterized by that the choice of N number of radiation emitting units (113.1-113. N), and the choice of the wavelengths (Xi) to (AN.) from the radiation emitting units (113.1-113. N), are carried out by a sub-procedure where spectral data are collected (71) for the materials to be classified and/or characterized with the material imaging analyzer and where collection is carried out with at least one spectrograph, alternatively at least one hyperspectral camera or similar, after which a principal component analysis (72) of the collected spectral data is carried out so that projected data and the projection's coefficients are calculated, said coefficients providing guidance on which wavelengths/wavelength ranges are characterized by Xi (73) within the total analyzed spectral range that are most important for the classification and/or characterization of the different materials, which in turn provides guidance on how the radiation emitting device is to be equipped (76) and how the modulator (106) and the spectral demodulator (109) are to be configured. The method for the use of the material imaging analyzer (114) in accordance with claim 20 characterized by that the collected spectral dataset is filtered with simulated optical filters (74) according to the choices made by the coefficients, which means that a spectral description of the materials with significantly lower spectral resolution than that obtained with the spectrograph is obtained, after which half of the filtered dataset is used to train a classifier and/or for regression analysis of the
properties of the material, while the other half of the filtered spectral dataset is used for a simulated classification and/or characterization (75) of the materials. The method for the use of the material imaging analyzer (114) in accordance with the claims 20 to 21 characterized by that if the classification and/or characterization (75) is not acceptable, operations (73, 74, 75) must be repeated, if the result of the classification and/or characterization is acceptable, the radiation emitting device (104) is equipped (76) with a set of N radiation emitting units (113.1-113. N) that are characterized by the previously selected (73) wavelengths ( i) to (AN.) and that the modulator (106) and the spectral demodulator (109) are configured. The method for the use of the material imaging analyzer (114) in accordance with at least one of the claims 15 to 22 characterized by that the configuration of modulator (106) and spectral demodulator (109) is respectively done by modification of source code for software and/or firmware, after which the material imaging analyzer (114) is updated with new machine code and/or new configuration files for programmable logic. The method for the use of the material imaging analyzer (114) in accordance with at least one of the claims 15 to 24 characterized by that during the demodulation of the digital signals (103) with the spectral demodulator (109), the sine and cosines terms (31) and (32) are multiplied with the video signal V (33) in digital multipliers (39) to the resulting signals (34) and (35) which are then filtered in a bank of linear low-pass digital filters (36) whose pass bands have a width selected to B, which filters operate temporally, i.e. filtered in the time domain of each pixel data, after which an arithmetic calculation (37) of the intensity Ri to RN (38) of the reflected electromagnetic radiation for each wavelength ( i) to (AN) is performed. The use of the material imaging analyzer (114) in accordance with at least one of the claims 1 to 14 in a vehicle alternatively in an autonomous craft. The use of the material imaging analyzer (114) in accordance with at least one of the claims 1 to 14 in a stationary application.
27. The use of the material imaging analyzer (114) in accordance with at least one of the claims 1 to 14 characterized by that the material imaging analyzer (114) is used manually.
Applications Claiming Priority (2)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
SE2030308A SE544814C2 (en) | 2020-10-12 | 2020-10-12 | Imaging material analyzer and procedure for using it |
SE2030308-7 | 2020-10-12 |
Publications (1)
Publication Number | Publication Date |
---|---|
WO2022081057A1 true WO2022081057A1 (en) | 2022-04-21 |
Family
ID=81208408
Family Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
PCT/SE2021/000008 WO2022081057A1 (en) | 2020-10-12 | 2021-10-11 | Material imaging analyzer and method for its use |
Country Status (2)
Country | Link |
---|---|
SE (1) | SE544814C2 (en) |
WO (1) | WO2022081057A1 (en) |
Citations (5)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
WO1994002812A1 (en) * | 1992-07-28 | 1994-02-03 | Patchen California, Inc. | Structure and method for differentiating one object from another object |
US5310260A (en) * | 1990-04-10 | 1994-05-10 | Luxtron Corporation | Non-contact optical techniques for measuring surface conditions |
EP0710817A1 (en) * | 1993-06-29 | 1996-05-08 | Omron Corporation | Road-surface examining device and device using it |
US20070058170A1 (en) * | 2005-09-12 | 2007-03-15 | Lodder Robert A | Method and system for in situ spectroscopic evaluation of an object |
US20130278749A1 (en) * | 2012-04-13 | 2013-10-24 | Andreas Mandelis | Method and apparatus for performing heterodyne lock-in imaging and quantitative non-contact measurements of electrical properties |
Family Cites Families (5)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
WO2005017996A1 (en) * | 2003-03-14 | 2005-02-24 | Andreas Mandelis | Method of photocarrier radiometry of semiconductors |
AU2013308186B9 (en) * | 2012-08-30 | 2017-09-21 | Iti Scotland - Scottish Enterprise | Long wavelength infrared detection and imaging with long wavelength infrared source |
DE102016113703A1 (en) * | 2016-07-26 | 2018-02-01 | Eberhard-Karls-Universität Tübingen | Method and device for the spatially resolved measurement of radiation signals |
CN114812808A (en) * | 2016-11-29 | 2022-07-29 | 光热光谱股份有限公司 | Method and apparatus for enhanced photothermographic and spectroscopic imaging |
JP2021526633A (en) * | 2018-06-01 | 2021-10-07 | フォトサーマル・スペクトロスコピー・コーポレーション | Wide-area optical photothermal infrared spectroscopy |
-
2020
- 2020-10-12 SE SE2030308A patent/SE544814C2/en unknown
-
2021
- 2021-10-11 WO PCT/SE2021/000008 patent/WO2022081057A1/en active Application Filing
Patent Citations (5)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US5310260A (en) * | 1990-04-10 | 1994-05-10 | Luxtron Corporation | Non-contact optical techniques for measuring surface conditions |
WO1994002812A1 (en) * | 1992-07-28 | 1994-02-03 | Patchen California, Inc. | Structure and method for differentiating one object from another object |
EP0710817A1 (en) * | 1993-06-29 | 1996-05-08 | Omron Corporation | Road-surface examining device and device using it |
US20070058170A1 (en) * | 2005-09-12 | 2007-03-15 | Lodder Robert A | Method and system for in situ spectroscopic evaluation of an object |
US20130278749A1 (en) * | 2012-04-13 | 2013-10-24 | Andreas Mandelis | Method and apparatus for performing heterodyne lock-in imaging and quantitative non-contact measurements of electrical properties |
Non-Patent Citations (4)
Title |
---|
JONSSON P. ET AL.: "Road Surface Status Classification Using Spectral Analysis of NIR Camera Images", IEEE SENSORS JOURNAL, vol. 15, 2015, pages 1641 - 1656, XP011569668, DOI: 10.1109/JSEN.2014.2364854 * |
JONSSON P.: "Surface classification status, utilizing image sensor technology and computer models", PHD THESIS, vol. 219, 2015, pages 1 - 90, XP055934727, ISSN: 1652-893X * |
LI J. ET AL.: "Early detection of decay on apples using hyperspectral reflectance imaging combining both principal component analysis and improved watershed segmentation method", POSTHARVEST BIOLOGY AND TECHNOLOGY, vol. 149, 2019, pages 235 - 246, XP085574881, DOI: 10.1016/j.postharvbio.2018.12.007 * |
LIU C. ET AL.: "Discriminative Illumination: Per-Pixel Classification of Raw Materials Based on Optimal Projections of Spectral BRDF", IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE, vol. 36, 2014, pages 86 - 98, XP011532091, DOI: 10.1109/TPAMI.2013.110 * |
Also Published As
Publication number | Publication date |
---|---|
SE2030308A1 (en) | 2022-04-13 |
SE544814C2 (en) | 2022-11-29 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
US6466961B1 (en) | Methods for adapative spectral, spatial and temporal sensing for imaging applications | |
Cao et al. | A prism-mask system for multispectral video acquisition | |
US6392748B1 (en) | Radiation filter, spectrometer and imager using a micro-mirror array | |
US6909095B2 (en) | System and method for terahertz imaging using a single terahertz detector | |
US6859275B2 (en) | System and method for encoded spatio-spectral information processing | |
US8587657B2 (en) | Determining a number of objects in an IR image | |
Abouraddy et al. | Demonstration of the complementarity of one-and two-photon interference | |
US20040218172A1 (en) | Application of spatial light modulators for new modalities in spectrometry and imaging | |
US7385709B2 (en) | Microscopy imaging apparatus and method for generating an image | |
US9869633B2 (en) | Spectral imaging of a sample using a plurality of discrete mid-infrared wavelengths | |
US11704886B2 (en) | Coded light for target imaging or spectroscopic or other analysis | |
KR20120066604A (en) | Determining a total number of people in an ir image obtained via an ir imaging system | |
US20050083521A1 (en) | System and method for detection and identification of optical spectra | |
KR20150088879A (en) | Detection systems and methods using coherent anti-stokes raman spectroscopy | |
WO2021009280A1 (en) | Spectrometer device | |
WO2022081057A1 (en) | Material imaging analyzer and method for its use | |
Sergiyenko et al. | Developing and applying optoelectronics in machine vision | |
JP6826972B2 (en) | Imaging device | |
US10630907B2 (en) | Methods of producing video images that are independent of the background lighting | |
Rousset | Single-pixel imaging: Development and applications of adaptive methods | |
Thörnberg | The Material Imaging Analyzer MIA | |
CN212748719U (en) | Remote measuring system for smoke intensity of tail gas of black smoke vehicle | |
US20240362883A1 (en) | Wearable device providing glucose measurement | |
RU2061223C1 (en) | Method of measurement of sizes of microparticles | |
CN117561426A (en) | Coded light for target imaging or analysis |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
121 | Ep: the epo has been informed by wipo that ep was designated in this application |
Ref document number: 21880663 Country of ref document: EP Kind code of ref document: A1 |
|
NENP | Non-entry into the national phase |
Ref country code: DE |
|
122 | Ep: pct application non-entry in european phase |
Ref document number: 21880663 Country of ref document: EP Kind code of ref document: A1 |