BG113145A - Method and device for processing and interpretation of images in the electromagnetic spectrum - Google Patents

Abstract

The present invention relates to a method and device for processing and interpreting images in the electromagnetic spectrum, which will find application in the field of broad spectrum thermography for spectral image analysis for remote recognition of objects. The created device comprises a CMOS sensor (1) connected to a graphics processing unit (3) and a thermal head (2) connected to an interface converter (4). The graphics processing unit (3) and the interface converter (4) are connected to a microprocessor (5), which in turn is connected to a second interface converter (6) connected to the video display (7). The microprocessor (5) is also connected to a peripheral microprocessor (10), which is connected to a compass (8), an accelerometer (9), a remote control module (11), a control unit (12) and an interface connection connector for connection with external devices (13). The microprocessor (5) on the other hand is also connected to a USB connector (14), an SD card connector (15), a WiFi module (16), a Bluetooth module (17) and a GPS module (18).

Description

Method and device for processing and interpreting images in the electromagnetic spectrum

Field of technique

The present invention relates to a method and device for processing and interpreting images in the wide electromagnetic spectrum, in the ultraviolet, light and infrared ranges, and will find application for the identification of chemical content of substances and in particular for the survey of buildings, for fire prevention, production and quality control and others. This invention also relates to the use of spectral imaging analysis of emitted and received light spectra for non-invasive and remote detection of the presence, location and/or amount of a selected target substance by spectral imaging analysis.

Prior art

Spectral analysis is a branch of spectroscopy and photography in which at least one spectral information is collected from an image plane (eg, a two-dimensional image) about an object in a certain scene. The image capture device may be aimed at a scene to capture image information for that scene. Various spectral imaging methods are known. Examples include hyperspectral imaging, multispectral imaging, full spectral imaging, imaging spectroscopy, chemical imaging, and the like. Historically, hyperspectral imaging and multispectral image analysis have been associated with satellite, airborne, or large-scale operations using large, expensive camera systems that are not suitable for handheld or everyday business as well as consumer applications.

Spectral analysis generally involves capturing spectral information from one or more parts of the electromagnetic spectrum. Although spectral information can be used for any wavelength in the electromagnetic spectrum, spectroscopy often uses spectral information for wavelengths in the range of about 100 nm to about 14,000 nm. The range of the electromagnetic spectrum is divided into the following ranges: ultraviolet band (UV) from 100 to 400 nm; visible band (VIS) from 400 to 700 nm; near infrared range (NIR) from 700 to 1500 nm; shortwave infrared range (SWIR) from 1500 to 3000 nm; mid-wave infrared (MWIR) from 3000 to 5000 nm; and longwave infrared (LWIR) from 5000 to 14000 nm.

The ultraviolet band includes the following subbands: far ultraviolet band (FUV) from 122 to 200 nm; middle ultraviolet band (MUV) from 200 to 300 nm; and the near ultraviolet (NUV) band from 300 to 400 nm. The ultraviolet band is also divided into the following sub-bands: ultraviolet C band (UVC) from 100 to 280 nm; ultraviolet B band (UVB) from 280 to 315 nm; and the ultraviolet A band (UVA) from 315 to 400 nm.

Infrared thermography is a method of visualizing an object by interpreting the infrared rays it emits compared to a black body. The higher the temperature of the object, the greater the radiation from it. The need to see in complete darkness or in reduced visibility - smoke or fog - provoked the development and implementation of technologies with the ability to interpret rays from the broad electromagnetic spectrum in many industries. A thermal image is a conversion of the thermal signature of an observed object, i.e. its infrared energy produced by a scene in the frequency band 8 pm to 14 pm, in digital data. This data can be used to create a visible image or input into a computer for interpretation or post-processing. By converting heat radiation into an electrical signal, infrared cameras create an image that is visible to the human eye. Because the thermal energy of a scene is independent of reflected light, and because it can travel through objects with small particle sizes, thermal processing is a technology that is suitable for receiving images in any condition where light is limited or absent.

The spectral information captured for a given scene can be represented as superimposed images that form a three-dimensional array. In spectroscopy, a data array is obtained when the spectrally resolved image is represented in a three-dimensional volume, in which the captured image is represented by multiple two-dimensional images, while the spectral information associated with individual pixels or groups of pixels is represented as a third coordinate in the array of the data.

Conventional broadband imaging is a powerful but expensive analysis technology for remote sensing of surface chemistry. For example, wide-spectral spectral imaging usually generates the dataset with two spatial dimensions and one spectral dimension, for which 2D images for captured at a speed of 100 to 400 frames per second and the 3rd dimension have about 256 frames for spectral analysis, usually formed from one camera through many filters. These frames must be processed in real time, which requires extremely fast and expensive electronics.

Technical essence of the invention

The task of the invention is to create a method and a device for processing and interpreting images in the wide electromagnetic spectrum, which provide spectral image analysis for non-invasive and remote recognition of objects, regardless of the surrounding environment and light intensity, as well as perform identification of chemical content of the substances and checking for the presence of organic matter.

The task was solved by creating a device for processing and interpreting images in the wide electromagnetic spectrum, which includes a first CMOS sensor connected to a graphics processor and a thermal head with a second CMOS sensor in the thermal range connected to an interface converter. The graphics processor and interface converter are connected to a microprocessor, which in turn is connected to a second interface converter connected to a video display. The microprocessor is also connected to a peripheral microprocessor, which in turn is connected to a compass, to an accelerometer, to a remote control module, to a control unit and to an interface connector for connection to external devices. The microprocessor on the other hand is also connected to a USB connector, an SD card connector, a WiFi module, a Bluetooth module and a GPS module.

A third CMOS sensor in the short and mid-infrared SWIR spectral range is connected to the microprocessor via the interface connector for connection to external devices.

An external illuminator is connected to the interface connector for connecting external devices to calibrate the CMOS sensor according to the captured scene.

The external illuminator has a changing spectral range in the range of 0.2 nm to 12 nm, during the recording of the spectral pictures.

The task was solved by creating a method for processing and interpreting images in the electromagnetic spectrum, for application to the created device, and the method includes the following operations:

calibrating the CMOS sensor and the second CMOS sensor in the thermal head for each spectral range;

scanning the full spectrum of all sensors and recording all spectral pictures;

analyzing the spectral characteristics and their normalization for each pixel or zone;

compiling a complex spectral picture for each scene.

In addition, the method also includes the operations:

creating a scene reference picture for each broadcaster;

analyzing the reference picture;

recognition of the target element from the reference picture.

The created device visualizes a complex picture based on correlation between spectral pictures of all ranges; calculates the correlation between areas in the spectral images to digitally obtain the spectral picture at the corresponding wavelength; records the spectral images using the pixel frequency characteristics or areas of the CMOS sensor itself; calculates and visualizes the complex numbers for each zone or pixel; and uses normalized optics to provide uniform angular detection for each pixel or area.

An advantage of the created device is that it provides high picture quality and image detail when recognizing stationary or moving objects, regardless of the environment, from intense light to complete darkness. In addition, by using broad-spectrum chromatography, recognition of the chemical content of substances is carried out. An advantage is also the ability to connect the created device with a portable meteorological device that provides information on wind speed, humidity, altitude, temperature and others, as a result of which the location of the observed object can be corrected.

Description of attached figures

The present invention is illustrated in the accompanying figures, where

Fig. 1 is a schematic diagram of the device for processing and interpreting images in the electromagnetic spectrum, according to the invention;

Fig. 2 shows a two-dimensional spectral image matrix with recognition zones plotted;

Fig. 3 shows a superimposed three-dimensional matrix of the same image with superimposed areas of all spectral ranges;

Fig. 4 shows a scene irradiated with different wavelengths and captured by a thermal camera;

Fig. 5 shows exemplary material recognition using differential and conditional correlation between two images;

Fig. 6 shows a diagram of a day and night vision CMOS sensor where scanning is performed on the basis of frequency correlation;

Fig. 7 shows pixel-level or zone-level material recognition by broad-spectrum spectrography and capturing 22 frames at different wavelengths; and

Fig. 8 shows a diagram of the detection method.

Examples of implementation of the invention

The created device for processing and interpreting images in the electromagnetic spectrum, shown in figure 1, includes a CMOS sensor 1 connected to a graphics processor 3 and a thermal head 2 with a second CMOS sensor in the thermal range connected to an interface converter 4. The graphics processor 3 and the interface converter 4 are connected to a microprocessor 5, which in turn is connected to a second interface converter 6, connected to a video display 7. The microprocessor 5 is also connected to a peripheral microprocessor 10, which in turn is connected to a compass 8, to an accelerometer 9, with a remote control module 11, with a control unit 12 and with an interface connector for connection to external devices 13. The microprocessor 5 on the other hand is also connected with a USB connector 14, with an SD card connector 15, with a WiFi module 16, with a Bluetooth module 17 and with GPS module 18.

A third CMOS sensor in the short and mid-infrared SWIR spectral range is connected to the microprocessor 5 through the interface connector for connection with external devices 13.

An external illuminator is connected to the external device connection interface connector 13 for calibrating the CMOS sensor 1 according to the captured scene.

The external illuminator has a changing spectral range in the range of 0.2 nm to 12 nm, during the recording of the spectral pictures.

The CMOS sensor 1 is a type of semiconductor image sensor, representing a 13 megapixel matrix, which converts the beams of photons falling on its surface into an electrical signal in digital form. This digital signal, by means of a MIPI interface (Mobile Industry Processor Interface), is transmitted to the graphic processor 3 for processing and changing its parameters in order to extract a quality image. In the graphics processor 3, the resolution of the image, the frequency of data transmission, as well as contrast, brightness, gamma corrections, saturation, sharpness and other parameters of the picture are determined. After performing the necessary processing, the digital data is fed to the microprocessor 5, specialized in the processing of digital video signals.

At the same time, the thermal head 2 measures the electromagnetic radiation power of this same scene observed by the CMOS sensor 1, taking into account the heating of the materials with known variable electrical resistance depending on the temperature. By means of a specialized processor and a second CMOS thermal sensor built into the thermal head 2, the measured power of electromagnetic radiation is visualized. Like any digital image, thermal images are made up of pixels, each of which represents a specific temperature value. Images are usually interpreted in grayscale, where dark areas indicate cooler temperatures and light areas warmer. Each point in the captured temperature range can be interpreted with a unique color or shade, based on its temperature value.

Before proceeding to image extraction, however, it is necessary to correct the non-uniform data obtained from the different pixels. This calibration is realized both in production conditions with reference to a reference body, and subsequently in real time accompanying the operation of the device. After performing the calibration, some of the pixels do not behave as expected and generate anomalous values. Through bi-linear interpolation implemented in the specialized processor of the thermal head 2, a high resolution of the signal is achieved, without data loss.

After processing the data from the thermal head 2, the finished image is fed for conversion to the interface converter 4. The interface converter 4 serves to convert the signal from RGB (Red Green Blue) to a MIPI format suitable for the digital video input of the microprocessor 5. By means of a suitable pixel-level blending the data from the CMOS sensor 1 and the sensor in the thermal head 2 are superimposed on each other, thus achieving an image containing the detail of the observed object data in both the visible spectrum and its temperature characteristics and changes.

The integrated overlay (combination of different layers of data) in the output image of the microprocessor 5, allows visualization of text and graphics on the image received by the thermal head 2 and the CMOS sensor 1. The second interface converter 6 converts the M1PI format of the output image of the microprocessor 5 to the LVDS format (Low-voltage differential signaling) suitable for visualization on the display 7.

The compass 8 and accelerometer 9 provide information about geographic coordinates, the position the device is pointed at, its orientation, the angle at which it is pointed and changes in acceleration, if applied.

The interface connector for connection with external devices 13 is a UART connector (Universal asynchronous receiver-transmitter), which enables updating the software of the device, as well as adding external modules such as a rangefinder, external power supply and others.

The created device for processing and interpreting images in the electromagnetic spectrum combines the data from the compass 8, the accelerometer 9 and other external devices connected through the interface connector for connection with external devices 13, and these data are collected and processed by the peripheral microprocessor 10. After being processed by means of a serial protocol, the data is sent to the microprocessor 5, where it is superimposed in the form of text and graphics on the already mixed thermal and visible images.

The user interface implemented in the graphics layer of the microprocessor 5 gives access to settings of the video processing modules of the CMOS sensor 1, the graphics processor 3, the sensor from the thermal head 2 and the display 7, as well as the compass 8, the accelerometer 9 and the remote control module 11. Through the user interface of the microprocessor 5, settings are also made on the USB connector 14, the SD card connector 15, the WiFi module 16, the Bluetooth module 17 and the GPS module 18. Depending on the application of the device, the realized user interface allows different configurations.

By means of the Wi-Fi module 16, the video signal can be transmitted wirelessly in real time to various devices supporting this functionality. Video recording software is implemented in the microprocessor 5, both in the device's built-in memory and in external memory accessible via the SD card connector 15. The recording visualizes both the information collected by the CMOS sensor 1 and the thermal head sensor 2, and and the overlay added in the graphics layer of the microprocessor 5 of the device. The records are available to the user through the USB connector 14 for connection to a computer, or for direct viewing on the display 7 of the device.

Through the Bluetooth module 17, a connection with a phone is implemented, with a special application that allows two-way communication between the phone and the device. The device sends the values from all its sensors in real time. In turn, the phone sends to the device GPS coordinates, as well as commands allowing remote handling of the user interface through the remote control module 11, which also shows the orientation of the device on a geographical map.

To obtain an accurate idea of the position of the monitored object in certain applications, a portable weather station can be connected to the device via the Bluetooth module 17. Based on parameters such as air humidity, temperature, altitude, wind speed and direction, functions for ballistic calculations are implemented, and data from a rangefinder attached to the device can also be used. Through the created interface and the use of a compass and automatic distance measurement, an accurate determination of the position of the recognized material is carried out.

The created device allows the observation of absorption spectra of compounds, which are a unique reflection of their molecular structure, allowing their identification. The photon energies associated with this part of the infrared flux (from 1 to 15 kcal/mol) are not large enough to excite electrons, but can cause vibrational excitation of covalently bonded atoms and groups. Molecules experience a wide variety of vibrational motions characteristic of their atoms. Therefore, almost all organic compounds will absorb the infrared radiation that corresponds to the energy of these vibrations.

The device uses information from the daytime and thermal image of a given object, obtained in completely independent paths, to reproduce and interpret various characteristics of that object visible in the waveband between 0.25 pm to 12 pm. Through appropriate blending and digital processing, the best object recognition is performed regardless of the environment - from intense light to complete darkness. Based on infrared chromatography, the device recognizes chemical contents of the observed object, identifies the materials from which it is made, as well as the presence of organic matter.

The variety of interfaces implemented in the device allows easy reconfiguration, addition of additional sensors capturing different bands of the electromagnetic spectrum, external peripheral devices depending on the specific requirements for monitoring and detection of certain objects. The design and coupling with a specially designed optical part allows the development of interferometers with applications in materials science and industrial quality control, in building inspection, in fire safety, in agriculture and others.

The first wide-spectrum CMOS sensor used 1 covers 3 light ranges: ultraviolet 0.15 to 0.45 pm, light 0.45 to 0.95 pm and infrared 0.95 to 1.5 rt. The second CMOS thermal sensor used covers the mid-infrared range from 1.5 to 5 pm. A third CMOS thermal sensor covering the far infrared range from 5 to 12 pm was also used.

Each of these sensors is connected to a configuration of lenses that provide the same angular resolution for each pixel. In order to provide uniform two-dimensional images of the scene, a uniform field of view (FOV) must be provided, so the optical parameters must be chosen appropriately, such as a 13 megapixel CMOS sensor with a resolution of 4208x3120, a pixel size of 1.1 microns, and an objective lens with focal length 22 mm field of view is 12 x 9 degrees and resolution 0.17 minutes.

To provide the same area of analysis with the thermal CMOS sensor with a resolution of 320x256, a pixel size of 12 microns, an objective lens with a focal length of 18 mm should be used, where the resolution will be 2.3 minutes.

To change the sensitivity of the CMOS sensor 1 from the light to the near-infrared region, the pixels are combined into larger binning arrays. For example, to achieve identical resolution of the first sensor as the thermal one, the pixels of the first sensor are combined into a 16x16 pixel array. Thanks to binning, the first sensor is possible to operate with sensitivities proportional to the pixel size of 1.1, 2.2, 4.4, 8.8 and 17.6 microns and a corresponding resolution of 0.17, 0.34, 0.69, 1.38 and 2.75 minutes while keeping the same area and viewing angle. Captured two-dimensional images are based on minimal information bits, the minimum size of which is the size of a pixel, but it is possible to be larger and combine multiple pixels, which determines the resolution of recognition.

In addition, to ensure a quality spectral picture, it is necessary to perform calibration of all sensors in the spectral ranges in which they work. To perform this calibration, standardized light sources in the respective spectral bands, UV, light green, red, blue, near, mid and far UV are used. A uniform scene irradiated with the corresponding spectral band is used for calibration.

In image detection, it is possible to use a similar light source if a greater gain of the detection effect is to be obtained. With such irradiation, the light source in each spectrum is absorbed and reflected with different intensity, which changes the data for each pixel along the spectral axis, which in the corresponding analysis depicts the characteristics of the materials in the scene being analyzed.

The present invention can be used to detect any type of target substance that has spectral characteristics that can be viewed in at least portions of the fields of view of the spectral image array. Individual pixels or groups of pixels of captured image information can be analyzed to assess whether the pixels exhibit spectral characteristics of the target substance. In embodiments that use image capture elements of high resolution, this allows even minute traces of a target substance to be detected if even just one pixel in the image corresponds to the target substance. Moreover, by recognizing which pixel/s of the captured image information corresponds to the target substance, the exact location of the target substance in the image can be located and identified.

The recognition of the substances itself takes place as a matrix of values is formed for each pixel, scanned in different modes of the sensors used, during the wide-spectrum scanning. These modes are mainly dependent on the spectral band in which the scene was captured. When these values are analyzed in the mode of frequency, quantitative and qualitative evaluation, the independence characteristic of the chemical evaluation of the material is obtained.

To determine the sensitivity of the first CMOS sensor 1, its ability to receive 3 independent images with wavelengths of 450 nm, 550 nm and 650 nm is used. Similar selectivity can be deduced for 350 nm, 750 nm, 850 nm and 950 nm in most of these sensors. An example of sensor characterization for such recognition is shown in fig. 6, where images captured at each frequency are analyzed by correlation.

In order for each spectral band to be identified and isolated with the highest precision in the thermal range from each signal arriving from a wide-spectrum sensor, a special calibration mode is required. This calibration is performed based on the following 2 conditions: an emitter is used in the corresponding band and the image received by the sensor is analyzed by a sensitivity histogram with and without the reference light from the emitter. In this way, the sensitivity of the respective detector of the respective wavelength is determined. An example shown in fig. 4, where 4 irradiation methods were used with beams of 5, 7, 9 and 12 micrometers respectively.

The emitter may not be constantly emitting and perform frame-synchronized flashes. The most efficient way to analyze is if the flash is through a frame of a progressive frame image so that 2 pictures are identified to find the difference between. Such a detection method is shown in fig. 5 where an image with and without irradiation is correlated with each other by a difference filter and an XOR filter. In this way, areas of interest are defined and visualized instantly.

The completeness of the method for determining target substances is achieved when the specific values of the scanned wavelengths are taken, normalized to unity or another selected value and presented in a numerical series for each pixel or area of the image, for example dividing the two-dimensional image into areas is shown in Fig. 2. The normalized values of each zone or normalized pixel as explained above in calculating the angular viewing area are superimposed in a row as shown in fig. 7 or represented as a complex number and displayed as a 3 dimensional array as shown in fig. 3.

In fig. 3 shows a three-dimensional image of the spectral model where each zone or in this case pixel is depicted vertically with its corresponding value normalized to 255 or in 8 bit resolution. This resolution can be 16, 32, 48, etc. depending on the complexity of target recognition.

An advantage of the created device is that it provides recognition of substances with standard sensor matrices and without the need to use external filters, special frequency-dependent sensors requiring a special pixel geometry, which makes many recognition devices more expensive. For this purpose, the frequency-dependent image is created by analytical transformation and correlation between the array of spectral images and in particular by analyzing the dependence between the change of a pixel or an area.

Material detection using NIR and LWIR spectroscopy is aided by an emission consisting of a wavelength capable of absorption by the target compound. Materials are detected across the full electromagnetic spectrum from 100 nm to about 14,000 nm using a dual camera array, NIR and LWIR, while the SWIR range is extrapolated.

The created device for processing and interpreting images in the electromagnetic spectrum is used according to the following method.

The sensors are calibrated by the electronic interface described above for each range using a reference scene and possible external illuminator if required. The sensors are then programmed to receive at the appropriate wavelength by direct sensing, mode shifting, correlation, binning, or using a digital filter. Corresponding images are also recorded for each working spectral range and the 3D array is created in which each zone or pixel has a value depending on the frequency/spectral picture. These values are normalized to the dynamic range of the sensors.

The values for each pixel are formed by stacking the normalized values from each spectral image of the scene sequentially, forming a complex number. This complex number from all zones is visualized in a complex picture that shows the complex characteristic of the scene. Different chemical compounds are correlated to this complex number for each pixel or zone.

Recognized elements are identified with an overlay and colored in zones as they are plotted on the current spectral image whether it is video or thermal. In this way, the required chemical element is imposed on the basis of the picture that is displayed at the relevant moment.

Claims (8)

1. A device for processing and interpreting images in the electromagnetic spectrum, characterized in that it includes a CMOS sensor (1) connected to a graphics processor (3) and a thermal head (2) with a second CMOS sensor in the thermal range connected to an interface converter (4), wherein the graphics processor (3) and the interface converter (4) are connected to a microprocessor (5), which in turn is connected to a second interface converter (6) connected to a video display (7), wherein the microprocessor (5) is also connected to a peripheral microprocessor (10), which in turn is connected to a compass (8), to an accelerometer (9), to a remote control module (11), to a control unit (12) and to an interface connector for connection to external devices (13), the microprocessor (5) on the other hand is also connected to a USB connector (14), to an SD card connector (15), to a WiFi module (16), to a Bluetooth module (17) and with GPS module (18). 2. A device according to claim 1, characterized in that a third CMOS sensor in the short and mid-infrared SWIR spectral range is connected to the microprocessor (5) via the interface connector for connection with external devices (13). 3. A device according to claim 1, characterized in that an external illuminator for calibrating the CMOS sensor (1) according to the captured scene is connected to the interface connector for connection with external devices (13). 4. Device according to claim 3, characterized in that the external illuminator has a changing spectral range in the range of 0.2 nm to 12 nm, during the recording of the spectral pictures. 5. A method for processing and interpreting images in the electromagnetic spectrum, for application to the device of claim 1, characterized in that it includes the following operations: calibration of the CMOS sensor (1) and the CMOS sensor in the thermal head (2) for each spectral range; scanning the full spectrum of all sensors and recording all spectral pictures; analyzing the spectral characteristics and their normalization for each pixel or area; compilation of a complex spectral picture for each scene; imaging the complex spectral picture and correlating the areas of interest to target substances. 6. A method according to claim 5, characterized in that a scene reference picture is created for each emitter. 7. Method according to claim 5 and 6, characterized in that the reference picture is analyzed. 8. Method according to claim 5, 6 and 7, characterized in that recognition of the target element from the reference picture is performed.