Classifications

• • G—PHYSICS
  • G01—MEASURING; TESTING
  • G01J—MEASUREMENT OF INTENSITY, VELOCITY, SPECTRAL CONTENT, POLARISATION, PHASE OR PULSE CHARACTERISTICS OF INFRARED, VISIBLE OR ULTRAVIOLET LIGHT; COLORIMETRY; RADIATION PYROMETRY
  • G01J3/00—Spectrometry; Spectrophotometry; Monochromators; Measuring colours
  • G01J3/28—Investigating the spectrum
  • G01J3/30—Measuring the intensity of spectral lines directly on the spectrum itself
  • G01J3/32—Investigating bands of a spectrum in sequence by a single detector
• • G—PHYSICS
  • G06—COMPUTING; CALCULATING OR COUNTING
  • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
  • G06T3/00—Geometric image transformations in the plane of the image
  • G06T3/40—Scaling of whole images or parts thereof, e.g. expanding or contracting
  • G06T3/4053—Scaling of whole images or parts thereof, e.g. expanding or contracting based on super-resolution, i.e. the output image resolution being higher than the sensor resolution
  • G06T3/4061—Scaling of whole images or parts thereof, e.g. expanding or contracting based on super-resolution, i.e. the output image resolution being higher than the sensor resolution by injecting details from different spectral ranges
• • H—ELECTRICITY
  • H04—ELECTRIC COMMUNICATION TECHNIQUE
  • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
  • H04N25/00—Circuitry of solid-state image sensors [SSIS]; Control thereof
  • H04N25/70—SSIS architectures; Circuits associated therewith
  • H04N25/701—Line sensors
• • G—PHYSICS
  • G01—MEASURING; TESTING
  • G01J—MEASUREMENT OF INTENSITY, VELOCITY, SPECTRAL CONTENT, POLARISATION, PHASE OR PULSE CHARACTERISTICS OF INFRARED, VISIBLE OR ULTRAVIOLET LIGHT; COLORIMETRY; RADIATION PYROMETRY
  • G01J3/00—Spectrometry; Spectrophotometry; Monochromators; Measuring colours
  • G01J3/28—Investigating the spectrum
  • G01J3/2803—Investigating the spectrum using photoelectric array detector
  • G01J2003/2806—Array and filter array
• • G—PHYSICS
  • G01—MEASURING; TESTING
  • G01J—MEASUREMENT OF INTENSITY, VELOCITY, SPECTRAL CONTENT, POLARISATION, PHASE OR PULSE CHARACTERISTICS OF INFRARED, VISIBLE OR ULTRAVIOLET LIGHT; COLORIMETRY; RADIATION PYROMETRY
  • G01J3/00—Spectrometry; Spectrophotometry; Monochromators; Measuring colours
  • G01J3/28—Investigating the spectrum
  • G01J3/2823—Imaging spectrometer
  • G01J2003/2826—Multispectral imaging, e.g. filter imaging
• • G—PHYSICS
  • G01—MEASURING; TESTING
  • G01N—INVESTIGATING OR ANALYSING MATERIALS BY DETERMINING THEIR CHEMICAL OR PHYSICAL PROPERTIES
  • G01N21/00—Investigating or analysing materials by the use of optical means, i.e. using sub-millimetre waves, infrared, visible or ultraviolet light
  • G01N21/17—Systems in which incident light is modified in accordance with the properties of the material investigated
  • G01N2021/1793—Remote sensing

Definitions

• the invention relates to the field of space imaging, and more particularly to a method for acquiring multispectral images and panchromatic images intended for Earth observation.
• Space imagery also called satellite imagery, refers to the taking of images from space by matrix sensors placed on satellites. There are essentially three spatial image acquisition techniques respectively called: “push broom”, “single shot” and “push frame”.
• the “push broom” technique also called “scanning”, consists of using at least one linear sensor of the CCD (Charge Coupled Device) strip type and the displacement of the satellite around the earth to construct bands of an image. Its operating principle is identical to that of scanners and photocopiers which include a reading bar moved along the page to be scanned.
• This technique is typically implemented on large Earth observation satellites such as the so-called SPOT and Pleiades satellites.
• the linear sensor can be equipped with a filter allowing the production of spectral images.
• the “single shot” technique consists of using a matrix sensor to build a 2D image in a single pass. Its operating principle is identical to that of cameras and is frequently used on microsatellites and nanosatellites. Although it is possible to construct color images by equipping the matrix sensor with a Bayer filter, for example, this acquisition technique makes it difficult to produce multispectral images and presents a low flux and a long time. high integration leading to produce blurring on the image.
• the “push frame” technique combines the “push broom” technique with the “frame” technique. It consists in using a matrix sensor to acquire images shifted in time along a vertical axis of the sensor due to the movement of the satellite around the Earth. This technique allows the realization of multispectral and panchromatic images by placing in front of the sensor a filter having several high horizontal bands of a few hundred pixels and extending over the entire width of the sensor, each band covering a different wavelength interval. . The acquisitions are shifted in time so that the resulting spatial shift is slightly less than the height of the bands of the filter, so that for each spectral band a complete image of a scene can be reconstituted by concatenation.
• the object of the invention is therefore to propose a method for the acquisition of multispectral images and panchromatic images obviating at least in part the aforementioned problems.
• the invention proposes a method for the acquisition of multispectral images and panchromatic images by a monochrome matrix sensor on board a mobile carrier in a direction of movement above a space comprising a scene to be observed.
• the matrix sensor has a field covering an extent of space greater than the scene in the direction of movement and is provided with a filter comprising a panchromatic filtering band followed by at least two spectral filtering bands each covering an interval of different wavelength.
• the panchromatic filtering band and the spectral filtering bands are parallel and extend in a direction substantially orthogonal to the direction of movement of the wearer.
• the method comprises the following steps: a) when the scene is included in one of the filtering bands of the filter, acquisition of a multispectral image by the sensor at least through the two spectral filtering bands; b) transfer of the multispectral image to an electronic processing unit; c) acquisition of a panchromatic imagette by the sensor only through the panchromatic filter band; d) transfer of the panchromatic imagette to the electronic processing unit; e) as long as the scene is not included in another of the filtering bands, repeating steps c) and d); and f) repeating steps a) to e) until obtaining a multispectral image of the scene through each band of spectral filtering.
• the time necessary for the scene to pass from one to the other of the spectral filtering bands is used to acquire partially overlapping panchromatic images on which a so-called “super- resolution” allowing the generation of a high resolution panchromatic image
• the acquisition of the multispectral image produced in step a) is done through the panchromatic filtering band and the spectral filtering bands, which has the effect of increasing the number of acquisitions by panchromatic thumbnails and therefore the resolution of the panchromatic image resulting from the “super-resolution” processing method.
• the acquisition of the multispectral image produced in step a) is done only through the spectral filtering bands.
• the acquisition of the multispectral image produced in step a) is done in “2 ⁇ 2 binning” mode.
• the method comprises the steps of: correcting the defects of all the thumbnails of a band over a given duration, aligning said thumbnails, merging the thumbnails to form a reconstituted image and cropping the reconstituted image to retain a useful field .
• the invention also relates to an image acquisition device arranged to be mounted on a carrier vehicle.
• the device includes an image sensor connected to an electronic control unit programmed to implement such a method.
• the invention also relates to a carrier vehicle equipped with such an acquisition device.
• FIG. 1 is a schematic view of the method according to a particular embodiment of the invention
• Figure 2 is a schematic view of the carrier equipped with the matrix sensor used in the method illustrated in Figure 1
• Figure 3 is a schematic view of the filter fitted to the matrix sensor used in the method illustrated in Figure 1;
• Figure 4 shows different successive acquisitions made by the sensor according to the method illustrated in Figure 1;
• FIG.5 shows a timing diagram of the various acquisitions made by the sensor according to the process illustrated in Figure 1.
• a monochrome matrix sensor C is on board a carrier A moving above a space comprising a scene Sc to be observed.
• the carrier A is here a satellite in orbit around the Earth and the space comprising the scene Sc to be observed is the terrestrial surface.
• the satellite transmits image signals to a station M on the ground by radiocommunication means in a manner known per se.
• the sensor C comprises a detection matrix zone comprising unitary detectors, or pixels (from the English “picture element”), arranged in rows extending along a horizontal X axis substantially perpendicular to a direction of movement of carrier A and in columns extending along a Y axis substantially parallel to the direction of movement of carrier A.
• Sensor C here has a resolution of 5120 x 5120 pixels, i.e. approximately 26 megapixels, and is preferably of the CMOS (Complementary Metal Oxide Semi-Conductor) type.
• the sensor C is here arranged behind an optical group comprising one or more lenses and/or one or more mirrors, and thus has a field covering an extent of the terrestrial surface greater than the scene to be observed.
• An electronic processing unit U connected to the sensor C receives signals generated by the various pixels and from these signals supplies data representative of an image acquired by the sensor.
• the signals are representative of the power of a light flux received by each pixel and will be translated into gray levels in the images created from the sensor signals.
• the electronic processing unit U comprises for example at least one calculation module, such as a processor, and a memory containing a sensor control and image processing computer program executable by the processor.
• the computer program includes instructions arranged to implement the method of the invention.
• Sensor C is equipped with the “area of interest” functionality, also called “Area Of Interest” (AOI), which makes it possible to select and read only part of the matrix area of said sensor C such as a windowing.
• AOI Area Of Interest
• Such functionality has the advantage of reducing the number of pixels read and therefore of increasing the acquisition speed of the sensor and of reducing the quantity of data transmitted to the electronic processing unit U.
• the transfer of data between the sensor C and the electronic processing unit U is typically done by one or several LVDS (“Low Voltage Differential Signaling”) channels in digital form and the time for transferring an image is proportional to the quantity of data, in other words to the number of pixels used and therefore to the dimensions of the area of interest.
• LVDS Low Voltage Differential Signaling
• the sensor C is covered with a filter F comprising six filter bands P, S1, S2, S3, S4, S5 of rectangular shape and each spaced apart by a transition zone T.
• the bands of filtering P, S1, S2, S3, S4, S5 have substantially identical dimensions and extend along the X axis over an entire width of the sensor C.
• Each filter band P, S1, S2, S3, S4, S5 covers a different wavelength interval and here has a height of approximately 800 pixels along the Y axis.
• the transition zones T are substantially identical and here have a height equal to a few pixels.
• the first filtering band P is a panchromatic filtering band, in other words a band transmitting all the wavelengths of the visible domain of the light spectrum and blocking all the other wavelengths.
• the other five filter bands S1, S2, S3, S4, S5 are spectral filter bands, in other words bands each transmitting a predetermined color of the light spectrum and absorbing all the other colors. It could for example be chosen the color red for the second band S1, the color magenta for the third band S2, the color orange for the fourth band S3, the color green for the fifth band S4 and the color blue for the sixth band S5 .
• each filter band P, S1, S2, S3, S4, S5 corresponds to a different wavelength interval and partly covers the detection matrix zone of the sensor.
• sensor C By moving in the same direction as carrier A, sensor C scans along the Y axis the scene Sc which, given of the sensor C, has dimensions smaller than those of the filter bands P, S1, S2, S3, S4, S5, so that the filter F allows the acquisition of an image of the scene Sc for different wavelengths . To do this, it is necessary to wait for the scene Sc to pass from one filtering band to another to trigger the acquisition of an image by the sensor C.
• the waiting time depends on the speed of movement of the carrier A and is here equal to 119 milliseconds.
• the principle of the invention is to take advantage of this waiting time to make panchromatic thumbnails of the scene Sc via the panchromatic P filtering band of the filter F.
• FIGS. 4 and 5 illustrate a particular embodiment of the invention which will now be detailed.
• the scene Sc appears for the first time in one of the filter bands P, S1, S2, S3, S4, S5, covering the sensor, namely here the panchromatic filter band P.
• the acquisition of a first multispectral image IMS1 is carried out by the sensor C through all the filtering bands P, S1, S2, S3, S4 , S5, in other words through the panchromatic filtering band P and the spectral filtering bands S1, S2, S3, S4, S5 (step 10). It is then understood that the zone of interest of the sensor is maximum and covers all of the filtering bands P, S1, S2, S3, S4, S5.
• the multispectral image IMS1 is then transferred to the electronic processing unit U (step 20) then the area of interest of the sensor is modified via an appropriate link (RS232, I2C, SPI or equivalent serial type link) so as to select only the pixels of the matrix zone of sensor C corresponding to the panchromatic P filter band.
• the transfer time is here equal to 38 milliseconds.
• sensor C continues to scan scene Sc. carried out by the sensor C through the panchromatic filter band P (step 30) as they are transferred to the electronic processing unit U (step 40).
• the acquisition speed of the sensor C and the speed at which the signals generated by the pixels of the sensor C are transmitted to the electronic processing unit U allow here the acquisition and the transfer of three first panchromatic thumbnails IP1.1, IP1.2, IP1.3, before the scene Sc is entirely included in the filtering band S1.
• the acquisition time and the transfer time are here respectively 18 milliseconds and 4.8 milliseconds for each of the first panchromatic imagettes IP1.1, IP1.2, IP1.3.
• the first panchromatic images IP1.1, IP1.2, IP1.3 strongly overlap with a non-integer number of pixels.
• the zone of interest of the sensor is then again modified so as to become maximal again and cover all of the filtering bands P, S1, S2, S3, S4, S5.
• the acquisition of a second multispectral image IMS2 is carried out by the sensor through all the filter bands P, S1, S2, S3, S4, S5 (step 10), then the second multispectral image IMS2 is transferred to the electronic processing unit U (step 20).
• Steps 30 and 40 are then repeated (steps 50) and result in the acquisition and transfer of three second panchromatic images IP2.1, IP2.2, IP2.3, before the scene Sc is entirely comprised in the third filtering band S2.
• the second panchromatic thumbnails IP2.1, IP2.2, IP2.3 strongly overlap by a non-integer number of pixels.
• the second multispectral image IMS2 for its part slightly overlaps the first multispectral image IMS1 by a generally non-integer number of pixels.
• steps 10 to 50 are finally repeated until a multispectral image of the scene Sc is obtained through each of the filter bands P, S1, S2, S3, S4, S5 (step 60).
• FIG. 5 illustrates in the form of a chronogram all of the steps 10, 20, 30, 40, 50, 50 of the method for acquiring the multispectral images IMS1, IMS2 and the panchromatic images IP1.1, IP1.2, IP1 .3, IP2.1, IP2.2, IP2.3.
• panchromatic imagette IP1.1, IP1.2, IP1.3, IP2.1 , IP2.2, IP2.3 is here six times lighter than an IMS1, IMS2 multispectral image.
• Each multispectral image IMS1, IMS2 is first of all divided into six imagettes IMS1.1, IMS1.2 ... IMS1.6, IMS2.1, IMS2.2, ... IMS2.6 each corresponding to one of the filtering bands P, S1, S2, S3, S4, S5 after removal of the transition zones T.
• IMS1, IMS2 a panchromatic imagette IMS1.1, IMS2.1 and five spectral imagettes IMS1.
• the thumbnails IMS1.1, IMS1.2, ... IMS1.6, IMS2.1, IMS2.2, ... IMS2.6 are possibly corrected for defects such as distortion, then those from the same band of filtering are concatenated by aligning them on an entire pixel after calculating their overlap.
• the calculation of the overlap can result from the a priori knowledge of the overlap obtained for example from navigation data (GPS position, pointing angle, etc.) or from an analysis of a common zone between the thumbnails such as a image correlation.
• the integer part of the number of pixels is used for the concatenation, and the fractional part is added to the distortion defects to be corrected.
• the fractional part is added to the distortion defects to be corrected.
• the images generated by concatenation also called concatenated or reconstituted images, are clipped in order to in particular to eliminate the stepped zones caused by the lateral offset of the thumbnails between them.
• panchromatic images IP1.1, IP1.2, IP1.3, IP2.1, IP2.2, IP2.3 obtained between two acquisitions of multispectral images are clipped and then grouped by rank of acquisition between two multispectral images IMS1, IMS2.
• the panchromatic imagettes IP1.2, IP1.3, IP2.1, IP2.2, IP2.3 of the same rank are then, in the same way as the panchromatic imagettes IMS1.1, IMS2.1 resulting from the multispectral images IMS1, IMS2, corrected for defects and concatenated.
• All of the concatenated panchromatic images are then processed by the “super resolution” algorithm as if they were native images to generate a high resolution panchromatic image of the scene Sc and of what surrounds it.
• the high-resolution image is then cropped to match the concatenated images from the multispectral images.
• IMS1, IMS2 is carried out through all the filtering bands P, S1, S2, S3, S4, S5, S6, it can also be carried out only through a only part of the filter bands, for example through the spectral filter bands S1, S2, S3, S4, S5, S6.
• the “2 ⁇ 2 binning” mode of the sensor C which consists in grouping during the acquisition the signal of four adjacent pixels and therefore in reducing by two the resolution of the multispectral images. Consequently, the acquisition and transfer times will be substantially divided by four, which will allow the acquisition of a greater number of panchromatic imagettes between two acquisitions of multispectral images, the acquisition of panchromatic images using the classic sensor C mode (“binning 1x1”).
• panchromatic images IP1.1, IP1.2, IP1.3, IP2.1, IP2.2, IP2.3 are used to increase the resolution of the panchromatic image resulting from the multispectral images IMS1, IMS2, they can also be used to increase its signal-to-noise ratio (or “SNR” for “Signal-to-Noise Ratio”), by carrying out a simple summation of said panchromatic imagettes.
• SNR signal-to-noise ratio
• a different exposure time for the acquisition of panchromatic images (IP1.1, IP1.2, IP1.3, IP2.1, IP2.2, IP2.3) and the acquisition of multispectral images (IMS1, IMS2) can be used to increase their signal to noise ratio (SNR).
• SNR signal to noise ratio
• the method of the invention can of course be used for the observation of planets other than the Earth.
• the method of the invention can be used in fields other than spatial imaging provided wearer A can generate a scan of the scene Sc by sensor C and said sensor C is sensitive in intervals of wavelengths selectable by a filter and that its zone of interest is configurable: observation of the ground by a drone, by a high-altitude pseudo-satellite of the HAPS "High-altitude platform station” type, etc.
• the carrier A can be any aircraft such as an airplane, a space vehicle or a satellite.
• LVDS channels Low Voltage Differential Signaling
• MIPI-CSI2 Mobile Industry Processor Interface - Camera Serial Interface 2
• Camera Link CXP (CoaXPress).

Landscapes

• Physics & Mathematics (AREA)
• Spectroscopy & Molecular Physics (AREA)
• General Physics & Mathematics (AREA)
• Engineering & Computer Science (AREA)
• Theoretical Computer Science (AREA)
• Multimedia (AREA)
• Signal Processing (AREA)
• Color Television Image Signal Generators (AREA)
• Image Processing (AREA)

Applications Claiming Priority (2)

Publications (1)

Family

ID=80595288

Family Applications (1)

Country Status (4)

Family Cites Families (1)

• 2021
  • 2021-09-23 FR FR2110050A patent/FR3127298A1/fr active Pending
• 2022
  • 2022-09-14 CA CA3231635A patent/CA3231635A1/fr active Pending
  • 2022-09-14 WO PCT/EP2022/075527 patent/WO2023046551A1/fr active Application Filing
  • 2022-09-14 EP EP22786325.5A patent/EP4405648A1/de active Pending

Also Published As

Similar Documents

Legal Events