CN114820317B - Super-resolution time delay integral imaging method for pixel offset - Google Patents
Super-resolution time delay integral imaging method for pixel offset Download PDFInfo
- Publication number
- CN114820317B CN114820317B CN202210460299.3A CN202210460299A CN114820317B CN 114820317 B CN114820317 B CN 114820317B CN 202210460299 A CN202210460299 A CN 202210460299A CN 114820317 B CN114820317 B CN 114820317B
- Authority
- CN
- China
- Prior art keywords
- super
- resolution
- pixel
- imaging
- image shift
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 238000003384 imaging method Methods 0.000 title claims abstract description 44
- 238000000034 method Methods 0.000 claims abstract description 19
- 238000005070 sampling Methods 0.000 claims description 6
- 238000001914 filtration Methods 0.000 claims description 4
- 230000003287 optical effect Effects 0.000 description 6
- 230000010354 integration Effects 0.000 description 5
- 230000000694 effects Effects 0.000 description 4
- 238000004364 calculation method Methods 0.000 description 3
- 238000010586 diagram Methods 0.000 description 2
- 238000006073 displacement reaction Methods 0.000 description 2
- 238000005516 engineering process Methods 0.000 description 2
- 244000007853 Sarothamnus scoparius Species 0.000 description 1
- 238000004458 analytical method Methods 0.000 description 1
- 238000003491 array Methods 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 238000001514 detection method Methods 0.000 description 1
- 229910003460 diamond Inorganic materials 0.000 description 1
- 239000010432 diamond Substances 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 230000006641 stabilisation Effects 0.000 description 1
- 238000011105 stabilization Methods 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T3/00—Geometric image transformations in the plane of the image
- G06T3/40—Scaling of whole images or parts thereof, e.g. expanding or contracting
- G06T3/4053—Scaling of whole images or parts thereof, e.g. expanding or contracting based on super-resolution, i.e. the output image resolution being higher than the sensor resolution
- G06T3/4069—Scaling of whole images or parts thereof, e.g. expanding or contracting based on super-resolution, i.e. the output image resolution being higher than the sensor resolution by subpixel displacements
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F17/00—Digital computing or data processing equipment or methods, specially adapted for specific functions
- G06F17/10—Complex mathematical operations
- G06F17/11—Complex mathematical operations for solving equations, e.g. nonlinear equations, general mathematical optimization problems
- G06F17/12—Simultaneous equations, e.g. systems of linear equations
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/20—Analysis of motion
- G06T7/277—Analysis of motion involving stochastic approaches, e.g. using Kalman filters
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/10—Image acquisition modality
- G06T2207/10032—Satellite or aerial image; Remote sensing
Landscapes
- Engineering & Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Mathematical Physics (AREA)
- Theoretical Computer Science (AREA)
- Pure & Applied Mathematics (AREA)
- Mathematical Analysis (AREA)
- Computational Mathematics (AREA)
- Mathematical Optimization (AREA)
- Data Mining & Analysis (AREA)
- Databases & Information Systems (AREA)
- Algebra (AREA)
- Operations Research (AREA)
- Software Systems (AREA)
- General Engineering & Computer Science (AREA)
- Multimedia (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Studio Devices (AREA)
- Transforming Light Signals Into Electric Signals (AREA)
Abstract
The super-resolution TDI push-broom imaging method with slightly mismatched image moving speed and direction is provided, so that the same ground object area image acquired by a push-broom imaging system from four discrete windows has sub-pixel level offset in the object space, the imaging quality of each window can be ensured not to be obviously reduced, the imaging modes such as digital domain TDI and the like can be realized on the premise of meeting the super-resolution reconstruction condition, and the method is suitable for high-resolution imaging requirements and realizes flexible and efficient super-resolution imaging.
Description
Technical Field
The invention relates to the field of aerospace optical remote sensing imaging, in particular to a super-resolution time delay integral imaging method of pixel offset. The method is suitable for Time Delay Integration (TDI) imaging in the field of TDI push-broom imaging or industrial TDI push-broom imaging, breaks through the resolution limit of an optical system, and acquires a higher resolution image through pixel offset and super-resolution reconstruction. The invention can be used in the fields of space optical remote sensing or display panel detection in the industrial field, and the like, and can realize super-resolution imaging of a light and small camera.
Background
In the industrial field, the linear HS 32K TDI introduced by DALSA company adopts two TDI arrays with pixel offset, two images are collected at the same time, super-resolution reconstruction is realized, the technology completely depends on the architecture of the sensor, and super-resolution imaging in two directions of rows and columns cannot be really realized due to only pixel offset between columns, so that the application effect is limited.
In the field of space optical remote sensing, US HERA SYSTEM business satellite company 2019 applies for super-resolution imaging patent (Image Stabilization and pixel shifting for a nanosatellite image system, patent number US010338376B 2) about pixel offset, two groups of images are respectively acquired by optical light splitting and two 5120×5120 pixel image sensors, the method relies on a high-precision micro-displacement mechanism to realize sub-pixel offset of the two sensors, and only area array imaging can be performed, but TDI push broom imaging cannot be realized.
The patent (a method and a system for acquiring a sub-pixel displacement sequence image, publication No. CN 106791405B) applies for super-resolution imaging about pixel offset in the beginning of university of teachers and universities, adopts an inclined mode sampling mode, forms a certain included angle with the row direction of a sensor to generate sub-pixel offset (2 times of super-resolution corresponds to 45 degrees, 3 times of super-resolution corresponds to 26.5 degrees, 4 times of super-resolution corresponds to 18.4 degrees), adopts the same window super-sampling mode, couples the window size, the included angle and a digital domain integration processing algorithm with super-resolution, is too complex in processing process, is unfavorable for flexible and efficient application, and the large-angle inclined application of the sensor can enable a sampling window to be changed into a diamond shape from a square shape, reduces MTF of a rail and a vertical rail, so that the super-resolution effect cannot reach expectations.
Disclosure of Invention
The invention provides a super-resolution time delay integral imaging method for pixel offset, which aims to solve the problems that the existing super-resolution imaging technology for pixel offset can only perform area array imaging, cannot realize TDI push-broom imaging, has complex processing process, is unfavorable for flexible and efficient application, reduces MTF of a track and a vertical track, cannot reach the expected super-resolution effect and the like.
A super-resolution time delay integral imaging method of pixel offset is realized by the following steps:
step one, according to the set super resolution ratio beta, integral series M, pixel size d and line transfer time T L MTF sloughing margin eta, image shift speed v f And the image shift speed stability psi, calculating a bias current mismatch angle delta theta, an image shift mismatch rate delta v, the pixel shift windowing quantity N and the windowing interval L 0 ；
The pixel offset windowing quantity N is more than or equal to beta 2 ；
According to an along-track MTF tolerance constraint equation, constraining the maximum value of the image shift mismatch rate delta v; expressed by the following formula:
calculating a corresponding windowing interval L according to the image shift mismatch rate delta v 0 Expressed by the following formula:
according to a vertical rail MTF tolerance constraint equation, constraining the maximum value of a bias current mismatch angle delta theta; expressed by the following formula:
calculating to obtain a corresponding windowing interval according to the drift angle mismatch angle delta theta;
calculating line transfer time according to the image transfer mismatch rate; the following formula is adopted:
adopting an image shift stability constraint equation to characterize the constraint of the platform stability to the window interval; the following formula is adopted:
wherein GSD is satellite pixel resolution, and H is orbit height.
Step two, performing pixel offset sampling and super-resolution reconstruction according to the imaging parameters obtained by calculation in the step one;
acquiring four windowed regions to obtain an image frame P according to the constrained pixel dislocation rule 1 ～P 4 The super-resolution pixel thereof reconstructs the following equation set:
the equation has multiple solutions, and the existing optimal estimation methods such as wiener filtering, kalman filtering and the like can be adopted to realize the optimal solution solving of super-resolution reconstruction; a super-resolution reconstructed image P is obtained.
The invention has the beneficial effects that: the imaging method of the invention is characterized in that on a single-chip high-speed CMOS image sensor, the basic characteristics of stable image shift speed in space remote sensing TDI push-broom imaging and industrial TDI push-broom imaging are utilized, the inter-column pixel shift is generated in a bias current micro-mismatch mode, the inter-row pixel shift is generated in an image shift micro-mismatch mode (the existing method is realized by adopting large bias current and image shift mismatch, as described by a reference standard, the super-resolution effect and the flexibility of an imaging mode are limited obviously), and therefore, the flexible and efficient super-resolution imaging is realized under the condition of not adding any hardware device.
The method of the invention provides a super-resolution TDI push-broom imaging method of introducing pixel offset between columns and rows by micro mismatch of the image shift speed and direction, so that the same ground object region image acquired by a push-broom imaging system from four discrete windows has sub-pixel level offset in the object space, the imaging quality of each window can be ensured not to be obviously reduced, and the imaging modes such as digital domain TDI and the like can be realized on the premise of meeting the super-resolution reconstruction condition, thereby being suitable for the high-resolution imaging requirement.
Drawings
FIG. 1 is a schematic diagram of a mechanism for generating pixel shift in the row direction and column direction in a super-resolution time delay integral imaging method of pixel shift according to the present invention;
FIG. 2 is a super-resolution multi-window schematic;
fig. 3 is a super-resolution pixel correspondence diagram.
Detailed Description
The present embodiment will be described with reference to fig. 1 to 3, which illustrate a super-resolution time delay integral imaging method with pixel offsetThe method for solving several key parameters of super-resolution imaging is as follows: bias current mismatch angle Δθ: the too large bias current mismatch angle can affect the MTF of the TDI integral image, and the too small bias current mismatch angle can lead to the windowing interval L 0 The requirement on the stability of the image shift speed is increased, and the engineering application is not facilitated.
Image shift mismatch ratio Δν: the relative error rate between the component of the image shift speed in the digital domain TDI integration direction and the charge transfer speed of the digital domain TDI integration process is too high, so that the MTF of the TDI integration image can be influenced, the image shift mismatch rate is too small, the windowing interval can be increased, the requirement on the stability of the image shift speed is high, and the engineering application is not facilitated;
pixel offset windowing number N and windowing pitch L 0 : and determining the number of windows and the number of lines of the spacing of each window according to the super-resolution ratio beta, so that the number of windows and the number of lines of the spacing of each window just meet the pixel offset.
The system inputs parameters: super-resolution ratio beta, integral progression M, pixel size d, line transfer time T L MTF sloughing margin eta, image shift speed v f The image shift speed stability psi is as follows: equation (1-1) is an information quantity conservation equation, and only if the acquired information quantity is increased by not less than the square of the super-resolution ratio, the super-resolution image quality can be ensured; equation (1-2) is an along-track MTF tolerance constraint equation, a given MTF tolerance will constrain the maximum of the image shift mismatch rate; equation (1-3) is a windowing interval calculation equation, and the corresponding windowing interval can be calculated according to the image shift mismatch rate; equations (1-4) are vertical rail MTF margin constraint equations, a given MTF margin will constrain the maximum value of the drift angle mismatch angle; equation (1-5) is a windowing interval calculation equation, and the corresponding windowing interval can be calculated according to the drift angle mismatch angle; equations (1-6) are image shift mismatch rate definition equations from which line shift time parameters can be calculated; equations (1-7) are image shift stability constraint equations used to characterize the constraints of platform stability versus window spacing.
Pixel offset sampling and super-resolution reconstruction: obtaining an image frame P according to the pixel dislocation rule of four windowing acquisitions 1 ～P 4 The object space pixel overlapping rule is shown as a formula (2), the super-resolution reconstruction process is a solving process of the equation set, and because the equation set has an iteration relationship, an analysis solution cannot be directly obtained, and an optimal estimation method can be adopted to obtain the super-resolution reconstruction image P.
In this embodiment, taking a high-speed planar array CMOS space optical camera supporting multiple windows as an example, the original resolution is 2 meters, the resolution of 1 meter is realized by super-resolution imaging, the satellite platform stability is 0.0001 °/s, the MTF ecdysis margin is 0.97, the image shift mismatch margin is 0.00124, the bias current mismatch margin is 0.0071 °, and the window interval L is calculated 0 Should not be less than 202 rows;
according to the platform stability index, the deviation introduced by the platform stability in the imaging time difference range of each windowing should not exceed 1/3 of the pixel offset between windowing so as to ensure the super-resolution processing precision and obtain the windowing interval L 0 No more than 228 lines, so the windowing interval L in engineering practice 0 Any integer between 202 and 228 can be taken as L 0 Let 220 be L 0 And (3) calculating a deviation mismatch angle of 0.0651 degrees and an image shift mismatch rate of 0.001136 as a setting basis of working parameters of the camera.
The technical features of the above-described embodiments may be arbitrarily combined, and all possible combinations of the technical features in the above-described embodiments are not described for brevity of description, however, as long as there is no contradiction between the combinations of the technical features, they should be considered as the scope of the description.
The above examples illustrate only a few embodiments of the invention, which are described in detail and are not to be construed as limiting the scope of the invention. It should be noted that it will be apparent to those skilled in the art that several variations and modifications can be made without departing from the spirit of the invention, which are all within the scope of the invention. Accordingly, the scope of protection of the present invention is to be determined by the appended claims.
Claims (2)
1. A super-resolution time delay integral imaging method of pixel offset is characterized in that: the method is realized by the following steps:
step one, according to the set super resolution ratio beta, integral series M, pixel size d and line transfer time T L MTF sloughing margin eta, image shift speed v f And the image shift speed stability psi, imaging parameters are calculated, wherein the imaging parameters comprise bias current mismatch angle delta theta, image shift mismatch rate delta v, pixel shift windowing quantity N and windowing interval L 0 L and L 1 ；
The pixel offset windowing quantity N is more than or equal to beta 2 ；
According to an along-track MTF tolerance constraint equation, constraining the maximum value of the image shift mismatch rate delta v; expressed by the following formula:
calculating a corresponding windowing interval L according to the image shift mismatch rate delta v 0 Expressed by the following formula:
according to a vertical rail MTF tolerance constraint equation, constraining the maximum value of a bias current mismatch angle delta theta; expressed by the following formula:
calculating to obtain a corresponding windowing interval according to the drift angle mismatch angle delta theta;
according to the line transfer time T L Calculating the image shift mismatch rate; the following formula is adopted:
adopting an image shift stability constraint equation to characterize the constraint of the platform stability to the window interval; the following formula is adopted:
wherein GSD is satellite pixel resolution, H is orbit height;
step two, according to the calculated mismatch angle delta theta, the image shift mismatch rate delta v, the pixel shift windowing quantity N and the windowing interval L in the step one 0 Performing pixel offset sampling and super-resolution reconstruction;
acquiring four windowed regions according to the constrained pixel dislocation rule to obtain an image frame P 1 ～P 4 The super-resolution pixel thereof reconstructs the following equation set:
solving the equation set by adopting an optimal estimation method to realize the solving of the super-resolution reconstruction optimal solution; a super-resolution reconstructed image P is obtained.
2. The method of pixel-shifted super-resolution time-delay integral imaging of claim 1, wherein: the optimal estimation method in the second step comprises a wiener filtering or Kalman filtering method.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210460299.3A CN114820317B (en) | 2022-04-28 | 2022-04-28 | Super-resolution time delay integral imaging method for pixel offset |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210460299.3A CN114820317B (en) | 2022-04-28 | 2022-04-28 | Super-resolution time delay integral imaging method for pixel offset |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN114820317A CN114820317A (en) | 2022-07-29 |
| CN114820317B true CN114820317B (en) | 2023-04-21 |
Family
ID=82509857
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202210460299.3A Active CN114820317B (en) | 2022-04-28 | 2022-04-28 | Super-resolution time delay integral imaging method for pixel offset |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN114820317B (en) |
Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN104122549A (en) * | 2014-07-21 | 2014-10-29 | 电子科技大学 | Deconvolution based radar angle super-resolution imaging method |
| CN109348126A (en) * | 2018-11-07 | 2019-02-15 | 中国科学院光电研究院 | A kind of face battle array continuous push-scanning image method of number TDI for space camera |
| CN114201489A (en) * | 2021-11-30 | 2022-03-18 | 深圳市魔方卫星科技有限公司 | Fast parallel on-orbit image motion matching method and device and storage medium |
Family Cites Families (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101562698B (en) * | 2009-04-07 | 2011-02-23 | 中国科学院长春光学精密机械与物理研究所 | Method for intermittently adjusting drift angle of space camera in real time |
| DE102014112648A1 (en) * | 2014-08-29 | 2016-03-03 | Carl Zeiss Ag | Image pickup device and method for image acquisition |
| CN107888836B (en) * | 2017-12-05 | 2020-09-29 | 中国科学院长春光学精密机械与物理研究所 | Push-broom type remote sensing camera focusing method based on auxiliary focal plane |
| US11688039B2 (en) * | 2019-03-25 | 2023-06-27 | Teledyne Digital Imaging, Inc. | Method for generating a super-resolution image and related device |
| CN112498746B (en) * | 2020-11-16 | 2022-06-28 | 长光卫星技术股份有限公司 | Method for automatically planning push-scanning time and posture of satellite along longitude line |
-
2022
- 2022-04-28 CN CN202210460299.3A patent/CN114820317B/en active Active
Patent Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN104122549A (en) * | 2014-07-21 | 2014-10-29 | 电子科技大学 | Deconvolution based radar angle super-resolution imaging method |
| CN109348126A (en) * | 2018-11-07 | 2019-02-15 | 中国科学院光电研究院 | A kind of face battle array continuous push-scanning image method of number TDI for space camera |
| CN114201489A (en) * | 2021-11-30 | 2022-03-18 | 深圳市魔方卫星科技有限公司 | Fast parallel on-orbit image motion matching method and device and storage medium |
Non-Patent Citations (1)
| Title |
|---|
| 姚烨 ; 乔彦峰 ; 钟兴 ; 于树海 ; 戴路 ; 白杨.凝视成像降质模型的超分辨率重建.光学学报.2017,全文. * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN114820317A (en) | 2022-07-29 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN103115631B (en) | System and method for correcting imaging parameter error of remote sensing camera | |
| CN114820317B (en) | Super-resolution time delay integral imaging method for pixel offset | |
| Wang et al. | A fine image motion compensation method for the panoramic TDI CCD camera in remote sensing applications | |
| CN111595332B (en) | Full-environment positioning method integrating inertial technology and visual modeling | |
| CN102607815B (en) | Method and device for measuring lateral magnification of optical system based on jointing of point target images | |
| CN103248838A (en) | CMOS camera image bias adjusting method based on digital field TDI (time delayed and integration) algorithm | |
| Tao et al. | Realize the image motion self-registration based on TDI in digital domain | |
| FRIEDMANN | Two-dimensional resampling of line scan imagery by one-dimensional processing | |
| CN113920046B (en) | Multi-fragment satellite image stitching and geometric model construction method | |
| CN102620914B (en) | Method adopting line source for measuring transverse magnification of optical system | |
| Zhang et al. | Precise calibration of dynamic geometric parameters cameras for aerial mapping | |
| US20150304653A1 (en) | Mapping electrical crosstalk in pixelated sensor arrays | |
| CN105046674A (en) | Nonuniformity correction method of multi-pixel parallel scanning infrared CCD images | |
| CN114862723A (en) | Astronomical telescope image field distortion calibration method based on measurement of dense star field | |
| Pál et al. | A Hexapod Design for All-sky Sidereal Tracking | |
| CN106568437A (en) | Geostationary satellite imaging navigation and registration fixed star north and south parameter extraction method, and system | |
| CN108519161B (en) | A kind of infrared focal plane asymmetric correction method | |
| CN102607813A (en) | Method and device for measuring transverse magnifying power of optical system based on linear source | |
| Zhang et al. | Theoretic studies of full constraints on a star tracker's influential error sources for in-orbit calibration | |
| CN109870155B (en) | Estimation method for arrival time difference of direct solar light/planet reflected light | |
| CN111062976A (en) | FMT-based low-orbit satellite solar telescope remote sensing image registration method | |
| Seshadri et al. | Mapping electrical crosstalk in pixelated sensor arrays | |
| CN106528498B (en) | The point spread function extracting method of fixed star remote sensing image | |
| LoPresto et al. | The center-to-limb wavelength shift of a number of Fraunhofer lines | |
| CN115164938B (en) | Linear array swing scanning type optical remote sensing satellite on-orbit geometric calibration method |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant | ||
| PE01 | Entry into force of the registration of the contract for pledge of patent right |
Denomination of invention: A Super Resolution Time Delay Integration Imaging Method with Pixel Shift Granted publication date: 20230421 Pledgee: Changchun Rural Commercial Bank Co.,Ltd. Exhibition Center sub branch Pledgor: Changguang Satellite Technology Co.,Ltd. Registration number: Y2024220000024 |
|
| PE01 | Entry into force of the registration of the contract for pledge of patent right |