CN115689887A - Automatic driving RGBIR image resampling method, system, terminal and medium - Google Patents
Automatic driving RGBIR image resampling method, system, terminal and medium Download PDFInfo
- Publication number
- CN115689887A CN115689887A CN202211336962.5A CN202211336962A CN115689887A CN 115689887 A CN115689887 A CN 115689887A CN 202211336962 A CN202211336962 A CN 202211336962A CN 115689887 A CN115689887 A CN 115689887A
- Authority
- CN
- China
- Prior art keywords
- image
- channel
- rgbir
- interpolation
- point
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000000034 method Methods 0.000 title claims abstract description 74
- 238000012952 Resampling Methods 0.000 title claims abstract description 56
- 238000005070 sampling Methods 0.000 claims abstract description 53
- 230000006870 function Effects 0.000 claims description 29
- 230000003044 adaptive effect Effects 0.000 claims description 26
- 239000011159 matrix material Substances 0.000 claims description 26
- 230000003595 spectral effect Effects 0.000 claims description 24
- 230000015654 memory Effects 0.000 claims description 20
- 238000001228 spectrum Methods 0.000 claims description 20
- 238000004364 calculation method Methods 0.000 claims description 12
- 238000012937 correction Methods 0.000 claims description 12
- 238000004590 computer program Methods 0.000 claims description 11
- 238000001914 filtration Methods 0.000 claims description 8
- 238000004422 calculation algorithm Methods 0.000 claims description 5
- 230000009471 action Effects 0.000 claims description 4
- 238000010606 normalization Methods 0.000 claims description 4
- 239000000126 substance Substances 0.000 claims description 2
- 230000008569 process Effects 0.000 abstract description 20
- 238000010586 diagram Methods 0.000 description 7
- 238000012545 processing Methods 0.000 description 4
- 230000000007 visual effect Effects 0.000 description 4
- 238000012986 modification Methods 0.000 description 3
- 230000004048 modification Effects 0.000 description 3
- 238000004566 IR spectroscopy Methods 0.000 description 2
- 238000002329 infrared spectrum Methods 0.000 description 2
- 230000008447 perception Effects 0.000 description 2
- 230000009286 beneficial effect Effects 0.000 description 1
- CLOMYZFHNHFSIQ-UHFFFAOYSA-N clonixin Chemical compound CC1=C(Cl)C=CC=C1NC1=NC=CC=C1C(O)=O CLOMYZFHNHFSIQ-UHFFFAOYSA-N 0.000 description 1
- 239000000470 constituent Substances 0.000 description 1
- 230000007547 defect Effects 0.000 description 1
- 238000013461 design Methods 0.000 description 1
- 238000009499 grossing Methods 0.000 description 1
- 238000005192 partition Methods 0.000 description 1
- 238000012805 post-processing Methods 0.000 description 1
- 230000003068 static effect Effects 0.000 description 1
- 230000001360 synchronised effect Effects 0.000 description 1
Images
Landscapes
- Color Television Image Signal Generators (AREA)
Abstract
The invention provides an automatic driving RGBIR image resampling method and system, wherein the method comprises the following steps: interpolating a G channel in the RGBIR image to obtain a complete G channel image; respectively interpolating R, B and IR channels in the RGBIR image by taking the G channel as a guide to obtain a complete R channel image, a complete B channel image and an IR channel image; wherein, the IR channel image is an IR image; and respectively carrying out down-sampling on the G channel image, the R channel image and the B channel image according to a Bayer format to obtain a Bayer image, and completing the re-sampling of the RGBIR image. The method can reconstruct the RGBIR image into the traditional Bayer image and the IR image with high quality, then fully utilize ISP to process the Bayer image to obtain the RGB image with high quality, and utilize the obtained IR image with high quality to assist the automatic driving of a low-brightness scene.
Description
Technical Field
The invention relates to the technical field of automatic driving and image processing, in particular to a method, a system, a terminal and a medium for resampling an automatic driving RGBIR image.
Background
In order to simultaneously perceive a scene with sufficient light and a scene with dim light in the visual environment perception of automatic driving, sensors of an rgbiir filter array (CFA) are often used to simultaneously acquire a visible light image and an infrared image, wherein a more common rgbiir filter array design is shown in fig. 1.
However, a conventional Image Signal Processor (ISP) is designed for a conventional Bayer filter array image, and it needs to resample an image collected by an rgbiir filter array into image data in Bayer format and IR image data in a single channel, and in a scene with sufficient light, the image data in Bayer format is used to enter the ISP for processing to obtain a visible RGB image, and then the visible RGB image is sensed and assisted by automatic driving, and in a scene with insufficient light, the IR image is used for assisting by automatic driving, which is shown in fig. 2, wherein a blue-blue dotted line region is the rgbiir image resampling process in the present invention.
For the rgbiir image, in the conventional technical scheme, for the IR image, the IR data is directly sampled into a low-resolution image according to the arrangement mode of the rgbiir filter array, or the IR image is directly used for interpolation to obtain an image with the same size as the rgbiir resolution, in such a scheme, the IR image is reduced in resolution or poor in image quality obtained by interpolation;
for RGB data in RGBIR, in the traditional scheme, data corresponding to the position of IR data is directly interpolated to obtain an RGB image arranged by 4x4 CFA, then an interpolation algorithm aiming at the arrangement of the 4x4 CFA is designed to obtain an RGB three-channel image, and then the RGB three-channel image is down sampled to form Bayer data.
Disclosure of Invention
Aiming at the defects in the prior art, the invention provides an automatic driving RGBIR image resampling method, an automatic driving RGBIR image resampling system, an automatic driving RGBIR image resampling terminal and an automatic driving RGBIR image resampling medium.
The invention is realized by the following technical scheme.
According to an aspect of the present invention, there is provided an automatic driving rgbiir image resampling method, including:
interpolating a G channel in the RGBIR image to obtain a G channel image;
after the G channel image is obtained, respectively interpolating R, B and IR channels in the RGBIR image by taking the G channel as a guide to obtain a corresponding R channel image, a corresponding B channel image and a corresponding IR channel image; wherein the IR channel image is an IR image having the same resolution as the input image;
and respectively carrying out downsampling on the G channel image, the R channel image and the B channel image according to a given Bayer format to obtain corresponding Bayer images, and completing the resampling of the RGBIR images.
Optionally, the interpolating the G channel in the rgbiir image to obtain a G channel image includes:
constructing an adaptive spectrum correlation kernel function based on the spectral correlation of each channel of R, G, B and IR in the RGBIR image;
and performing Gaussian up-sampling on the positions of the R, B and IR points in the G channel according to the adaptive spectrum correlation kernel function, and completing interpolation on G pixels of the R, B and IR points in the G channel of the RGBIR image to obtain a G channel image.
Optionally, the constructing an adaptive spectral correlation kernel function based on spectral correlations of R, G, B, and IR channels in the rgbiir image includes:
if the derivatives of the spectrums of all channels in the RGBIR image are approximately equal, the derivatives are obtained through subtraction calculation of diagonal pixels; constructing the position x using the calculated derivatives in the diagonal direction p Adaptive spectral correlation kernel of
Comprises the following steps:
covariance matrix of the Gaussian kernel
By at position x p The derivatives of nearby diagonal directions are calculated, then:
wherein z is u And z v Is the derivative in the diagonal direction and,
represents x p The number of pixels in the neighborhood of the location,
is shown as
The number of pixels in the neighborhood.
Optionally, the gaussian upsampling the positions of the R, B and IR points in the G channel according to the adaptive spectral correlation kernel function includes:
performing Gaussian up-sampling on R, B and IR point positions in a G channel according to the self-adaptive spectrum correlation kernel function, and setting the point needing up-sampling as x p The calculation formula of the up-sampling
Comprises the following steps:
wherein, the first and the second end of the pipe are connected with each other,
representing the point x that needs to be upsampled p The number of pixels in the neighborhood of the location,
is x i The value after the point re-sampling is,
is x i A matrix of point binary masks is used to generate,
is a weight normalization factor, is obtained by multiplying and summing mask points of the adaptive spectrum correlation kernel function,
is an adaptive spectrum correlation kernel function;
and G pixel interpolation of the R, B and IR point positions in the G channel is realized after Gaussian up-sampling is carried out on the R, B and IR point positions in the G channel.
Optionally, the interpolating the R, B, and IR channels in the rgbeir image with the G channel as a guide to obtain a corresponding R channel image, B channel image, and IR channel image includes:
let X be: r, B or IR;
taking the G channel as a guide image, and performing pre-interpolation on the X channel to obtain an X channel pre-interpolation image
Selecting X sampling points in an X channel of an RGBIR image, and calculating a pre-interpolation image of the X channel
Obtaining an error image delta X from the error between the X sampling point image and the image;
interpolating the error image delta X to obtain a complete error plane image delta X full ；
Using said error plane image DeltaX full For X channel pre-interpolation image
Point-to-point correction is carried out to obtain the final X channel interpolation image X final I.e., the corresponding complete R-channel image, B-channel image, or IR-channel image.
Optionally, the G channel is used as a guide image, and the X channel is pre-interpolated to obtain an X channel pre-interpolated image
The method comprises the following steps:
adopting a guide filtering method, taking the G channel as a guide image, taking the X channel as an image to be processed, and carrying out guide filtering to obtain an X channel pre-interpolation image
Optionally, the selecting an X sampling point in an X channel of the rgbiir image, and calculating an error between the X channel pre-interpolation image and an X sampling point image to obtain an error image Δ X, where the error image Δ X includes:
the mask selects a binary matrix for the sampling points, 1 is taken at the X sampling point, 0 is taken at other points, and the dot elements are multiplied by the points of the matrix.
Optionally, the error image is interpolated to obtain a complete error plane image Δ X full The method comprises the following steps:
interpolating the error image delta X plane by adopting a BiCubic interpolation algorithm to obtain a complete error plane image delta X full 。
Optionally, the error plane image Δ X is used full The pair of X-channel pre-interpolation images
Point-to-point correction is carried out to obtain the final X channel interpolation graphImage X final The method comprises the following steps:
at the plane of obtaining the complete error image DeltaX full Thereafter, the image is pre-interpolated for the X channel as follows
And (3) performing point-to-point correction:
obtaining the final X channel interpolation image X final 。
Optionally, the down-sampling the G-channel image, the R-channel image, and the B-channel image according to a given Bayer format to obtain corresponding Bayer images includes:
in each channel image, 2x2 block areas are taken as units, in each 2x2 block area, according to a given Bayer format, channels corresponding to a corresponding sequence in the Bayer format are respectively reserved according to the sequence of a left upper corner point, a right upper corner point, a left lower corner point and a right lower corner point, and other channels are set to be zero;
and combining the non-zero points in the 4 points into an image of a single channel according to the positions of the non-zero points to obtain a Bayer image in a given format.
According to another aspect of the present invention, there is provided an autopilot rgbiir image resampling system comprising:
the G channel interpolation module is used for interpolating a G channel in the RGBIR image to obtain a complete G channel image;
the R/B/IR channel interpolation module is used for respectively interpolating R, B and IR channels in the RGBIR image by taking the G channel as a guide after the G channel image is obtained so as to obtain a corresponding complete R channel image, B channel image and IR channel image;
the IR image reconstruction module is used for acquiring the IR channel image, and the IR channel image is an IR image with the same resolution as the input image;
and the Bayer image reconstruction module is used for respectively carrying out down-sampling on the G channel image, the R channel image and the B channel image according to a given Bayer format to obtain corresponding Bayer images.
According to a third aspect of the present invention, there is provided a terminal comprising a memory, a processor and a computer program stored in the memory and executable on the processor, the processor being operable to execute the method of any one of the above, or to operate the system of any one of the above, when the processor executes the program.
According to a fourth aspect of the invention, there is provided a computer readable storage medium having stored thereon a computer program which, when executed by a processor, is operable to perform the method of any one of the above or to operate the system described above.
Due to the adoption of the technical scheme, compared with the prior art, the invention has the following beneficial effects:
the automatic driving RGBIR image resampling method, the system, the terminal and the medium provided by the invention fully utilize the correlation between the visible light RGB spectrum and the IR spectrum, can reconstruct the RGBIR image into the traditional Bayer image and IR image with high quality, and then fully utilize ISP to process the Bayer image to obtain the RGB image with high quality; meanwhile, in the resampling process, the high-quality IR image is obtained to assist automatic driving of a low-brightness scene.
The automatic driving RGBIR image resampling method, the automatic driving RGBIR image resampling system, the automatic driving RGBIR image resampling terminal and the automatic driving RGBIR image resampling medium can obtain the high-resolution and high-quality IR image under the condition that the resolution of a sensor is not upgraded, can better assist automatic driving of a night scene, and accordingly save the cost of an automatic driving solution.
The automatic driving RGBIR image resampling method, the automatic driving RGBIR image resampling system, the automatic driving RGBIR image resampling terminal and the automatic driving RGBIR image resampling medium can resample the RGBIR image to obtain image data in a specified Bayer format, so that the existing ISP system designed for the conventional Bayer format can be fully utilized to perform image processing to obtain a high-quality visible light image, visual automatic driving perception of the visible light image is assisted, and an image processing system does not need to be redesigned.
According to the automatic driving RGBIR image resampling method, the automatic driving RGBIR image resampling system, the automatic driving RGBIR image resampling terminal and the automatic driving RGBIR image resampling medium, high-quality images which are suitable for daytime/low-brightness/night scenes can be obtained by using the RGBIR sensor, and further the image acquisition requirements of auxiliary visual driving of different scenes can be met under the condition of saving cost.
Drawings
Other features, objects and advantages of the invention will become more apparent upon reading of the detailed description of non-limiting embodiments with reference to the following drawings:
fig. 1 is a schematic diagram of a 4 × 4 rgbirr color filter array in the prior art.
Fig. 2 is a schematic diagram of rgbirr image resampling and post-processing in the prior art.
Fig. 3 is a flowchart illustrating an automatic driving rgbiir image resampling method according to an embodiment of the present invention.
FIG. 4 is a schematic diagram of an interpolation process of RGBIR image according to a preferred embodiment of the present invention.
FIG. 5 is a schematic diagram of RGBIR image resampling process in a preferred embodiment of the present invention.
Fig. 6 is a schematic diagram of G channel data according to a preferred embodiment of the present invention.
Fig. 7 is a schematic diagram of the constituent modules of the automatic driving rgbiir image resampling system according to an embodiment of the present invention.
Detailed Description
The following examples illustrate the invention in detail: the embodiment is implemented on the premise of the technical scheme of the invention, and a detailed implementation mode and a specific operation process are given. It should be noted that various changes and modifications can be made by those skilled in the art without departing from the spirit of the invention, and these changes and modifications are all within the scope of the invention.
Fig. 3 is a flowchart of an automatic driving rgbiir image resampling method according to an embodiment of the present invention.
As shown in fig. 3, the method for resampling an autopilot rgbiir image provided by the present invention in this embodiment may include:
s1, carrying out interpolation on a G channel in an RGBIR image to obtain a G channel image;
s2, after a G channel image is obtained, respectively interpolating R, B and IR channels in the RGBIR image by taking the G channel as a guide to obtain a corresponding R channel image, a corresponding B channel image and a corresponding IR channel image; wherein the IR channel image is an IR image having the same resolution as the input image;
and S3, respectively carrying out downsampling on the G channel image, the R channel image and the B channel image according to a given Bayer format to obtain corresponding Bayer images, and finishing resampling the RGBIR images.
In a preferred embodiment of S1, interpolating the G channel in the rgbiir image to obtain a G channel image may include:
s11, constructing an adaptive spectrum correlation kernel function based on the spectral correlation of each channel of R, G, B and IR in the RGBIR image;
s12, performing Gaussian up-sampling on the positions of the R, B and IR points in the G channel according to the adaptive spectrum correlation kernel function, and completing interpolation on G pixels of the R, B and IR points in the G channel of the RGBIR image to obtain a G channel image.
In a preferred embodiment of S11, constructing the adaptive spectral correlation kernel based on the spectral correlation of each channel of R, G, B and IR in the rgbiir image may include:
if the derivatives of the spectra of all channels in the RGBIR image are approximately equal, the derivatives are obtained through the subtraction calculation of diagonal pixels; constructing the position x using the calculated derivatives in the diagonal direction p Adaptive spectral correlation kernel function of
Comprises the following steps:
covariance matrix of Gaussian kernel
By at position x p The derivative of nearby diagonal directions is calculated, then:
wherein z is u And z v Is the derivative in the diagonal direction and,
denotes x p The number of pixel points in the neighborhood of the location,
is shown as
The number of pixels in the neighborhood.
In a preferred embodiment of S12, gaussian upsampling the positions of the R, B and IR points in the G channel according to the adaptive spectral correlation kernel function may include:
performing Gaussian up-sampling on R, B and IR point positions in a G channel according to a self-adaptive spectrum correlation kernel function, and setting the point needing up-sampling as x p The calculation formula of the up-sampling
Comprises the following steps:
wherein the content of the first and second substances,
representing the point x that needs to be upsampled p The number of pixels in the neighborhood of the location,
is x i The value after the point re-sampling,
is x i A matrix of a binary mask of points is formed,
is a weight normalization factor, is obtained by multiplying and summing the mask points of the adaptive spectrum correlation kernel function,
is an adaptive spectral correlation kernel function;
and G pixel interpolation of the R, B and IR point positions in the G channel is realized after Gaussian up-sampling is carried out on the R, B and IR point positions in the G channel.
In a preferred embodiment of S2, taking the G channel as a guide, respectively interpolating R, B, and IR channels in the rgbiir image to obtain a corresponding complete R channel image, a complete B channel image, and an IR channel image, which may include:
s20, setting X as: r, B or IR;
s21, taking the G channel as a guide image, and performing pre-interpolation on the X channel to obtain an X channel pre-interpolation image
S22, selecting an X sampling point in an X channel of the RGBIR image, and calculating an error between an X channel pre-interpolation image and an X sampling point image to obtain an error image delta X;
s23, interpolating the error image delta X to obtain a complete error plane image delta X full ；
S24, adopting the error plane image delta X full For X channel pre-interpolation image
Point-to-point correction is carried out to obtain a final X channel interpolation image X final I.e. the corresponding complete R-channel image, B-channel image or IR-channel image.
In a preferred embodiment of S21, the G channel is used as a guide image to perform pre-interpolation on the X channel to obtain an X channel pre-interpolation image
The method can comprise the following steps:
adopting a guide filtering method, taking the G channel as a guide image, taking the X channel as an image to be processed, and carrying out guide filtering to obtain an X channel pre-interpolation image
In a preferred embodiment of S22, selecting X sampling points in an X channel of the rgbiir image, and calculating an error between the X channel pre-interpolated image and the X sampling point image to obtain an error image Δ X, where the method may include:
and the mask selects a binary matrix for the sampling points, the sampling points of the X are 1, other points are 0, and the point elements are multiplied by each other by using the matrix.
In a preferred embodiment of S23, the error image is interpolated to obtain a complete error plane image Δ X full The method comprises the following steps:
interpolating the error image delta X plane by adopting a BiCubic interpolation algorithm to obtain a complete error plane image delta X full 。
In a preferred embodiment of S24, an error plane image Δ X is used full For X channel pre-interpolation image
Performing point-to-point correction to obtainThe final X-channel interpolated image may include:
at the plane of obtaining the complete error image DeltaX full Thereafter, the image is pre-interpolated for the X channel as follows
And (3) performing point-to-point correction:
obtaining the final X channel interpolation image X final 。
In a preferred embodiment of S3, respectively down-sampling the G-channel image, the R-channel image, and the B-channel image according to a given Bayer format to obtain corresponding Bayer images, which may include:
s31, in each channel image, taking a 2x2 block area as a unit, respectively reserving channels corresponding to a corresponding sequence in a Bayer format according to a given Bayer format and the sequence of a left upper corner point, a right upper corner point, a left lower corner point and a right lower corner point in each 2x2 block area, and setting other channels to be zero;
and S32, combining the non-zero points in the 4 points into an image of a single channel according to the positions of the non-zero points to obtain a Bayer image with a given format.
The automatic driving RGBIR image resampling method provided by the above embodiment of the present invention, wherein the most important is the G channel interpolation, and then the R/B/IR channel interpolation is assisted by the G channel and the channel spectrum correlation theory; therefore, the whole framework of the auto-driving rgbiir image resampling method provided by the above embodiment of the present invention can be divided into an interpolation process and a resampling process, where the interpolation process is shown in fig. 4, and the resampling process is shown in fig. 5.
The technical solutions provided by the above embodiments of the present invention are further described below with reference to the accompanying drawings.
The method for resampling the automatic driving rgbiir image provided by the above embodiment of the present invention may include the following steps:
step 1, G channel interpolation
In the rgbiir array, data of a G channel is as shown in fig. 6, and since spectra of R/G/B/IR channels have correlation, in the interpolation process of the G channel, a G pixel of an R point in the rgbiir image is interpolated by using the spectral correlation between G and R, and similarly, a G pixel of a B point in the rgbiir image is interpolated in a similar manner, and a G pixel of an IR point in the rgbiir image is also interpolated in a similar manner.
Further:
step 1.1, calculating the adaptive spectral correlation kernel function
In the RGBIR interpolation theory to which embodiments of the present invention relate, the estimation of the adaptive spectral correlation kernel is based on the assumption that the derivatives of the spectral channels are approximately equal. Based on this assumption, the derivative can be calculated from the diagonal pixel subtraction. Using the derivatives of these diagonal directions, at position x p Adaptive spectral correlation kernel of
The estimation can be directly carried out from the original data, and the calculation formula is as follows:
wherein z is u And z v Is the derivative in the diagonal direction and,
denotes x p The number of pixels in the neighborhood of the location,
is shown as
The number of pixels in the neighborhood.
By the above calculation, the adaptive spectral correlation kernel can be estimated. The kernel function estimation can model and estimate other anisotropic kernel functions, so that a better edge reconstruction result is obtained.
Step 1.2, adaptive upsampling
From the adaptive spectral correlation kernel estimated in step 1.1
Gaussian up-sampling is carried out on positions of R, B and IR points in the G channel, and the point needing up-sampling is set as x p The calculation formula of the up-sampling
Comprises the following steps:
Step 2, interpolation of R channel
After the step 1 is finished, the G channel is completely recovered through interpolation, then the G channel is used as a guide to assist the R channel to carry out pre-interpolation, data estimated by the pre-interpolation and R data obtained by sampling are subtracted from each other at the position of an R sampling point to obtain an interpolation error, then the full-error plane interpolation is carried out on an interpolation error plane, and finally the point-to-point pre-interpolation image is subjected to error correction;
further:
step 2.1, R channel Pre-interpolation
For convenience, the embodiment of the invention can adopt a guide filtering method to carry out R channel pre-interpolation, takes the G channel as a guide image and the R channel as an image to be processed, and carries out guide filtering to obtain an R channel pre-interpolation image
Step 2.2, R channel pre-interpolation error calculation
Calculating a pre-interpolation image at R sampling points of the RGBIR image
And obtaining an error image delta R by the error of the R sampling point image, wherein the calculation formula is as follows:
the mask selects a binary matrix for the sampling points, 1 is taken as the R sampling point, 0 is taken as the other points, and the dot elements are multiplied by the points of the matrix.
Step 2.3, R error plane interpolation
Because the error image delta R only has a value at the R sampling point, and other points are all 0, then the error image delta R is interpolated, the embodiment of the invention can select bicubic interpolation algorithm to interpolate the error image delta R plane to obtain the complete error image plane delta R full 。
Step 2.4, R channel interpolation result correction
After obtaining the complete error image DeltaR full Then, the pre-interpolation image is processed
Point-to-point correction is carried out, and the calculation formula is as follows:
thus, a final R channel interpolation image R is obtained final 。
Step 3, B channel interpolation
For the interpolation of the B channel, an interpolation step similar to the R channel may be adopted for the interpolation, and details are not repeated here.
Step 4, IR channel interpolation
For the interpolation of the IR channel, an interpolation step similar to that of the R channel may be adopted for the interpolation, and details are not repeated here.
After the IR channel is subjected to interpolation, obtaining an IR image with high quality and the same resolution as the input image, and for night scenes and low-brightness scenes in the daytime, directly transmitting the IR image to a sensing system to assist a driving assisting system of a vehicle to automatically drive;
step 5, RGB three-channel resampling is carried out to obtain a standard bayer image
Through the interpolation process, an RGB three-channel true color image is obtained, in order to fully utilize an ISP system to process the image and then perform visual auxiliary driving, the RGB three-channel image must be resampled, a Bayer format must be given in the sampling process, and here, the resampling process is described by taking a Bayer format of BGGR as an example, and is shown in fig. 5.
The resampling process takes 2x2 block areas as a unit, and in each 2x2 block area: the upper left corner point, the B channel is reserved, and the other two channels are set to be zero; the upper right corner point, the G channel is reserved, and the other two channels are set to zero; the left lower corner point, the G channel is reserved, and the other two channels are set to be zero; the R channel is reserved at the right lower corner point, and other channels are set to be zero; combining the non-zero points in the 4 points into an image of a single channel according to the positions of the non-zero points to obtain a bayer image in a BGGR format; the resampling process for other Bayer formats is similar and will not be described herein.
Fig. 7 is a schematic diagram of component modules of an automatic driving rgbiir image resampling system according to an embodiment of the present invention.
As shown in fig. 7, the automatic driving rgbiir image resampling system provided by this embodiment may include:
the G channel interpolation module is used for interpolating a G channel in the RGBIR image to obtain a complete G channel image;
the R/B/IR channel interpolation module is used for respectively interpolating R, B and IR channels in the RGBIR image by taking the G channel as a guide after the G channel image is obtained so as to obtain a corresponding complete R channel image, B channel image and IR channel image;
the IR image reconstruction module acquires an IR channel image, wherein the IR channel image is an IR image with the same resolution as the input image;
and the Bayer image reconstruction module is used for respectively carrying out downsampling on the G channel image, the R channel image and the B channel image according to a given Bayer format to obtain the corresponding Bayer image.
It should be noted that, the steps in the method provided by the present invention may be implemented by using corresponding modules, devices, units, and the like in the system, and those skilled in the art may implement the composition of the system with reference to the technical solution of the method, that is, the embodiment in the method may be understood as a preferred embodiment of constructing the system, and details are not described herein.
An embodiment of the present invention provides a terminal, including a memory, a processor, and a computer program stored in the memory and executable on the processor, where the processor is configured to execute the method in any one of the above embodiments or execute the system in any one of the above embodiments when executing the program.
Optionally, a memory for storing a program; a Memory, which may include a volatile Memory (RAM), such as a Static Random Access Memory (SRAM), a Double Data Rate Synchronous Dynamic Random Access Memory (DDR SDRAM), and the like; the memory may also comprise a non-volatile memory, such as a flash memory. The memories are used to store computer programs (e.g., applications, functional modules, etc. that implement the above-described methods), computer instructions, etc., which may be stored in partition on the memory or memories. And the computer programs, computer instructions, data, etc. described above may be invoked by a processor.
The computer programs, computer instructions, etc. described above may be stored in one or more memories in a partitioned manner. And the computer programs, computer instructions, data, etc. described above may be invoked by a processor.
A processor for executing the computer program stored in the memory to implement the steps of the method according to the above embodiments. Reference may be made in particular to the description relating to the preceding method embodiment.
The processor and the memory may be separate structures or may be an integrated structure integrated together. When the processor and the memory are separate structures, the memory and the processor may be coupled by a bus.
An embodiment of the present invention further provides a computer-readable storage medium, on which a computer program is stored, which, when executed by a processor, is operable to perform the method of any one of the above embodiments, or to run the system of any one of the above embodiments.
The automatic driving RGBIR image resampling method, the automatic driving RGBIR image resampling system, the automatic driving RGBIR image resampling terminal and the automatic driving RGBIR image resampling medium provided by the embodiment of the invention fully utilize the correlation between the visible light RGB spectrum and the IR spectrum, can reconstruct the RGBIR image into the traditional Bayer image and the traditional IR image with high quality, and then fully utilize ISP to process the Bayer image to obtain the RGB image with high quality; meanwhile, in the resampling process, the obtained high-quality IR image is utilized to assist automatic driving of a low-brightness scene.
The above embodiments of the present invention are not exhaustive of the techniques known in the art.
The foregoing description has described specific embodiments of the present invention. It is to be understood that the present invention is not limited to the specific embodiments described above, and that various changes and modifications may be made by one skilled in the art within the scope of the appended claims without departing from the spirit of the invention.
Claims (9)
1. An autopilot RGBIR image resampling method, comprising:
interpolating a G channel in the RGBIR image to obtain a complete G channel image;
after the G channel image is obtained, respectively interpolating R, B and IR channels in the RGBIR image by taking the G channel as a guide to obtain a corresponding complete R channel image, a complete B channel image and an IR channel image; wherein the IR channel image is an IR image having the same resolution as the input image;
and respectively carrying out downsampling on the G channel image, the R channel image and the B channel image according to a given Bayer format to obtain corresponding Bayer images, and completing the resampling of the RGBIR images.
2. The autopilot RGBIR image resampling method of claim 1, wherein said interpolating the G channel in the RGBIR image resulting in a G channel image comprises:
constructing an adaptive spectrum correlation kernel function based on the spectral correlation of each channel of R, G, B and IR in the RGBIR image;
and performing Gaussian up-sampling on the positions of the R, B and IR points in the G channel according to the self-adaptive spectrum correlation kernel function, and completing interpolation on G pixels of the R, B and IR points in the G channel of the RGBIR image to obtain a complete G channel image.
3. The autopilot RGBIR image resampling method of claim 2, characterized in that it further comprises any one or more of the following:
-said constructing an adaptive spectral correlation kernel based on spectral correlations of R, G, B and IR channels in the rgbiir image, comprising:
if the derivatives of the spectra of all channels in the RGBIR image are approximately equal, the derivatives are obtained through the subtraction calculation of diagonal pixels; constructing position x by using calculated derivatives in diagonal direction p Adaptive spectral correlation kernel function of
Comprises the following steps:
wherein, x is the space of the pixel point,
the method comprises the following steps of (1) obtaining a covariance matrix of a Gaussian kernel, H being a parameter for controlling a kernel function action region and controlling smoothness, and H being a rotation matrix for aligning the direction of a derivative;
covariance matrix of the Gaussian kernel
By at position x p The derivatives of nearby diagonal directions are calculated, then:
wherein z is u And z v Is the derivative in the diagonal direction and,
denotes x p The number of pixel points in the neighborhood of the location,
is shown as
The number of pixels in the neighborhood;
-said gaussian up-sampling the positions of the R, B and IR points in the G-channel according to the adaptive spectral correlation kernel function, comprising:
performing Gaussian up-sampling on R, B and IR point positions in a G channel according to the self-adaptive spectrum correlation kernel function, and setting the point needing up-sampling as x p Calculation formula of upsampling
Comprises the following steps:
wherein the content of the first and second substances,
representing the point x that needs to be upsampled p The number of pixel points in the neighborhood of the location,
is x i The value after the point re-sampling,
is x i A matrix of a binary mask of points is formed,
is a weight normalization factor, is obtained by multiplying and summing mask points of the adaptive spectrum correlation kernel function,
is an adaptive spectral correlation kernel function;
and G pixel interpolation of the R, B and IR point positions in the G channel is realized after Gaussian up-sampling is carried out on the R, B and IR point positions in the G channel.
4. The method of claim 1, wherein interpolating R, B and IR channels in the RGBIR image respectively using the G channel as a guide to obtain a corresponding complete R channel image, a complete B channel image and an IR channel image comprises:
let X be: r, B or IR;
taking the G channel as a guide image, and performing pre-interpolation on the X channel to obtain an X channel pre-interpolation image
Selecting X sampling points in an X channel of an RGBIR image, and calculating a pre-interpolation image of the X channel
Obtaining an error image delta X with the error between the X sampling point image and the error image;
interpolating the error image delta X to obtain a complete error plane image delta X full ；
Using said error plane image DeltaX full For X channel pre-interpolation image
Point-to-point correction is carried out to obtain the final X channel interpolation image X final I.e. byCorresponding full R channel images, full B channel images, or IR channel images.
5. The autopilot RGBIR image resampling method of claim 4, characterized in that it further comprises any one or more of the following:
-the G channel is used as a guide image, and the X channel is pre-interpolated to obtain an X channel pre-interpolated image
The method comprises the following steps:
adopting a guide filtering method, taking the G channel as a guide image, taking the X channel as an image to be processed, and carrying out guide filtering to obtain an X channel pre-interpolation image
Selecting X sampling points in an X channel of an RGBIR image, calculating an error between an X channel pre-interpolation image and an X sampling point image, and obtaining an error image delta X, wherein the method comprises the following steps:
selecting a binary matrix for the sampling points by the mask, taking 1 at the X sampling point, taking 0 at other points, and multiplying the point elements by the points of the matrix;
-said interpolation of said error image resulting in a complete error plane image Δ X full The method comprises the following steps:
interpolating the error image delta X plane by adopting a BiCubic interpolation algorithm to obtain a complete error plane image delta X full ；
-using said error plane image ax full Pre-interpolating the image of the pair of X channels
Performing point-to-point correction to obtain final productX channel interpolation image X final The method comprises the following steps:
at the time of obtaining a complete error image plane DeltaX full Thereafter, the image is pre-interpolated for the X channel as follows
And (3) performing point-to-point correction:
obtaining the final X channel interpolation image X final 。
6. The autopilot RGBIR image resampling method of claim 1, wherein said downsampling said G-channel image, R-channel image and B-channel image, respectively, according to a given Bayer format to obtain corresponding Bayer images comprises:
in each channel image, 2x2 block regions are taken as units, in each 2x2 block region, according to a given Bayer format, channels corresponding to a corresponding sequence in the Bayer format are respectively reserved according to the sequence of a left upper angular point, a right upper angular point, a left lower angular point and a right lower angular point, and other channels are set to be zero;
and combining the non-zero points in the 4 points into an image of a single channel according to the positions of the non-zero points to obtain a Bayer image with a given format.
7. An autonomous driving RGBIR image resampling system, comprising:
the G channel interpolation module is used for interpolating a G channel in the RGBIR image to obtain a complete G channel image;
the R/B/IR channel interpolation module is used for respectively interpolating R, B and IR channels in the RGBIR image by taking the G channel as a guide after the G channel image is obtained so as to obtain a corresponding complete R channel image, a B channel image and an IR channel image;
the IR image reconstruction module is used for acquiring the IR channel image, and the IR channel image is an IR image with the same resolution as the input image;
and the Bayer image reconstruction module is used for respectively carrying out downsampling on the G channel image, the R channel image and the B channel image according to a given Bayer format to obtain the corresponding Bayer image.
8. A terminal comprising a memory, a processor and a computer program stored on the memory and executable on the processor, wherein the processor when executing the program is operable to perform the method of any one of claims 1 to 6 or to operate the system of claim 7.
9. A computer-readable storage medium, on which a computer program is stored which, when being executed by a processor, is adapted to carry out the method of any one of claims 1 to 6 or to carry out the system of claim 7.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202211336962.5A CN115689887A (en) | 2022-10-28 | 2022-10-28 | Automatic driving RGBIR image resampling method, system, terminal and medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202211336962.5A CN115689887A (en) | 2022-10-28 | 2022-10-28 | Automatic driving RGBIR image resampling method, system, terminal and medium |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN115689887A true CN115689887A (en) | 2023-02-03 |
Family
ID=85046721
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202211336962.5A Pending CN115689887A (en) | 2022-10-28 | 2022-10-28 | Automatic driving RGBIR image resampling method, system, terminal and medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN115689887A (en) |
Citations (10)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102778201A (en) * | 2011-05-13 | 2012-11-14 | 索尼公司 | Image processing device, image processing method, and program |
| CN103716606A (en) * | 2013-12-30 | 2014-04-09 | 上海富瀚微电子有限公司 | Bayer domain image downsampling method and device and camera equipment |
| US20170374299A1 (en) * | 2016-06-28 | 2017-12-28 | Intel Corporation | Color correction of rgbir sensor stream based on resolution recovery of rgb and ir channels |
| CN107967668A (en) * | 2016-10-20 | 2018-04-27 | 上海富瀚微电子股份有限公司 | A kind of image processing method and device |
| CN108377340A (en) * | 2018-05-10 | 2018-08-07 | 杭州雄迈集成电路技术有限公司 | One kind being based on RGB-IR sensor diurnal pattern automatic switching methods and device |
| CN110852953A (en) * | 2019-11-15 | 2020-02-28 | 展讯通信（上海）有限公司 | Image interpolation method and device, storage medium, image signal processor and terminal |
| CN111182242A (en) * | 2019-12-20 | 2020-05-19 | 翱捷智能科技(上海)有限公司 | RGB-IR image correction method and device |
| WO2020139493A1 (en) * | 2018-12-28 | 2020-07-02 | Qualcomm Incorporated | Systems and methods for converting non-bayer pattern color filter array image data |
| CN112233019A (en) * | 2020-10-14 | 2021-01-15 | 长沙行深智能科技有限公司 | ISP color interpolation method and device based on self-adaptive Gaussian kernel |
| US20220198604A1 (en) * | 2020-12-21 | 2022-06-23 | Texas Instruments Incorporated | Method and Apparatus for Processing RGB-Infrared (RGB-IR) Sensor Data |
-
2022
- 2022-10-28 CN CN202211336962.5A patent/CN115689887A/en active Pending
Patent Citations (10)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102778201A (en) * | 2011-05-13 | 2012-11-14 | 索尼公司 | Image processing device, image processing method, and program |
| CN103716606A (en) * | 2013-12-30 | 2014-04-09 | 上海富瀚微电子有限公司 | Bayer domain image downsampling method and device and camera equipment |
| US20170374299A1 (en) * | 2016-06-28 | 2017-12-28 | Intel Corporation | Color correction of rgbir sensor stream based on resolution recovery of rgb and ir channels |
| CN107967668A (en) * | 2016-10-20 | 2018-04-27 | 上海富瀚微电子股份有限公司 | A kind of image processing method and device |
| CN108377340A (en) * | 2018-05-10 | 2018-08-07 | 杭州雄迈集成电路技术有限公司 | One kind being based on RGB-IR sensor diurnal pattern automatic switching methods and device |
| WO2020139493A1 (en) * | 2018-12-28 | 2020-07-02 | Qualcomm Incorporated | Systems and methods for converting non-bayer pattern color filter array image data |
| CN110852953A (en) * | 2019-11-15 | 2020-02-28 | 展讯通信（上海）有限公司 | Image interpolation method and device, storage medium, image signal processor and terminal |
| CN111182242A (en) * | 2019-12-20 | 2020-05-19 | 翱捷智能科技(上海)有限公司 | RGB-IR image correction method and device |
| CN112233019A (en) * | 2020-10-14 | 2021-01-15 | 长沙行深智能科技有限公司 | ISP color interpolation method and device based on self-adaptive Gaussian kernel |
| US20220198604A1 (en) * | 2020-12-21 | 2022-06-23 | Texas Instruments Incorporated | Method and Apparatus for Processing RGB-Infrared (RGB-IR) Sensor Data |
Non-Patent Citations (2)
| Title |
|---|
| YUSUKE MONNO ET AL: "Multispectral demosaicking using guided filter", DIGITAL PHOTOGRAPHY VIII, pages 1 - 7 * |
| 汤漫 等: "基于快速残差插值和卷积神经网络的去马赛克算法", 南华大学学报(自然科学版), pages 68 - 76 * |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| EP4050558A1 (en) | Image fusion method and apparatus, storage medium, and electronic device | |
| CN106651938A (en) | Depth map enhancement method blending high-resolution color image | |
| CN107492066B (en) | Image processing apparatus and method for performing preprocessing to obtain an image with improved sharpness | |
| US20130301933A1 (en) | Method and device for generating a super-resolution version of a low resolution input data structure | |
| EP1751708B1 (en) | Super-resolution image processing | |
| US20080112612A1 (en) | Noise reduction of panchromatic and color image | |
| US20100079609A1 (en) | Apparatus and method of obtaining high resolution image | |
| US11720999B2 (en) | Method, device and non-transitory computer-readable storage medium for increasing the resolution and dynamic range of a sequence of respective top view images of a same terrestrial location | |
| US10410314B2 (en) | Systems and methods for crossfading image data | |
| WO2005122083A1 (en) | Image pickup apparatus, and method for enhancing resolution of images | |
| CN102053804B (en) | Image processing apparatus and control method | |
| CN111539893A (en) | Bayer image joint demosaicing denoising method based on guided filtering | |
| JP2008079026A (en) | Image processor, image processing method, and program | |
| CN110555805B (en) | Image processing method, device, equipment and storage medium | |
| CN113454687A (en) | Image processing method, apparatus and system, computer readable storage medium | |
| US20070253626A1 (en) | Resizing Raw Image Data Before Storing The Data | |
| CN115689887A (en) | Automatic driving RGBIR image resampling method, system, terminal and medium | |
| JP2002305751A (en) | Reconstruction of color filter array images | |
| US7995107B2 (en) | Enhancement of images | |
| CN115004220A (en) | Neural network for raw low-light image enhancement | |
| KR20230124699A (en) | Circuitry for Combined Downsampling and Correction of Image Data | |
| WO2017159137A1 (en) | Image processing device and control method and program therefor | |
| CN116416132A (en) | Image reconstruction method based on multi-view reference image, computer equipment and medium | |
| CN114312577B (en) | Vehicle chassis perspective method and device and electronic equipment | |
| CN115082352A (en) | Method and device for generating full-focus image |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20230203 |