WO2022135588A1 - 图像校正方法、装置及系统、电子设备 - Google Patents

图像校正方法、装置及系统、电子设备 Download PDF

Info

Publication number
WO2022135588A1
WO2022135588A1 PCT/CN2021/141355 CN2021141355W WO2022135588A1 WO 2022135588 A1 WO2022135588 A1 WO 2022135588A1 CN 2021141355 W CN2021141355 W CN 2021141355W WO 2022135588 A1 WO2022135588 A1 WO 2022135588A1
Authority
WO
WIPO (PCT)
Prior art keywords
image
correction
target
pair
resolution
Prior art date
Application number
PCT/CN2021/141355
Other languages
English (en)
French (fr)
Inventor
刘梦晗
郁理
凤维刚
王进
Original Assignee
虹软科技股份有限公司
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by 虹软科技股份有限公司 filed Critical 虹软科技股份有限公司
Priority to KR1020237021758A priority Critical patent/KR20230110618A/ko
Publication of WO2022135588A1 publication Critical patent/WO2022135588A1/zh

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/30—Determination of transform parameters for the alignment of images, i.e. image registration
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/50—Depth or shape recovery
    • G06T7/55—Depth or shape recovery from multiple images
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F18/00—Pattern recognition
    • G06F18/20—Analysing
    • G06F18/22—Matching criteria, e.g. proximity measures
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T5/00—Image enhancement or restoration
    • G06T5/80—Geometric correction
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/97—Determining parameters from multiple pictures
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/40—Extraction of image or video features
    • G06V10/44—Local feature extraction by analysis of parts of the pattern, e.g. by detecting edges, contours, loops, corners, strokes or intersections; Connectivity analysis, e.g. of connected components
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
    • G06V10/74—Image or video pattern matching; Proximity measures in feature spaces
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/20—Special algorithmic details
    • G06T2207/20016—Hierarchical, coarse-to-fine, multiscale or multiresolution image processing; Pyramid transform

Definitions

  • the present invention relates to the technical field of image processing, and in particular, to an image correction method, device and system, and electronic equipment.
  • depth imaging has become more and more accurate, and applications based on depth information have also developed rapidly.
  • common depth imaging methods are mainly divided into three types: binocular stereo imaging and structured light. Imaging and Time of Flight (ToF).
  • Structured light imaging is a laser that emits a specific pattern (speckle or dot matrix) through a camera.
  • the measured object reflects these patterns
  • the reflected patterns are captured by the camera, and the size of the speckle or point above is calculated.
  • the distance between the measured object and the camera is not affected by the texture of the object, but the laser speckle will be submerged under strong light, so it is not suitable for outdoor use.
  • ToF camera imaging by calculating the time difference between the transmitted signal and the reflected signal, the depth information of the measured point can be directly obtained.
  • the advantage is that it has high real-time performance and is not affected by lighting changes and object textures, but the image resolution is generally not high, the module is relatively large, and the hardware cost is relatively high.
  • the alignment accuracy between the depth image and the visible light image will be affected, and the subsequent algorithm effects that rely on depth information will be affected.
  • the present disclosure provides an image correction method, device and system, and electronic equipment, so as to at least solve the problem that dynamic correction between two different cameras cannot be realized in the related art, and the adaptability to the environment is low, resulting in poor image alignment effect, which easily affects the use of users. technical issues of interest.
  • an image correction method comprising: acquiring a visible light image and a depth image captured on a target object, and after transformation, a base image pair is formed, wherein the base image pair includes a first image and a second image image; perform correction processing on the above-mentioned basic image pair by using a preset correction mode to obtain a plurality of correction parameters; perform alignment and correction on the above-mentioned basic image pair based on each of the correction parameters to obtain a target image pair.
  • the steps of performing correction processing on the above-mentioned basic image pair using a preset correction mode to obtain a plurality of correction parameters include: scaling the above-mentioned basic image pair to a preset resolution, and performing pyramidal correction processing to obtain the above-mentioned basic image pair. Multiple calibration parameters.
  • the step of obtaining a visible light image and a depth image captured on the target object, and forming a base image pair after transformation includes: transforming the above-mentioned depth image into the image coordinate system of the above-mentioned visible light image based on preset calibration parameters, Adjust to obtain a preliminary aligned depth map with the same resolution as the above-mentioned visible light image, wherein the above-mentioned visible light image and the above-mentioned preliminary aligned depth map are combined to form the above-mentioned basic image pair, the above-mentioned first image is the above-mentioned visible light image, and the above-mentioned second image is all Describe the preliminary alignment depth map.
  • the step of performing correction processing on the above-mentioned basic image pair using a preset correction mode to obtain a plurality of correction parameters further includes: determining a target translation parameter and a target scaling coefficient between the above-mentioned first image and the above-mentioned second image; Based on the above-mentioned target translation parameters and the above-mentioned target scaling coefficients, a plurality of correction parameters are determined.
  • the above-mentioned image correction method further includes: preprocessing the preliminary aligned depth map in the above-mentioned basic image pair to obtain the above-mentioned image correction method.
  • a first image filtering the visible light image in the above-mentioned basic image pair to obtain the above-mentioned second image.
  • the step of determining the target translation parameter and the target zoom factor between the first image and the second image includes: calculating the target translation parameter of the first image relative to the second image, and based on the target The translation parameter translates the first image to obtain a third image; selects a plurality of scaling coefficients, and scales the third image with each of the scaling coefficients, and calculates an image matching score between the third image and the second image; The scaling coefficient corresponding to the smallest score among the plurality of above-mentioned image matching scores is used as the target scaling coefficient.
  • the step of determining the target translation parameter and the target zoom factor between the first image and the second image includes: calculating the target translation parameter of the first image relative to the second image, and based on the target The translation parameter translates the first image to obtain a fourth image; selects a plurality of scaling coefficients, and scales the fourth image with each of the scaling coefficients, and calculates an image matching score between the fourth image and the second image; The scaling coefficient is adjusted until the score change in the image matching score is less than the first threshold, and the scaling coefficient corresponding to the image matching score is used as the target scaling coefficient.
  • the step of determining a target translation parameter and a target scaling coefficient between the first image and the second image includes: selecting multiple scaling coefficients, and scaling the first image with each scaling coefficient; Based on the above-mentioned first image scaled by each of the above-mentioned scaling coefficients, slide on the above-mentioned second image, and calculate the image matching score between the above-mentioned second image; The corresponding zoom coefficient and translation amount are used as the target zoom coefficient and the target translation parameter.
  • the step of preprocessing the preliminary aligned depth map in the above-mentioned basic image pair to obtain the above-mentioned first image includes: mapping the depth value of each pixel in the preliminary aligned depth map in the above-mentioned basic image pair to and/or, adjusting the image contrast of the above-mentioned preliminary aligned depth map to obtain the above-mentioned first image.
  • the step of determining a target translation parameter and a target scaling coefficient between the first image and the second image includes: extracting image features of the first image to obtain a first feature subset, wherein the first The feature subset includes a first range image, a first boundary pattern and first mask information; the image features of the second image are extracted to obtain a second feature subset, wherein the second feature subset includes a second range image and a second boundary direction map; based on the first feature subset and the second feature subset, calculate the target translation parameter of the first image relative to the second image.
  • the step of extracting image features of the first image to obtain a first feature subset includes: extracting all boundary pixels of each target object in the first image to obtain a first edge image; The image is subjected to inverse color processing to obtain a second edge image; the contour is extracted from the first edge image to obtain a first contour array, and the pixel point direction corresponding to each pixel is calculated based on the first contour array to obtain a first contour direction array ; Based on the first preset distance threshold, perform preset distance transformation processing on the above-mentioned second edge image to obtain the above-mentioned first distance image; Based on the above-mentioned first contour direction array, calculate the corresponding boundary of each target object in the above-mentioned second edge image.
  • the first boundary pattern of the based on the first range image and the first boundary pattern, the first feature subset is determined.
  • the step of performing a preset distance transformation process on the second edge image based on the first preset distance threshold to obtain a first distance image includes: determining the first mask based on the first preset distance threshold information, wherein the first mask information is set to shield part of the edge information in the second image; the first mask information is added to the first feature subset.
  • the step of extracting image features of the second image to obtain a second feature subset includes: extracting all boundary pixels of each target object in the second image to obtain a third edge image; using the first mask
  • the film information performs a subtraction process on the contour in the above-mentioned third edge image; performs inverse color processing on the above-mentioned third edge image after the subtraction process to obtain a fourth edge image; Extracts the contour from the above-mentioned fourth edge image to obtain a second Contour array, calculating the pixel point direction corresponding to each pixel point based on the second contour array to obtain a second contour direction array; based on the second preset distance threshold, performing preset distance transformation processing on the fourth edge image to obtain the above the second distance image; based on the above-mentioned second contour direction array, calculate the above-mentioned second boundary pattern corresponding to the boundary of each target object in the above-mentioned fourth edge image; based on the above-mentioned second distance image and the above-mentione
  • the step of calculating the target translation parameter of the first image relative to the second image based on the first feature subset and the second feature subset includes: using a first judgment condition to extract the first distance The contour pixel points whose pixel distance is less than the first distance threshold in the image and the above-mentioned second distance image are obtained, and the first contour pixel point set participating in the image matching is obtained; using the second judgment condition, the above-mentioned first boundary pattern and the above-mentioned second boundary are extracted.
  • the contour pixel points whose pixel distance in the orientation map is less than the second distance threshold value are obtained, and the second contour pixel point set participating in the image matching is obtained; based on the first contour pixel point set and the second contour pixel point set, the first image and The chamfering distance score, directional map distance and image adjustment factor between the above-mentioned second images, wherein the above-mentioned image adjustment factor is set to adjust the above-mentioned chamfering distance score and the above-mentioned directional map distance proportion; slide the above-mentioned first image on the above-mentioned first image.
  • the above-mentioned chamfering distance score, the above-mentioned directional map distance and the above-mentioned image adjustment factor into the first preset formula to calculate the image sliding score; determine the target sliding position corresponding to the minimum score in all the image sliding scores; The above target sliding position determines the target translation parameter.
  • the step of scaling the above-mentioned base image pair to a preset resolution and performing pyramidal correction processing to obtain the above-mentioned multiple correction parameters includes: acquiring an alignment precision value of the terminal application, based on the above-mentioned alignment precision value and the above-mentioned alignment precision value.
  • the resolution of the base image pair, and multiple correction resolutions are determined, wherein the multiple correction resolutions include at least: a preset resolution, and the preset resolution is the smallest resolution among the multiple correction resolutions; Zoom to the above preset resolution, and perform pyramidal correction processing until the above alignment precision value is satisfied, to obtain the above plurality of correction parameters.
  • the above-mentioned image correction method further includes: determining the image alignment requirement accuracy of the terminal application under the target image resolution; step S1, judging the current alignment accuracy corresponding to the above-mentioned target image pair under the above-mentioned first image resolution Whether the image meets the above-mentioned image alignment requirement accuracy; Step S2, if it is determined that the current alignment accuracy image corresponding to the above-mentioned target image pair does not meet the above-mentioned image alignment requirement accuracy, adjust the image resolution to the second image resolution, wherein the above-mentioned second image resolution The resolution value of the image resolution is higher than the above-mentioned first image resolution; Step S3, performing correction processing on the above-mentioned basic image pair by using a preset correction mode to obtain a plurality of correction parameters; Step S4, performing a step based on each of the above-mentioned basic images The steps of aligning and correcting the above-mentioned basic image pair with the correction parameters to obtain the target image pair; repeating
  • the above-mentioned image correction method further includes: comparing the image resolution of the above-mentioned visible light image and the image resolution of the above-mentioned depth image, and obtaining a comparison result with the smallest resolution; and a correction process based on the image resolution obtained by the comparison result and the initial setting.
  • the maximum resolution is used to calculate the threshold for the number of corrections; during the alignment correction process, if the number of image corrections reaches the above threshold for the number of corrections, the correction process is stopped.
  • an image correction device comprising: an acquisition unit configured to acquire a visible light image and a depth image captured on a target object, and to form a basic image pair after transformation, wherein the above basic image pair It includes a first image and a second image; a first correction unit, configured to perform correction processing on the above-mentioned basic image pair using a preset correction mode to obtain a plurality of correction parameters; a second correction unit, set to be based on each of the above-mentioned correction parameter pairs The above-mentioned basic image pairs are aligned and corrected to obtain target image pairs.
  • the above-mentioned first correction unit includes: a first correction module configured to scale the above-mentioned base image pair to a preset resolution, and perform pyramidal correction processing to obtain the above-mentioned plurality of correction parameters.
  • the above-mentioned acquisition unit includes: a first transformation module, configured to transform the above-mentioned depth image into the image coordinate system of the above-mentioned visible light image based on preset calibration parameters, and after adjustment, obtain the same resolution as the above-mentioned visible light image.
  • a first transformation module configured to transform the above-mentioned depth image into the image coordinate system of the above-mentioned visible light image based on preset calibration parameters, and after adjustment, obtain the same resolution as the above-mentioned visible light image.
  • Preliminary alignment of the depth map wherein the visible light image and the preliminary aligned depth map are combined to form the basic image pair, the first image is the visible light image, and the second image is the preliminary aligned depth map.
  • the above-mentioned first correction unit further includes: a first determination module configured to determine target translation parameters and target zoom coefficients between the above-mentioned first image and the above-mentioned second image; a second determination module, configured to be based on the above-mentioned target The translation parameter and the above-mentioned target scaling factor determine a plurality of correction parameters.
  • the above-mentioned image correction apparatus further includes: a first processing unit, configured to perform correction processing on the above-mentioned basic image pair using a preset correction mode to obtain a plurality of correction parameters, to perform preliminary alignment depth in the above-mentioned basic image pair.
  • the image is preprocessed to obtain the above-mentioned first image;
  • the second processing unit is configured to perform filtering processing on the visible light image in the above-mentioned basic image pair to obtain the above-mentioned second image.
  • the above-mentioned first determination module includes: a first calculation module, configured to calculate the above-mentioned target translation parameter of the above-mentioned first image relative to the above-mentioned second image, and to translate the above-mentioned first image based on the above-mentioned target translation parameter, to obtain a third an image; a first scaling module, configured to select a plurality of scaling coefficients, and respectively scale the above-mentioned third image with each of the above-mentioned scaling coefficients, and calculate an image matching score between the above-mentioned third image and the above-mentioned second image; a second determining module , which is set to take the scaling coefficient corresponding to the smallest score among the multiple above-mentioned image matching scores as the target scaling coefficient.
  • a first calculation module configured to calculate the above-mentioned target translation parameter of the above-mentioned first image relative to the above-mentioned second image, and to translate the above-mentioned first image
  • the above-mentioned first determination module further includes: a second calculation module, configured to calculate the above-mentioned target translation parameter of the above-mentioned first image relative to the above-mentioned second image, and translate the above-mentioned first image based on the above-mentioned target translation parameter to obtain the first image.
  • a second scaling module configured to select a plurality of scaling coefficients, and respectively scale the fourth image with each of the scaling coefficients, to calculate the image matching score between the fourth image and the second image
  • the third determination The module is configured to adjust the scaling coefficient until the score change in the image matching score is less than the first threshold, then the scaling coefficient corresponding to the image matching score is used as the target scaling coefficient.
  • the above-mentioned first determining module further includes: a third scaling module, configured to select a plurality of scaling coefficients, and respectively scale the above-mentioned first image with each of the above-mentioned scaling coefficients;
  • the above-mentioned first images scaled by the above-mentioned scaling coefficients are slid on the above-mentioned second images, and the image matching scores between the above-mentioned second images and the above-mentioned second images are calculated;
  • the fourth determination module is set to match the scores of the plurality of above-mentioned images.
  • the zoom coefficient and translation amount corresponding to the smallest median score are used as the target zoom coefficient and the target translation parameter.
  • the above-mentioned first processing unit includes: a first mapping module, configured to map the depth value of each pixel point in the preliminary aligned depth map in the above-mentioned basic image pair to a preset pixel range; and/or, the first An adjustment module, configured to adjust the image contrast of the preliminary aligned depth map to obtain the first image.
  • a first mapping module configured to map the depth value of each pixel point in the preliminary aligned depth map in the above-mentioned basic image pair to a preset pixel range
  • the first An adjustment module configured to adjust the image contrast of the preliminary aligned depth map to obtain the first image.
  • the first determining module further includes: a first extracting module, configured to extract image features of the first image to obtain a first feature subset, wherein the first feature subset includes a first distance image, and the first feature subset includes a first range image. a boundary orientation map and first mask information; a second extraction module configured to extract image features of the second image to obtain a second feature subset, wherein the second feature subset includes a second distance image and a second feature subset. A boundary direction map; and a fourth calculation module, configured to calculate the target translation parameter of the first image relative to the second image based on the first feature subset and the second feature subset.
  • the above-mentioned first extraction module includes: a first extraction sub-module, configured to extract all boundary pixels of each target object in the above-mentioned first image, to obtain a first edge image; a first inversion sub-module, configured to The first edge image is subjected to inverse color processing to obtain a second edge image; the second extraction sub-module is configured to extract contours from the first edge image, obtain a first contour array, and calculate each pixel point based on the first contour array The corresponding pixel point direction is to obtain a first contour direction array; the first transformation sub-module is set to perform preset distance transformation processing on the above-mentioned second edge image based on the first preset distance threshold to obtain a first distance image; the first The calculation submodule is set to calculate the first boundary direction map corresponding to the boundary of each target object in the above-mentioned second edge image based on the above-mentioned first contour direction array; the first determination sub-module is set to be based on the above-ment
  • the above-mentioned first transformation sub-module includes: a second determination sub-module, configured to determine first mask information based on the above-mentioned first preset distance threshold, wherein the above-mentioned first mask information is set to shield the above-mentioned second Part of the edge information in the image; an adding submodule, configured to add the above-mentioned first mask information to the above-mentioned first feature subset.
  • the above-mentioned second extraction module includes: a second extraction sub-module, configured to extract all boundary pixels of each target object in the above-mentioned second image, to obtain a third edge image; a subtraction sub-module, configured to use the above-mentioned first image.
  • a mask information is used to perform subtraction processing on the outline in the above-mentioned third edge image; the second inversion sub-module is set to perform inverse color processing on the above-mentioned third edge image after the subtraction processing, so as to obtain a fourth edge image;
  • the second calculation sub-module is set to extract the contour of the fourth edge image to obtain a second contour array, and calculate the pixel point direction corresponding to each pixel point based on the second contour array to obtain a second contour direction array;
  • the second transform sub-module The module is set to perform preset distance transformation processing on the above-mentioned fourth edge image based on the second preset distance threshold to obtain a second distance image;
  • the third calculation sub-module is set to calculate the above-mentioned first contour direction array based on the above-mentioned second contour direction array.
  • the second boundary pattern corresponding to the boundary of each target object in the four-edge image;
  • the third determination submodule is configured to obtain a second feature subset
  • the above-mentioned fourth calculation module includes: a third extraction sub-module, configured to use the first judgment condition to extract the contour pixels whose pixel distances in the above-mentioned first distance image and the above-mentioned second distance image are less than the first distance threshold, Obtain the first set of contour pixel points participating in the image matching; the fourth extraction sub-module is set to adopt the second judgment condition to extract the contour whose pixel distance between the above-mentioned first boundary pattern and the above-mentioned second boundary pattern is less than the second distance threshold pixel points to obtain the second outline pixel point set participating in the image matching; the fifth determination sub-module is set to determine the above-mentioned first image and the above-mentioned second image based on the above-mentioned first outline pixel point set and the above-mentioned second outline pixel point set The chamfering distance score, the directional map distance and the image adjustment factor between, wherein, the above-mentioned image adjustment factor is set to
  • the sixth determination sub-module is set to determine the sliding score of all images The target sliding position corresponding to the smallest median score; the seventh determination sub-module is set to determine the target translation parameter based on the above-mentioned target sliding position.
  • the above-mentioned first correction module includes: a first acquisition sub-module, configured to acquire the alignment precision value of the terminal application, and determine a plurality of correction resolutions based on the above-mentioned alignment precision value and the resolution of the above-mentioned basic image pair, wherein,
  • the multiple correction resolutions at least include: a preset resolution, where the preset resolution is the smallest resolution among the multiple correction resolutions; a first correction sub-module configured to scale the base image pair to the preset resolution , and perform pyramidal correction processing until the above-mentioned alignment precision value is satisfied, and the above-mentioned multiple correction parameters are obtained.
  • the above-mentioned image correction device further includes: a determination unit, configured to determine the image alignment requirement accuracy of the terminal application under the target image resolution; a first determination unit, configured to execute step S1, to determine the first image resolution at the above-mentioned first image resolution.
  • the first adjustment unit is set to perform step S2, if it is determined that the current alignment accuracy image corresponding to the above-mentioned target image pair does not reach the above-mentioned image alignment accuracy
  • the image alignment requires accuracy, and the image resolution is adjusted to be the second image resolution, wherein the resolution value of the second image resolution is higher than the first image resolution
  • the first execution unit is configured to execute step S3, execute the use of The preset correction mode performs correction processing on the above-mentioned basic image pairs, and obtains a plurality of correction parameters
  • the second execution unit is set to perform step S4, and performs alignment and correction on the above-mentioned basic image pairs based on each of the above-mentioned correction parameters, and obtains the target
  • the step of image pairing Steps S1 to S4 are repeatedly performed until the current alignment accuracy image reaches the above-mentioned image alignment requirement accuracy, and the process ends
  • the above-mentioned image correction device further includes: a comparison unit, configured to compare the image resolution of the visible light image and the image resolution of the above-mentioned depth image, and obtain a comparison result with the smallest resolution; a calculation unit, configured to obtain based on the comparison result.
  • the image resolution and the initially set maximum resolution of the correction processing calculate the correction times threshold; the stopping unit is set to stop the correction processing if the image correction times reaches the above correction times threshold during the alignment correction process.
  • an image correction system comprising: a first image capture device configured to capture a visible light image of a target object; a second image capture device configured to capture a depth image of the target object; correction The device is configured to obtain a visible light image and a depth image captured on the target object, and form a basic image pair after the change, wherein the basic image pair includes a first image and a second image; the basic image pair is performed using a preset correction mode.
  • the correction process is performed to obtain a plurality of correction parameters; the above-mentioned basic image pair is aligned and corrected based on each of the above-mentioned correction parameters to obtain a target image pair; the result output device is configured to output the aligned target image pair to a preset terminal display interface.
  • an electronic device comprising: a processor; and a memory configured to store executable instructions of the processor; wherein the processor is configured to execute by executing the executable instructions Any one of the image correction methods described above.
  • a computer-readable storage medium is also provided, where the computer-readable storage medium includes a stored computer program, wherein when the computer program is executed, the device where the computer-readable storage medium is located is controlled to execute the above-mentioned Any one of the above image correction methods.
  • a visible light image and a depth image captured on a target object are acquired, and transformed to form a base image pair, wherein the base image pair includes a first image and a second image, and a preset correction mode is used to correct the base image pair
  • a plurality of correction parameters are obtained, and the above-mentioned base image pair is aligned and corrected based on each correction parameter to obtain a target image pair.
  • an alignment operation can be performed on images captured by a variety of cameras to achieve dynamic calibration, the calibration environment is simple, and the alignment and calibration can be completed by using the images actually captured by the device, thereby solving the problem that the related art cannot realize the alignment between two different cameras. Dynamic correction, low adaptability to the environment, resulting in poor alignment of images and technical problems that easily affect the user's interest in use.
  • FIG. 1 is a flowchart of an optional image correction method according to an embodiment of the present invention.
  • FIG. 2 is a schematic diagram of an optional preliminary aligned depth map according to an embodiment of the present invention.
  • FIG. 3 is an optional superimposed image of the aligned depth image and the visible light image according to an embodiment of the present invention
  • FIG. 4 is a schematic diagram of an optional image correction apparatus according to an embodiment of the present invention.
  • RGB Red Green Blue
  • this article also refers to the usual color image
  • RGB-D Red Green Blue-Depth, color-depth map
  • VGA resolution 640*480 resolution.
  • the application scenarios of the following embodiments of the present invention include but are not limited to: 3D reconstruction, unmanned driving, face recognition, object measurement, 3D modeling, background blur, fusion of visible light image and infrared light image, visible light image and depth image Fusion, VR glasses, vehicle imaging equipment, etc.
  • the alignment parameters are calculated according to the images obtained by a variety of cameras. image, or an image capture device with a depth camera that can provide visible light images, infrared light images and depth images, and there is no requirement for the type of depth camera, which can be ToF cameras, infrared cameras, structured light cameras and/or binocular depth cameras.
  • the present invention has a simple correction environment, does not require a specific environment or a specific shooting pattern, and only needs a visible light image and a depth image that are preliminarily aligned according to preset calibration parameters to achieve dynamic correction.
  • the present invention only needs to perform dynamic alignment correction at regular intervals to achieve image alignment; in addition, for correction processing that requires high alignment accuracy but low real-time requirements, the present invention can choose to perform image alignment at high resolution. It can be used for 3D modeling, background blur, fusion of visible light image and infrared light image, etc.
  • the present invention can also be used for detection and matching of some objects, and common applications include gesture detection and pedestrian detection.
  • the present invention can be used to periodically automatically calibrate or after-sell the terminal equipment equipped with the depth camera; for VR glasses and vehicle imaging equipment, the difference between the visible light image and the depth image caused by vibration is Alignment errors can also be corrected using the present invention.
  • the present invention will be described below with reference to various embodiments.
  • an embodiment of an image correction method is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings may be executed in a computer system such as a set of computer-executable instructions, and, although in A logical order is shown in the flowcharts, but in some cases steps shown or described may be performed in an order different from that herein.
  • An embodiment of the present invention provides an image correction method, which can be applied to an image correction system.
  • the image correction system includes: a first image capture device capable of capturing a depth image/infrared image (in this embodiment of the present invention, a depth camera Schematic illustration) and a second image capturing device capable of capturing a visible light image (the embodiment of the present invention is schematically illustrated with a visible light camera).
  • OIS mechanism, AF mechanism, frame asynchrony or frame rate difference between equipment drop and image capture device will change the internal and external parameters of the camera.
  • the embodiment of the present invention proposes a high-precision alignment , An image correction method with fast processing speed, real-time, and applicable to most practical use scenarios.
  • the embodiments of the present invention can improve the practicability of image alignment, and perform alignment operations on images captured by various cameras.
  • the calibration parameters can be calculated from the visible light image and the infrared image, and the calibration parameters can also be calculated from the visible light image and the depth image. It is suitable for various devices equipped with ToF depth cameras or structured light depth cameras. Usually, the images collected are different from visible light images in texture. The common key point matching scheme is not feasible, but the technical scheme provided by the embodiment of the present invention can still obtain a relatively accurate alignment effect.
  • FIG. 1 is a flowchart of an optional image correction method according to an embodiment of the present invention. As shown in FIG. 1 , the method includes the following steps:
  • Step S102 obtaining a visible light image and a depth image captured on the target object, and after transformation, a base image pair is formed, wherein the base image pair includes a first image and a second image;
  • Step S104 using a preset correction mode to perform correction processing on the base image pair to obtain a plurality of correction parameters
  • Step S106 performing alignment correction on the above-mentioned base image pair based on each correction parameter to obtain a target image pair.
  • the visible light image and the depth image captured on the target object can be obtained, and after transformation, a basic image pair can be formed, wherein the basic image pair includes a first image and a second image, and a preset correction mode is used to perform the calibration on the basic image pair.
  • a plurality of correction parameters are obtained, and the above-mentioned basic image pair is aligned and corrected based on each correction parameter to obtain a target image pair.
  • an alignment operation can be performed on images captured by a variety of cameras to achieve dynamic calibration, the calibration environment is simple, and the alignment and calibration can be completed by using the images actually captured by the device, thereby solving the problem that the related art cannot realize the alignment between two different cameras. Dynamic correction, low adaptability to the environment, resulting in poor alignment of images and technical problems that easily affect the user's interest in use.
  • Step S102 acquiring a visible light image and a depth image captured on the target object, and after transformation, a base image pair is formed, wherein the base image pair includes a first image and a second image.
  • the image capturing device used in the embodiment of the present invention may include: a depth camera and a visible light camera, the depth camera can obtain a depth image and a corresponding infrared image, and the visible light camera can obtain a visible light image.
  • the depth camera does not need to obtain a depth image and an infrared image at the same time, and the embodiment of the present invention is applicable not only to a device that can provide an infrared image and a depth image, but also to a device that can only provide a depth image or an infrared image.
  • obtaining a visible light image and a depth image captured on a target object, and forming a base image pair after transformation includes: transforming the depth image into the image coordinate system of the visible light image based on preset calibration parameters , after adjustment to obtain a preliminary aligned depth map with the same resolution as the visible light image, wherein the visible light image and the preliminary aligned depth map are combined to form a base image pair, the first image is the visible light image, and the second image is the preliminary aligned depth map.
  • the above-mentioned preset calibration parameters are parameters determined when the depth camera and the visible light camera are initially calibrated, such as the results of factory calibration parameters.
  • the resolution of the depth map is smaller than that of the visible light image.
  • the depth map is convenient for subsequent image alignment and correction processing.
  • the image correction method before using a preset correction mode to perform correction processing on the base image pair to obtain a plurality of correction parameters, the image correction method further includes: preprocessing the preliminary aligned depth map in the base image pair to obtain the first image; A second image is obtained by filtering the visible light image in the basic image pair.
  • the step of preprocessing the preliminary aligned depth map in the base image pair to obtain the first image includes: mapping the depth value of each pixel in the preliminary aligned depth map in the base image pair to a preset pixel and/or, adjusting the image contrast of the preliminary aligned depth map to obtain a first image.
  • a visible light image and a depth image are obtained from the visible light camera and the depth camera respectively, and the depth image is transformed into the visible light image coordinate system according to the preset calibration parameters to obtain a preliminary alignment with the same resolution as the visible light image.
  • depth map Preprocess the preliminary aligned depth map, map the depth value of each pixel of the preliminary aligned depth map to the preset pixel range [0, 255], and adjust the image contrast to restore the lost details for the existing overexposure problem. Strengthen the texture information weakened by overexposure to more effectively perform subsequent alignment and correction processing, and record the pre-processed preliminary aligned depth map as the first image or template image.
  • Another visible light image is filtered, and the filtering can help remove high-frequency noise signals that appear due to sampling when the resolution of the visible light image changes, and the obtained image is recorded as a second image or a matching image.
  • the correction parameter may be calculated according to the visible light image and the depth map, and the correction parameter may also be calculated according to the visible light image and the infrared image.
  • the implementation process is described by taking the visible light image and the depth image to calculate the correction parameter as an example, and this implementation process is also applicable to the visible light image and the infrared image to calculate the correction parameter.
  • FIG. 2 is a schematic diagram of an optional preliminary aligned depth map according to an embodiment of the present invention. As shown in FIG. 2 , the preliminary aligned depth map includes outline and edge information of main objects that have been extracted.
  • FIG. 3 is an optional superimposed image of the aligned depth image and the visible light image according to an embodiment of the present invention. As shown in FIG. 3 , after the visible light image is added, the visible light image needs to be deviated from the object in the preliminary aligned depth image. The shifted parts are aligned, and the further aligned images in Figure 3 have very little offset difference, and can basically achieve pixel-by-pixel alignment.
  • the image correction scheme or the dynamic correction scheme involved in the embodiments of the present invention is applicable to a device that has been calibrated, that is, the initial camera internal parameters and external parameters are known. Due to factors such as OIS mechanism, AF mechanism, and equipment drop, the camera internal parameters and inter-camera external parameters may be affected. Therefore, there may be alignment errors between the depth map and the visible light map aligned with the known calibration parameters, mainly manifested in the presence of translation and For the zoom problem, this embodiment of the present invention corrects the translation and zoom problem according to the input visible light map and depth map.
  • step S104 a preset correction mode is used to perform correction processing on the base image pair to obtain a plurality of correction parameters.
  • a preset correction mode is used to perform correction processing on the base image pair to obtain a plurality of correction parameters, including: determining target translation parameters and target scaling coefficients between the first image and the second image; The target translation parameter and the target zoom factor determine multiple correction parameters.
  • the step of determining the target translation parameter and the target scaling coefficient between the first image and the second image includes: extracting image features of the first image to obtain a first feature subset, wherein the first feature subset includes: the first range image, the first boundary pattern and the first mask information; the image features of the second image are extracted to obtain a second feature subset, wherein the second feature subset includes the second range image and the second boundary pattern ; Calculate the target translation parameter of the first image relative to the second image based on the first feature subset and the second feature subset.
  • Feature extraction is performed on the preprocessed first image and the second image respectively below.
  • the embodiment of the present invention extracts image features of the first image to obtain a first feature subset.
  • the step of extracting image features of the first image to obtain the first feature subset includes: extracting all boundary pixels of each target object in the first image to obtain a first edge image; inverting the first edge image. color processing to obtain a second edge image; extract the contour of the first edge image to obtain a first contour array, calculate the pixel point direction corresponding to each pixel point based on the first contour array, and obtain a first contour direction array; Set a distance threshold, perform preset distance transformation processing on the second edge image, and obtain a first distance image; based on the first contour direction array, calculate the first boundary direction map corresponding to the boundary of each target object in the second edge image; A range image and a first boundary pattern to determine a first subset of features.
  • extracting all the boundary pixels of each target object in the first image to obtain the first edge image refers to extracting the main edge in the first image
  • the edge refers to the edge of at least one object in the image.
  • the present invention does not specifically limit the algorithm for extracting edges, for example, edge detection methods based on gradient difference value detection, depth difference value detection and deep learning-based edge detection.
  • a first contour array By extracting contours from the first edge image, a first contour array is obtained.
  • the first contour array records the contour division information and the positional relationship of the pixels included in each contour in a multi-dimensional array, so the pixel point direction corresponding to each pixel point can be calculated based on the first contour array to obtain the first contour direction array.
  • calculating the first boundary direction map corresponding to the boundary of each target object in the second edge image may refer to: the first contour direction array records the contour division information and the direction values of the pixels included in each contour, The information contained in the first contour direction array is mapped to the edge image of the second image, and the directions corresponding to the boundaries of each target object in the second edge image are calculated and saved as the first boundary direction map.
  • the step of performing a preset distance transformation process on the second edge image based on the first preset distance threshold to obtain the first distance image includes: determining a first mask based on the first preset distance threshold information, wherein the first mask information is set to mask part of the edge information in the second image; the first mask information is added to the first feature subset.
  • the second edge image is subjected to Euclidean distance distance transform (distance transform) to obtain the first distance image.
  • distance transform distance transform
  • a mask for subsequently removing redundant edge information in the second image/query image can also be generated.
  • the mask information is set to perform region screening and masking on the second image or the matching image, so that it does not participate in processing.
  • extracting the first image/template image features includes:
  • Step 1 Extract the main edge of the first image, and record the acquired edge image as C1, where the gray value of the edge pixel position is 255, and the gray value of the non-edge pixel position is 0;
  • Step 2 Invert the edge image C1 to obtain the edge image E1, where the gray value of the edge pixel position is 0, and the gray value of the non-edge pixel position is 255;
  • Step 3 Extract the contour in C1 and record it as V1;
  • Step 4 Based on the preset distance threshold, the edge image E1 is subjected to Euclidean distance distance transform (distance transform) to obtain the distance image DT1 and the mask for subsequent removal of redundant edge information in the second image/query image;
  • distance transform distance transform
  • Step5 Calculate the orientation map OM1 of the edge in the edge image E1. First calculate the direction of the pixels on the contour V1, denoted as O1; calculate the direction of the contour based on the mapping of O1 to the edge image E1, and save it as the direction map OM1.
  • the embodiment of the present invention extracts image features of the second image to obtain a second feature subset.
  • the step of extracting image features of the second image to obtain the second feature subset includes: extracting all boundary pixels of each target object in the second image to obtain a third edge image; using the first mask information to The contour in the third edge image is subtracted; the third edge image after the subtraction is processed by inverse color processing to obtain the fourth edge image; the contour is extracted from the fourth edge image to obtain the second contour array, which is based on the second
  • the contour array calculates the pixel point direction corresponding to each pixel point to obtain a second contour direction array; based on the second preset distance threshold, the fourth edge image is subjected to preset distance transformation processing to obtain a second distance image; based on the second contour
  • the direction array is used to calculate the second boundary direction map corresponding to the boundary of each target object in the fourth edge image; based on the second distance image and the second boundary direction map, a second feature subset is obtained.
  • Extract the main edges of the objects in the second image to obtain the third edge image then use the mask information to delete the edges in the third edge image, and then invert the processed edge image to obtain the fourth edge image Or primary edge images.
  • the mask information obtained by the distance transformation of the simple contour image to process the second image with complex contour the redundant contour can be removed and the calculation amount of image processing can be reduced.
  • extract the second image/match image features including:
  • Step1 Extract the main edge of the image, and record the acquired edge image as C2;
  • Step2 Use the mask information obtained by the above calculation to delete the edge in C2, and then invert the processed edge image to obtain the main edge image E2 of the image, in which the gray value of the edge pixel position is 0, and the non-edge pixel position The gray value is 255;
  • Step3 Extract the contour in the image C2 and record it as V2;
  • Step4 Perform the distance transform of the Euclidean distance on the edge image E2 to obtain the distance image DT2; set the distance threshold to generate the DT2 that actually participates in the calculation.
  • Step5 Calculate the orientation map OM2 of the edge in the edge image E2. First calculate the direction of the pixel points on the contour V2, denoted as O2; calculate the direction of the contour based on the mapping of O2 to the edge image E2, and save it as the direction map OM2.
  • the image target translation parameter and the target zoom factor can be calculated.
  • the step of calculating the target translation parameter of the first image relative to the second image based on the first feature subset and the second feature subset includes: using a first judgment condition to extract the first distance image and the second distance.
  • the contour pixel points whose pixel distance is less than the first distance threshold value are obtained, and the first contour pixel point set participating in the image matching is obtained; the second judgment condition is used to extract the pixel distance between the first boundary pattern and the second boundary pattern that is smaller than the second boundary pattern.
  • the contour pixels of the distance threshold are obtained to obtain the second contour pixel set participating in the image matching; based on the first contour pixel set and the second contour pixel set, determine the chamfering distance score between the first image and the second image, Direction map distance and image adjustment factor, where the image adjustment factor is set to adjust the chamfering distance score and the proportion of the direction map distance; slide the second image on the first image, and adjust the chamfer distance score, the direction map distance and the image adjustment factor Input into the first preset formula, calculate the image sliding score; determine the target sliding position corresponding to the smallest score among all the image sliding scores; determine the target translation parameter based on the target sliding position.
  • target translation parameters include:
  • Step1 Extract the position of contour pixels participating in image matching.
  • the judgment conditions are as follows:
  • th1 and th2 are two preset distance thresholds respectively,
  • is the pixel distance of a pixel in the first distance image and the second distance image,
  • is the pixel distance between the first boundary pattern and the second boundary pattern of a certain pixel.
  • Step2 Calculate the chamfering distance score. By sliding the first image on the second image, an image sliding score can be obtained by calculating each sliding position.
  • the first preset formula for the calculation is as follows:
  • score EDT+s*ODT; wherein, score is the image sliding score of a certain sliding position, EDT is the chamfering distance of the first distance image and the second distance image; ODT is the first boundary direction map and the second boundary direction map The distance between, the directional map distance; s is an image adjustment factor that adjusts the weight of EDT and ODT.
  • Step3 Select the translation value in x and y direction. Determine the target sliding position corresponding to the minimum score among all the image sliding scores, and the translation amount corresponding to this position is the target translation parameters required for alignment, that is, the translation values in the x and y directions: dx and dy.
  • the sequence of performing scaling and translation is not limited.
  • the translation correction amount and the zoom correction amount may be calculated first, and then the alignment may be realized based on the correction amount, or the translation correction may be calculated first. and complete the translation correction, and then realize the zoom correction on this basis.
  • the step of determining the target translation parameter and the target zoom factor between the first image and the second image includes: calculating the target translation parameter of the first image relative to the second image, and translating the first image based on the target translation parameter , obtain the third image; select multiple scaling coefficients, and scale the third image with each scaling coefficient respectively, and calculate the image matching score between the third image and the second image; The corresponding scaling factor is used as the target scaling factor.
  • a third image is obtained; then scaling the third image, adjusting the scaling factor, and calculating the image between the third image and the second image under the scaling factor
  • the matching score by selecting the scaling coefficient corresponding to the minimum image matching score score as the target scaling coefficient, and using the target scaling coefficient to scale and process the third image, the correction processing between the two images is realized.
  • the above-mentioned target scaling factor includes, but is not limited to, a scaling width factor, a scaling length factor, a scaling factor, and the like.
  • the step of determining the target translation parameter and the target zoom coefficient between the first image and the second image includes: calculating the target translation parameter of the first image relative to the second image, and translating the first image based on the target translation parameter , obtain the fourth image; select multiple scaling coefficients, and scale the fourth image with each scaling coefficient respectively, and calculate the image matching score between the fourth image and the second image; based on the image matching score, adjust the scaling coefficient until the images match If the score change in the score is less than the first threshold, the scaling factor corresponding to the image matching score is used as the target scaling factor.
  • a fourth image is obtained; then, the fourth image is zoomed, and the zoom factor is adjusted until the image matching score between the fourth image and the second image under the zoom factor is reached If the change in the middle score is less than the first threshold, the scaling factor corresponding to the image matching score is taken as the target scaling factor.
  • the first image is scaled and processed by using the target scaling factor, so as to realize the correction processing between the two images.
  • the step of determining the target translation parameter and the target scaling factor between the first image and the second image includes: selecting multiple scaling coefficients, and scaling the first image with each scaling factor; The coefficient-scaled first image is slid on the second image, and the image matching score between it and the second image is calculated; the zoom coefficient and translation amount corresponding to the minimum score among the multiple image matching scores are used as the target Zoom factor and target translation parameters.
  • the first image scaled based on each zoom factor can be slid on the second image to calculate the image matching score between it and the second image, and the image matching score in the image matching score is the smallest.
  • the corresponding zoom factor and translation amount are used as the target zoom factor and the target translation parameter to realize the correction processing between the two images.
  • Step S106 performing alignment correction on the above-mentioned base image pair based on each correction parameter to obtain a target image pair.
  • the above-mentioned basic image pair is aligned and corrected to obtain a target image pair that meets the alignment requirements.
  • a preset correction mode is used to perform correction processing on the base image pair, and when multiple correction parameters are obtained, the base image pair can be scaled to a preset resolution, and pyramidal correction processing is performed to obtain multiple correction parameters. Correction parameters.
  • the chamfering distance matching is performed on the basis of image edge feature extraction, it still needs to traverse the edge pixel positions, resulting in a huge amount of computation. If the image to be aligned has high resolution and complex edge information, it will further affect the real-time performance of image alignment. Therefore, when performing alignment correction for high-resolution basic image pairs, it is necessary to use multi-resolution dynamic correction methods from coarse to Align finely.
  • this embodiment of the present invention in the process of correcting an image, first downsampling to a low resolution is adopted, and then upsampling layer by layer for dynamic correction.
  • This method is a pyramidal algorithm, which can reduce calculation time and run at the lowest resolution. The time is the smallest, the preliminary rough results are found, and the fine-tuning calculation is performed according to the results at the lowest resolution, without all recalculation. If the accuracy requirements are low, the alignment accuracy requirements can be met by performing the correction processing at the low resolution; if the accuracy requirements are high, the calibration processing at the low resolution cannot meet the accuracy requirements, then upsampling to the high resolution for correction until it meets the requirements. Alignment accuracy requirements.
  • the image correction method further includes: determining the image alignment requirement accuracy of the terminal application under the target image resolution Step S1, judge under the first image resolution, whether the current alignment accuracy image corresponding to the target image reaches the image alignment requirement accuracy; Step S2, if it is determined that the current alignment accuracy image corresponding to the target image pair does not reach the image Align the required accuracy, and adjust the image resolution to be the second image resolution, where the resolution value of the second image resolution is higher than the first image resolution; step S3, performing correction processing on the base image pair using a preset correction mode , the steps of obtaining a plurality of correction parameters; Step S4, performing alignment and correction on the above-mentioned basic image pair based on each correction parameter to obtain the target image pair; Repeating steps S1 to S4 until the current alignment accuracy image reaches the image alignment Ends when precision is required.
  • a visible light image and a depth map image are obtained from the visible light camera and the depth camera respectively, the depth map is transformed into the visible light image coordinate system according to the preset calibration parameters, and adjusted to obtain a preliminary alignment with the same resolution as the visible light image. depth map.
  • Scale the initially aligned image pair P1 to a low resolution p for dynamic correction obtain the correction parameters (dx_1, dy_1, scale_1) and correct the input image pair to obtain a new image pair, denoted as the aligned image pair P2;
  • the alignment image pair obtained by the correction is dynamically corrected at the resolution p*s2 (s is an amplification factor, generally 2), and the input image pair P2 is corrected after obtaining the correction parameters (dx_2, dy_2, scale_2), and a new image is obtained.
  • the image pair is recorded as the alignment image pair P3; continue to increase the resolution used in the correction process, and repeat the dynamic correction process until the alignment accuracy required by the application is met.
  • the accuracy of the image correction process is improved.
  • the error can be corrected from 30 pixels to within 4 pixels under VGA resolution in various scenarios, and the alignment accuracy is very high .
  • the image correction method further includes: comparing the image resolution of the visible light image and the image resolution of the depth image, and obtaining a comparison result with the smallest resolution; and correcting and processing the maximum resolution based on the image resolution obtained by the comparison result and the initial setting. , and calculate the correction times threshold; during the alignment correction process, if the image correction times reaches the correction times threshold, the correction process is stopped.
  • the resolution of the initial alignment and the number of required dynamic corrections can be determined according to the resolution of the input image. For example, the resolution of the smaller image between the two images (depth image and visible light image) is Tw*Th, and usually the visible light image resolution is much larger than the depth map resolution.
  • Tw*Th the resolution of the smaller image between the two images
  • the visible light image resolution is much larger than the depth map resolution.
  • the correction parameters can be calculated according to the visible light image and the depth map (or the visible light image and the infrared image), which is not only suitable for only providing the visible light image and the infrared image, or only providing the visible light image and the depth image, or a device with a depth camera that can provide three types of visible light map, infrared light map and depth map; it can also be applied to the situation where the contents of the images captured by the two cameras are similar but the textures are quite different, and those that cannot be matched based on feature points.
  • the embodiment of the present invention can also improve the alignment error of images captured by binocular devices caused by problems such as OIS, drop, frame asynchrony, and different frame rates, etc., and the correction environment is very simple, no specific environment is required, With a specific shooting pattern, the image correction process can be quickly completed and an image that is satisfactory to the user can be obtained.
  • An embodiment of the present invention provides an image correction apparatus, and the plurality of implementation units included in the apparatus correspond to the implementation steps in the foregoing first embodiment.
  • FIG. 4 is a schematic diagram of an optional image correction apparatus according to an embodiment of the present invention.
  • the image correction apparatus may include: an acquisition unit 41 , a first correction unit 43 , and a second correction unit 45 , wherein ,
  • the acquisition unit 41 is configured to acquire the visible light image and the depth image captured by the target object, and after transformation, form a basic image pair, wherein the basic image pair includes a first image and a second image;
  • the first correction unit 43 is configured to perform correction processing on the base image pair using a preset correction mode to obtain a plurality of correction parameters
  • the second correction unit 45 is configured to perform alignment correction on the above-mentioned base image pair based on each correction parameter to obtain a target image pair.
  • the above-mentioned image correction device can acquire the visible light image and the depth image of the target object through the acquisition unit 41, and after transformation, form a basic image pair, wherein the basic image pair includes a first image and a second image, and the first correction unit 45
  • the base image pair is corrected by using the preset correction mode to obtain a plurality of correction parameters
  • the second correction unit 47 performs alignment and correction on the base image pair based on each correction parameter to obtain the target image pair.
  • an alignment operation can be performed on images captured by a variety of cameras to achieve dynamic calibration, the calibration environment is simple, and the alignment and calibration can be completed by using the images actually captured by the device, thereby solving the problem that the related art cannot realize the alignment between two different cameras. Dynamic correction, low adaptability to the environment, resulting in poor alignment of images and technical problems that easily affect the user's interest in use.
  • the first correction unit includes: a first correction module, configured to scale the base image pair to a preset resolution, and perform pyramidal correction processing to obtain multiple correction parameters.
  • a first correction module configured to scale the base image pair to a preset resolution, and perform pyramidal correction processing to obtain multiple correction parameters.
  • the acquisition unit includes: a first transformation module, configured to transform the depth image into the image coordinate system of the visible light image based on preset calibration parameters, and after adjustment, obtain a preliminary aligned depth map with the same resolution as the visible light image. , wherein the visible light image and the preliminary aligned depth map are combined to form a base image pair, the first image is the visible light image, and the second image is the preliminary aligned depth map.
  • a first transformation module configured to transform the depth image into the image coordinate system of the visible light image based on preset calibration parameters, and after adjustment, obtain a preliminary aligned depth map with the same resolution as the visible light image.
  • the first correction unit further includes: a first determination module, configured to determine the target translation parameter and the target zoom coefficient between the first image and the second image; a second determination module, configured to be based on the target translation parameter and the target. Scaling factor, which determines multiple correction parameters.
  • the image correction device further includes: a first processing unit, configured to perform correction processing on the base image pair by using a preset correction mode to obtain a plurality of correction parameters, preliminarily pre-align the depth map in the base image pair. processing to obtain a first image; and a second processing unit configured to perform filtering processing on the visible light image in the basic image pair to obtain a second image.
  • a first processing unit configured to perform correction processing on the base image pair by using a preset correction mode to obtain a plurality of correction parameters, preliminarily pre-align the depth map in the base image pair. processing to obtain a first image
  • a second processing unit configured to perform filtering processing on the visible light image in the basic image pair to obtain a second image.
  • the first determination module includes: a first calculation module configured to calculate a target translation parameter of the first image relative to the second image, and to translate the first image based on the target translation parameter to obtain a third image; a first scaling module , set to select a plurality of scaling coefficients, and scale the third image with each scaling coefficient respectively, and calculate the image matching score between the third image and the second image; the second determining module is set to match the scores of the multiple images.
  • the scaling factor corresponding to the smallest score is used as the target scaling factor.
  • the first determination module further includes: a second calculation module, configured to calculate a target translation parameter of the first image relative to the second image, and to translate the first image based on the target translation parameter to obtain a fourth image; the second zoom module, set to select a plurality of scaling coefficients, and scale the fourth image with each scaling coefficient respectively, and calculate the image matching score between the fourth image and the second image; the third determining module is set to adjust the above-mentioned scaling coefficients until the above-mentioned If the score change in the image matching score is less than the first threshold, the scaling coefficient corresponding to the above-mentioned image matching score is used as the target scaling coefficient.
  • a second calculation module configured to calculate a target translation parameter of the first image relative to the second image, and to translate the first image based on the target translation parameter to obtain a fourth image
  • the second zoom module set to select a plurality of scaling coefficients, and scale the fourth image with each scaling coefficient respectively, and calculate the image matching score between the fourth image and the second image
  • the first determination module further includes: a third scaling module, configured to select multiple scaling coefficients, and scale the first image with each scaling coefficient; a third computing module, configured to scale based on each scaling coefficient. After the first image, slide on the second image, and calculate the image matching score between it and the second image; the fourth determination module is set to the scaling factor corresponding to the minimum score in the multiple image matching scores and translation amount as target zoom factor and target translation parameter.
  • a third scaling module configured to select multiple scaling coefficients, and scale the first image with each scaling coefficient
  • a third computing module configured to scale based on each scaling coefficient.
  • the first processing unit includes: a first mapping module configured to map the depth value of each pixel in the preliminary aligned depth map in the base image pair to a preset pixel range; and/or, a first adjustment A module configured to adjust the image contrast of the preliminary aligned depth map to obtain a first image.
  • a first mapping module configured to map the depth value of each pixel in the preliminary aligned depth map in the base image pair to a preset pixel range
  • a first adjustment A module configured to adjust the image contrast of the preliminary aligned depth map to obtain a first image.
  • the first determination module further includes: a first extraction module, configured to extract image features of the first image to obtain a first feature subset, wherein the first feature subset includes a first distance image, a first boundary orientation map and first mask information; a second extraction module configured to extract image features of the second image to obtain a second feature subset, wherein the second feature subset includes a second range image and a second boundary orientation map ; a fourth calculation module, configured to calculate the target translation parameter of the first image relative to the second image based on the first feature subset and the second feature subset.
  • a first extraction module configured to extract image features of the first image to obtain a first feature subset, wherein the first feature subset includes a first distance image, a first boundary orientation map and first mask information
  • a second extraction module configured to extract image features of the second image to obtain a second feature subset, wherein the second feature subset includes a second range image and a second boundary orientation map
  • a fourth calculation module configured to calculate the target translation parameter of the first image relative to the second
  • the first extraction module includes: a first extraction sub-module, configured to extract all boundary pixels of each target object in the first image to obtain a first edge image; a first inversion sub-module, configured to extract the first edge image; Perform inverse color processing on the edge image to obtain a second edge image; the second extraction sub-module is set to extract the contour of the first edge image to obtain a first contour array, and calculate the pixel point direction corresponding to each pixel point based on the first contour array , obtains the first contour direction array; the first transformation submodule is set to perform preset distance transformation processing on the second edge image based on the first preset distance threshold to obtain the first distance image; the first calculation submodule is set to Based on the first contour direction array, calculate the first boundary direction map corresponding to the boundary of each target object in the second edge image; the first determination sub-module is set to determine the first feature based on the first distance image and the first boundary direction map Subset.
  • the first transformation sub-module includes: a second determination sub-module, configured to determine first mask information based on a first preset distance threshold, wherein the first mask information is set to shield a portion in the second image edge information; adding a submodule, configured to add the first mask information to the first feature subset.
  • the second extraction module includes: a second extraction sub-module, configured to extract all boundary pixels of each target object in the second image, to obtain a third edge image; a deletion sub-module, configured to use the first mask The information performs subtraction processing on the contour in the third edge image; the second inversion sub-module is set to perform inverse color processing on the deleted third edge image to obtain a fourth edge image; the second calculation submodule, Set to extract the contour of the fourth edge image to obtain a second contour array, calculate the pixel point direction corresponding to each pixel point based on the second contour array, and obtain a second contour direction array; the second transformation sub-module is set to be based on the second contour array.
  • the preset distance threshold is used to perform preset distance transformation processing on the fourth edge image to obtain a second distance image;
  • the third calculation sub-module is set to calculate the corresponding boundary of each target object in the fourth edge image based on the second contour direction array The second boundary direction map of ;
  • the third determination sub-module is set to obtain a second feature subset based on the second distance image and the second boundary direction map.
  • the fourth calculation module includes: a third extraction sub-module, configured to use the first judgment condition to extract contour pixels whose pixel distances in the first distance image and the second distance image are less than the first distance threshold to obtain a participating image.
  • the set of matched first contour pixels the fourth extraction sub-module is set to adopt the second judgment condition to extract the contour pixels whose pixel distance between the first boundary pattern and the second boundary pattern is less than the second distance threshold, and obtain participation
  • the fifth determination sub-module is set to determine the chamfering distance score and direction between the first image and the second image based on the first set of contour pixels and the second set of contour pixels
  • the image distance and the image adjustment factor wherein the image adjustment factor is set to adjust the chamfering distance score and the proportion of the direction map distance
  • the fourth calculation sub-module is set to slide the second image on the first image, and adjust the chamfering distance score,
  • the direction map distance and the image adjustment factor are input into the first preset formula,
  • the first correction module includes: a first acquisition sub-module, configured to acquire the alignment precision value of the terminal application, and determine a plurality of correction resolutions based on the alignment precision value and the resolution of the basic image pair, wherein a plurality of correction resolutions are The resolution includes at least: a preset resolution, the preset resolution is the smallest resolution among the multiple correction resolutions; the first correction sub-module is set to scale the base image pair to the preset resolution and perform pyramid correction Process until the alignment accuracy value is met, resulting in a number of correction parameters.
  • a first acquisition sub-module configured to acquire the alignment precision value of the terminal application, and determine a plurality of correction resolutions based on the alignment precision value and the resolution of the basic image pair, wherein a plurality of correction resolutions are The resolution includes at least: a preset resolution, the preset resolution is the smallest resolution among the multiple correction resolutions; the first correction sub-module is set to scale the base image pair to the preset resolution and perform pyramid correction Process until the alignment accuracy value is met
  • the image correction device further includes: a determination unit, configured to determine the image alignment requirement accuracy of the terminal application under the target image resolution; a first determination unit, configured to execute step S1, and determine that under the first image resolution, Whether the current alignment accuracy image corresponding to the target image pair meets the required image alignment accuracy; the first adjustment unit is set to perform step S2, if it is determined that the current alignment accuracy image corresponding to the target image pair does not meet the required image alignment accuracy, adjust The image resolution is the second image resolution, wherein the resolution value of the second image resolution is higher than the first image resolution; the first execution unit is set to execute step S3, and executes the pairing of the base image using the preset correction mode performing correction processing to obtain a plurality of correction parameters; the second execution unit is set to perform step S4, performing alignment and correction on the above-mentioned basic image pair based on each correction parameter, and obtaining a target image pair; repeating steps S1 to S4 Step S4, the process ends when the current alignment accuracy image reaches the required image alignment accuracy.
  • a determination unit configured
  • the image correction device further includes: a comparison unit, configured to compare the image resolution of the visible light image and the image resolution of the depth image, and obtain a comparison result with the smallest resolution; a calculation unit, configured to obtain an image resolution based on the comparison result. rate and the initially set maximum resolution of correction processing, and calculate the threshold of correction times; the stopping unit is set to stop the correction processing if the number of image corrections reaches the threshold of correction times during the alignment correction process.
  • a comparison unit configured to compare the image resolution of the visible light image and the image resolution of the depth image, and obtain a comparison result with the smallest resolution
  • a calculation unit configured to obtain an image resolution based on the comparison result. rate and the initially set maximum resolution of correction processing, and calculate the threshold of correction times
  • the stopping unit is set to stop the correction processing if the number of image corrections reaches the threshold of correction times during the alignment correction process.
  • the above-mentioned image correction device may also include a processor and a memory, and the above-mentioned acquisition unit 41, the first correction unit 43, the second correction unit 45, etc. are all stored in the memory as program units, and the processor executes the above-mentioned programs stored in the memory. unit to achieve the corresponding function.
  • the above-mentioned processor includes a kernel, and the corresponding program unit is called from the memory by the kernel.
  • One or more kernels can be set, and the target image pair can be obtained by adjusting the kernel parameters to perform alignment correction on the above-mentioned base image pair based on each correction parameter.
  • the above-mentioned memory may include non-persistent memory in computer readable medium, random access memory (RAM) and/or non-volatile memory, such as read only memory (ROM) or flash memory (flash RAM), the memory includes at least a memory chip.
  • RAM random access memory
  • ROM read only memory
  • flash RAM flash memory
  • an image correction system including: a first image capture device configured to capture a visible light image of a target object; a second image capture device configured to capture a depth image of the target object
  • the correction device is set to obtain the visible light image and the depth image of the target object, and form a basic image pair after transformation, wherein, the above-mentioned basic image pair includes a first image and a second image, and adopts a preset correction mode to the basic image.
  • the result output device is configured to output the aligned target image pair to a preset terminal display interface .
  • an electronic device comprising: a processor; and a memory configured to store executable instructions of the processor; wherein the processor is configured to execute the above-mentioned execution of the executable instructions Any image correction method.
  • a computer-readable storage medium is also provided, where the computer-readable storage medium includes a stored computer program, wherein when the computer program runs, the device where the computer-readable storage medium is located is controlled to execute any of the above A method of image correction.
  • the present application also provides a computer program product, which, when executed on a data processing device, is suitable for executing a program initialized with the following method steps: acquiring a visible light image and a depth image captured on a target object, and after transformation, a basic image pair is formed , wherein the basic image pair includes a first image and a second image; a preset correction mode is used to perform correction processing on the basic image pair to obtain multiple correction parameters; alignment correction is performed on the basic image pair based on each correction parameter , get the target image pair.
  • the disclosed technical content can be implemented in other ways.
  • the device embodiments described above are only illustrative.
  • the division of the above-mentioned units may be a logical function division.
  • multiple units or components may be combined or integrated. to another coefficient, or some features can be ignored, or not implemented.
  • the shown or discussed mutual coupling or direct coupling or communication connection may be through some interfaces, indirect coupling or communication connection of units or modules, and may be in electrical or other forms.
  • the units described above as separate components may or may not be physically separated, and components shown as units may or may not be physical units, that is, may be located in one place, or may be distributed to multiple units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution in this embodiment.
  • each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically alone, or two or more units may be integrated into one unit.
  • the above-mentioned integrated units may be implemented in the form of hardware, or may be implemented in the form of software functional units.
  • the above-mentioned integrated units are implemented in the form of software functional units and sold or used as independent products, they may be stored in a computer-readable storage medium.
  • the technical solution of the present invention is essentially or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium , which includes several instructions for causing a computer device (which may be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the above-mentioned methods in various embodiments of the present invention.
  • the aforementioned storage medium includes: U disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), mobile hard disk, magnetic disk or optical disk and other media that can store program codes .
  • the solutions provided by the embodiments of the present application can realize automatic correction of images captured by cameras.
  • the technical solutions provided in the present disclosure can be applied to electronic devices having at least one image capture unit, for example, applicable to various types of mobile devices and mobile platforms.
  • vehicle chip, embedded chip, etc. the correction environment is simple, no specific environment, specific shooting pattern is required, only the visible light image and depth image that are initially aligned according to the preset calibration parameters can be dynamically corrected.
  • the position or its own parameters change, it is necessary to further correct the visible light image and the depth image aligned with the preset calibration parameters to reduce the alignment error and solve the problem that the dynamic correction between two different cameras cannot be realized in the related technology, and the adaptability to the environment is low.

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • General Physics & Mathematics (AREA)
  • Data Mining & Analysis (AREA)
  • Multimedia (AREA)
  • Artificial Intelligence (AREA)
  • Evolutionary Computation (AREA)
  • General Engineering & Computer Science (AREA)
  • Evolutionary Biology (AREA)
  • Bioinformatics & Computational Biology (AREA)
  • Bioinformatics & Cheminformatics (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • Health & Medical Sciences (AREA)
  • Computing Systems (AREA)
  • Databases & Information Systems (AREA)
  • General Health & Medical Sciences (AREA)
  • Medical Informatics (AREA)
  • Software Systems (AREA)
  • Image Processing (AREA)
  • Geometry (AREA)

Abstract

本发明公开了一种图像校正方法、装置及系统、电子设备。其中，该图像校正方法包括：获取对目标对象拍摄的可见光图像和深度图像，经变换后形成基础图像对，其中，所述基础图像对包括第一图像和第二图像；采用预设校正模式对基础图像对进行校正处理，得到多个校正参数；基于每个校正参数对上述基础图像对进行对齐校正，得到目标图像对。

Description

图像校正方法、装置及系统、电子设备
本申请要求于2020年12月25日提交中国专利局、申请号为202011567624.3、申请名称“图像校正方法、装置及系统、电子设备”的中国专利申请的优先权，其全部内容通过引用结合在本申请中。
技术领域
本发明涉及图像处理技术领域，具体而言，涉及一种图像校正方法、装置及系统、电子设备。
背景技术
相关技术中，随着硬件技术的不断提升，深度成像变得越来越准确，基于深度信息的应用也迅速发展起来，目前常见的深度成像方式主要分为三种：双目立体成像、结构光成像和飞行时间成像(ToF，Time of Flight)。
双目立体成像，需要使用两个RGB相机同时采集图像，然后通过双目匹配，使用三角测量方法获得深度信息。优点是成本低、功耗低且图像的分辨率也比较高，但是由于深度计算完全通过算法计算获得，所以对于计算资源要求高，实时性较差，对于成像环境也比较敏感。
结构光成像，是通过相机发射特定图形(散斑或者点阵)的激光，当被测物体反射这些图案，通过相机捕捉到这些反射回来的图案，计算上面散斑或者点的大小，从而测算出被测物体到相机之间的距离。优点主要是不受物体纹理影响，但是在强光下激光散斑会被淹没，所以不适合在户外使用。
ToF相机成像，通过计算发射信号和反射信号的时间差，直接获得被测点的深度信息。优点是实时性较高，不受光照变化和物体纹理影响，但是图像分辨率普遍不高，且模块比较大，硬件成本也比较高。
在使用各种终端设备时，需要对齐深度相机和可见光相机的相对位置，而当前为了降低拍摄时的抖动，很多可见光相机会采用光学防抖技术(OIS，Optical Image Stabilization)和自动对焦(AF，Auto Focus)提升拍摄图像清晰度，这些机制会导致两个相机的相对位置改变，相机的焦距和光心改变。另外，设备跌落、相机之间帧不同步或者帧率不同也会导致两个相机的相对位置改变。当发生这些变化时，两个相 机的内参和外参已经发生改变，若仍然使用已标定的参数会影响深度图像与可见光图像之间对齐精度，进而影响后续依赖深度信息的算法效果。很多时候不具有重新标定的环境，无法实现动态校正，导致图像对齐效果较差，影响用户的使用兴趣。
针对上述的问题，目前尚未提出有效的解决方案。
发明内容
本公开提供了一种图像校正方法、装置及系统、电子设备，以至少解决相关技术中无法实现两种不同相机间的动态校正，适应环境低，导致对齐图像效果较差，容易影响用户的使用兴趣的技术问题。
根据本公开的一个方面，提供了一种图像校正方法，包括：获取对目标对象拍摄的可见光图像和深度图像，经变换后形成基础图像对，其中，上述基础图像对包括第一图像和第二图像；采用预设校正模式对上述基础图像对进行校正处理，得到多个校正参数；基于每个所述校正参数对上述基础图像对进行对齐校正，得到目标图像对。
可选地，采用预设校正模式对上述基础图像对进行校正处理，得到多个校正参数的步骤，包括：将上述基础图像对缩放至预设分辨率下，并进行金字塔化校正处理，得到上述多个校正参数。
可选地，获取对目标对象拍摄的可见光图像和深度图像，经变换后形成基础图像对的步骤，包括：基于预设标定参数，将上述深度图像变换到上述可见光图像的图像坐标系下，经过调整以得到与上述可见光图像具有相同分辨率的初步对齐深度图，其中，上述可见光图像与上述初步对齐深度图组合形成上述基础图像对，上述第一图像为上述可见光图像，上述第二图像为所述初步对齐深度图。
可选地，采用预设校正模式对上述基础图像对进行校正处理，得到多个校正参数的步骤，还包括：确定上述第一图像与上述第二图像之间的目标平移参数和目标缩放系数；基于上述目标平移参数和上述目标缩放系数，确定多个校正参数。
可选地，在采用预设校正模式对上述基础图像对进行校正处理，得到多个校正参数之前，上述图像校正方法还包括：对上述基础图像对中的初步对齐深度图进行预处理，得到上述第一图像；对上述基础图像对中可见光图像进行滤波处理，得到上述第二图像。
可选地，确定上述第一图像与上述第二图像之间的目标平移参数和目标缩放系数的步骤，包括：计算上述第一图像相对于上述第二图像的上述目标平移参数，并基于上述目标平移参数平移上述第一图像，获得第三图像；选取多个缩放系数，并以每个 上述缩放系数分别缩放上述第三图像，计算上述第三图像与上述第二图像之间的图像匹配得分；将多个上述图像匹配得分中分值最小所对应的缩放系数作为目标缩放系数。
可选地，确定上述第一图像与上述第二图像之间的目标平移参数和目标缩放系数的步骤，包括：计算上述第一图像相对于上述第二图像的上述目标平移参数，并基于上述目标平移参数平移上述第一图像，获得第四图像；选取多个缩放系数，并以每个上述缩放系数分别缩放上述第四图像，计算上述第四图像与上述第二图像之间的图像匹配得分；调整上述缩放系数直至上述图像匹配得分中分数变化小于第一阈值，则上述图像匹配得分所对应的缩放系数作为目标缩放系数。
可选地，确定上述第一图像与上述第二图像之间的目标平移参数和目标缩放系数的步骤，包括：选取多个缩放系数，并以每个上述缩放系数分别缩放上述第一图像；对基于上述每个上述缩放系数缩放后的上述第一图像，在上述第二图像上滑动，并计算其与上述第二图像之间的图像匹配的得分；将多个上述图像匹配得分中分值最小所对应的缩放系数和平移量作为上述目标缩放系数和上述目标平移参数。
可选地，对上述基础图像对中的初步对齐深度图进行预处理，得到上述第一图像的步骤，包括：将上述基础图像对中的初步对齐深度图中每个像素点的深度值映射至预设像素范围内；和/或，调整上述初步对齐深度图的图像对比度，得到上述第一图像。
可选地，确定上述第一图像与上述第二图像之间的目标平移参数和目标缩放系数的步骤，包括：提取上述第一图像的图像特征，得到第一特征子集，其中，上述第一特征子集包括第一距离图像，第一边界方向图和第一掩膜信息；提取上述第二图像的图像特征，得到第二特征子集，其中，上述第二特征子集包括第二距离图像和第二边界方向图；基于上述第一特征子集和上述第二特征子集，计算上述第一图像相对于上述第二图像的目标平移参数。
可选地，提取上述第一图像的图像特征，得到第一特征子集的步骤，包括：提取上述第一图像中各个目标物体的所有边界像素点，得到第一边缘图像；对上述第一边缘图像进行反色处理，得到第二边缘图像；对上述第一边缘图像提取轮廓，得到第一轮廓数组，基于上述第一轮廓数组计算每个像素点对应的像素点方向，得到第一轮廓方向数组；基于第一预设距离阈值，对上述第二边缘图像进行预设距离变换处理，得到上述第一距离图像；基于上述第一轮廓方向数组，计算上述第二边缘图像中各个目标物体边界所对应的第一边界方向图；基于上述第一距离图像和上述第一边界方向图，确定上述第一特征子集。
可选地，基于第一预设距离阈值，对上述第二边缘图像进行预设距离变换处理， 得到第一距离图像的步骤，包括：基于上述第一预设距离阈值，确定上述第一掩膜信息，其中，上述第一掩膜信息设置为屏蔽上述第二图像中的部分边缘信息；将上述第一掩膜信息添加至上述第一特征子集。
可选地，提取上述第二图像的图像特征，得到第二特征子集的步骤，包括：提取上述第二图像中各个目标物体的所有边界像素点，得到第三边缘图像；采用上述第一掩膜信息对上述第三边缘图像中的轮廓进行删减处理；对删减处理后的上述第三边缘图像进行反色处理，得到第四边缘图像；对上述第四边缘图像提取轮廓，得到第二轮廓数组，基于上述第二轮廓数组计算每个像素点对应的像素点方向，得到第二轮廓方向数组；基于第二预设距离阈值，对上述第四边缘图像进行预设距离变换处理，得到上述第二距离图像；基于上述第二轮廓方向数组，计算上述第四边缘图像中各个目标物体边界所对应的上述第二边界方向图；基于上述第二距离图像和上述第二边界方向图，得到上述第二特征子集。
可选地，基于上述第一特征子集和上述第二特征子集，计算上述第一图像相对于上述第二图像的目标平移参数的步骤，包括：采用第一判断条件，提取上述第一距离图像与上述第二距离图像中像素距离小于第一距离阈值的轮廓像素点，得到参与图像匹配的第一轮廓像素点集合；采用第二判断条件，提取上述第一边界方向图与上述第二边界方向图中像素距离小于第二距离阈值的轮廓像素点，得到参与图像匹配的第二轮廓像素点集合；基于上述第一轮廓像素点集合和上述第二轮廓像素点集合，确定上述第一图像和上述第二图像之间的倒角距离得分、方向图距离和图像调节因子，其中，上述图像调节因子设置为调节上述倒角距离得分和上述方向图距离比重；在上述第一图像上滑动上述第二图像，并将上述倒角距离得分、上述方向图距离和上述图像调节因子输入至第一预设公式，计算图像滑动得分；确定所有图像滑动得分中分值最小所对应的目标滑动位置；基于上述目标滑动位置，确定目标平移参数。
可选地，将上述基础图像对缩放至预设分辨率下，并进行金字塔化校正处理，得到上述多个校正参数的步骤，包括：获取终端应用的对齐精度值，基于上述对齐精度值和上述基础图像对的分辨率，确定多个校正分辨率，其中，上述多个校正分辨率至少包括：预设分辨率，上述预设分辨率为多个校正分辨率中最小分辨率；将上述基础图像对缩放至上述预设分辨率下，并进行金字塔化校正处理直到满足上述的对齐精度值，得到上述多个校正参数。
可选地，上述图像校正方法还包括：确定终端应用在目标图像分辨率下的图像对齐需求精度；步骤S1，判断在上述第一图像分辨率下，与上述目标图像对所对应的当前对齐精度图像是否达到上述图像对齐需求精度；步骤S2，若确定与上述目标图像对 所对应的当前对齐精度图像未达到上述图像对齐需求精度，调整图像分辨率为第二图像分辨率，其中，上述第二图像分辨率的分辨率数值高于上述第一图像分辨率；步骤S3，执行采用预设校正模式对上述基础图像对进行校正处理，得到多个校正参数的步骤；步骤S4，执行基于每个上述校正参数对上述基础图像对进行对齐校正，得到目标图像对的步骤；重复执行步骤S1至步骤S4，直至当前对齐精度图像达到上述图像对齐需求精度时结束。
可选地，上述图像校正方法还包括：比较上述可见光图像的图像分辨率和上述深度图像的图像分辨率，得到分辨率最小的比较结果；基于比较结果得到的图像分辨率和初始设置的校正处理最大分辨率，计算校正次数阈值；在进行对齐校正过程中，若图像校正次数达到上述校正次数阈值，则停止校正处理。
根据本公开的另一方面，还提供了一种图像校正装置，包括：获取单元，设置为获取对目标对象拍摄的可见光图像和深度图像，经变换后形成基础图像对，其中，上述基础图像对包括第一图像和第二图像；第一校正单元，设置为采用预设校正模式对上述基础图像对进行校正处理，得到多个校正参数；第二校正单元，设置为基于每个上述校正参数对上述基础图像对进行对齐校正，得到目标图像对。
可选地，上述第一校正单元包括：第一校正模块，设置为将上述基础图像对缩放至预设分辨率下，并进行金字塔化校正处理，得到上述多个校正参数。
可选地，上述获取单元包括：第一变换模块，设置为基于预设标定参数，将上述深度图像变换到上述可见光图像的图像坐标系下，经过调整以得到与上述可见光图像具有相同分辨率的初步对齐深度图，其中，上述可见光图像与上述初步对齐深度图组合形成上述基础图像对，上述第一图像为上述可见光图像，上述第二图像为上述初步对齐深度图。
可选地，上述第一校正单元还包括：第一确定模块，设置为确定上述第一图像与上述第二图像之间的目标平移参数和目标缩放系数；第二确定模块，设置为基于上述目标平移参数和上述目标缩放系数，确定多个校正参数。
可选地，上述图像校正装置还包括：第一处理单元，设置为在采用预设校正模式对上述基础图像对进行校正处理，得到多个校正参数之前，对上述基础图像对中的初步对齐深度图进行预处理，得到上述第一图像；第二处理单元，设置为对上述基础图像对中可见光图像进行滤波处理，得到上述第二图像。
可选地，上述第一确定模块包括：第一计算模块，设置为计算上述第一图像相对于上述第二图像的上述目标平移参数，并基于上述目标平移参数平移上述第一图像， 获得第三图像；第一缩放模块，设置为选取多个缩放系数，并以每个上述缩放系数分别缩放上述第三图像，计算上述第三图像与上述第二图像之间的图像匹配得分；第二确定模块，设置为将多个上述图像匹配得分中分值最小所对应的缩放系数作为目标缩放系数。
可选地，上述第一确定模块还包括：第二计算模块，设置为计算上述第一图像相对于上述第二图像的上述目标平移参数，并基于上述目标平移参数平移上述第一图像，获得第四图像；第二缩放模块，设置为选取多个缩放系数，并以每个上述缩放系数分别缩放上述第四图像，计算上述第四图像与上述第二图像之间的图像匹配得分；第三确定模块，设置为调整上述缩放系数直至上述图像匹配得分中分数变化小于第一阈值，则上述图像匹配得分所对应的缩放系数作为目标缩放系数。
可选地，上述第一确定模块还包括：第三缩放模块，设置为选取多个缩放系数，并以每个上述缩放系数分别缩放上述第一图像；第三计算模块，设置为对基于上述每个上述缩放系数缩放后的上述第一图像，在上述第二图像上滑动，并计算其与上述第二图像之间的图像匹配的得分；第四确定模块，设置为将多个上述图像匹配得分中分值最小所对应的缩放系数和平移量作为上述目标缩放系数和上述目标平移参数。
可选地，上述第一处理单元包括：第一映射模块，设置为将上述基础图像对中的初步对齐深度图中每个像素点的深度值映射至预设像素范围内；和/或，第一调整模块，设置为调整上述初步对齐深度图的图像对比度，得到上述第一图像。
可选地，上述第一确定模块还包括：第一提取模块，设置为提取上述第一图像的图像特征，得到第一特征子集，其中，上述第一特征子集包括第一距离图像，第一边界方向图和第一掩膜信息；第二提取模块，设置为提取上述第二图像的图像特征，得到第二特征子集，其中，上述第二特征子集包括第二距离图像和第二边界方向图；第四计算模块，设置为基于上述第一特征子集和上述第二特征子集，计算上述第一图像相对于上述第二图像的目标平移参数。
可选地，上述第一提取模块包括：第一提取子模块，设置为提取上述第一图像中各个目标物体的所有边界像素点，得到第一边缘图像；第一反色子模块，设置为对上述第一边缘图像进行反色处理，得到第二边缘图像；第二提取子模块，设置为对上述第一边缘图像提取轮廓，得到第一轮廓数组，基于上述第一轮廓数组计算每个像素点对应的像素点方向，得到第一轮廓方向数组；第一变换子模块，设置为基于第一预设距离阈值，对上述第二边缘图像进行预设距离变换处理，得到第一距离图像；第一计算子模块，设置为基于上述第一轮廓方向数组，计算上述第二边缘图像中各个目标物体边界所对应的第一边界方向图；第一确定子模块，设置为基于上述第一距离图像和 上述第一边界方向图，确定第一特征子集。
可选地，上述第一变换子模块包括：第二确定子模块，设置为基于上述第一预设距离阈值，确定第一掩膜信息，其中，上述第一掩膜信息设置为屏蔽上述第二图像中的部分边缘信息；添加子模块，设置为将上述第一掩膜信息添加至上述第一特征子集。
可选地，上述第二提取模块包括：第二提取子模块，设置为提取上述第二图像中各个目标物体的所有边界像素点，得到第三边缘图像；删减子模块，设置为采用上述第一掩膜信息对上述第三边缘图像中的轮廓进行删减处理；第二反色子模块，设置为对删减处理后的上述第三边缘图像进行反色处理，得到第四边缘图像；第二计算子模块，设置为对上述第四边缘图像提取轮廓，得到第二轮廓数组，基于上述第二轮廓数组计算每个像素点对应的像素点方向，得到第二轮廓方向数组；第二变换子模块，设置为基于第二预设距离阈值，对上述第四边缘图像进行预设距离变换处理，得到第二距离图像；第三计算子模块，设置为基于上述第二轮廓方向数组，计算上述第四边缘图像中各个目标物体边界所对应的第二边界方向图；第三确定子模块，设置为基于上述第二距离图像和上述第二边界方向图，得到第二特征子集。
可选地，上述第四计算模块包括：第三提取子模块，设置为采用第一判断条件，提取上述第一距离图像与上述第二距离图像中像素距离小于第一距离阈值的轮廓像素点，得到参与图像匹配的第一轮廓像素点集合；第四提取子模块，设置为采用第二判断条件，提取上述第一边界方向图与上述第二边界方向图中像素距离小于第二距离阈值的轮廓像素点，得到参与图像匹配的第二轮廓像素点集合；第五确定子模块，设置为基于上述第一轮廓像素点集合和上述第二轮廓像素点集合，确定上述第一图像和上述第二图像之间的倒角距离得分、方向图距离和图像调节因子，其中，上述图像调节因子设置为调节上述倒角距离得分和上述方向图距离比重；第四计算子模块，设置为在上述第一图像上滑动上述第二图像，并将上述倒角距离得分、上述方向图距离和上述图像调节因子输入至第一预设公式，计算图像滑动得分；第六确定子模块，设置为确定所有图像滑动得分中分值最小所对应的目标滑动位置；第七确定子模块，设置为基于上述目标滑动位置，确定目标平移参数。
可选地，上述第一校正模块包括：第一获取子模块，设置为获取终端应用的对齐精度值，基于上述对齐精度值和上述基础图像对的分辨率，确定多个校正分辨率，其中，上述多个校正分辨率至少包括：预设分辨率，上述预设分辨率为多个校正分辨率中最小分辨率；第一校正子模块，设置为将上述基础图像对缩放至上述预设分辨率下，并进行金字塔化校正处理直到满足上述的对齐精度值，得到上述多个校正参数。
可选地，上述图像校正装置还包括：确定单元，设置为确定终端应用在目标图像 分辨率下的图像对齐需求精度；第一判断单元，设置为执行步骤S1，判断在上述第一图像分辨率下，与上述目标图像对所对应的当前对齐精度图像是否达到上述图像对齐需求精度；第一调整单元，设置为执行步骤S2，若确定与上述目标图像对所对应的当前对齐精度图像未达到上述图像对齐需求精度，调整图像分辨率为第二图像分辨率，其中，上述第二图像分辨率的分辨率数值高于上述第一图像分辨率；第一执行单元，设置为执行步骤S3，执行采用预设校正模式对上述基础图像对进行校正处理，得到多个校正参数的步骤；第二执行单元，设置为执行步骤S4，执行基于每个上述校正参数对上述基础图像对进行对齐校正，得到目标图像对的步骤；重复执行步骤S1至步骤S4，直至当前对齐精度图像达到上述图像对齐需求精度时结束。
可选地，上述图像校正装置还包括：比较单元，设置为比较上述可见光图像的图像分辨率和上述深度图像的图像分辨率，得到分辨率最小的比较结果；计算单元，设置为基于比较结果得到的图像分辨率和初始设置的校正处理最大分辨率，计算校正次数阈值；停止单元，设置为在进行对齐校正过程中，若图像校正次数达到上述校正次数阈值，则停止校正处理。
根据本公开的另一方面，还提供了一种图像校正系统，包括：第一图像捕获装置，设置为拍摄目标对象的可见光图像；第二图像捕获装置，设置为拍摄目标对象的深度图像；校正装置，设置为获取对目标对象拍摄的可见光图像和深度图像，经变化后形成基础图像对，其中，上述基础图像对包括第一图像和第二图像；采用预设校正模式对上述基础图像对进行校正处理，得到多个校正参数；基于每个上述校正参数对上述基础图像对进行对齐校正，得到目标图像对；结果输出装置，设置为将对齐后的目标图像对输出至预设终端展示界面。
根据本公开的另一方面，还提供了一种电子设备，包括：处理器；以及存储器，设置为存储上述处理器的可执行指令；其中，上述处理器配置为经由执行上述可执行指令来执行上述任意一项上述的图像校正方法。
根据本公开的另一方面，还提供了一种计算机可读存储介质，上述计算机可读存储介质包括存储的计算机程序，其中，在上述计算机程序运行时控制上述计算机可读存储介质所在设备执行上述任意一项上述的图像校正方法。
本公开中，获取对目标对象拍摄的可见光图像和深度图像，经变换后形成基础图像对，其中，上述基础图像对包括第一图像和第二图像，采用预设校正模式对基础图像对进行校正处理，得到多个校正参数，基于每个校正参数对上述基础图像对进行对齐校正，得到目标图像对。在该实施例中，可以对多种相机拍摄图像进行对齐操作，实现动态校正，校正环境简单，利用设备实拍的图像就可以完成对齐校正，从而解决 相关技术中无法实现两种不同相机间的动态校正，适应环境低，导致对齐图像效果较差，容易影响用户的使用兴趣的技术问题。
附图说明
此处所说明的附图用来提供对本发明的进一步理解，构成本申请的一部分，本发明的示意性实施例及其说明用于解释本发明，并不构成对本发明的不当限定。在附图中：
图1是根据本发明实施例的一种可选的图像校正方法的流程图；
图2是根据本发明实施例的一种可选的初步对齐深度图的示意图；
图3是根据本发明实施例的一种可选的完成对齐的深度图像与可见光图像叠加图像；
图4是根据本发明实施例的一种可选的图像校正装置的示意图。
具体实施方式
为了使本技术领域的人员更好地理解本发明方案，下面将结合本发明实施例中的附图，对本发明实施例中的技术方案进行清楚、完整地描述，显然，所描述的实施例仅仅是本发明一部分的实施例，而不是全部的实施例。基于本发明中的实施例，本领域普通技术人员在没有做出创造性劳动前提下所获得的所有其他实施例，都应当属于本发明保护的范围。
需要说明的是，本发明的说明书和权利要求书及上述附图中的术语“第一”、“第二”等是用于区别类似的对象，而不必用于描述特定的顺序或先后次序。应该理解这样使用的数据在适当情况下可以互换，以便这里描述的本发明的实施例能够以除了在这里图示或描述的那些以外的顺序实施。此外，术语“包括”和“具有”以及他们的任何变形，意图在于覆盖不排他的包含，例如，包含了一系列步骤或单元的过程、方法、系数、产品或设备不必限于清楚地列出的那些步骤或单元，而是可包括没有清楚地列出的或对于这些过程、方法、产品或设备固有的其它步骤或单元。
为便于本领域技术人员能够理解本发明，下面对本发明各实施例中涉及的部分术语进行解释：
RGB，Red Green Blue，一种颜色标准，本文中亦指通常的彩色图像；
RGB-D，Red Green Blue-Depth，彩色-深度图；
ToF，Time of flight，飞行时间；
OIS，Optical Image Stabilization，光学图像防抖；
AF，Automatic Focus，自动对焦；
FF，Fixed-focus，定焦；
VGA分辨率：640*480分辨率。
本发明下述各实施例应用的场景包括但不限于：三维重建、无人驾驶、人脸识别、物体测量、3D建模、背景虚化、可见光图像与红外光图像融合、可见光图像与深度图像融合、VR眼镜、车载成像设备等。为了能够应对复杂拍摄环境带来的图像差异，高效的完成图像对齐，根据多种相机获取的图像计算对齐参数，本发明适用于只能提供可见光图像和红外图像，或者只能提供可见光图像和深度图像，或者可以提供可见光图像，红外光图像和深度图像的含深度相机的图像捕获设备，并且对深度相机种类无要求，可以是ToF相机，红外相机，结构光相机和/或双目深度相机。
本发明校正环境简单，无需特定的环境，特定的拍摄图案，只需要根据预设标定参数初步对齐的可见光图像和深度图像即可实现动态校正。对于没有使用OIS的相机设备，本发明只需要每隔一段时间进行一次动态对齐校正即可实现图像对齐；另外对于对齐精度要求高但是实时性要求不高的校正处理，本发明可以选择在高分辨率上进行对齐校正处理，后续可用于3D建模，背景虚化、可见光图像与红外光图像融合等。
本发明还可用于一些物体的检测和匹配，常见的应用有手势检测，行人检测。因跌落等外力因素造成手机相机出现的标定误差，可使用本发明对搭载深度相机的终端设备定期自动校准或者售后标定；对于VR眼镜、车载成像设备，由于震动造成的可见光图像与深度图像间的对齐误差也可使用本发明进行校正。下面结合各个实施例来说明本发明。
实施例一
根据本发明实施例，提供了一种图像校正方法实施例，需要说明的是，在附图的流程图示出的步骤可以在诸如一组计算机可执行指令的计算机系统中执行，并且，虽然在流程图中示出了逻辑顺序，但是在某些情况下，可以以不同于此处的顺序执行所示出或描述的步骤。
本发明实施例提供了一种图像校正方法，该图像校正方法可以应用于图像校正系统中，图像校正系统包括：能够捕获深度图像/红外图像的第一图像捕获装置(本发明实施例以深度相机示意说明)和能够捕获可见光图像的第二图像捕获装置(本发明实 施例以可见光相机示意说明)。OIS机制，AF机制、设备跌落和图像捕获装置之间帧不同步或者帧率等不同，会使相机内参和外参改变，使用预设标定参数对齐两个相机的图像就产生误差，主要表现为初步对齐图像对存在平移和缩放问题，这对于精度要求严格的三维重建、背景虚化等图像处理技术是不可以接受的。所以，当图像捕获装置的相对位置或自身参数改变时，需要对使用预设标定参数对齐的可见光图像和深度图像做进一步的校正处理，降低对齐误差，本发明实施例提出了一种对齐精度高、处理速度快、可实时、可应用于绝大部分的实际使用场景的图像校正方法。
本发明实施例能提升图像对齐的实用性，对多种相机拍摄图像进行对齐操作。可以根据可见光图和红外图计算校准参数，也可以根据可见光图和深度图计算校准参数，适用于各种搭载ToF深度相机或者结构光深度相机的设备，通常其所采集的图像与可见光图像纹理差异大，常见的关键点匹配方案不可行，但是本发明实施例提供的技术方案依然可以获得较为准确的对齐效果。
图1是根据本发明实施例的一种可选的图像校正方法的流程图，如图1所示，该方法包括如下步骤：
步骤S102，获取对目标对象拍摄的可见光图像和深度图像，经变换后形成基础图像对，其中，上述基础图像对包括第一图像和第二图像；
步骤S104，采用预设校正模式对基础图像对进行校正处理，得到多个校正参数；
步骤S106，基于每个校正参数对上述基础图像对进行对齐校正，得到目标图像对。
通过上述步骤，可以获取对目标对象拍摄的可见光图像和深度图像，经变换后形成基础图像对，其中，上述基础图像对包括第一图像和第二图像，采用预设校正模式对基础图像对进行校正处理，得到多个校正参数，基于每个校正参数对上述基础图像对进行对齐校正，得到目标图像对。在该实施例中，可以对多种相机拍摄图像进行对齐操作，实现动态校正，校正环境简单，利用设备实拍的图像就可以完成对齐校正，从而解决相关技术中无法实现两种不同相机间的动态校正，适应环境低，导致对齐图像效果较差，容易影响用户的使用兴趣的技术问题。
下面结合上述各实施步骤进行详细说明。
步骤S102，获取对目标对象拍摄的可见光图像和深度图像，经变换后形成基础图像对，其中，上述基础图像对包括第一图像和第二图像。
本发明实施例使用的图像捕获装置可以包括：深度相机和可见光相机，深度相机可以获得深度图像和相对应的红外图像，可见光相机获取可见光图像。本发明实施例 中，深度相机不需要同时获得深度图像和红外图像，本发明实施例既适用于可以提供红外图像和深度图像的设备，也适用于只能提供深度图像或红外图像的设备。
作为本发明可选的实施方式，获取对目标对象拍摄的可见光图像和深度图像，经变换后形成基础图像对时，包括：基于预设标定参数，将深度图像变换到可见光图像的图像坐标系下，经过调整以得到与可见光图像具有相同分辨率的初步对齐深度图，其中，可见光图像与初步对齐深度图组合形成基础图像对，第一图像为可见光图像，第二图像为初步对齐深度图。
上述预设标定参数是基于深度相机和可见光相机进行初始标定时确定的参数，例如出厂标定参数结果。通常深度图的分辨率小于可见光图的分辨率，将深度图变换到可见光图像坐标系下后，可经过传统插值算法或深度学习超分模型调整，以得到与可见光图像具有相同分辨率的初步对齐深度图，方便后续图像对齐校正处理。
可选的，在采用预设校正模式对基础图像对进行校正处理，得到多个校正参数之前，图像校正方法还包括：对基础图像对中的初步对齐深度图进行预处理，得到第一图像；对基础图像对中可见光图像进行滤波处理，得到第二图像。
可选的，对基础图像对中的初步对齐深度图进行预处理，得到第一图像的步骤，包括：将基础图像对中的初步对齐深度图中每个像素点的深度值映射至预设像素范围内；和/或，调整初步对齐深度图的图像对比度，得到第一图像。
本发明实施例中，分别从可见光相机和深度相机获得一张可见光图像和一张深度图像，根据预设标定参数将深度图变换到可见光图像坐标系下，获得与可见光图像相同分辨率的初步对齐深度图。对初步对齐深度图进行预处理，将初步对齐深度图每个像素点的深度值映射至预设像素范围[0，255]中，以及针对存在的过曝问题，调整图像对比度恢复丢失的细节，加强因过曝导致减弱的纹理信息，以更有效地进行后续对齐校正处理，将预处理后的初步对齐深度图记为第一图像或者模板图像。对另一可见光图像进行滤波处理，滤波可帮助去除可见光图像分辨率变化时因为采样出现的高频噪点信号，并将获得的图像记为第二图像或者匹配图像。
实际应用中，例如三维重建，通常需要对齐可见光图像和深度图像，对齐误差越小对后续算法效果越有利。本发明实施例的目标是，减少可见光图像和初始对齐深度图之间的误差。本发明实施例中，可根据可见光图像和深度图计算校正参数，亦可以根据可见光图像和红外图像计算校正参数。本发明实施例中以可见光图像和深度图像计算校正参数为例描述实施流程，此实施流程也适用于可见光图像和红外图像计算校正参数。
图2是根据本发明实施例的一种可选的初步对齐深度图的示意图，如图2所示，该初步对齐深度图中包含有已经提取的主要的物体的轮廓和边缘信息。
图3是根据本发明实施例的一种可选的完成对齐的深度图像与可见光图像叠加图像，如图3所示，添加了可见光图像后，需要将可见光图像与初步对齐深度图中物体发生偏移的部分对齐，图3中进一步对齐的图像对偏移差异非常小，基本能够做到逐像素对齐。
本发明实施例中涉及的图像校正方案或者动态校正方案适用于已做过标定的设备，即初始的相机内参和外参已知。由于OIS机制，AF机制和设备跌落等因素会影响相机内参和相机间外参，故用已知的标定参数对齐的深度图与可见光图可能存在对齐误差，主要表现为初步对齐图像对存在平移和缩放问题，本发明实施例根据输入的可见光图和深度图，对这种平移和缩放问题进行校正。
步骤S104，采用预设校正模式对基础图像对进行校正处理，得到多个校正参数。
作为本发明可选的实施方式，采用预设校正模式对基础图像对进行校正处理，得到多个校正参数，包括：确定第一图像与第二图像之间的目标平移参数和目标缩放系数；基于目标平移参数和目标缩放系数，确定多个校正参数。
可选的，确定第一图像与第二图像之间的目标平移参数和目标缩放系数的步骤，包括：提取第一图像的图像特征，得到第一特征子集，其中，第一特征子集包括第一距离图像，第一边界方向图和第一掩膜信息；提取第二图像的图像特征，得到第二特征子集，其中，第二特征子集包括第二距离图像和第二边界方向图；基于第一特征子集和第二特征子集，计算第一图像相对于第二图像的目标平移参数。
下面分别对预处理后的得到第一图像和第二图像进行特征提取。
首先，本发明实施例对第一图像的图像特征进行提取，得到第一特征子集。
可选的，提取第一图像的图像特征，得到第一特征子集的步骤，包括：提取第一图像中各个目标物体的所有边界像素点，得到第一边缘图像；对第一边缘图像进行反色处理，得到第二边缘图像；对第一边缘图像提取轮廓，得到第一轮廓数组，基于第一轮廓数组计算每个像素点对应的像素点方向，得到第一轮廓方向数组；基于第一预设距离阈值，对第二边缘图像进行预设距离变换处理，得到第一距离图像；基于第一轮廓方向数组，计算第二边缘图像中各个目标物体边界所对应的第一边界方向图；基于第一距离图像和第一边界方向图，确定第一特征子集。
在提取第一图像的图像特征时，提取第一图像中各个目标物体的所有边界像素点 获得第一边缘图像，是指提取第一图像中的主要边缘，边缘指的是图像中至少一个物体的边界像素点，本发明对提取边缘的算法不做具体限定，例如，基于梯度差值检测，基于深度差值检测和基于深度学习边缘检测方法。
通过对第一边缘图像提取轮廓，得到第一轮廓数组。第一轮廓数组以多维数组记载轮廓划分信息和每个轮廓包含像素的位置关系，故基于第一轮廓数组可计算每个像素点对应的像素点方向，得到第一轮廓方向数组。基于第一轮廓方向数组，计算第二边缘图像中各个目标物体边界所对应的第一边界方向图可以是指：第一轮廓方向数组记载轮廓划分信息和每个轮廓包含像素点的方向值，将第一轮廓方向数组包含的信息映射至第二图像边缘图像中，计算第二边缘图像中各个目标物体边界所对应的方向，并保存为第一边界方向图。
在本发明实施例中，基于第一预设距离阈值，对第二边缘图像进行预设距离变换处理，得到第一距离图像的步骤，包括：基于第一预设距离阈值，确定第一掩膜信息，其中，第一掩膜信息设置为屏蔽第二图像中的部分边缘信息；将第一掩膜信息添加至第一特征子集。
基于第一预设距离阈值，对第二边缘图像进行欧氏距离的距离变换(distance transform)，获得第一距离图像，同时，还可以产生后续去除第二图像/query图像中多余边缘信息的掩膜，掩膜信息设置为对第二图像或者匹配图像上进行区域筛选与屏蔽，使其不参加处理。通过使用简单轮廓图像距离变换得到的掩膜信息对轮廓复杂的第二图像进行处理，可去除多余的轮廓，降低图像处理的计算量。
例如，提取第一图像/template图像特征包括：
步骤1：提取第一图像主要边缘，将获取边缘图像记为C1，其中边缘像素位置灰度值为255，非边缘像素位置灰度值为0；
步骤2：对边缘图像C1反色处理，获得边缘图像E1，其中边缘像素位置灰度值为0，非边缘像素位置灰度值为255；
步骤3：提取C1中的轮廓，记做V1；
步骤4：基于预设距离阈值，对边缘图像E1进行欧氏距离的距离变换(distance transform)，获得距离图像DT1以及后续去除第二图像/query图像中多余边缘信息的掩膜；
Step5：计算边缘图像E1中边缘的方向图OM1。首先计算轮廓V1上像素点的方向，记做O1；基于O1映射至边缘图像E1计算轮廓的方向，并保存为方向图OM1。
然后，本发明实施例对第二图像的图像特征进行提取，得到第二特征子集。
可选的，提取第二图像的图像特征，得到第二特征子集的步骤，包括：提取第二图像中各个目标物体的所有边界像素点，得到第三边缘图像；采用第一掩膜信息对第三边缘图像中的轮廓进行删减处理；对删减处理后的第三边缘图像进行反色处理，得到第四边缘图像；对第四边缘图像提取轮廓，得到第二轮廓数组，基于第二轮廓数组计算每个像素点对应的像素点方向，得到第二轮廓方向数组；基于第二预设距离阈值，对第四边缘图像进行预设距离变换处理，得到第二距离图像；基于第二轮廓方向数组，计算第四边缘图像中各个目标物体边界所对应的第二边界方向图；基于第二距离图像和第二边界方向图，得到第二特征子集。
提取第二图像中各物体的主要边缘，得到第三边缘图像，然后利用掩膜信息对第三边缘图像中的边缘进行删减，然后对处理后的边缘图像反色处理，获得第四边缘图像或者主要边缘图像。通过使用简单轮廓图像距离变换得到的掩膜信息对轮廓复杂的第二图像进行处理，可去除多余的轮廓，降低图像处理的计算量。
例如，提取第二图像/匹配图像特征，包括：
Step1：提取图像主要边缘，将获取边缘图像记为C2；
Step2：利用上述计算得到的掩膜信息对C2中的边缘进行删减，然后对处理后的边缘图像反色处理，获得图像主要边缘图像E2其中边缘像素位置灰度值为0，非边缘像素位置灰度值为255；
Step3：提取图像C2中的轮廓，记做V2；
Step4：对边缘图像E2进行欧氏距离的距离变换(distance transform)，获得距离图像DT2；设置距离阈值，产生实际参与计算的DT2。
Step5：计算边缘图像E2中边缘的方向图OM2。首先计算轮廓V2上像素点的方向，记做O2；基于O2映射至边缘图像E2计算轮廓的方向，并保存为方向图OM2。
在完成第一图像和第二图像的特征提取操作后，可以计算图像目标平移参数和目标缩放系数。
可选的，基于第一特征子集和第二特征子集，计算第一图像相对于第二图像的目标平移参数的步骤，包括：采用第一判断条件，提取第一距离图像与第二距离图像中像素距离小于第一距离阈值的轮廓像素点，得到参与图像匹配的第一轮廓像素点集合；采用第二判断条件，提取第一边界方向图与第二边界方向图中像素距离小于第二距离阈值的轮廓像素点，得到参与图像匹配的第二轮廓像素点集合；基于第一轮廓像素点 集合和第二轮廓像素点集合，确定第一图像和第二图像之间的倒角距离得分、方向图距离和图像调节因子，其中，图像调节因子设置为调节倒角距离得分和方向图距离比重；在第一图像上滑动第二图像，并将倒角距离得分、方向图距离和图像调节因子输入至第一预设公式，计算图像滑动得分；确定所有图像滑动得分中分值最小所对应的目标滑动位置；基于目标滑动位置，确定目标平移参数。
例如，计算目标平移参数时，包括：
Step1:提取参与图像匹配的轮廓像素点位置，判断条件如下：
||DT1(i，j)-DT2(i，j)||<th1并且||OM1(i，j)-OM2(i，j)||<th2；
其中，th1和th2分别是两个预设距离阈值，||DT1(i，j)-DT2(i，j)||为某像素点在第一距离图像与第二距离图像中像素距离，||OM1(i，j)-OM2(i，j)||为某像素点第一边界方向图与第二边界方向图中像素距离。
对于不满足条件的像素位置则不会参与倒角距离得分的计算，减少计算量。
Step2:计算倒角距离得分。通过在第二图像上滑动第一图像，计算每一个滑动位置均可获得一个图像滑动得分，其计算的第一预设公式如下：
score＝EDT+s*ODT；其中，score是某一个滑动位置的图像滑动得分，EDT是第一距离图像和第二距离图像的倒角距离；ODT是第一边界方向图和第二边界方向图之间的距离，方向图距离；s是一个调节EDT和ODT比重的图像调节因子。
Step3:选取x和y方向平移值。确定所有图像滑动得分中分值最小所对应的目标滑动位置，该位置对应的平移量为对齐所需目标平移参数，即x和y方向平移值:dx和dy。
本发明实施例中，对基础图像对进行校正处理时，并未限定缩放与平移的先后执行顺序，可以先计算平移校正量和缩放校正量，然后基于校正量实现对齐，也可以先计算平移校正量并完成平移校正，再在此基础上实现缩放校正。
下面分别说明几种确定第一图像与第二图像之间的目标平移参数和目标缩放系数的实施方式。
第一种，确定第一图像与第二图像之间的目标平移参数和目标缩放系数的步骤，包括：计算第一图像相对于第二图像的目标平移参数，并基于目标平移参数平移第一图像，获得第三图像；选取多个缩放系数，并以每个缩放系数分别缩放第三图像，计算第三图像与第二图像之间的图像匹配得分；将多个图像匹配得分中分值最小所对应 的缩放系数作为目标缩放系数。
通过先确定目标平移参数，采用目标平移参数平移第一图像，获得第三图像；然后对第三图像进行缩放处理，调整缩放系数并计算该缩放系数下第三图像与第二图像之间的图像匹配得分，通过选取图像匹配得分分值最小对应的缩放系数作为目标缩放系数，采用目标缩放系数缩放处理第三图像，实现两张图像之间的校正处理。
上述目标缩放系数包括但不限于：缩放宽度系数、缩放长度系数、缩放比例系数等。
第二种，确定第一图像与第二图像之间的目标平移参数和目标缩放系数的步骤，包括：计算第一图像相对于第二图像的目标平移参数，并基于目标平移参数平移第一图像，获得第四图像；选取多个缩放系数，并以每个缩放系数分别缩放第四图像，计算第四图像与第二图像之间的图像匹配得分；基于图像匹配得分，调整缩放系数直至图像匹配得分中分数变化小于第一阈值，则图像匹配得分所对应的缩放系数作为目标缩放系数。
通过先确定平移参数，采用目标平移参数平移第一图像，获得第四图像；然后对第四图像进行缩放处理，调整缩放系数直至该缩放系数下第四图像与第二图像之间的图像匹配得分中分数变化小于第一阈值，将该图像匹配得分所对应的缩放系数作为目标缩放系数。采用目标缩放系数缩放处理第一图像，实现两张图像之间的校正处理。
第三种，确定第一图像与第二图像之间的目标平移参数和目标缩放系数的步骤，包括：选取多个缩放系数，并以每个缩放系数分别缩放第一图像；对基于每个缩放系数缩放后的第一图像，在第二图像上滑动，并计算其与第二图像之间的图像匹配的得分；将多个图像匹配得分中分值最小所对应的缩放系数和平移量作为目标缩放系数和目标平移参数。
在选取缩放系数时，可以对基于每个缩放系数缩放后的第一图像，在第二图像上滑动，计算其与第二图像之间的图像匹配的得分，将图像匹配得分中分值最小所对应的缩放系数和平移量作为目标缩放系数和目标平移参数，以实现两张图像之间的校正处理。
步骤S106，基于每个校正参数对上述基础图像对进行对齐校正，得到目标图像对。
具体地，根据上述步骤获取的多个校正参数，包括目标平移参数和目标缩放系数，对上述基础图像对进行对齐校正，得到满足对齐需求的目标图像对。
在本发明实施例中，采用预设校正模式对基础图像对进行校正处理，得到多个校 正参数时，可以将基础图像对缩放至预设分辨率下，并进行金字塔化校正处理，得到多个校正参数。
尽管倒角距离匹配是在图像边缘特征提取的基础上进行匹配，其仍需遍历边缘像素位置，导致运算量巨大。若需要对齐的图像本身分辨率较高，边缘信息复杂，则进一步会影响图像对齐的实时性，故针对高分辨率基础图像对进行对齐校正时，需要采用多分辨率动态矫正的方法从粗到细进行对齐。
本发明实施例，在校正图像过程中，采用先下采样到低分辨率，再逐层上采样进行动态校正，这样做的方式，是金字塔化算法，可以降低计算时间，在最低分辨率上运行时间最小，找到初步粗糙结果，根据最低分辨率上的结果进行微调计算，不需要全部重新计算。若精度要求低，在低分辨率上进行校正处理即可满足对齐的精度要求；若精度要求高，在低分辨率上进行校正处理不能满足精度要求，则上采样至高分辨率上进行校正直至满足对齐精度要求。
作为本发明可选的实施方式，在基于每个校正参数对上述基础图像对进行对齐校正，得到目标图像对之后，图像校正方法还包括：确定终端应用在目标图像分辨率下的图像对齐需求精度；步骤S1，判断在第一图像分辨率下，与目标图像对所对应的当前对齐精度图像是否达到图像对齐需求精度；步骤S2，若确定与目标图像对所对应的当前对齐精度图像未达到图像对齐需求精度，调整图像分辨率为第二图像分辨率，其中，第二图像分辨率的分辨率数值高于第一图像分辨率；步骤S3，执行采用预设校正模式对基础图像对进行校正处理，得到多个校正参数的步骤；步骤S4，执行基于每个校正参数对上述基础图像对进行对齐校正，得到目标图像对的步骤；重复执行步骤S1至步骤S4，直至当前对齐精度图像达到图像对齐需求精度时结束。
例如，分别从可见光相机和深度相机获得一张可见光图像和一张深度图图像，根据预设标定参数将深度图变换到可见光图像坐标系下，经过调整以获得与可见光图像相同分辨率的初步对齐深度图。将初步对齐的图像对P1缩放到低分辨率p下进行动态校正，获得校正参数(dx_1,dy_1,scale_1)后对输入的图像对进行校正，得到新的图像对，记做对齐图像对P2；将校正获得对齐图像对在分辨率p*s2(s是一个放大系数，一般为2)下进行动态校正，获得校正参数(dx_2,dy_2,scale_2)后对输入的图像对P2进行校正，得到新的图像对，记做对齐图像对P3；继续提升校正过程中使用的分辨率，重复动态校正流程直到满足应用所需的对齐精度。
通过上述实施方式，提升了图像校正过程的精确度，本发明实施例中，可以在多种场景下，在VGA分辨率下，将误差从30个像素校正到4个像素以内，对齐精度非常高。
可选的，图像校正方法还包括：比较可见光图像的图像分辨率和深度图像的图像分辨率，得到分辨率最小的比较结果；基于比较结果得到的图像分辨率和初始设置的校正处理最大分辨率，计算校正次数阈值；在进行对齐校正过程中，若图像校正次数达到校正次数阈值，则停止校正处理。
初始对齐的分辨率以及需要动态校正次数可以根据输入图像的分辨率确定。例如，两个图像(深度图像和和可见光图像)之间分辨率较小的图像分辨率为Tw*Th，通常可见光图像分辨率远大于深度图分辨率。设置初始对齐的最大分辨率为Mw*Mh(通常Mw取320)，进而得到动态校正次数t的计算公式为：
Figure PCTCN2021141355-appb-000001
其中Sa＝Tw/Mw，其中，
公式中
Figure PCTCN2021141355-appb-000002
表示向上取整；s表示单次放大倍数，通常取2。由动态校正次数t可以得到初始值m 0＝Tw/s t-1；整个对齐过程分辨率金字塔可以用下面的公式表示：
m n＝Tw/s t-1-n，n＝0，1，2...t-1；
n n＝Th/s t-1-n，n＝0，1，2...t-1。
若某一低分辨率(上采样了t’次)上的结果已经满足精度要求，则可以选择不继续上采样，t为动态校正最大次数，优化的过程可以根据实际需要提前终止，采样次数t’<＝t。
通过上述实施例，可以根据可见光图像和深度图(或者可见光图像和红外图像)计算校正参数(或者是对齐参数)，不仅适用于只能提供可见光图像和红外图像、或者只能提供可见光图像和深度图像、或者可以提供可见光图，红外光图和深度图三种的含深度相机的设备；还可以适用于两个相机采集的画面内容相近但纹理差异较大的情况和无法根据特征点进行匹配的情况；同时，本发明实施例还可以改善相机由OIS、跌落、帧不同步和帧率不同等问题造成的双目设备采集图像的对齐误差的情况，且校正环境十分简单，无需特定的环境，特定的拍摄图案，即可快速完成图像校正处理，得到令用户满意的图像。
下面结合另一种可选的实施例来说明本发明。
实施例二
本发明实施例提供了一种图像校正装置，其包括的多个实施单元对应于上述实施例一中的各个实施步骤。
图4是根据本发明实施例的一种可选的图像校正装置的示意图，如图4所示，该 图像校正装置可以包括：获取单元41，第一校正单元43，第二校正单元45，其中，
获取单元41，设置为获取对目标对象拍摄的可见光图像和深度图像，经变换后形成基础图像对，其中，上述基础图像对包括第一图像和第二图像；
第一校正单元43，设置为采用预设校正模式对基础图像对进行校正处理，得到多个校正参数；
第二校正单元45，设置为基于每个校正参数对上述基础图像对进行对齐校正，得到目标图像对。
上述图像校正装置，可以通过获取单元41获取对目标对象拍摄的可见光图像和深度图像，经变换后形成基础图像对，其中，基础图像对包括第一图像和第二图像，通过第一校正单元45采用预设校正模式对基础图像对进行校正处理，得到多个校正参数，通过第二校正单元47基于每个校正参数对上述基础图像对进行对齐校正，得到目标图像对。在该实施例中，可以对多种相机拍摄图像进行对齐操作，实现动态校正，校正环境简单，利用设备实拍的图像就可以完成对齐校正，从而解决相关技术中无法实现两种不同相机间的动态校正，适应环境低，导致对齐图像效果较差，容易影响用户的使用兴趣的技术问题。
可选的，第一校正单元包括：第一校正模块，设置为将基础图像对缩放至预设分辨率下，并进行金字塔化校正处理，得到多个校正参数。
可选的，获取单元包括：第一变换模块，设置为基于预设标定参数，将深度图像变换到可见光图像的图像坐标系下，经过调整以得到与可见光图像具有相同分辨率的初步对齐深度图，其中，可见光图像与初步对齐深度图组合形成基础图像对，第一图像为可见光图像，第二图像为初步对齐深度图。
可选的，第一校正单元还包括：第一确定模块，设置为确定第一图像与第二图像之间的目标平移参数和目标缩放系数；第二确定模块，设置为基于目标平移参数和目标缩放系数，确定多个校正参数。
可选的，图像校正装置还包括：第一处理单元，设置为在采用预设校正模式对基础图像对进行校正处理，得到多个校正参数之前，对基础图像对中的初步对齐深度图进行预处理，得到第一图像；第二处理单元，设置为对基础图像对中可见光图像进行滤波处理，得到第二图像。
可选的，第一确定模块包括：第一计算模块，设置为计算第一图像相对于第二图像的目标平移参数，并基于目标平移参数平移第一图像，获得第三图像；第一缩放模 块，设置为选取多个缩放系数，并以每个缩放系数分别缩放第三图像，计算第三图像与第二图像之间的图像匹配得分；第二确定模块，设置为将多个图像匹配得分中分值最小所对应的缩放系数作为目标缩放系数。
可选的，第一确定模块还包括：第二计算模块，设置为计算第一图像相对于第二图像的目标平移参数，并基于目标平移参数平移第一图像，获得第四图像；第二缩放模块，设置为选取多个缩放系数，并以每个缩放系数分别缩放第四图像，计算第四图像与第二图像之间的图像匹配得分；第三确定模块，设置为调整上述缩放系数直至上述图像匹配得分中分数变化小于第一阈值，则上述图像匹配得分所对应的缩放系数作为目标缩放系数。
可选的，第一确定模块还包括：第三缩放模块，设置为选取多个缩放系数，并以每个缩放系数分别缩放第一图像；第三计算模块，设置为对基于每个缩放系数缩放后的第一图像，在第二图像上滑动，并计算其与第二图像之间的图像匹配的得分；第四确定模块，设置为将多个图像匹配得分中分值最小所对应的缩放系数和平移量作为目标缩放系数和目标平移参数。
可选的，第一处理单元包括：第一映射模块，设置为将基础图像对中的初步对齐深度图中每个像素点的深度值映射至预设像素范围内；和/或，第一调整模块，设置为调整初步对齐深度图的图像对比度，得到第一图像。
可选的，第一确定模块还包括：第一提取模块，设置为提取第一图像的图像特征，得到第一特征子集，其中，上述第一特征子集包括第一距离图像，第一边界方向图和第一掩膜信息；第二提取模块，设置为提取第二图像的图像特征，得到第二特征子集，其中，上述第二特征子集包括第二距离图像和第二边界方向图；第四计算模块，设置为基于上述第一特征子集和上述第二特征子集，计算第一图像相对于第二图像的目标平移参数。
可选的，第一提取模块包括：第一提取子模块，设置为提取第一图像中各个目标物体的所有边界像素点，得到第一边缘图像；第一反色子模块，设置为对第一边缘图像进行反色处理，得到第二边缘图像；第二提取子模块，设置为对第一边缘图像提取轮廓，得到第一轮廓数组，基于第一轮廓数组计算每个像素点对应的像素点方向，得到第一轮廓方向数组；第一变换子模块，设置为基于第一预设距离阈值，对第二边缘图像进行预设距离变换处理，得到第一距离图像；第一计算子模块，设置为基于第一轮廓方向数组，计算第二边缘图像中各个目标物体边界所对应的第一边界方向图；第一确定子模块，设置为基于第一距离图像和第一边界方向图，确定第一特征子集。
可选的，第一变换子模块包括：第二确定子模块，设置为基于第一预设距离阈值，确定第一掩膜信息，其中，第一掩膜信息设置为屏蔽第二图像中的部分边缘信息；添加子模块，设置为将第一掩膜信息添加至第一特征子集。
可选的，第二提取模块包括：第二提取子模块，设置为提取第二图像中各个目标物体的所有边界像素点，得到第三边缘图像；删减子模块，设置为采用第一掩膜信息对第三边缘图像中的轮廓进行删减处理；第二反色子模块，设置为对删减处理后的第三边缘图像进行反色处理，得到第四边缘图像；第二计算子模块，设置为对第四边缘图像提取轮廓，得到第二轮廓数组，基于第二轮廓数组计算每个像素点对应的像素点方向，得到第二轮廓方向数组；第二变换子模块，设置为基于第二预设距离阈值，对第四边缘图像进行预设距离变换处理，得到第二距离图像；第三计算子模块，设置为基于第二轮廓方向数组，计算第四边缘图像中各个目标物体边界所对应的第二边界方向图；第三确定子模块，设置为基于第二距离图像和第二边界方向图，得到第二特征子集。
可选的，第四计算模块包括：第三提取子模块，设置为采用第一判断条件，提取第一距离图像与第二距离图像中像素距离小于第一距离阈值的轮廓像素点，得到参与图像匹配的第一轮廓像素点集合；第四提取子模块，设置为采用第二判断条件，提取第一边界方向图与第二边界方向图中像素距离小于第二距离阈值的轮廓像素点，得到参与图像匹配的第二轮廓像素点集合；第五确定子模块，设置为基于第一轮廓像素点集合和第二轮廓像素点集合，确定第一图像和第二图像之间的倒角距离得分、方向图距离和图像调节因子，其中，图像调节因子设置为调节倒角距离得分和方向图距离比重；第四计算子模块，设置为在第一图像上滑动第二图像，并将倒角距离得分、方向图距离和图像调节因子输入至第一预设公式，计算图像滑动得分；第六确定子模块，设置为确定所有图像滑动得分中分值最小所对应的目标滑动位置；第七确定子模块，设置为基于目标滑动位置，确定目标平移参数。
可选的，第一校正模块包括：第一获取子模块，设置为获取终端应用的对齐精度值，基于对齐精度值和基础图像对的分辨率，确定多个校正分辨率，其中，多个校正分辨率至少包括：预设分辨率，预设分辨率为多个校正分辨率中最小分辨率；第一校正子模块，设置为将基础图像对缩放至预设分辨率下，并进行金字塔化校正处理直到满足的对齐精度值，得到多个校正参数。
可选的，图像校正装置还包括：确定单元，设置为确定终端应用在目标图像分辨率下的图像对齐需求精度；第一判断单元，设置为执行步骤S1，判断在第一图像分辨率下，与目标图像对所对应的当前对齐精度图像是否达到图像对齐需求精度；第一调 整单元，设置为执行步骤S2，若确定与目标图像对所对应的当前对齐精度图像未达到图像对齐需求精度，调整图像分辨率为第二图像分辨率，其中，第二图像分辨率的分辨率数值高于第一图像分辨率；第一执行单元，设置为执行步骤S3，执行采用预设校正模式对基础图像对进行校正处理，得到多个校正参数的步骤；第二执行单元，设置为执行步骤S4，执行基于每个校正参数对上述基础图像对进行对齐校正，得到目标图像对的步骤；重复执行步骤S1至步骤S4，直至当前对齐精度图像达到图像对齐需求精度时结束。
可选的，图像校正装置还包括：比较单元，设置为比较可见光图像的图像分辨率和深度图像的图像分辨率，得到分辨率最小的比较结果；计算单元，设置为基于比较结果得到的图像分辨率和初始设置的校正处理最大分辨率，计算校正次数阈值；停止单元，设置为在进行对齐校正过程中，若图像校正次数达到校正次数阈值，则停止校正处理。
上述的图像校正装置还可以包括处理器和存储器，上述获取单元41，第一校正单元43，第二校正单元45等均作为程序单元存储在存储器中，由处理器执行存储在存储器中的上述程序单元来实现相应的功能。
上述处理器中包含内核，由内核去存储器中调取相应的程序单元。内核可以设置一个或以上，通过调整内核参数来基于每个校正参数对上述基础图像对进行对齐校正，得到目标图像对。
上述存储器可能包括计算机可读介质中的非永久性存储器，随机存取存储器(RAM)和/或非易失性内存等形式，如只读存储器(ROM)或闪存(flash RAM)，存储器包括至少一个存储芯片。
根据本发明实施例的另一方面，还提供了一种图像校正系统，包括：第一图像捕获装置，设置为拍摄目标对象的可见光图像；第二图像捕获装置，设置为拍摄目标对象的深度图像；校正装置，设置为将获取对目标对象拍摄的可见光图像和深度图像，经变换后形成基础图像对，其中，上述基础图像对包括第一图像和第二图像，采用预设校正模式对基础图像对进行校正处理，得到多个校正参数，基于每个校正参数对上述基础图像对进行对齐校正，得到目标图像对；结果输出装置，设置为将对齐后的目标图像对输出至预设终端展示界面。
根据本发明实施例的另一方面，还提供了一种电子设备，包括：处理器；以及存储器，设置为存储处理器的可执行指令；其中，处理器配置为经由执行可执行指令来执行上述任意一项的图像校正方法。
根据本发明实施例的另一方面，还提供了一种计算机可读存储介质，计算机可读存储介质包括存储的计算机程序，其中，在计算机程序运行时控制计算机可读存储介质所在设备执行上述任意一项的图像校正方法。
本申请还提供了一种计算机程序产品，当在数据处理设备上执行时，适于执行初始化有如下方法步骤的程序：获取对目标对象拍摄的可见光图像和深度图像，经变换后形成基础图像对，其中，所述基础图像对包括第一图像和第二图像；采用预设校正模式对基础图像对进行校正处理，得到多个校正参数；基于每个校正参数对所述基础图像对进行对齐校正，得到目标图像对。
上述本发明实施例序号仅仅为了描述，不代表实施例的优劣。
在本发明的上述实施例中，对各个实施例的描述都各有侧重，某个实施例中没有详述的部分，可以参见其他实施例的相关描述。
在本申请所提供的几个实施例中，应该理解到，所揭露的技术内容，可通过其它的方式实现。其中，以上所描述的装置实施例仅仅是示意性的，例如上述单元的划分，可以为一种逻辑功能划分，实际实现时可以有另外的划分方式，例如多个单元或组件可以结合或者可以集成到另一个系数，或一些特征可以忽略，或不执行。另一点，所显示或讨论的相互之间的耦合或直接耦合或通信连接可以是通过一些接口，单元或模块的间接耦合或通信连接，可以是电性或其它的形式。
上述作为分离部件说明的单元可以是或者也可以不是物理上分开的，作为单元显示的部件可以是或者也可以不是物理单元，即可以位于一个地方，或者也可以分布到多个单元上。可以根据实际的需要选择其中的部分或者全部单元来实现本实施例方案的目的。
另外，在本发明各个实施例中的各功能单元可以集成在一个处理单元中，也可以是各个单元单独物理存在，也可以两个或两个以上单元集成在一个单元中。上述集成的单元既可以采用硬件的形式实现，也可以采用软件功能单元的形式实现。
上述集成的单元如果以软件功能单元的形式实现并作为独立的产品销售或使用时，可以存储在一个计算机可读取存储介质中。基于这样的理解，本发明的技术方案本质上或者说对现有技术做出贡献的部分或者该技术方案的全部或部分可以以软件产品的形式体现出来，该计算机软件产品存储在一个存储介质中，包括若干指令用以使得一台计算机设备(可为个人计算机、服务器或者网络设备等)执行本发明各个实施例上述方法的全部或部分步骤。而前述的存储介质包括：U盘、只读存储器(ROM，Read-Only Memory)、随机存取存储器(RAM，Random Access Memory)、移动硬盘、磁碟或者光盘 等各种可以存储程序代码的介质。
以上所述仅是本发明的优选实施方式，应当指出，对于本技术领域的普通技术人员来说，在不脱离本发明原理的前提下，还可以做出若干改进和润饰，这些改进和润饰也应视为本发明的保护范围。
工业实用性
本申请实施例提供的方案可以实现相机拍摄图像的自动校正，在本公开提供的技术方案中，可以应用于具有至少一个图像捕获单元的电子设备中，例如，适用于各类移动设备、移动平台、车载芯片、嵌入式芯片等，校正环境简单，无需特定的环境，特定的拍摄图案，只需要根据预设标定参数初步对齐的可见光图像和深度图像即可实现动态校正，当图像捕获装置的相对位置或自身参数改变时，需要对使用预设标定参数对齐的可见光图像和深度图像做进一步的校正处理，降低对齐误差，解决相关技术中无法实现两种不同相机间的动态校正，适应环境低，导致对齐图像效果较差，容易影响用户的使用兴趣的技术问题。

Claims (21)

  1. 一种图像校正方法，包括：
    获取对目标对象拍摄的可见光图像和深度图像，经变换后形成基础图像对，其中，所述基础图像对包括第一图像和第二图像；
    采用预设校正模式对所述基础图像对进行校正处理，得到多个校正参数；
    基于每个所述校正参数对所述基础图像对进行对齐校正，得到目标图像对。
  2. 根据权利要求1所述的图像校正方法，其中，采用预设校正模式对所述基础图像对进行校正处理，得到多个校正参数的步骤，包括：
    将所述基础图像对缩放至预设分辨率下，并进行金字塔化校正处理，得到所述多个校正参数。
  3. 根据权利要求1所述的图像校正方法，其中，获取对目标对象拍摄的可见光图像和深度图像，经变换后形成基础图像对的步骤，包括：
    基于预设标定参数，将所述深度图像变换到所述可见光图像的图像坐标系下，经过调整以得到与所述可见光图像具有相同分辨率的初步对齐深度图，其中，所述可见光图像与所述初步对齐深度图组合形成所述基础图像对，所述第一图像为所述可见光图像，所述第二图像为所述初步对齐深度图。
  4. 根据权利要求1所述的图像校正方法，其中，采用预设校正模式对所述基础图像对进行校正处理，得到多个校正参数的步骤，还包括：
    确定所述第一图像与所述第二图像之间的目标平移参数和目标缩放系数；
    基于所述目标平移参数和所述目标缩放系数，确定多个校正参数。
  5. 根据权利要求1所述的图像校正方法，其中，在采用预设校正模式对所述基础图像对进行校正处理，得到多个校正参数之前，所述图像校正方法还包括：
    对所述基础图像对中的初步对齐深度图进行预处理，得到所述第一图像；
    对所述基础图像对中的可见光图像进行滤波处理，得到所述第二图像。
  6. 根据权利要求4所述的图像校正方法，其中，确定所述第一图像与所述第二图像之间的目标平移参数和目标缩放系数的步骤，包括：
    计算所述第一图像相对于所述第二图像的所述目标平移参数，并基于所述目 标平移参数平移所述第一图像，获得第三图像；
    选取多个缩放系数，并以每个所述缩放系数分别缩放所述第三图像，计算所述第三图像与所述第二图像之间的图像匹配得分；
    将多个所述图像匹配得分中分值最小所对应的缩放系数作为目标缩放系数。
  7. 根据权利要求4所述的图像校正方法，其中，确定所述第一图像与所述第二图像之间的目标平移参数和目标缩放系数的步骤，包括：
    计算所述第一图像相对于所述第二图像的所述目标平移参数，并基于所述目标平移参数平移所述第一图像，获得第四图像；
    选取多个缩放系数，并以每个所述缩放系数分别缩放所述第四图像，计算所述第四图像与所述第二图像之间的图像匹配得分；
    调整所述缩放系数直至所述图像匹配得分中分数变化小于第一阈值，则所述图像匹配得分所对应的缩放系数作为目标缩放系数。
  8. 根据权利要求4所述的图像校正方法，其中，确定所述第一图像与所述第二图像之间的目标平移参数和目标缩放系数的步骤，包括：
    选取多个缩放系数，并以每个所述缩放系数分别缩放所述第一图像；
    对基于每个所述缩放系数缩放后的所述第一图像，在所述第二图像上滑动，并计算其与所述第二图像之间的图像匹配的得分；
    将多个所述图像匹配得分中分值最小所对应的缩放系数和平移量作为所述目标缩放系数和所述目标平移参数。
  9. 根据权利要求5所述的图像校正方法，其中，对所述基础图像对中的初步对齐深度图进行预处理，得到所述第一图像的步骤，包括：
    将所述基础图像对中的初步对齐深度图中每个像素点的深度值映射至预设像素范围内；和/或，
    调整所述初步对齐深度图的图像对比度，得到所述第一图像。
  10. 根据权利要求4所述的图像校正方法，其中，确定所述第一图像与所述第二图像之间的目标平移参数和目标缩放系数的步骤，包括：
    提取所述第一图像的图像特征，得到第一特征子集，其中，所述第一特征子集包括第一距离图像，第一边界方向图和第一掩膜信息；
    提取所述第二图像的图像特征，得到第二特征子集，其中，所述第二特征子集包括第二距离图像和第二边界方向图；
    基于所述第一特征子集和所述第二特征子集，计算所述第一图像相对于所述第二图像的目标平移参数。
  11. 根据权利要求10所述的图像校正方法，其中，提取所述第一图像的图像特征，得到第一特征子集的步骤，包括：
    提取所述第一图像中各个目标物体的所有边界像素点，得到第一边缘图像；
    对所述第一边缘图像进行反色处理，得到第二边缘图像；
    对所述第一边缘图像提取轮廓，得到第一轮廓数组，基于所述第一轮廓数组计算每个像素点对应的像素点方向，得到第一轮廓方向数组；
    基于第一预设距离阈值，对所述第二边缘图像进行预设距离变换处理，得到所述第一距离图像；
    基于所述第一轮廓方向数组，计算所述第二边缘图像中各个目标物体边界所对应的所述第一边界方向图；
    基于所述第一距离图像和所述第一边界方向图，确定所述第一特征子集。
  12. 根据权利要求11所述的图像校正方法，其中，基于第一预设距离阈值，对所述第二边缘图像进行预设距离变换处理，得到所述第一距离图像的步骤，包括：
    基于所述第一预设距离阈值，确定所述第一掩膜信息，其中，所述第一掩膜信息设置为屏蔽所述第二图像中的部分边缘信息；
    将所述第一掩膜信息添加至所述第一特征子集。
  13. 根据权利要求12所述的图像校正方法，其中，提取所述第二图像的图像特征，得到第二特征子集的步骤，包括：
    提取所述第二图像中各个目标物体的所有边界像素点，得到第三边缘图像；
    采用所述第一掩膜信息对所述第三边缘图像中的轮廓进行删减处理；
    对删减处理后的所述第三边缘图像进行反色处理，得到第四边缘图像；
    对所述第四边缘图像提取轮廓，得到第二轮廓数组，基于所述第二轮廓数组计算每个像素点对应的像素点方向，得到第二轮廓方向数组；
    基于第二预设距离阈值，对所述第四边缘图像进行预设距离变换处理，得到所述第二距离图像；
    基于所述第二轮廓方向数组，计算所述第四边缘图像中各个目标物体边界所对应的所述第二边界方向图；
    基于所述第二距离图像和所述第二边界方向图，得到所述第二特征子集。
  14. 根据权利要求10所述的图像校正方法，其中，基于所述第一特征子集和所述第二特征子集，计算所述第一图像相对于所述第二图像的目标平移参数的步骤，包括：
    采用第一判断条件，提取所述第一距离图像与所述第二距离图像中像素距离小于第一距离阈值的轮廓像素点，得到参与图像匹配的第一轮廓像素点集合；
    采用第二判断条件，提取所述第一边界方向图与所述第二边界方向图中像素距离小于第二距离阈值的轮廓像素点，得到参与图像匹配的第二轮廓像素点集合；
    基于所述第一轮廓像素点集合和所述第二轮廓像素点集合，确定所述第一图像和所述第二图像之间的倒角距离得分、方向图距离和图像调节因子，其中，所述图像调节因子设置为调节所述倒角距离得分和所述方向图距离比重；
    在所述第一图像上滑动所述第二图像，并将所述倒角距离得分、所述方向图距离和所述图像调节因子输入至第一预设公式，计算图像滑动得分；
    确定所有图像滑动得分中分值最小所对应的目标滑动位置；
    基于所述目标滑动位置，确定目标平移参数。
  15. 根据权利要求2所述的图像校正方法，其中，将所述基础图像对缩放至预设分辨率下，并进行金字塔化校正处理，得到所述多个校正参数的步骤，包括：
    获取终端应用的对齐精度值，基于所述对齐精度值和所述基础图像对的分辨率，确定多个校正分辨率，其中，所述多个校正分辨率至少包括：预设分辨率，所述预设分辨率为多个校正分辨率中最小分辨率；
    将所述基础图像对缩放至所述预设分辨率下，并进行金字塔化校正处理直到满足所述的对齐精度值，得到所述多个校正参数。
  16. 根据权利要求15所述的图像校正方法，其中，所述图像校正方法还包括：
    确定终端应用在目标图像分辨率下的图像对齐需求精度；
    步骤S1，判断在所述第一图像分辨率下，与所述目标图像对所对应的当前对 齐精度图像是否达到所述图像对齐需求精度；
    步骤S2，若确定与所述目标图像对所对应的当前对齐精度图像未达到所述图像对齐需求精度，调整图像分辨率为第二图像分辨率，其中，所述第二图像分辨率的分辨率数值高于所述第一图像分辨率；
    步骤S3，执行采用预设校正模式对所述基础图像对进行校正处理，得到多个校正参数的步骤；
    步骤S4，执行基于每个所述校正参数对所述基础图像对进行对齐校正，得到目标图像对的步骤；
    重复执行步骤S1至步骤S4，直至当前对齐精度图像达到所述图像对齐需求精度时结束。
  17. 根据权利要求1所述的图像校正方法，其中，所述图像校正方法还包括：
    比较所述可见光图像的图像分辨率和所述深度图像的图像分辨率，得到分辨率最小的比较结果；
    基于比较结果得到的图像分辨率和初始设置的校正处理最大分辨率，计算校正次数阈值；
    在进行对齐校正过程中，若图像校正次数达到所述校正次数阈值，则停止校正处理。
  18. 一种图像校正装置，包括：
    获取单元，设置为获取对目标对象拍摄的可见光图像和深度图像，经变化后形成基础图像对，其中，所述基础图像对包括第一图像和第二图像；
    第一校正单元，设置为采用预设校正模式对所述基础图像对进行校正处理，得到多个校正参数；
    第二校正单元，设置为基于每个所述校正参数对所述基础图像对进行对齐校正，得到目标图像对。
  19. 一种图像校正系统，包括：
    第一图像捕获装置，设置为拍摄目标对象的可见光图像；
    第二图像捕获装置，设置为拍摄目标对象的深度图像；
    校正装置，设置为获取对目标对象拍摄的可见光图像和深度图像，经变化后 形成基础图像对，其中，所述基础图像对包括第一图像和第二图像；采用预设校正模式对所述基础图像对进行校正处理，得到多个校正参数；基于每个所述校正参数对所述基础图像对进行对齐校正，得到目标图像对；
    结果输出装置，设置为将对齐后的目标图像对输出至预设终端展示界面。
  20. 一种电子设备，包括：
    处理器；以及
    存储器，设置为存储所述处理器的可执行指令；
    其中，所述处理器配置为经由执行所述可执行指令来执行权利要求1至17中任意一项所述的图像校正方法。
  21. 一种计算机可读存储介质，所述计算机可读存储介质包括存储的计算机程序，其中，在所述计算机程序运行时控制所述计算机可读存储介质所在设备执行权利要求1至17中任意一项所述的图像校正方法。
PCT/CN2021/141355 2020-12-25 2021-12-24 图像校正方法、装置及系统、电子设备 WO2022135588A1 (zh)

Priority Applications (1)

Application Number Priority Date Filing Date Title
KR1020237021758A KR20230110618A (ko) 2020-12-25 2021-12-24 영상 보정 방법, 장치 및 시스템, 전자 장치

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
CN202011567624.3A CN114693760A (zh) 2020-12-25 2020-12-25 图像校正方法、装置及系统、电子设备
CN202011567624.3 2020-12-25

Publications (1)

Publication Number Publication Date
WO2022135588A1 true WO2022135588A1 (zh) 2022-06-30

Family

ID=82130825

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2021/141355 WO2022135588A1 (zh) 2020-12-25 2021-12-24 图像校正方法、装置及系统、电子设备

Country Status (3)

Country Link
KR (1) KR20230110618A (zh)
CN (1) CN114693760A (zh)
WO (1) WO2022135588A1 (zh)

Cited By (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN115293971A (zh) * 2022-09-16 2022-11-04 荣耀终端有限公司 图像拼接方法及装置
CN115861088A (zh) * 2022-10-20 2023-03-28 国科天成科技股份有限公司 红外摄像头的非均匀性漂移实时校正方法和系统
CN117670880A (zh) * 2024-01-31 2024-03-08 中成空间（深圳）智能技术有限公司 一种柔性光伏电池的检测校正方法及系统
CN117689813A (zh) * 2023-12-08 2024-03-12 华北电力大学（保定） 一种变电站高精度电力变压器红外三维建模方法及系统
CN118015677A (zh) * 2024-01-09 2024-05-10 深圳市中研安创科技发展有限公司 一种用于手持式人脸识别终端的抖动修复系统

Families Citing this family (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN114842003B (zh) * 2022-07-04 2022-11-01 杭州健培科技有限公司 一种医学影像随访目标配对方法、装置及应用
CN117994121A (zh) * 2022-10-28 2024-05-07 华为技术有限公司 一种图像处理方法及电子设备

Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20180040107A1 (en) * 2015-04-23 2018-02-08 Fujifilm Corporation Image processing device, imaging device, image processing method, and program
CN109035193A (zh) * 2018-08-29 2018-12-18 成都臻识科技发展有限公司 一种基于双目立体相机的成像处理方法及成像处理系统
CN111741281A (zh) * 2020-06-30 2020-10-02 Oppo广东移动通信有限公司 图像处理方法、终端及存储介质
CN111757086A (zh) * 2019-03-28 2020-10-09 杭州海康威视数字技术股份有限公司 有源双目相机、rgb-d图像确定方法及装置

Patent Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20180040107A1 (en) * 2015-04-23 2018-02-08 Fujifilm Corporation Image processing device, imaging device, image processing method, and program
CN109035193A (zh) * 2018-08-29 2018-12-18 成都臻识科技发展有限公司 一种基于双目立体相机的成像处理方法及成像处理系统
CN111757086A (zh) * 2019-03-28 2020-10-09 杭州海康威视数字技术股份有限公司 有源双目相机、rgb-d图像确定方法及装置
CN111741281A (zh) * 2020-06-30 2020-10-02 Oppo广东移动通信有限公司 图像处理方法、终端及存储介质

Cited By (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN115293971A (zh) * 2022-09-16 2022-11-04 荣耀终端有限公司 图像拼接方法及装置
CN115861088A (zh) * 2022-10-20 2023-03-28 国科天成科技股份有限公司 红外摄像头的非均匀性漂移实时校正方法和系统
CN117689813A (zh) * 2023-12-08 2024-03-12 华北电力大学（保定） 一种变电站高精度电力变压器红外三维建模方法及系统
CN118015677A (zh) * 2024-01-09 2024-05-10 深圳市中研安创科技发展有限公司 一种用于手持式人脸识别终端的抖动修复系统
CN117670880A (zh) * 2024-01-31 2024-03-08 中成空间（深圳）智能技术有限公司 一种柔性光伏电池的检测校正方法及系统
CN117670880B (zh) * 2024-01-31 2024-05-07 中成空间（深圳）智能技术有限公司 一种柔性光伏电池的检测校正方法及系统

Also Published As

Publication number Publication date
KR20230110618A (ko) 2023-07-24
CN114693760A (zh) 2022-07-01

Similar Documents

Publication Publication Date Title
WO2022135588A1 (zh) 图像校正方法、装置及系统、电子设备
CN109410207B (zh) 一种基于ncc特征的无人机巡线图像输电线路检测方法
CN111260543B (zh) 一种基于多尺度图像融合和sift特征的水下图像拼接方法
WO2019042216A1 (zh) 图像虚化处理方法、装置及拍摄终端
WO2019105261A1 (zh) 背景虚化处理方法、装置及设备
CN108537782B (zh) 一种基于轮廓提取的建筑物图像匹配与融合的方法
EP3158532A1 (en) Local adaptive histogram equalization
WO2020119467A1 (zh) 高精度稠密深度图像的生成方法和装置
CN108765476B (zh) 一种偏振图像配准方法
CN111160232B (zh) 正面人脸重建方法、装置及系统
CN113822942B (zh) 一种基于二维码的单目摄像头测量物体尺寸的方法
CN110956661A (zh) 基于双向单应矩阵的可见光与红外相机动态位姿计算方法
CN107895375A (zh) 基于视觉多特征的复杂道路线提取方法
CN112233189B (zh) 多深度相机外部参数标定方法、装置及存储介质
CN109559353A (zh) 摄像模组标定方法、装置、电子设备及计算机可读存储介质
CN111382722A (zh) 车牌图像优选方法、图像处理装置及具有存储功能的装置
CN111739071A (zh) 基于初始值的快速迭代配准方法、介质、终端和装置
EP4266250A1 (en) Image processing method and chip, and electronic device
JP2020095621A (ja) 画像処理装置および画像処理方法
CN117058183A (zh) 一种基于双摄像头的图像处理方法、装置、电子设备及存储介质
CN110910457B (zh) 基于角点特征的多光谱立体相机外参计算方法
CN117152330A (zh) 一种基于深度学习的点云3d模型贴图方法和装置
WO2021044186A1 (en) Multi-view image fusion by image space equalization and stereo-based rectification from two different cameras
CN113808185B (zh) 图像深度恢复方法、电子设备及存储介质
CN115661258A (zh) 标定方法及装置、畸变校正方法及装置、存储介质、终端

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 21909583

Country of ref document: EP

Kind code of ref document: A1

ENP Entry into the national phase

Ref document number: 20237021758

Country of ref document: KR

Kind code of ref document: A

NENP Non-entry into the national phase

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 21909583

Country of ref document: EP

Kind code of ref document: A1