WO2020043155A1 - 图像的多尺度融合方法、装置、存储介质及终端 - Google Patents

图像的多尺度融合方法、装置、存储介质及终端 Download PDF

Info

Publication number
WO2020043155A1
WO2020043155A1 PCT/CN2019/103226 CN2019103226W WO2020043155A1 WO 2020043155 A1 WO2020043155 A1 WO 2020043155A1 CN 2019103226 W CN2019103226 W CN 2019103226W WO 2020043155 A1 WO2020043155 A1 WO 2020043155A1
Authority
WO
WIPO (PCT)
Prior art keywords
image
matching
telephoto
reference block
points
Prior art date
Application number
PCT/CN2019/103226
Other languages
English (en)
French (fr)
Inventor
方璐
戴琼海
刘烨斌
Original Assignee
清华-伯克利深圳学院筹备办公室
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by 清华-伯克利深圳学院筹备办公室 filed Critical 清华-伯克利深圳学院筹备办公室
Publication of WO2020043155A1 publication Critical patent/WO2020043155A1/zh

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T5/00—Image enhancement or restoration
    • G06T5/50—Image enhancement or restoration using two or more images, e.g. averaging or subtraction
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/30—Determination of transform parameters for the alignment of images, i.e. image registration
    • G06T7/33—Determination of transform parameters for the alignment of images, i.e. image registration using feature-based methods
    • G06T7/344—Determination of transform parameters for the alignment of images, i.e. image registration using feature-based methods involving models

Definitions

  • Embodiments of the present application relate to the field of image processing technologies, for example, to a method, a device, a storage medium, and a terminal for multi-scale image fusion.
  • the gigapixel image technology uses a short-focus camera and multiple telephoto cameras to record the scene image at the same time, and then replaces the image obtained by the telephoto camera with the corresponding position of the short-focus camera, so that the range can be obtained.
  • the effect of large and clear images requires technical means of positioning and image combination.
  • not only high requirements are imposed on the hardware, but also a great challenge on the software side.
  • computing efficiency and robustness have become pain points in the industry.
  • Embodiments of the present application provide a multi-scale image fusion method, device, storage medium, and terminal, which can be integrated into an unstructured camera array to reduce hardware requirements and improve image fusion speed and robustness.
  • an embodiment of the present application provides an image multi-scale fusion method, including: acquiring a short-focus image and at least one tele-focus image; and using a block matching algorithm to determine a short-focus reference block; wherein, the The short-focus reference block is obtained by performing image block matching and structural edge block matching on the short-focus image and the tele-focus image; taking 4 pairs of matching points from the tele-focus image and the short-focus reference block.
  • an embodiment of the present application further provides an image fusion apparatus, the apparatus includes: an image acquisition module configured to acquire a short-focus image and at least one telephoto image; a short-focus reference block determination module configured to Use a block matching algorithm to determine a short focus reference block; wherein the short focus reference block is obtained by performing image block matching and structural edge block matching on the short focus image and the telephoto image; coarsely aligned telephoto An image determination module configured to take 4 pairs of matching points from the telephoto image and the short focus reference block, and determine a homography matrix from the telephoto image to the short focus reference block; according to the homography The matrix is used to obtain a coarsely aligned telephoto image.
  • the image fusion module is configured to use the key points of the short focus reference block to fuse the grid control points of the coarsely aligned telephoto image to obtain a fused image.
  • an embodiment of the present application provides a computer-readable storage medium on which a computer program is stored.
  • the program is executed by a processor, a multi-scale image fusion method according to the embodiment of the present application is implemented.
  • an embodiment of the present application provides a terminal, including a memory, a processor, and a computer program stored in the memory and executable by the processor.
  • the processor executes the computer program, the implementation is implemented as in the embodiment of the present application.
  • the image multi-scale fusion method is implemented as in the embodiment of the present application.
  • FIG. 1 is a flowchart of an image multi-scale fusion method according to an embodiment of the present application
  • FIG. 2 is a flowchart of an image multi-scale fusion method according to another embodiment of the present application.
  • FIG. 3 is a flowchart of a multi-scale image fusion method according to another embodiment of the present application.
  • FIG. 4 is a flowchart of an image multi-scale fusion method according to another embodiment of the present application.
  • FIG. 5 is a schematic structural diagram of an image fusion apparatus according to an embodiment of the present application.
  • FIG. 6 is a schematic structural diagram of a terminal according to an embodiment of the present application.
  • FIG. 1 is a flowchart of a multi-scale image fusion method according to an embodiment of the present application. This embodiment is applicable to the situation of image or video acquisition.
  • the method may be performed by an image fusion device provided in the embodiment of the present application.
  • the device may be implemented by software and / or hardware, and may be integrated into an unstructured camera.
  • Array image acquisition terminal may be implemented by software and / or hardware, and may be integrated into an unstructured camera.
  • the multi-scale image fusion method includes steps S120 to S140.
  • step S110 one short-focus image and at least one long-focus image are acquired.
  • the short-focus image and the long-focus image may be relative.
  • the image obtained by the camera with the shortest focal length may be used as the short-focus image, and the images obtained by other cameras are used as A telephoto image.
  • the obtained short-focus image is one
  • the telephoto image is multiple
  • the telephoto image can be at least one.
  • the short-focus image and the telephoto image can also be defined according to the focal length of the camera. For example, if the focal length of the camera is 4-16mm, the acquired image is used as the short-focus image, and the focal length of the camera is 25-135mm. Even larger, the acquired image is used as a telephoto image.
  • the technical solution provided in this embodiment is to fuse a telephoto image and a short focus image to obtain a high-resolution image, only one short focus image is required, so only one can be set in a group of camera arrays. Short-focus cameras, and multiple telephoto cameras. In this way, by integrating the telephoto images obtained by all the telephoto cameras into the short focus camera, an image with pixels higher than that captured by a single camera can be obtained, and finally a high pixel image or video is formed.
  • a short-focus reference block is determined by using a block matching algorithm; wherein the short-focus reference block is obtained by performing image block matching and structural edge block matching on the short focus image and the telephoto image. .
  • the block matching algorithm can be a zero-mean normalized template cross-correlation algorithm (ZNCC).
  • ZNCC zero-mean normalized template cross-correlation algorithm
  • the use of the ZNCC algorithm has more stable performance and strong anti-interference ability.
  • the short focus reference block may be a reference tile corresponding to the long focus image in the short focus image.
  • the short focus reference block is obtained by performing image block matching and structural edge block matching on the short focus image and the telephoto image. These include the results obtained by performing block matching on the short focus image and the telephoto image, and the results obtained by performing block matching on the structural edge map of the short focus image and the structural edge map of the telephoto image.
  • This setting in this embodiment can more accurately determine the position of the telephoto image in the short-focus image.
  • a poster with a length of 10 meters and a width of 8 meters, in a group of camera arrays at a certain distance the result obtained by the short-focus camera may be a tenth of the range of the image, and The result captured by a telephoto camera may be the full range of the image.
  • the specific position of the image in the telephoto image in the short focus image can be determined, and then the short focus reference block is determined.
  • the short focus reference block is determined.
  • a more accurate short focus reference block can be obtained, which can improve the accuracy of the image summation of this solution and has strong robustness.
  • step S130 four pairs of matching points are taken from the telephoto image and the short focus reference block to determine a homography matrix from the telephoto image to the short focus reference block; according to the homography Matrix to get a coarsely aligned telephoto image.
  • the homography matrix is the image of the same object in two images, which show different images due to different viewing distances and perspectives.
  • an object can obtain two different photos by rotating the camera lens (the content of the two photos does not have to correspond exactly, only a partial correspondence is required), we can set the homography to a two-dimensional matrix M, then the photo 1 times M is the photo 2.
  • This has many practical applications, such as image correction, image alignment, or calculation of camera motion (rotation and translation) between two images.
  • the operation of the homography matrix can be realized by obtaining the coordinates of 4 pairs of matching points.
  • the transformed position of each pixel in the image is known to form a transformed image. For example, you can first get the coordinates pts_src of the four vertices of the book; then we need to know the aspect ratio of the book.
  • the aspect ratio of this book is 3: 4, so the size of the output image can be 300 ⁇ 400
  • the coordinates of its four points can be set to (0, 0), (299, 0), (299, 399), and (0, 399) are stored in pts_dst; the homography matrix (homography) is obtained through pts_src and pts_dst; Apply homography to the original image to get a new output image.
  • a homography matrix of the telephoto image to the short focus reference block is determined; according to the homography Matrix to obtain a coarsely aligned telephoto image.
  • a coarsely aligned telephoto image can be understood as an image output using a homography matrix based on 4 pairs of matching points.
  • the thus obtained coarsely aligned telephoto image will not cause image distortion due to the parallax of the shooting camera.
  • the image is more accurate than the broken cross reference block.
  • the homography matrix can be used to perform a coarse alignment of the telephoto image, which can reduce the amount of fine alignment calculation in the subsequent steps and improve the accuracy of the fine alignment process.
  • step S140 the grid control points of the coarsely aligned telephoto image are fused by using key points matching the short focus reference block to obtain a fused image.
  • the grid of the coarse-aligned telephoto image may be an 8 * 8 grid, a 16 ⁇ 16 grid, or a 32 ⁇ 32 grid.
  • the number of grids is set to a positive integer power of 2, so that the setting is beneficial to data calculation.
  • the key point matching result between the coarsely aligned telephoto image and the short focus reference block can be used to determine the position of each grid control point to obtain the grid transformed image and compare it with the original short focus.
  • the images are fused to form a fused image.
  • the present application performs coarse alignment of the telephoto image and the short focus reference block by estimating a global homography, but in the actual situation, there is parallax between the cameras due to the depth change of the scene. It is difficult to align the two with a global homography matrix. Therefore, in the third step of fine alignment, this application uses a grid deformation method to perform a non-uniform deformation on the input telephoto camera image to obtain a better alignment effect.
  • a short-focus image and at least one tele-focus image are obtained; a block matching algorithm is used to determine a short-focus reference block; Obtained by performing image block matching and structural edge block matching on the telephoto image and the telephoto image; taking 4 pairs of matching points from the telephoto image and the short focus reference block to determine the telephoto image to the The homography matrix of the short focus reference block; a coarsely aligned telephoto image is obtained according to the homography matrix; the grid control of the coarsely aligned telephoto image is matched by using the key points of the short focus reference block Points are fused to obtain a fused image.
  • the technical solution provided in the present application can be integrated into an unstructured camera array to reduce hardware requirements and improve image fusion speed and robustness.
  • FIG. 2 is a flowchart of a multi-scale image fusion method according to an embodiment of the present application. Based on the above embodiment, this embodiment is detailed as follows: the determining a short focus reference block by using a block matching algorithm includes: using a block matching algorithm to perform block matching on the short focus image and the telephoto image. To obtain a first response map; perform block matching on the structural edge map of the short focus image and the structural edge map of the telephoto image to obtain a second response map; combine the first response map and the second response The map performs a dot product operation to locate a position in the telephoto image corresponding to the short focus image; and a portion corresponding to the telephoto image is cut out from the short focus image and determined as a short focus reference block.
  • the multi-scale image fusion method includes steps S210 to S260.
  • step S210 one short-focus image and at least one long-focus image are acquired.
  • step S220 a block matching algorithm is used to perform block matching on the short focus image and the telephoto image to obtain a first response map.
  • the size of the telephoto image can be reduced according to the focal distance difference between the telephoto image and the short focus image. This can make the same object have the same pixel size in the short-focus camera image and the long-focus camera image. For example, if an object is 50 pixels in a 16mm short-focus camera, the size of an object in a 135mm telephoto camera is approximately Pixels. The telephoto camera image can be reduced so that the size of the object in the telephoto camera image is also 50 pixels.
  • ZNCC block matching is performed on the short-focus image and the reduced telephoto image to obtain a first response map.
  • step S230 block matching is performed on the structure edge map of the short focus image and the structure edge map of the telephoto image to obtain a second response map.
  • the structural edge map of the reduced telephoto image may be calculated, and the structural edge map of the short focus image may be calculated.
  • the structural edge maps of the two images are subjected to ZNCC block matching to obtain a second response map.
  • the calculation method of the structure edge map may adopt a random forest algorithm or other algorithms capable of obtaining the structure edge map. For example, first perform bilateral filtering on the image, erase small details, leave structural information, and then calculate the edge map.
  • step S240 performing a dot product operation on the first response map and the second response map to locate a position in the telephoto image corresponding to the short focus image; and extracting from the short focus image A part corresponding to the telephoto image is determined and determined as a short focus reference block.
  • a response map can be obtained.
  • the position with the largest value on the response map is the corresponding position of the telephoto camera image.
  • this block matching algorithm is not robust and is prone to mismatches. Therefore, in this application, a structural edge map is added to calculate another response map, and then the two response maps are merged to obtain the final result.
  • the fusion method is to perform a dot-product operation on the two response graphs to obtain a new response graph.
  • Ir, Il are input short-focus camera images and telephoto camera images (after reduction), and Er, El represent calculated structural edge maps.
  • step S250 4 pairs of matching points are taken from the telephoto image and the short focus reference block, and a homography matrix of the telephoto image to the short focus reference block is determined; according to the homography Matrix to get a coarsely aligned telephoto image.
  • step S260 the grid control points of the coarsely aligned telephoto image are fused by using key points matching the short focus reference block to obtain a fused image.
  • This embodiment provides an implementation method for determining a short-focus reference block on the basis of the foregoing embodiment.
  • This embodiment adopts matching and performing dot product operations on two types of images to determine a short-focus reference block in the short-focus image. This position can improve the accuracy and robustness of the technical solution provided by this application.
  • FIG. 3 is a flowchart of an image multi-scale fusion method according to an embodiment of the present application. Based on the foregoing embodiment, this embodiment is refined as follows: 4 pairs of matching points are taken from the telephoto image and the short focus reference block, and the method includes: segmenting the short focus reference block in a preset manner. To obtain a short-focus reference block sub-block as a search area; segment the tele-focus image according to the preset method to obtain a tele-focus image sub-image, and select a preset area as a middle of each tele-focus image sub-image Template; matching the template and the search area to obtain 4 pairs of matching points.
  • the multi-scale image fusion method includes steps S310 to S370.
  • step S310 one short-focus image and at least one long-focus image are acquired.
  • a short-focus reference block is determined by using a block matching algorithm.
  • the short-focus reference block is obtained by performing image block matching and structural edge block matching on the short focus image and the telephoto image. .
  • step S330 the short focus reference block is sliced in a preset manner to obtain a short focus reference block sub-block as a search area.
  • the preset method may be to divide the short-focus reference block into 4 sub-blocks of 2 ⁇ 2 according to a length direction and a width direction, and obtain 4 short-focus reference block sub-blocks as 4 search areas.
  • the preset method may also be other methods, and the specific method may be set here according to requirements.
  • step S340 the telephoto image is segmented according to the preset method to obtain a telephoto image sub-image, and a preset area is selected as a template in the middle of each telephoto image sub-image.
  • the preset manner is divided into 4 image regions divided into 2 ⁇ 2 according to a length direction and a width direction; correspondingly, matching the template and the search region to obtain 4 Matching points includes matching the 4 templates and the 4 search areas at corresponding positions to obtain a pair of matching points for each of the template and each of the search areas, and a total of 4 pairs of matching points.
  • the telephoto image is segmented in the same way as the short focus reference block. This setting is convenient to find a template corresponding to each search area, quickly determine 4 pairs of key points, reduce the calculation amount, and increase the calculation speed. So here is also divided according to the preset method.
  • step S350 the template and the search area are matched to obtain 4 pairs of matching points.
  • Each template is matched with each corresponding search area, and 4 matching points are determined.
  • a smaller range may be determined for each telephoto image sub-image of the telephoto image as a template. For example, you can determine the 75% or even 50% of the length and width of each telephoto image as a template. This setting can reduce the amount of calculation and find matching points more quickly. However, this range should not be set too small. If it is set too small, there may be situations where there are no obvious features in the template for matching, which affects the accuracy of the matching points.
  • step S360 a homography matrix from the telephoto image to the short focus reference block is determined; according to the homography matrix, a coarsely aligned telephoto image is obtained.
  • the homography matrix from the telephoto image to the short focus reference block is determined according to the four pairs of matching points, and then the coarsely aligned telephoto image is obtained based on the homography matrix.
  • the external contour of the telephoto image obtained in this way may be slightly distorted, the distortion of the image content due to the parallax between the telephoto camera and the short focus camera can be avoided, so the image content of the coarsely aligned telephoto image is relative to the telephoto image In terms of a certain correction effect.
  • step S370 the grid control points of the coarsely-aligned telephoto image are fused by using key points matching the short focus reference block to obtain a fused image.
  • This embodiment provides a method for coarsely aligning a telephoto image on the basis of the foregoing embodiment. Such a setting can improve the accuracy of the image while reducing the workload and improving the image fusion process. Speed and robustness.
  • FIG. 4 is a flowchart of a multi-scale image fusion method according to an embodiment of the present application. Based on the foregoing embodiment, this embodiment is refined as follows: using the key points matching the short focus reference block to fuse the grid control points of the coarsely aligned telephoto image to obtain a fused image, including: Performing key point detection on the short focus reference block; determining a corresponding key point image block for each of the key points; calculating a key point score from a structure edge map of the key point image block, and performing a key point score Key points below the preset threshold are filtered; key points are matched with the coarsely aligned telephoto image using the filtered key points; grid control of the coarsely aligned telephoto image is performed based on the key point matching results Points are fused to obtain a fused image.
  • the image multi-scale fusion method includes steps S410 to S480.
  • step S410 one short-focus image and at least one long-focus image are acquired.
  • a short focus reference block is determined by using a block matching algorithm; wherein the short focus reference block is obtained by performing image block matching and structural edge block matching on the short focus image and the telephoto image. .
  • step S430 4 pairs of matching points are taken from the telephoto image and the short focus reference block, and a homography matrix of the telephoto image to the short focus reference block is determined; according to the homography Matrix to get a coarsely aligned telephoto image.
  • step S440 keypoint detection is performed on the short focus reference block.
  • the key point refers to a point that contains structural texture information.
  • the reason is that based on the ZNCC block matching algorithm, enough structural texture information is needed to give reliable results.
  • a classic corner detection algorithm (goodfeaturestotrack algorithm) is used to detect key points, and many key points with rich structural texture information can be found by using this algorithm.
  • step S450 for each of the key points, a key point image block corresponding to each key point is determined.
  • the keypoint image block is an image block containing the keypoint.
  • the keypoint image block is determined by using the center of the keypoint and a fixed number of pixels as the radius. For example, a key point image block is determined with the detected key point as the center and 32 pixel points as the side length of the key point image block.
  • step S460 a keypoint score is calculated from the structural edge map of the keypoint image block, and keypoints whose keypoint score is lower than a preset threshold are filtered.
  • calculating a key point score by using a structural edge map of the key point image block includes: calculating a sum of pixel values of all pixels through the structural edge map of the key point image block, and dividing all The sum of the pixel values of the pixels is determined as the key point score.
  • a key point image block can be cut out based on the key point, and the structural edge map of the key point image block is used to calculate the key point score.
  • S i represents the score of the i-th key point
  • p i represents the i-th key point
  • E (p i ) represents the structure edge map calculated from the corresponding key point image block that was cut out. Sum of all pixel values on the structure edge map.
  • the short focus reference block may be divided into a certain number of grids with priority, for example, it may be 16.
  • priority for example, it may be 16.
  • all keypoints are reordered according to the score size, and finally the keypoints with the highest scores in each grid are left.
  • the key point selection only a small number of key points will be left in each grid, such as 1 to 3, which can greatly reduce the number of key points and improve the calculation speed.
  • After screening the number of key points is much less. Some grids have no key points left because no key points are detected or the key point score is too low. But this does not affect the determination of key points for high scores.
  • step S470 key points are matched with the coarsely aligned telephoto image by using the filtered key points.
  • performing key point matching with the coarsely aligned telephoto image by using the filtered key points includes: in the short focus reference block, taking the filtered key points as the center and M number of key points.
  • a block with a pixel length of an edge is used as a key point matching template; in the coarsely aligned telephoto image, the filtered key points corresponding to the coordinate position of the short focus reference block are corresponding to the coarsely aligned telephoto.
  • the pixel point in the image is the center, and the block with N pixels as the side length is used as the key point matching search area; where M is less than N; matching the key point matching template and the key point matching search area to obtain Key points in the coarsely aligned telephoto image that match the filtered key points.
  • M is a positive integer power of 2 and N is a positive integer power of 2.
  • the size of the template block is preferably set to 2n, this size of the block can speed up the calculation of ZNCC template matching).
  • a 512 ⁇ 512 block with the same key point coordinate as the center is selected as the key point matching search area (the key point matching search area is a large image, and the size is adjusted accordingly with the image size).
  • ZNCC matching can get the key points of matching.
  • the keypoint matching template and the keypoint matching search area are matched to obtain key points matching the filtered key points in the coarsely aligned telephoto image.
  • the method further includes: dividing the short focus reference block and the coarsely aligned telephoto image into a grid; using at least two different scales, filtering the key points of the short focus reference block and The keypoints in the coarsely aligned telephoto image that match the filtered keypoints are denoised using a random sampling consensus algorithm to obtain matching keypoints with high confidence.
  • the short focus image and the coarsely aligned telephoto image can be divided into a 16 ⁇ 16 grid, and three scales are selected (more fine grids can use more scales), for example: global scale, for all matching points
  • a random sampling consensus algorithm (RANSAC) calculation was performed to mark matching points with errors exceeding the threshold as noise
  • 2 ⁇ 2 scale the 16 ⁇ 16 grid was divided into 2 ⁇ 2 regions, each region was 8 ⁇ 8 grids, RANSAC is performed once for each matching point in each region, and matching points with errors exceeding the threshold are marked as noise
  • 4 ⁇ 4 scale the method is the same as 2 ⁇ 2 scale.
  • the size of the threshold can be adjusted according to the actual situation. Generally, for a 2000 ⁇ 1500 image, a threshold of about 3 to 7 pixels is more suitable.
  • the selection of several scales can be related to the number of grids divided by the short focus image and the coarsely aligned telephoto image. For example, if it is divided into 16 ⁇ 16 grids, two, three, or even four different grid scales can be used. If it is divided into 32 ⁇ 32 grids, more scales can be designed, which can be calculated based on actual calculations. The need for calculation speed and accuracy in the process is determined.
  • multi-scale RANSAC can be used to denoise the matching points and reduce the interference of noise on the image fusion result.
  • step S480 according to the key point matching result, the grid control points of the coarsely aligned telephoto image are fused to obtain a fused image.
  • this embodiment provides a method for detecting and filtering key points of a short focus reference block, so that the characteristics of the key points of the filtered short focus reference block are more obvious, which not only improves the solution. The accuracy of the calculation is also reduced.
  • fusing grid control points of the coarsely aligned telephoto image to obtain a fused image includes: dividing the short focus reference block and the coarsely aligned telephoto image into a grid; using at least Two different scales, determine the homography matrix corresponding to each scale; determine the position of the grid control point of the control grid according to the homography matrix corresponding to each scale; wherein, within the same scale, , The grid control points covered by multiple homography matrices, and the average value of the calculated position of each covering homography matrix is used as the grid covered by multiple homography matrices The position of the control point; the target grid at each scale is determined according to the obtained position of the grid control point; the target grid at each scale is fused to obtain a fused image.
  • a coarse-aligned telephoto image can be divided into a 16 ⁇ 16 grid, and there will be 17 ⁇ 17 grid control points.
  • At least two different scales can be global scale, 2 ⁇ 2 regional scale, 4 ⁇ 4 regional scales and 8 ⁇ 8 regional scales can also be other scales.
  • the 2 ⁇ 2 area scale divides the 16 ⁇ 16 grid into 4 areas, and each area is 4 ⁇ 4 grids, so that the 2 ⁇ 2 area scale covers the entire grid and does not appear in the area. There will be cases where the grids overlap, and the 4 ⁇ 4 area scale and 8 ⁇ 8 area scale are also determined in this way.
  • the corresponding homography matrix can be determined at each scale.
  • the specific method can be to select 4 pairs of matching points at each scale.
  • For the global scale select 4 pairs of matching points at the global scale.
  • In the 2 ⁇ 2 regional scale , Then within 4 regions, 4 pairs of matching points are selected for each region, and the homography matrix of each region is calculated, and so on.
  • each homography matrix is used to calculate the position coordinates of the corresponding area control points.
  • This technical solution provides a method for determining the control points of the target grid. This method is simple to calculate, and can have higher anti-interference ability, and the calculation result is more accurate.
  • each scale can also be divided into different regions correspondingly. For example, if it is divided into 16 ⁇ 16 grids, two, three, or even four different grid scales can be used.
  • the three scales can include global scale, 2 ⁇ 2 regional scale, and 4 ⁇ 4 regional scale. It can be global scale, 2 ⁇ 2 regional scale, and 8 ⁇ 8 regional scale; if it is divided into 32 ⁇ 32 grids, more scales can be designed, which can be based on the demand for calculation speed and accuracy in the actual calculation process. determine.
  • the target grids at each scale are fused to obtain a fused image, which includes: determining a confidence region centered on the control point of the target grid at the first scale; When the control points of the target grid are within the range of the confidence region, the control points of the target grid at the second scale shall prevail; traverse all the scales to determine the position of the control points of the final target grid; The position of the control points of the grid determines the fused image, wherein the first scale is larger than the second scale.
  • the confidence region may be determined by using a control point of the target grid of the first scale as a center and a preset length as a radius. In this embodiment, 10% of the smaller one of the length and width of the grid may be used as the radius of the confidence region.
  • the control points of the second-scale grid fall within the confidence region, the control points of the second-scale grid are accepted, and when the control points of the second-scale grid do not fall within the confidence region, the first Control points of a two-scale grid.
  • the final target mesh By traversing all the scales, the final target mesh can be determined, and image fusion is performed according to the position of the control points of the final target mesh. In this way, the confidence points of the control points can be used to fuse all the grids, and the effect of the mismatched points on the results can be further removed.
  • FIG. 5 is a schematic structural diagram of an image fusion apparatus according to an embodiment of the present application.
  • the image fusion device includes an image acquisition module 510, a short focus reference block determination module 520, a coarsely aligned telephoto image determination module 530, and an image fusion module 540.
  • the image acquisition module 510 is configured to acquire a short focus image and at least one telephoto image.
  • the short-focus reference block determination module 520 is configured to determine a short-focus reference block by using a block matching algorithm; wherein the short-focus reference block is obtained by performing image block matching and structural edges on the short-focus image and the tele-focus image. Tile matching.
  • the coarsely aligned telephoto image determination module 530 is configured to take 4 pairs of matching points from the telephoto image and the short focus reference block to determine a homography matrix from the telephoto image to the short focus reference block; According to the homography matrix, a coarsely aligned telephoto image is obtained.
  • the image fusion module 540 is configured to use the key points matched with the short focus reference block to fuse the grid control points of the coarsely aligned telephoto image to obtain a fused image.
  • a short-focus image and at least one tele-focus image are obtained; a block matching algorithm is used to determine a short-focus reference block; wherein the short-focus reference block is obtained by analyzing the short-focus reference block.
  • the above product can execute the method provided by any embodiment of the present application, and has corresponding function modules for executing the method.
  • An embodiment of the present application further provides a storage medium containing computer-executable instructions.
  • the method is configured to perform an image multi-scale fusion method.
  • the method includes: obtaining a short A focus image and at least one telephoto image; a short-focus reference block is determined using a block matching algorithm; wherein the short-focus reference block is obtained by performing image block matching and structural edges on the short-focus image and the telephoto image Obtained by tile matching; taking 4 pairs of matching points from the telephoto image and the short focus reference block to determine a homography matrix from the telephoto image to the short focus reference block; according to the homography To obtain a coarsely-aligned telephoto image; using grid matching points of the coarse-aligned telephoto image to match the key points of the short-focus reference block to obtain a fused image.
  • Storage medium any of a variety of types of memory devices or storage devices.
  • the term “storage medium” is intended to include: installation media, such as Compact Disc-Read-Only Memory (CD-ROM), floppy disks or magnetic tape devices; computer system memory or random access memory, such as dynamic random access memory ( Dynamic Random Access Memory (DRAM), Double Data Rate Random Access Memory (DDR RAM), Static Random Access Memory (Static Random Access Memory, SRAM), Extended Data Output Random Access Memory (Extended Data Output Random Access Memory (EDO RAM), Rambus RAM, etc .; non-volatile memory, such as flash memory, magnetic media (such as hard disk or optical storage); registers or other similar types of memory elements.
  • the storage medium may further include other types of memory or a combination thereof.
  • the storage medium may be located in a computer system in which the program is executed, or may be located in a different second computer system connected to the computer system through a network such as the Internet.
  • the second computer system may provide program instructions to a computer for execution.
  • the term "storage medium" may include two or more storage media that may reside in different locations, such as in different computer systems connected through a network.
  • the storage medium may store program instructions (for example, embodied as a computer program) executable by one or more processors.
  • a storage medium including computer-executable instructions provided in the embodiments of the present application is not limited to the operation of image fusion as described above, and may also execute multiple images provided by any embodiment of the present application. Relevant operations in the scale fusion method.
  • FIG. 6 is a schematic structural diagram of a terminal according to an embodiment of the present application.
  • the terminal may include: a memory 601, a central processing unit (CPU) 602 (also referred to as a processor, hereinafter referred to as a CPU), a circuit board (not shown in the figure), and a power supply circuit (figure Not shown).
  • CPU central processing unit
  • FIG. 6 the terminal may include: a memory 601, a central processing unit (CPU) 602 (also referred to as a processor, hereinafter referred to as a CPU), a circuit board (not shown in the figure), and a power supply circuit (figure Not shown).
  • CPU central processing unit
  • circuit board not shown in the figure
  • a power supply circuit figure Not shown
  • the circuit board is disposed in a space surrounded by the housing; the CPU 602 and the memory 601 are disposed on the circuit board; and the power supply circuit is configured to power a plurality of circuits or devices of the terminal
  • the memory 601 is configured to store executable program code; the CPU 602 runs a computer program corresponding to the executable program code by reading the executable program code stored in the memory 601 to implement the following steps: Acquiring a short-focus image and at least one telephoto image; determining a short-focus reference block by using a block matching algorithm; wherein the short-focus reference block is obtained by image-blocking the short-focus image and the telephoto image Obtained by matching and structural edge block matching; taking 4 pairs of matching points from the telephoto image and the short focus reference block to determine a homography matrix from the telephoto image to the short focus reference block; according to The homography matrix to obtain a coarsely-aligned telephoto image; using matching with key points of the short-focus reference block to fuse the grid control points of
  • the terminal also includes a peripheral interface 603, an RF (Radio Frequency) circuit 605, an audio circuit 606, a speaker 611, a power management chip 608, an input / output (I / O) subsystem 609, and a touch screen. 612, other input / control devices 610, and an external port 604. These components communicate through one or more communication buses or signal lines 607.
  • RF Radio Frequency
  • the illustrated terminal 600 is only an example of the terminal, and the terminal 600 may have more or fewer components than those shown in the figure, may combine two or more components, or may have Different component configurations.
  • the various components shown in the figures may be implemented in hardware, software, or a combination of hardware and software, including one or more signal processing and / or application specific integrated circuits.
  • the following describes the image fusion terminal provided in this embodiment in detail, and the terminal uses a mobile phone as an example.
  • Memory 601 which can be accessed by CPU 602, peripheral interface 603, etc.
  • the memory 601 can include high-speed random access memory, and can also include non-volatile memory, such as one or more disk storage devices, flash memory devices , Or other volatile solid-state storage devices.
  • Peripheral interface 603, which can connect the input and output peripherals of the device to the CPU 602 and the memory 601.
  • the I / O subsystem 609 which can connect input / output peripherals on the device, such as touch screen 612 and other input / control devices 610, to peripheral interface 603.
  • the I / O subsystem 609 may include a display controller 6091 and one or more input controllers 6092 configured to control other input / control devices 610. Among them, one or more input controllers 6092 receive electrical signals from or send electrical signals to other input / control devices 610.
  • Other input / control devices 610 may include physical buttons (press buttons, rocker buttons, etc.) ), Dial, slide switch, joystick, click wheel. It is worth noting that the input controller 6092 can be connected to any of the following: a keyboard, an infrared port, a USB interface, and a pointing device such as a mouse.
  • a touch screen 612 which is an input interface and an output interface between a user terminal and a user, and displays a visual output to the user.
  • the visual output may include graphics, text, icons, videos, and the like.
  • the display controller 6091 in the I / O subsystem 609 receives electrical signals from the touch screen 612 or sends electrical signals to the touch screen 612.
  • the touch screen 612 detects a contact on the touch screen, and the display controller 6091 converts the detected contact into interaction with a user interface object displayed on the touch screen 612, that is, realizes human-computer interaction.
  • the user interface object displayed on the touch screen 612 may be an operation Icons for games, icons connected to the appropriate network, etc.
  • the device may also include a light mouse, which is a touch-sensitive surface that does not display visual output, or an extension of the touch-sensitive surface formed by a touch screen.
  • the RF circuit 605 is mainly configured to establish communication between the mobile phone and the wireless network (that is, the network side), and realize data reception and transmission of the mobile phone and the wireless network. For example, send and receive text messages, e-mail, and so on.
  • the RF circuit 605 receives and sends an RF signal.
  • the RF signal is also referred to as an electromagnetic signal.
  • the RF circuit 605 converts an electrical signal into an electromagnetic signal or converts an electromagnetic signal into an electrical signal. Communication.
  • RF circuit 605 may include known circuits configured to perform these functions, including, but not limited to, antenna systems, RF transceivers, one or more amplifiers, tuners, one or more oscillators, digital signal processors, codecs (COder-DECoder, CODEC) chipset, Subscriber Identity Module (SIM), and so on.
  • the audio circuit 606 is mainly configured to receive audio data from the peripheral interface 603, convert the audio data into an electrical signal, and send the electrical signal to the speaker 611.
  • the speaker 611 is configured to restore a voice signal received by the mobile phone from the wireless network through the RF circuit 605 to a sound and play the sound to a user.
  • the power management chip 608 is configured to provide power and power management for the hardware connected to the CPU 602, the I / O subsystem, and peripheral interfaces.
  • the terminal provided in the embodiments of the present application can reduce hardware requirements and improve image fusion speed and robustness.
  • the image fusion apparatus, storage medium, and terminal provided in the foregoing embodiments can execute a multi-scale fusion method of images provided by any embodiment of the present application, and have corresponding function modules for executing the method.
  • a multi-scale image fusion method provided by any embodiment of the present application.

Landscapes

  • Engineering & Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Theoretical Computer Science (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Studio Devices (AREA)
  • Image Processing (AREA)

Abstract

本申请实施例公开了一种图像的多尺度融合方法、装置、存储介质及终端。该方法包括：获取一张短焦图像和至少一张长焦图像；利用块匹配算法，确定短焦参考块；其中，所述短焦参考块是通过对所述短焦图像和所述长焦图像进行图像块匹配和结构边缘图块匹配得到的；从所述长焦图像和所述短焦参考块中取4对匹配点，确定所述长焦图像到所述短焦参考块的单应性矩阵；根据所述单应性矩阵，得到粗对齐长焦图像；利用与所述短焦参考块的关键点匹配，对所述粗对齐长焦图像的网格控制点进行融合，得到融合图像。

Description

图像的多尺度融合方法、装置、存储介质及终端
本申请要求在2018年08月31日提交中国专利局、申请号为201811012085.X的中国专利申请的优先权，该申请的全部内容通过引用结合在本申请中。
技术领域
本申请实施例涉及图像处理技术领域，例如涉及一种图像的多尺度融合方法、装置、存储介质及终端。
背景技术
随着科技水平的发展，以及现阶段对图像或者视频的像素要求的提升，十亿像素图像已经成为行业中的需求。
十亿像素图像技术，采用的是一个短焦相机和多个长焦相机同时记录下场景图像，再利用将长焦相机拍摄得到的图像替换掉短焦相机相应位置的图像，这样就可以得到范围大而且图像清晰的效果，其中需要涉及到定位、图像结合的技术手段。相关技术中，不仅对硬件提出了极高的要求，同时对软件方面也是极大的挑战，由于需要在短时间内完成大量信息的处理，所以计算效率和鲁棒性都成为了行业的痛点。
发明内容
以下是对本文详细描述的主题的概述。本概述并非是为了限制权利要求的保护范围。
本申请实施例提供一种图像的多尺度融合方法、装置、存储介质及终端，可以集成于非结构化相机阵列中，用于实现降低硬件需求，提高图像融合速度与鲁棒性。
第一方面，本申请实施例提供了一种图像的多尺度融合方法，包括：获取一张短焦图像和至少一张长焦图像；利用块匹配算法，确定短焦参考块；其中，所述短焦参考块是通过对所述短焦图像和所述长焦图像进行图像块匹配和结构边缘图块匹配得到的；从所述长焦图像和所述短焦参考块中取4对匹配点，确定所述长焦图像到所述短焦参考块的单应性矩阵；根据所述单应性矩阵，得到粗对齐长焦图像；利用与所述短焦参考块的关键点匹配，对所述粗对齐长焦图像的网格控制点进行融合，得到融合图像。
第二方面，本申请实施例还提供了一种图像融合装置，该装置包括：图像获取模块，设置为获取一张短焦图像和至少一张长焦图像；短焦参考块确定模块，设置为利用块匹配算法，确定短焦参考块；其中，所述短焦参考块是通过 对所述短焦图像和所述长焦图像进行图像块匹配和结构边缘图块匹配得到的；粗对齐长焦图像确定模块，设置为从所述长焦图像和所述短焦参考块中取4对匹配点，确定所述长焦图像到所述短焦参考块的单应性矩阵；根据所述单应性矩阵，得到粗对齐长焦图像；图像融合模块，设置为利用与所述短焦参考块的关键点匹配，对所述粗对齐长焦图像的网格控制点进行融合，得到融合图像。
第三方面，本申请实施例提供了一种计算机可读存储介质，其上存储有计算机程序，该程序被处理器执行时实现如本申请实施例所述的图像的多尺度融合方法。
第四方面，本申请实施例提供了一种终端，包括存储器，处理器及存储在存储器上并可在处理器运行的计算机程序，所述处理器执行所述计算机程序时实现如本申请实施例所述的图像的多尺度融合方法。
在阅读并理解了附图和详细描述后，可以明白其他方面。
附图说明
图1是本申请一实施例提供的图像的多尺度融合方法的流程图；
图2是本申请另一实施例提供的图像的多尺度融合方法的流程图；
图3是本申请另一实施例提供的图像的多尺度融合方法的流程图；
图4是本申请另一实施例提供的图像的多尺度融合方法的流程图；
图5是本申请一实施例提供的图像融合装置的结构示意图；
图6为本申请一实施例提供的一种终端的结构示意图。
具体实施方式
下面结合附图和实施例对本申请作进一步的详细说明。可以理解的是，此处所描述的具体实施例仅仅用于解释本申请，而非对本申请的限定。另外还需要说明的是，为了便于描述，附图中仅示出了与本申请相关的部分而非全部结构。
在更加详细地讨论示例性实施例之前应当提到的是，一些示例性实施例被描述成作为流程图描绘的处理或方法。虽然流程图将多个步骤描述成顺序的处理，但是其中的许多步骤可以被并行地、并发地或者同时实施。此外，多个步骤的顺序可以被重新安排。在其操作完成时所述处理可以被终止，但是还可以具有未包括在附图中的附加步骤。所述处理可以对应于方法、函数、规程、子例程、子程序等等。
图1是本申请实施例提供的图像的多尺度融合方法的流程图。本实施例可适用图像或者视频采集的情况，该方法可以由本申请实施例所提供的图像融合装置来执行，该装置可以由软件和/或硬件的方式来实现，并可集成于非结构化 相机阵列图像获取终端中。
如图1所示，所述图像的多尺度融合方法包括步骤S120至步骤S140。
在步骤S110中，获取一张短焦图像和至少一张长焦图像。
其中，短焦图像和长焦图像可以是相对而言的，例如，在一组相机阵列中，可以是将焦距最短的相机所获取到的图像作为短焦图像，其他摄像头所获取到的图像作为长焦图像，这样，得到的短焦图像为一张，长焦图像为多张，长焦图像最少也可以是一张。除此之外，还可以将短焦图像和长焦图像根据相机的焦距来进行定义，如，相机的焦距为4-16mm的，获取到的图像作为短焦图像，相机的焦距为25-135mm甚至更大的，获取到的图像作为长焦图像。由于本实施例所提供的技术方案是将长焦图像和短焦图像进行融合，得到高分辨率的图像，所以短焦图像只要一张就可以，所以在一组相机阵列中，可以只设置一个短焦相机，再设置多个长焦相机。这样，将所有的长焦相机得到的长焦图像融合到短焦相机中，就可以得到一张像素高于单个相机所拍摄的到的图像，最终形成高像素图像或者视频。
在步骤S120中，利用块匹配算法，确定短焦参考块；其中，所述短焦参考块是通过对所述短焦图像和所述长焦图像进行图像块匹配和结构边缘图块匹配得到的。
其中，块匹配算法可以是去均值归一化模板互相关算法(zero-mean normalized 2d cross correlation，ZNCC)，采用ZNCC算法性能更加稳定，抗干扰能力强。短焦参考块可以是在短焦图像中与长焦图像相对应的参考图块。
在本实施例中，短焦参考块是通过对所述短焦图像和所述长焦图像进行图像块匹配和结构边缘图块匹配得到的。其中，包括通过短焦图像和长焦图像进行图像块匹配得到的结果，还有短焦图像的结构边缘图和长焦图像的结构边缘图进行块匹配得到的结果。本实施例这样设置可以更加准确的确定短焦图像中长焦图像的位置。示例性的，例如一块长10米、宽8米的宣传画，在一定距离处的一组相机阵列中，其在短焦相机拍摄到的结果中可能是图像十分之一的范围，而在一个长焦相机拍摄到的结果中可能是图像的全部范围，通过图像块匹配和结构边缘图块匹配就可以确定长焦图像中的影像在短焦图像中的具体位置，进而确定短焦参考块。本实施例中，通过将两次匹配结果进行融合，可以得到更加准确的短焦参考块，可以提高本方案的图像同和的准确性，鲁棒性强。
在步骤S130中，从所述长焦图像和所述短焦参考块中取4对匹配点，确定所述长焦图像到所述短焦参考块的单应性矩阵；根据所述单应性矩阵，得到粗对齐长焦图像。
其中，单应性矩阵是同一物体在两个图像中成像，由于视距视角的不同，而呈现出不同的图像。比如，一个物体可以通过旋转相机镜头获取两张不同的 照片(这两张照片的内容不一定要完全对应，部分对应即可)，我们可以把单应性设为一个二维矩阵M，那么照片1乘以M就是照片2。这有着很多实际应用，比如图像校正、图像对齐或两幅图像之间的相机运动计算(旋转和平移)等。
可以通过获取到4对匹配点的坐标来实现单应性矩阵的运算，并且通过单应性矩阵，已知图像中每个像素点在变换后的位置，形成变换后的图像。例如，可以首先获取书本四个顶点的坐标pts_src；然后我们需要知道书本的宽高比，此书的宽高比是3：4，所以可使输出图像的尺寸(size)为300×400，就可设其四个点的坐标为(0，0)，(299，0)，(299，399)，(0，399)保存在pts_dst中；通过pts_src和pts_dst获取单应性矩阵(homography)；对原图应用homography得到输出新图像。
在本实施例中，从所述长焦图像和所述短焦参考块中取4对匹配点，确定所述长焦图像到所述短焦参考块的单应性矩阵；根据所述单应性矩阵，得到粗对齐长焦图像。其中，粗对齐长焦图像可以理解为是根据4对匹配点，采用单应性矩阵输出的图像，这样得到的粗对齐长焦图像不会由于拍摄相机的视差造成图像的扭曲，其中的物体的图像相对于断交参考块是更加准确的，这样设置可以利用单应性矩阵对长焦图像进行一次粗对齐，可以降低后续步骤中精细对齐的计算量，并且提高精细对齐过程的准确性。
在步骤S140中，利用与所述短焦参考块的关键点匹配，对所述粗对齐长焦图像的网格控制点进行融合，得到融合图像。
其中，粗对齐长焦图像的网格可以是8*8的网格，也可以是16×16的网格，或者32×32的网格。在一实施例中，将网格的数量设置为2的正整数次幂，这样设置有利于数据计算。
以16×16的网格为例，共存在17×17个网格控制点。本实施例可以利用粗对齐长焦图像与短焦参考块之间的关键点匹配结果，确定每个网格控制点的位置，来得到网格变换后的图像，并将其与原来的短焦图像进行融合，形成融合后的图像。
本实施例，通过在粗对齐中，本申请通过估计一个全局的单应性将长焦图像和短焦参考块进行粗对齐，但是在实际情况中，由于场景存在深度变化，相机之间存在视差，一个全局的单应性矩阵难以将两者很好地对齐。因此本申请在第三步精细对齐中采用网格变形的方式来对输入的长焦相机图像进行一次非均匀变形，得到更好的对齐效果。
本申请实施例所提供的技术方案，通过获取一张短焦图像和至少一张长焦图像；利用块匹配算法，确定短焦参考块；其中，所述短焦参考块是通过对所述短焦图像和所述长焦图像进行图像块匹配和结构边缘图块匹配得到的；从所述长焦图像和所述短焦参考块中取4对匹配点，确定所述长焦图像到所述短焦 参考块的单应性矩阵；根据所述单应性矩阵，得到粗对齐长焦图像；利用与所述短焦参考块的关键点匹配，对所述粗对齐长焦图像的网格控制点进行融合，得到融合图像。本申请所提供的技术方案，可以集成于非结构化相机阵列中，实现降低硬件需求，提高图像融合速度与鲁棒性。
图2是本申请实施例提供的图像的多尺度融合方法的流程图。本实施例在上述实施例的基础上，细化为：所述利用块匹配算法，确定短焦参考块，包括：利用块匹配算法，对所述短焦图像和所述长焦图像进行块匹配，得到第一响应图；对所述短焦图像的结构边缘图和所述长焦图像的结构边缘图进行块匹配，得到第二响应图；将所述第一响应图和所述第二响应图进行点积运算以定位所述长焦图像对应到所述短焦图像中的位置；并从所述短焦图像中抠出和所述长焦图像对应的部分，确定为短焦参考块。
如图2所示，所述图像的多尺度融合方法包括步骤S210至步骤S260。
在步骤S210中，获取一张短焦图像和至少一张长焦图像。
在步骤S220中，利用块匹配算法，对所述短焦图像和所述长焦图像进行块匹配，得到第一响应图。
其中，首先可以根据长焦图像和短焦图像的焦距差距缩小长焦图像的尺寸。这样可以使得同一个物体在短焦相机图像和长焦相机图像中的像素大小一样。例如，一个物体在16mm的短焦相机中的大小为50个像素，那么在135mm的长焦相机中的大小约为
Figure PCTCN2019103226-appb-000001
个像素。可以缩小长焦相机图像，使得长焦相机图像中该物体的大小也为50个像素。
将短焦图像与缩小后的长焦图像进行ZNCC块匹配，从而得到第一响应图。
在步骤S230中，对所述短焦图像的结构边缘图和所述长焦图像的结构边缘图进行块匹配，得到第二响应图。
在本实施例中，可以计算缩小后的长焦图像的结构边缘图，并且计算短焦图像的结构边缘图，将两个图像的结构边缘图进行ZNCC块匹配，进而得到第二响应图。
其中，结构边缘图的计算方式可以采用随机森林算法，或者其他能够实现得到结构边缘图的算法。比如先将图像做一次双边滤波，抹去小的细节，留下结构信息，然后计算边缘图。
在步骤S240中，将所述第一响应图和所述第二响应图进行点积运算以定位所述长焦图像对应到所述短焦图像中的位置；并从所述短焦图像中抠出和所述长焦图像对应的部分，确定为短焦参考块。
每次ZNCC块匹配都可以得到一张响应图(response map)，响应图上值最大的位置就是长焦相机图像对应的位置。但是，这种块匹配算法并不鲁棒，很容易发生匹配错误的情况。所以本申请再加入结构边缘图来计算另一个响应 图，然后通过融合这两张响应图来得到最终的结果。融合的方式为对两张响应图进行点积(element-wise product)运算以得到新的响应图。其中Ir，Il为输入的短焦相机图像和长焦相机图像(缩小之后)，Er，El代表计算得到的结构边缘图。
在步骤S250中，从所述长焦图像和所述短焦参考块中取4对匹配点，确定所述长焦图像到所述短焦参考块的单应性矩阵；根据所述单应性矩阵，得到粗对齐长焦图像。
在步骤S260中，利用与所述短焦参考块的关键点匹配，对所述粗对齐长焦图像的网格控制点进行融合，得到融合图像。
本实施例在上述实施例的基础上，提供了确定短焦参考块的一种实现方式，本实施例采用将两种图像分别进行匹配并进行点积运算以确定短焦图像中短焦参考块的位置，这样设置可以提高本申请所提供的技术方案的准确性，鲁棒性好。
图3是本申请实施例提供的图像的多尺度融合方法的流程图。本实施例在上述实施例的基础上，细化为：从所述长焦图像和所述短焦参考块中取4对匹配点，包括：对所述短焦参考块按照预设方式切分，得到短焦参考块子块，作为搜索区域；对所述长焦图像按照所述预设方式切分，得到长焦图像子图像，并在每个长焦图像子图像中间选择预设区域作为模板；将所述模板和所述搜索区域进行匹配，得到4对匹配点。
如图3所示，所述图像的多尺度融合方法包括步骤S310至步骤S370。
在步骤S310中，获取一张短焦图像和至少一张长焦图像。
在步骤S320中，利用块匹配算法，确定短焦参考块；其中，所述短焦参考块是通过对所述短焦图像和所述长焦图像进行图像块匹配和结构边缘图块匹配得到的。
在步骤S330中，对所述短焦参考块按照预设方式切分，得到短焦参考块子块，作为搜索区域。
其中，预设方式可以是对所述短焦参考块按照长度方向和宽度方向划分为2×2的4个子块，得到4个短焦参考块子块，作为4个搜索区域。其中，预设方式还可以是其他方式，具体可以根据需求对此处进行设定。
在步骤S340中，对所述长焦图像按照所述预设方式切分，得到长焦图像子图像，并在每个长焦图像子图像中间选择预设区域作为模板。
在本实施例中，所述预设方式切分为按照长度方向和宽度方向划分为2×2的4个图像区域；相应的，所述将所述模板和所述搜索区域进行匹配，得到4对匹配点，包括：将所述4个模板和所述4个搜索区域进行对应位置匹配，得到每个所述模板和每个所述搜索区域的1对匹配点，共计得到4对匹配点。对于长 焦图像按照与短焦参考块相同的方式进行切分，这样设置便于找到与每个搜索区域对应的模板，快速地确定4对关键点，减小计算量，提高计算速度。所以此处也按照预设方式进行切分。
在步骤S350中，将所述模板和所述搜索区域进行匹配，得到4对匹配点。
将每个模板与对应的每个搜索区域进行匹配，确定4对匹配点。
在本实施例中，可以对将长焦图像的每个长焦图像子图像中，确定一个更小的范围，作为模板。例如，可以确定每个长焦图像的长和宽的75％，甚至50％的这一个区域作为模板，这样设置可以减小计算量，更加快速地找到匹配点。但是这个范围也不宜设置过小，如果设置的过小，可能会出现由于模板中没有比较明显的特征用来匹配的情况，影响匹配点的准确性。
在步骤S360中，确定所述长焦图像到所述短焦参考块的单应性矩阵；根据所述单应性矩阵，得到粗对齐长焦图像。
根据4对匹配点确定了由长焦图像到短焦参考块的单应性矩阵，再根据单应性矩阵，得到粗对齐的长焦图像。虽然这样得到的长焦图像的外部轮廓可能稍微有些扭曲，但是可以避免因为长焦相机和短焦相机之间的视差造成图像内容的扭曲，所以粗对齐长焦图像的图像内容相对于长焦图像而言，是由一定的校正效果的。
在步骤S370中，利用与所述短焦参考块的关键点匹配，对所述粗对齐长焦图像的网格控制点进行融合，得到融合图像。
本实施例在上述实施例的基础上，提供了一种对于长焦图像进行粗对齐的方法，这样设置可以在提高图像的准确性的同时，还能够减小工作量，提高图像融合过程中的速度，并且鲁棒性好。
图4是本申请实施例提供的图像的多尺度融合方法的流程图。本实施例在上述实施例的基础上，细化为：利用与所述短焦参考块的关键点匹配，对所述粗对齐长焦图像的网格控制点进行融合，得到融合图像，包括：对于所述短焦参考块进行关键点检测；对每个所述关键点，确定对应的关键点图像块；通过所述关键点图像块的结构边缘图计算关键点分数，对所述关键点分数低于预设阈值的关键点进行滤除；利用滤除后的关键点与所述粗对齐长焦图像进行关键点匹配；根据关键点匹配结果，对所述粗对齐长焦图像的网格控制点进行融合，得到融合图像。
如图4所示，所述图像的多尺度融合方法包括步骤S410至步骤S480。
在步骤S410中，获取一张短焦图像和至少一张长焦图像。
在步骤S420中，利用块匹配算法，确定短焦参考块；其中，所述短焦参考块是通过对所述短焦图像和所述长焦图像进行图像块匹配和结构边缘图块匹配得到的。
在步骤S430中，从所述长焦图像和所述短焦参考块中取4对匹配点，确定所述长焦图像到所述短焦参考块的单应性矩阵；根据所述单应性矩阵，得到粗对齐长焦图像。
在步骤S440中，对所述短焦参考块进行关键点检测。
在本实施例中，关键点指的是包含结构纹理信息丰富的点。原因是基于ZNCC块匹配算法需要足够多的结构纹理信息才能给出可靠的结果。在本实施例中，检测关键点使用的是经典的角点检测算法(goodfeaturestotrack算法)，采用该算法能够找到很多带有丰富结构纹理信息的关键点。
在步骤S450中，针对每个所述关键点，确定每个关键点对应的关键点图像块。
其中，关键点图像块为包含该关键点的图像块，在一实施例中，是以关键点位中心，以一个固定数量的像素点为半径来确定关键点图像块。例如，以检测出的关键点为中心，以32个像素点为关键点图像块的边长，来确定一个关键点图像块。
在步骤S460中，通过所述关键点图像块的结构边缘图计算关键点分数，对所述关键点分数低于预设阈值的关键点进行滤除。
在本实施例中，可以通过所述关键点图像块的结构边缘图计算关键点分数，包括：通过所述关键点图像块的结构边缘图中，计算所有像素点的像素值之和，将所有像素点的像素值之和确定为关键点分数。
其中，对于每个关键点，都可以以该关键点为中心，抠出一个关键点图像块，再利用关键点图像块的结构边缘图计算关键点分数。在本申请中，关键点的分数定义为：S i＝∑E(p i)；
S i代表第i个关键点的分数score，p i代表第i个关键点，E(p i)代表从抠出来的对应关键点图像块计算得到的结构边缘图，关键点的分数就定义为结构边缘图上所有像素值之和。
其中，可以优先对短焦参考块划分为一定数量的格子，例如可以是16个。在计算得到关键点分数之后，再对所有的关键点根据分数大小进行重排序，最后留下每个格子中分数最高的关键点。经过关键点筛选之后，每个格子中只会留下少量关键点，例如1到3个，这样可以大大减少了关键点的数量，提高计算速度。其中，也可以存在某些格子中并没有关键点的情况，在筛选之后，关键点的数量少了很多，有些格子没有关键点留下是因为没有检测到关键点或者是关键点分数太低，但这并不会影响高分数的关键点的确定。
在步骤S470中，利用滤除后的关键点与所述粗对齐长焦图像进行关键点匹配。
在本实施例中，利用滤除后的关键点与所述粗对齐长焦图像进行关键点匹 配，包括：在所述短焦参考块中，以滤除后的关键点为中心，以M个像素点为边长的块作为关键点匹配模板；在所述粗对齐长焦图像中，以滤除后的关键点在所述短焦参考块的坐标位置所对应的在所述粗对齐长焦图像中的像素点为中心，以N个像素点为边长的块作为关键点匹配搜索区域；其中M小于N；将所述关键点匹配模板和所述关键点匹配搜索区域进行匹配，得到在所述粗对齐长焦图像中与所述滤除后的关键点匹配的关键点。
在本实施例中，M为2的正整数次幂，N为2的正整数次幂。
可以对滤除后的关键点采用ZNCC匹配，如在短焦参考块上的以关键点为中心抠出256×256的块作为模板(在图像大小2000×1500的图像上使用256×256的块，如果是其他尺寸的输入图像可以对模板大小进行等比例调整。模板块的大小最好设置为2n，这种大小的块可以加速ZNCC模板匹配的计算)。在长焦相机图像上选取以同一个关键点坐标为中心，大小为512×512的块作为关键点匹配搜索区域(关键点匹配搜索区域为大图像，大小也随图像大小进行相应调整)。最后进行ZNCC匹配即可得到匹配的关键点。
在上述技术方案的基础上，在将所述关键点匹配模板和所述关键点匹配搜索区域进行匹配，得到在所述粗对齐长焦图像中与所述滤除后的关键点匹配的关键点之后，所述方法还包括：将所述短焦参考块和所述粗对齐长焦图像划分为网格；采用至少两种不同的尺度，对所述短焦参考块滤除后的关键点和所述粗对齐长焦图像中与所述滤除后的关键点匹配的关键点采用随机抽样一致算法进行去噪，得到高置信度的匹配关键点。
其中，可以对短焦图像和粗对齐长焦图像划分为16×16的网格，选取三种尺度(更加精细的网格则可以采用更多的尺度)，例如：全局尺度，对全部匹配点进行一次随机抽样一致算法(Random sample consensus，RANSAC)计算，将误差超过阈值的匹配点标记为噪声；2×2尺度，将16×16的网格分为2×2个区域，每个区域8×8个格子，分别对每个区域中的匹配点做一次RANSAC，将误差超过阈值的匹配点标记为噪声；4×4尺度，方法同2×2尺度。在这几个尺度的RANSAC中，阈值的大小可以根据实际情况进行调整。通常对于2000×1500的图像来说，选取3到7个像素左右的阈值比较合适。
在上述技术方案中，选择采用几种尺度可以根据对短焦图像和粗对齐长焦图像所划分的网格数有关系。例如如果划分为16×16的网格，则可以采用两种、三种甚至四种不同的网格尺度；如果划分为32×32的网格，则可以设计更多的尺度，可以根据实际计算过程中对计算速度和计算准确度的需求来确定。
最后，将所有被标记为噪声的点去除，剩下高置信度的匹配点，一些低置信度的匹配点就被去除了。这样可以利用多尺度的RANSAC来对匹配点进行去噪，降低噪声对图像融合结果的干扰，
在步骤S480中，根据关键点匹配结果，对所述粗对齐长焦图像的网格控制点进行融合，得到融合图像。
本实施例在上述实施例的基础上，提供了一种对于短焦参考块的关键点进行检测和过滤的方法，使得过滤后的短焦参考块关键点的特征更加明显，不仅提高了本方案的计算准确性，同时还减小了计算量。
在上述实施例中，对所述粗对齐长焦图像的网格控制点进行融合，得到融合图像，包括：将所述短焦参考块和所述粗对齐长焦图像划分为网格；采用至少两种不同的尺度，确定每个尺度所对应的单应性矩阵；根据所述每个尺度所对应的单应性矩阵，确定控制网格的网格控制点的位置；其中，在同一尺度内，被多个单应性矩阵覆盖的网格控制点，按照每个覆盖单应性矩阵的所计算出的位置的平均值，将所述平均值作为被多个单应性矩阵覆盖的网格控制点的位置；根据所得到的网格控制点的位置确定每个尺度下的目标网格；对所述每个尺度下的目标网格进行融合，得到融合图像。
其中，可以对粗对齐长焦图像划分为16×16的网格，则会有17×17个网格控制点，其中至少两种不同的尺度可以是全局尺度、2×2区域尺度、4×4区域尺度以及8×8区域尺度，还可以是其他尺度。其中，2×2区域尺度将16×16的网格切分为4个区域，每个区域为4×4个网格，这样使得2×2区域尺度刚好覆盖全部网格而且不会出现区域之间会有网格重合的情况，4×4区域尺度和8×8区域尺度也采用这种方式来确定。可以在每个尺度下确定对应的单应性矩阵，具体方式可以是选择每个尺度下的4对匹配点，如全局尺度则在全局范围内选择4对匹配点，在2×2区域尺度中，则在4个区域内，每个区域选择4对匹配点，并计算每个区域的单应性矩阵，以此类推。这样在每个尺度中，会有多个单应性矩阵，例如在在2×2区域尺度的4个区域中，以每个单应性矩阵来计算相应区域控制点的位置坐标，在同一个控制点被多个单应性矩阵控制的情况下，也就是控制点在所划分的区域的交界处时，可以通过求取所有覆盖这个控制点的单应性矩阵所确定的坐标的平均值，将平均值作为该控制点的坐标，从而确定该尺度下的目标网格。本技术方案给出了目标网格的控制点的确定方式，该方式计算简单，并且可以由较高的抗干扰能力，计算结果更加准确。
同样的，对于粗对齐长焦图像划分不同数量的网格，可以选取不同种的尺度，不仅选取尺度的数目不同，每种尺度还可以对应划分为不同的区域。例如如果划分为16×16的网格，则可以采用两种、三种甚至四种不同的网格尺度，其中三种尺度可以包括全局尺度、2×2区域尺度以及4×4区域尺度，还可以是全局尺度、2×2区域尺度以及8×8区域尺度；如果划分为32×32的网格，可以设计更多的尺度，可以根据实际计算过程中对计算速度和计算准确度的需求来确定。
在上述技术方案的基础上，对所述每个尺度下的目标网格进行融合，得到 融合图像，包括：以第一尺度的目标网格的控制点为中心确定置信区域；在第二尺度的目标网格的控制点在置信区域范围内的情况下，以第二尺度的目标网格的控制点为准；遍历所有的尺度，确定最终目标网格的控制点的位置；根据所述最终目标网格的控制点的位置确定融合图像，其中，所述第一尺度大于所述第二尺度。
其中，置信区域可以是以第一尺度的目标网格的控制点为中心，以预设长度为半径来确定。在本实施例中，可以以网格的长度和宽度较小的一个的10％作为置信区域的半径。在第二尺度网格的控制点落在置信区域内的情况下，接受第二尺度网格的控制点，在第二尺度网格的控制点没有落在置信区域内的情况下，不接受第二尺度网格的控制点。通过遍历所有的尺度，可确定最终目标网格，并根据最终目标网格的控制点的位置进行图像融合。这样可以利用控制点置信区间来对融合所有网格，进一步去除错误匹配点对结果造成的影响。
图5是本申请实施例提供的图像融合装置的结构示意图。如图5所示，所述图像融合装置，包括图像获取模块510，短焦参考块确定模块520，粗对齐长焦图像确定模块530以及图像融合模块540。
图像获取模块510，设置为获取一张短焦图像和至少一张长焦图像。
短焦参考块确定模块520，设置为利用块匹配算法，确定短焦参考块；其中，所述短焦参考块是通过对所述短焦图像和所述长焦图像进行图像块匹配和结构边缘图块匹配得到的。
粗对齐长焦图像确定模块530，设置为从所述长焦图像和所述短焦参考块中取4对匹配点，确定所述长焦图像到所述短焦参考块的单应性矩阵；根据所述单应性矩阵，得到粗对齐长焦图像。
图像融合模块540，设置为利用与所述短焦参考块的关键点匹配，对所述粗对齐长焦图像的网格控制点进行融合，得到融合图像。
本申请实施例所提供的技术方案，通过获取一张短焦图像和至少一张长焦图像；利用块匹配算法，确定短焦参考块；其中，所述短焦参考块是通过对所述短焦图像和所述长焦图像进行图像块匹配和结构边缘图块匹配得到的；从所述长焦图像和所述短焦参考块中取4对匹配点，确定所述长焦图像到所述短焦参考块的单应性矩阵；根据所述单应性矩阵，得到粗对齐长焦图像；利用与所述短焦参考块的关键点匹配，对所述粗对齐长焦图像的网格控制点进行融合，得到融合图像。通过采用本申请所提供的技术方案，可以实现降低硬件需求，提高图像融合速度与鲁棒性。
上述产品可执行本申请任意实施例所提供的方法，具备执行方法相应的功能模块。
本申请实施例还提供一种包含计算机可执行指令的存储介质，所述计算机 可执行指令在由计算机处理器执行时设置为执行一种图像的多尺度融合方法，该方法包括：获取一张短焦图像和至少一张长焦图像；利用块匹配算法，确定短焦参考块；其中，所述短焦参考块是通过对所述短焦图像和所述长焦图像进行图像块匹配和结构边缘图块匹配得到的；从所述长焦图像和所述短焦参考块中取4对匹配点，确定所述长焦图像到所述短焦参考块的单应性矩阵；根据所述单应性矩阵，得到粗对齐长焦图像；利用与所述短焦参考块的关键点匹配，对所述粗对齐长焦图像的网格控制点进行融合，得到融合图像。
存储介质——任何的多种类型的存储器设备或存储设备。术语“存储介质”旨在包括：安装介质，例如只读光盘(Compact Disc Read-Only Memory，CD-ROM)、软盘或磁带装置；计算机系统存储器或随机存取存储器，诸如动态随机存取存储器(Dynamic Random Access Memory，DRAM)、双倍数据速率随机存取存储器(Double Data Rate Random Access Memory，DDR RAM)、静态随机存取存储器(Static Random Access Memory，SRAM)、扩充数据输出随机存储器(Extended Data Output Random Access Memory，EDO RAM)，兰巴斯(Rambus)RAM等；非易失性存储器，诸如闪存、磁介质(例如硬盘或光存储)；寄存器或其它相似类型的存储器元件等。存储介质可以还包括其它类型的存储器或其组合。另外，存储介质可以位于程序在其中被执行的计算机系统中，或者可以位于不同的第二计算机系统中，第二计算机系统通过网络(诸如因特网)连接到计算机系统。第二计算机系统可以提供程序指令给计算机用于执行。术语“存储介质”可以包括可以驻留在不同位置中(例如在通过网络连接的不同计算机系统中)的两个或更多存储介质。存储介质可以存储可由一个或多个处理器执行的程序指令(例如具体实现为计算机程序)。
当然，本申请实施例所提供的一种包含计算机可执行指令的存储介质，其计算机可执行指令不限于如上所述的图像融合的操作，还可以执行本申请任意实施例所提供的图像的多尺度融合方法中的相关操作。
本申请实施例提供了一种终端，该终端中可集成本申请实施例提供的图像融合装置。图6为本申请实施例提供的一种终端的结构示意图。如图6所示，该终端可以包括：存储器601、中央处理器(Central Processing Unit，CPU)602(又称处理器，以下简称CPU)、电路板(图中未示出)和电源电路(图中未示出)。所述电路板安置在所述壳体围成的空间内部；所述CPU602和所述存储器601设置在所述电路板上；所述电源电路，设置为为所述终端的多个电路或器件供电；所述存储器601，设置为存储可执行程序代码；所述CPU602通过读取所述存储器601中存储的可执行程序代码来运行与所述可执行程序代码对应的计算机程序，以实现以下步骤：获取一张短焦图像和至少一张长焦图像；利用块匹配算法，确定短焦参考块；其中，所述短焦参考块是通过对所述短焦图像和 所述长焦图像进行图像块匹配和结构边缘图块匹配得到的；从所述长焦图像和所述短焦参考块中取4对匹配点，确定所述长焦图像到所述短焦参考块的单应性矩阵；根据所述单应性矩阵，得到粗对齐长焦图像；利用与所述短焦参考块的关键点匹配，对所述粗对齐长焦图像的网格控制点进行融合，得到融合图像。
所述终端还包括：外设接口603、RF(Radio Frequency，射频)电路605、音频电路606、扬声器611、电源管理芯片608、输入/输出(Input/Output，I/O)子系统609、触摸屏612、其他输入/控制设备610以及外部端口604，这些部件通过一个或多个通信总线或信号线607来通信。
应该理解的是，图示终端600仅仅是终端的一个范例，并且终端600可以具有比图中所示出的更多的或者更少的部件，可以组合两个或更多的部件，或者可以具有不同的部件配置。图中所示出的多种部件可以在包括一个或多个信号处理和/或专用集成电路在内的硬件、软件、或硬件和软件的组合中实现。
下面就本实施例提供的用于图像融合终端进行详细的描述，该终端以手机为例。
存储器601，所述存储器601可以被CPU602、外设接口603等访问，所述存储器601可以包括高速随机存取存储器，还可以包括非易失性存储器，例如一个或多个磁盘存储器件、闪存器件、或其他易失性固态存储器件。
外设接口603，所述外设接口603可以将设备的输入和输出外设连接到CPU602和存储器601。
I/O子系统609，所述I/O子系统609可以将设备上的输入输出外设，例如触摸屏612和其他输入/控制设备610，连接到外设接口603。I/O子系统609可以包括显示控制器6091和设置为控制其他输入/控制设备610的一个或多个输入控制器6092。其中，一个或多个输入控制器6092从其他输入/控制设备610接收电信号或者向其他输入/控制设备610发送电信号，其他输入/控制设备610可以包括物理按钮(按压按钮、摇臂按钮等)、拨号盘、滑动开关、操纵杆、点击滚轮。值得说明的是，输入控制器6092可以与以下任一个连接：键盘、红外端口、USB接口以及诸如鼠标的指示设备。
触摸屏612，所述触摸屏612是用户终端与用户之间的输入接口和输出接口，将可视输出显示给用户，可视输出可以包括图形、文本、图标、视频等。
I/O子系统609中的显示控制器6091从触摸屏612接收电信号或者向触摸屏612发送电信号。触摸屏612检测触摸屏上的接触，显示控制器6091将检测到的接触转换为与显示在触摸屏612上的用户界面对象的交互，即实现人机交互，显示在触摸屏612上的用户界面对象可以是运行游戏的图标、联网到相应网络的图标等。值得说明的是，设备还可以包括光鼠，光鼠是不显示可视输出的触摸敏感表面，或者是由触摸屏形成的触摸敏感表面的延伸。
RF电路605，主要设置为建立手机与无线网络(即网络侧)的通信，实现手机与无线网络的数据接收和发送。例如收发短信息、电子邮件等。例如，RF电路605接收并发送RF信号，RF信号也称为电磁信号，RF电路605将电信号转换为电磁信号或将电磁信号转换为电信号，并且通过该电磁信号与通信网络以及其他设备进行通信。RF电路605可以包括设置为执行这些功能的已知电路，其包括但不限于天线系统、RF收发机、一个或多个放大器、调谐器、一个或多个振荡器、数字信号处理器、编译码器(COder-DECoder，CODEC)芯片组、用户标识模块(Subscriber Identity Module，SIM)等等。
音频电路606，主要设置为从外设接口603接收音频数据，将该音频数据转换为电信号，并且将该电信号发送给扬声器611。
扬声器611，设置为将手机通过RF电路605从无线网络接收的语音信号，还原为声音并向用户播放该声音。
电源管理芯片608，设置为为CPU602、I/O子系统及外设接口所连接的硬件进行供电及电源管理。
本申请实施例提供的终端，可以实现降低硬件需求，提高图像融合速度与鲁棒性。
上述实施例中提供的图像融合装置、存储介质及终端可执行本申请任意实施例所提供的图像的多尺度融合方法，具备执行该方法相应的功能模块。未在上述实施例中详尽描述的技术细节，可参见本申请任意实施例所提供的图像的多尺度融合方法。

Claims (14)

  1. 一种图像的多尺度融合方法，包括：
    获取一张短焦图像和至少一张长焦图像；
    利用块匹配算法，确定短焦参考块；其中，所述短焦参考块是通过对所述短焦图像和所述长焦图像进行图像块匹配和结构边缘图块匹配得到的；
    从所述长焦图像和所述短焦参考块中取4对匹配点，确定所述长焦图像到所述短焦参考块的单应性矩阵；根据所述单应性矩阵，得到粗对齐长焦图像；
    利用与所述短焦参考块的关键点匹配，对所述粗对齐长焦图像的网格控制点进行融合，得到融合图像。
  2. 根据权利要求1所述的方法，其中，所述利用块匹配算法，确定短焦参考块，包括：
    利用块匹配算法，对所述短焦图像和所述长焦图像进行块匹配，得到第一响应图；
    对所述短焦图像的结构边缘图和所述长焦图像的结构边缘图进行块匹配，得到第二响应图；
    将所述第一响应图和所述第二响应图进行点积运算以定位所述长焦图像对应到所述短焦图像中的位置；并从所述短焦图像中抠出和所述长焦图像对应的部分，确定为短焦参考块。
  3. 根据权利要求1所述的方法，其中，从所述长焦图像和所述短焦参考块中取4对匹配点，包括：
    对所述短焦参考块按照预设方式进行切分，得到短焦参考块子块，作为搜索区域；
    对所述长焦图像按照所述预设方式进行切分，得到长焦图像子图像，并在每个长焦图像子图像中间选择预设区域作为模板；
    将所述模板和所述搜索区域进行匹配，得到4对匹配点。
  4. 根据权利要求3所述的方法，其中，所述预设方式进行切分为：按照长度方向和宽度方向划分为2×2的4个图像区域；
    所述将所述模板和所述搜索区域进行匹配，得到4对匹配点，包括：
    将所述4个模板和所述4个搜索区域进行对应位置匹配，得到每个所述模板和每个所述搜索区域的1对匹配点，共计得到4对匹配点。
  5. 根据权利要求1所述的方法，其中，利用与所述短焦参考块的关键点匹配，对所述粗对齐长焦图像的网格控制点进行融合，得到融合图像，包括：
    对所述短焦参考块进行关键点检测；
    针对每个所述关键点，确定所述每个关键点对应的关键点图像块；
    通过所述关键点图像块的结构边缘图计算关键点分数，对所述关键点分数低于预设阈值的关键点进行滤除；
    利用滤除后的关键点与所述粗对齐长焦图像进行关键点匹配；
    根据关键点匹配结果，对所述粗对齐长焦图像的网格控制点进行融合，得到融合图像。
  6. 根据权利要求5所述的方法，其中，通过所述关键点图像块的结构边缘图计算关键点分数，包括：
    通过所述关键点图像块的结构边缘图，计算所有像素点的像素值之和，将所有像素点的像素值之和确定为关键点分数。
  7. 根据权利要求5所述的方法，其中，利用滤除后的关键点与所述粗对齐长焦图像进行关键点匹配，包括：
    在所述短焦参考块中，以滤除后的关键点为中心，且以M个像素点为边长的块，作为关键点匹配模板；
    在所述粗对齐长焦图像中，以滤除后的关键点在所述短焦参考块的坐标位置所对应的在所述粗对齐长焦图像中的像素点为中心，且以N个像素点为边长的块，作为关键点匹配搜索区域；其中M小于N；
    将所述关键点匹配模板和所述关键点匹配搜索区域进行匹配，得到在所述粗对齐长焦图像中与所述滤除后的关键点匹配的关键点。
  8. 根据权利要求7所述的方法，其中，M为2的正整数次幂，N为2的正整数次幂。
  9. 根据权利要求7所述的方法，在将所述关键点匹配模板和所述关键点匹配搜索区域进行匹配，得到在所述粗对齐长焦图像中与所述滤除后的关键点匹配的关键点之后，还包括：
    将所述短焦参考块和所述粗对齐长焦图像划分为网格；
    采用至少两种不同的尺度，对所述短焦参考块滤除后的关键点和所述粗对齐长焦图像中与所述滤除后的关键点匹配的关键点，采用随机抽样一致算法进行去噪，得到高置信度的匹配关键点。
  10. 根据权利要求5所述的方法，其中，对所述粗对齐长焦图像的网格控制点进行融合，得到融合图像，包括：
    将所述短焦参考块和所述粗对齐长焦图像划分为网格；
    采用至少两种不同的尺度，确定每个尺度所对应的单应性矩阵；
    根据所述每个尺度所对应的单应性矩阵，确定控制网格的网格控制点的位置；其中，在同一尺度内，被多个单应性矩阵覆盖的网格控制点，按照每个覆盖单应性矩阵的所计算出的位置的平均值，将所述平均值作为被多个单应性矩阵覆盖的网格控制点的位置；
    根据所得到的网格控制点的位置确定每个尺度下的目标网格；
    对所述每个尺度下的目标网格进行融合，得到融合图像。
  11. 根据权利要求10所述的方法，其中，对所述每个尺度下的目标网格进行融合，得到融合图像，包括：
    以第一尺度的目标网格的控制点为中心确定置信区域；
    在第二尺度的目标网格的控制点在置信区域范围内的情况下，以第二尺度的目标网格的控制点为准；遍历所有的尺度，确定最终目标网格的控制点的位置；
    根据所述最终目标网格的控制点的位置确定融合图像；
    其中，所述第一尺度大于所述第二尺度。
  12. 一种图像融合装置，包括：
    图像获取模块，设置为获取一张短焦图像和至少一张长焦图像；
    短焦参考块确定模块，设置为利用块匹配算法，确定短焦参考块；其中，所述短焦参考块是通过对所述短焦图像和所述长焦图像进行图像块匹配和结构边缘图块匹配得到的；
    粗对齐长焦图像确定模块，设置为从所述长焦图像和所述短焦参考块中取4对匹配点，确定所述长焦图像到所述短焦参考块的单应性矩阵；根据所述单应性矩阵，得到粗对齐长焦图像；
    图像融合模块，设置为利用与所述短焦参考块的关键点匹配，对所述粗对齐长焦图像的网格控制点进行融合，得到融合图像。
  13. 一种计算机可读存储介质，其上存储有计算机程序，该程序被处理器执行时实现如权利要求1-11中任一所述的图像的多尺度融合方法。
  14. 一种终端，包括存储器，处理器及存储在存储器上并可在处理器运行的计算机程序，所述处理器执行所述计算机程序时实现如权利要求1-11中任一所述的图像的多尺度融合方法。
PCT/CN2019/103226 2018-08-31 2019-08-29 图像的多尺度融合方法、装置、存储介质及终端 WO2020043155A1 (zh)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
CN201811012085.XA CN109285136B (zh) 2018-08-31 2018-08-31 一种图像的多尺度融合方法、装置、存储介质及终端
CN201811012085.X 2018-08-31

Publications (1)

Publication Number Publication Date
WO2020043155A1 true WO2020043155A1 (zh) 2020-03-05

Family

ID=65183360

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2019/103226 WO2020043155A1 (zh) 2018-08-31 2019-08-29 图像的多尺度融合方法、装置、存储介质及终端

Country Status (2)

Country Link
CN (1) CN109285136B (zh)
WO (1) WO2020043155A1 (zh)

Cited By (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN111818304A (zh) * 2020-07-08 2020-10-23 杭州萤石软件有限公司 图像融合方法及装置
CN111882589A (zh) * 2020-06-23 2020-11-03 广州万维创新科技有限公司 基于图像的单目视觉slam的初始化方法

Families Citing this family (8)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN109285136B (zh) * 2018-08-31 2021-06-08 清华-伯克利深圳学院筹备办公室 一种图像的多尺度融合方法、装置、存储介质及终端
CN110217271A (zh) * 2019-05-30 2019-09-10 成都希格玛光电科技有限公司 基于图像视觉的快速轨道侵限识别监测系统及方法
CN110378850B (zh) * 2019-07-09 2021-03-30 浙江大学 一种结合块匹配和神经网络的变焦图像生成方法
CN111292278B (zh) * 2019-07-30 2023-04-07 展讯通信（上海）有限公司 图像融合方法及装置、存储介质、终端
CN111818262B (zh) * 2020-07-08 2021-12-03 杭州萤石软件有限公司 图像重建方法及装置
CN112261387B (zh) * 2020-12-21 2021-03-23 展讯通信（上海）有限公司 用于多摄像头模组的图像融合方法及装置、存储介质、移动终端
CN112702575B (zh) * 2020-12-23 2023-04-18 杭州中威电子股份有限公司 一种基于图像融合技术的多焦距图像采集装置及分析方法
CN113052900A (zh) * 2021-04-23 2021-06-29 深圳市商汤科技有限公司 一种位置确定方法及装置、电子设备和存储介质

Citations (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20160071238A1 (en) * 2013-04-30 2016-03-10 Sony Corporation Image processing device, image processing method, and program
CN107659774A (zh) * 2017-09-30 2018-02-02 深圳市未来媒体技术研究院 一种基于多尺度相机阵列的视频成像系统及视频处理方法
CN107734271A (zh) * 2017-10-16 2018-02-23 深圳市未来媒体技术研究院 高动态范围十亿像素视频生成方法
CN107959805A (zh) * 2017-12-04 2018-04-24 深圳市未来媒体技术研究院 基于混合相机阵列的光场视频成像系统及视频处理方法
CN108156383A (zh) * 2017-12-29 2018-06-12 清华大学 基于相机阵列的高动态十亿像素视频采集方法及装置
CN109285136A (zh) * 2018-08-31 2019-01-29 清华-伯克利深圳学院筹备办公室 一种图像的多尺度融合方法、装置、存储介质及终端

Family Cites Families (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN101866482B (zh) * 2010-06-21 2012-02-15 清华大学 基于摄像设备自标定技术的全景图拼接方法和装置
KR102170689B1 (ko) * 2014-01-14 2020-10-27 한화테크윈 주식회사 영상 정합을 위한 특징점 샘플링 방법
CN104200450B (zh) * 2014-08-25 2017-08-25 华南理工大学 一种红外热图像清晰度增强方法
CN105719239A (zh) * 2016-01-21 2016-06-29 科盾科技股份有限公司 一种用于图像拼接的方法及装置

Patent Citations (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20160071238A1 (en) * 2013-04-30 2016-03-10 Sony Corporation Image processing device, image processing method, and program
CN107659774A (zh) * 2017-09-30 2018-02-02 深圳市未来媒体技术研究院 一种基于多尺度相机阵列的视频成像系统及视频处理方法
CN107734271A (zh) * 2017-10-16 2018-02-23 深圳市未来媒体技术研究院 高动态范围十亿像素视频生成方法
CN107959805A (zh) * 2017-12-04 2018-04-24 深圳市未来媒体技术研究院 基于混合相机阵列的光场视频成像系统及视频处理方法
CN108156383A (zh) * 2017-12-29 2018-06-12 清华大学 基于相机阵列的高动态十亿像素视频采集方法及装置
CN109285136A (zh) * 2018-08-31 2019-01-29 清华-伯克利深圳学院筹备办公室 一种图像的多尺度融合方法、装置、存储介质及终端

Cited By (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN111882589A (zh) * 2020-06-23 2020-11-03 广州万维创新科技有限公司 基于图像的单目视觉slam的初始化方法
CN111818304A (zh) * 2020-07-08 2020-10-23 杭州萤石软件有限公司 图像融合方法及装置

Also Published As

Publication number Publication date
CN109285136A (zh) 2019-01-29
CN109285136B (zh) 2021-06-08

Similar Documents

Publication Publication Date Title
WO2020043155A1 (zh) 图像的多尺度融合方法、装置、存储介质及终端
US10043308B2 (en) Image processing method and apparatus for three-dimensional reconstruction
EP3614334B1 (en) Method for image processing and electronic device
CN110574026B (zh) 卷积引擎、电子设备和用于处理数据的方法
US10963727B2 (en) Method, device and storage medium for determining camera posture information
US10915998B2 (en) Image processing method and device
CN110574025B (zh) 用于合并交错通道数据的卷积引擎
CN110637297B (zh) 卷积引擎、数据处理方法和电子设备
CN109784250B (zh) 自动引导小车的定位方法和装置
US11392219B2 (en) Determining locations of electro-optical pens
WO2022156167A1 (zh) 图像处理方法及装置、电子设备、计算机可读存储介质、计算机程序及程序产品
CN110431563B (zh) 图像校正的方法和装置
EP3093822B1 (en) Displaying a target object imaged in a moving picture
WO2022042425A1 (zh) 视频数据处理方法、装置、计算机设备及存储介质
JP2014068264A (ja) 画像処理装置、画像処理方法、及びプログラム
CN110463177A (zh) 文档图像的校正方法及装置
WO2022001648A1 (zh) 图像处理方法、装置、设备及介质
CN114298902A (zh) 一种图像对齐方法、装置、电子设备和存储介质
CN111199169A (zh) 图像处理方法和装置
WO2018076720A1 (zh) 单手操控方法及操控系统
WO2022111461A1 (zh) 识别方法、装置及电子设备
CN111210506A (zh) 一种三维还原方法、系统、终端设备和存储介质
CN112102404B (zh) 物体检测追踪方法、装置及头戴显示设备
CN105049706A (zh) 一种图像处理方法及终端
CN110660091A (zh) 一种图像配准处理方法、装置和拍照批改作业系统

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 19854478

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 19854478

Country of ref document: EP

Kind code of ref document: A1