CN112017120A - Image synthesis method and device - Google Patents

Image synthesis method and device Download PDF

Info

Publication number
CN112017120A
CN112017120A CN202010923535.1A CN202010923535A CN112017120A CN 112017120 A CN112017120 A CN 112017120A CN 202010923535 A CN202010923535 A CN 202010923535A CN 112017120 A CN112017120 A CN 112017120A
Authority
CN
China
Prior art keywords
image
region
synthesized
pixel
detected
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Withdrawn
Application number
CN202010923535.1A
Other languages
Chinese (zh)
Inventor
徐小君
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Beijing Weijie Dongbo Information Technology Co ltd
Original Assignee
Beijing Weijie Dongbo Information Technology Co ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Beijing Weijie Dongbo Information Technology Co ltd filed Critical Beijing Weijie Dongbo Information Technology Co ltd
Priority to CN202010923535.1A priority Critical patent/CN112017120A/en
Publication of CN112017120A publication Critical patent/CN112017120A/en
Withdrawn legal-status Critical Current

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T3/00—Geometric image transformation in the plane of the image
    • G06T3/40—Scaling the whole image or part thereof
    • G06T3/4038—Scaling the whole image or part thereof for image mosaicing, i.e. plane images composed of plane sub-images
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N20/00—Machine learning
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/10—Segmentation; Edge detection
    • G06T7/11—Region-based segmentation
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/10—Segmentation; Edge detection
    • G06T7/13—Edge detection
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/30—Determination of transform parameters for the alignment of images, i.e. image registration
    • G06T7/33—Determination of transform parameters for the alignment of images, i.e. image registration using feature-based methods
    • G06T7/337—Determination of transform parameters for the alignment of images, i.e. image registration using feature-based methods involving reference images or patches
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/90—Determination of colour characteristics
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2200/00—Indexing scheme for image data processing or generation, in general
    • G06T2200/32—Indexing scheme for image data processing or generation, in general involving image mosaicing
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/10—Image acquisition modality
    • G06T2207/10004—Still image; Photographic image
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/20—Special algorithmic details
    • G06T2207/20081—Training; Learning

Abstract

The application discloses an image synthesis method and device. The method comprises the steps of respectively scratching the area to be detected from a plurality of images to be synthesized; acquiring edge feature points of the areas to be detected in each image to be synthesized, calculating the distance between the edge feature points of the areas to be detected in each image to be synthesized and the edge of the image to be synthesized, and determining the image synthesis sequence of the images to be synthesized according to the distance; and extracting characteristic points of each image to be synthesized, and registering and splicing the images according to the extracted characteristic points and the image synthesis sequence to obtain a final synthesized image. By adopting the technical scheme, the original image is preprocessed according to the size and the position of the area to be detected when the plurality of images are synthesized, the synthesizing sequence of the plurality of images is sorted out, the images are spliced and synthesized according to the synthesizing sequence of the images, and the accuracy and the speed of image synthesis are improved.

Description

Image synthesis method and device
Technical Field
The present application relates to the field of image processing, and in particular, to an image synthesis method and apparatus.
Background
In recent years, with the development of photographing apparatuses and the increase in the demand of people, a technique for synthesizing an image with a special effect has become popular.
However, in practical applications, after special effect synthesis is performed on an image input by a user, the image often has a splicing trace, and is difficult to be natural, so that better user experience cannot be provided.
Therefore, how to stitch images to realize automatic synthesis of images without stitching traces is a technical problem that needs to be solved urgently by those skilled in the art.
Disclosure of Invention
Based on the above, the application provides an image synthesis method and device, which determine the synthesis sequence of the images according to the positions of the areas to be detected in the images to be synthesized in the images, extract the feature points of the images to be synthesized, and then perform image splicing according to the synthesis sequence of the images, thereby improving the accuracy and speed of image synthesis.
The application provides an image synthesis method, which comprises the following steps:
respectively digging a region to be detected from a plurality of images to be synthesized;
acquiring edge feature points of the areas to be detected in each image to be synthesized, calculating the distance between the edge feature points of the areas to be detected in each image to be synthesized and the edge of the image to be synthesized, and determining the image synthesis sequence of the images to be synthesized according to the distance;
and extracting characteristic points of each image to be synthesized, and registering and splicing the images according to the extracted characteristic points and the image synthesis sequence to obtain a final synthesized image.
The image synthesis method described above, wherein the step of picking out the region to be detected from the plurality of images to be synthesized respectively, specifically includes the following sub-steps:
acquiring a known region and an unknown region in an image to be synthesized, and taking each point in the unknown region as a central point;
calculating the distance between the pixel color in the neighborhood taking each central point position as the center and taking the preset length as the radius and the pixel color at the position of the central point, and dividing the pixel point with the pixel distance larger than a set maximum threshold value and smaller than a set minimum threshold value into known regions so as to reduce the range of unknown regions;
and calculating the probability distribution of the known region types of the pixel points for the pixel points with the pixel distance between the set maximum threshold and the set minimum threshold, and dividing the unknown region into the corresponding known region types according to the probability.
The image synthesis method described above, wherein a pixel color distance is calculated with a certain pixel point in the unknown region as a center and a pixel in a neighborhood with a radius of a preset length, if the pixel color distance is greater than a given maximum threshold, the pixel point is divided into a foreground region of the known region, and if the pixel color distance is less than a given minimum threshold, the pixel point is divided into a background region of the known region.
The image synthesis method as described above, wherein the unknown region in the image is set to be a composition in which the foreground and the background are superimposed by weight, and the unknown region Ii(x, y) ═ α Fi + (1- α) Bi, where α denotes transparency, F is a foreground pixel, B is a background pixel, if the transparency α is 1, the unknown regions all belong to the foreground, if the transparency α is 0, the unknown regions all belong to the background, if the transparency α is between 0 and 1, the unknown regions are;
and establishing a relation model of alpha and a color characteristic vector in a machine learning mode, inputting the solved alpha into the relation model, if the mapping of the alpha and the color of the pixel point is a linear combination relation, setting the pixel point to belong to a foreground F, and otherwise, setting the pixel point to belong to a background B, thereby accurately scratching the area to be detected from the image to be synthesized.
The image synthesis method described above, wherein the illumination transition processing is performed on the junction of the stitched images to obtain the final synthesized image, and specifically: and after the spliced image is obtained by splicing, calculating the weighted average value of each color pixel at the boundary of the spliced image, and taking the weighted average value as the pixel after fusion and overlapping.
The present application also provides an image synthesizing apparatus including: the image registration and splicing system comprises an image segmentation module, an image synthesis sequence confirmation module and an image registration and splicing module;
the image segmentation module is used for respectively scratching the to-be-detected areas from the plurality of images to be synthesized;
the image synthesis sequence confirmation module is used for acquiring edge feature points of the areas to be detected in each image to be synthesized, calculating the distance between the edge feature points of the areas to be detected in each image to be synthesized and the edge of the image to be synthesized, and determining the image synthesis sequence of the images to be synthesized according to the distance;
and the image registration and splicing module is used for extracting the characteristic points of each image to be synthesized and performing image registration and splicing according to the extracted characteristic points and the image synthesis sequence to obtain a final synthesized image.
The image synthesis device as described above, wherein the image segmentation module specifically includes a selection sub-module, a search sub-module, and an area division sub-module;
the method comprises the steps that a submodule is selected to obtain a known area and an unknown area in an image to be synthesized, and each point in the unknown area is used as a central point; the searching submodule calculates the distance between the pixel color in the neighborhood with the preset length as the radius and the pixel color at the position of the central point by taking each central point as the center, and divides the pixel point with the pixel distance larger than a set maximum threshold value and smaller than a set minimum threshold value into known areas so as to narrow the range of unknown areas; and the region division submodule calculates the probability distribution of the known region type to which the pixel point belongs for the pixel point of which the pixel distance is between the set maximum threshold and the set minimum threshold, and divides the unknown region into the corresponding known region type according to the probability.
The image synthesis device as described above, wherein the region division submodule is specifically configured to calculate a pixel color distance with a pixel in a neighborhood of which a radius is a preset length and a certain pixel point in the unknown region as a center, divide the pixel point into a foreground region of the known region if the pixel color distance is greater than a given maximum threshold, and divide the pixel point into a background region of the known region if the pixel color distance is less than a given minimum threshold.
The image synthesizing apparatus as described above, wherein the region dividing sub-module,and the method is also specifically used for setting the unknown area in the image as a foreground and background which are superposed and formed according to the weight, wherein the unknown area Ii(x, y) ═ α Fi + (1- α) Bi, where α denotes transparency, F is a foreground pixel, B is a background pixel, if the transparency α is 1, the unknown regions all belong to the foreground, if the transparency α is 0, the unknown regions all belong to the background, if the transparency α is between 0 and 1, the unknown regions are;
and establishing a relation model of alpha and a color characteristic vector in a machine learning mode, inputting the solved alpha into the relation model, if the mapping of the alpha and the color of the pixel point is a linear combination relation, setting the pixel point to belong to a foreground F, and otherwise, setting the pixel point to belong to a background B, thereby accurately scratching the area to be detected from the image to be synthesized.
The image synthesis device described above, wherein the image synthesis module is configured to perform illumination transition processing on the spliced image boundary to obtain a final synthesized image, and specifically: and after the spliced image is obtained by splicing, calculating the weighted average value of each color pixel at the boundary of the spliced image, and taking the weighted average value as the pixel after fusion and overlapping.
The beneficial effect that this application realized is as follows: by adopting the technical scheme, the original image is preprocessed according to the size and the position of the area to be detected when the plurality of images are synthesized, the synthesizing sequence of the plurality of images is sorted out, the images are spliced and synthesized according to the synthesizing sequence of the images, and the accuracy and the speed of image synthesis are improved.
Drawings
In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the embodiments or the prior art will be briefly described below, it is obvious that the drawings in the following description are only some embodiments described in the present invention, and other drawings can be obtained by those skilled in the art according to the drawings.
Fig. 1 is a flowchart of an image synthesis method according to an embodiment of the present application;
FIG. 2 is a flowchart of a specific method for matting a region to be detected from an image to be synthesized;
FIG. 3 is a flow chart of a particular method of pre-processing an image to be synthesized;
FIG. 4 is a flowchart of a particular method of obtaining matching feature points from each image to be synthesized;
FIG. 5 is a flowchart of a specific method for registration and stitching of two images according to an image synthesis sequence based on extracted feature points;
fig. 6 is a schematic diagram of an image synthesizing apparatus according to the second embodiment of the present application.
Detailed Description
The technical solutions in the embodiments of the present invention are clearly and completely described below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are some, not all, embodiments of the present invention. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.
Example one
An embodiment of the present application provides an image synthesis method, as shown in fig. 1, including the following steps:
step 110, respectively digging out the areas to be detected from the plurality of images to be synthesized;
in the embodiment of the application, when a plurality of shot images at different angles in the same scene are subjected to synthesis processing (for example, a building under a certain landscape is shot, the building and the surrounding environment of the building are in each image, and the plurality of images are subjected to scene synthesis), firstly, an area to be detected is extracted from each image to be synthesized, the size of the area to be detected is calculated, and then, each image to be synthesized is zoomed to a fixed size, so that the synthesis of the images is facilitated;
specifically, for the extraction of the region to be detected, a region having a common feature point in each image to be synthesized may be extracted from all the images to be synthesized, and the region may be used as the region to be detected; or taking any image to be synthesized as an initial image, then obtaining a region of which the other images to be synthesized and the initial image have the least common characteristic points, and taking the region as a region to be detected; or the same object can be obtained from each image to be synthesized, and the region where the object is located is used as the region to be detected.
Referring to fig. 2, fig. 2 is a flowchart of a method for matting a same object from an image to be synthesized as a region to be detected according to an embodiment of the present application; the method specifically comprises the following substeps of scratching the region to be detected from the image to be synthesized:
step 210, acquiring a known region and an unknown region in an image to be synthesized, and taking each point in the unknown region as a central point;
the known region comprises a foreground region and a background region which can be clearly separated, the region between the foreground region and the background region is defined as an unknown region, the region to be detected is scratched, namely the foreground region is scratched, the unknown region is separated into the foreground region and the background region as far as possible, and the region to be detected is scratched.
Step 220, calculating the distance between the pixel color in the neighborhood with the preset length as the radius and the pixel color at the position of the central point by taking each central point as the center, and dividing the pixel points with the pixel distance larger than a set maximum threshold value and smaller than a set minimum threshold value into known areas so as to narrow the range of unknown areas;
with a certain pixel point I in the unknown regioni(x, y) as a center and a radius of a neighborhood of a predetermined length (e.g., r 5mm)
Figure BDA0002667527670000061
Inner pixel Ii(x ', y'), calculating the pixel color distance as:
Figure BDA0002667527670000062
if the pixel color distance is larger than a given maximum threshold value, the pixel point is divided into a foreground area of a known area, and if the pixel color distance is smaller than a given minimum threshold value, the pixel point is divided into a background area of the known area.
Step 230, for the pixel points with the pixel distance between the set maximum threshold and the set minimum threshold, calculating the probability distribution of the known region types to which the pixel points belong, and dividing the unknown region into the corresponding known region types according to the probability;
because not all the pixel points in the unknown region can be accurately divided into the known regions, the probability distribution of the pixel points in each unknown region is calculated, and whether the pixel points belong to a foreground region or a background region is determined according to the probability value;
specifically, an unknown region in an image to be synthesized is set as a foreground and a background which are superposed according to a certain weight, namely an unknown region Ii(x, y) ═ α Fi + (1- α) Bi, where α denotes transparency, F is a foreground pixel, B is a background pixel, if the transparency α is 1, the unknown regions all belong to the foreground, if the transparency α is 0, the unknown regions all belong to the background, if the transparency α is between 0 and 1, the unknown regions are;
carrying out derivation operation on the unknown region expression to obtain
Figure BDA0002667527670000063
Suppose that
Figure BDA0002667527670000064
And
Figure BDA0002667527670000065
very little, the above formula is simplified to the following form:
Figure BDA0002667527670000066
the energy equation is thus obtained (Ω is unknown region):
Figure BDA0002667527670000067
the above energy equation is expressed in bayesian form, i.e.:
argmaxF,B,α(F,B,α|C)＝argmaxF,B,αP(C|F,B,α)P(F)P(B)P(α)/P(C)＝argmaxF,B,αL(C|F,B,α)+L(F)+L(B)+L(α)
l is a logarithm, since the probabilities are all [0,1], if a large number of probabilities are multiplied together, and P (C) is a constant, the representation in the computer becomes 0; modeling L (C | F, B, alpha), L (F), L (B) and L (alpha), and for L (F), establishing probability distribution of foreground colors by using a clustering algorithm to show how large the probability of the currently selected F is, thereby realizing modeling of color distribution of the foreground; for L (B), the probability distribution of the background color is established by using a clustering algorithm to show how large the probability of the currently selected B is, so that the color distribution modeling of the background is realized; for L (alpha), assuming that the distribution of alpha is average, for the convex optimization problem, setting the derivative to be 0, and fixing F and B to solve alpha iteratively until the result is stable;
in the embodiment of the application, a relation model of alpha and a color feature vector can be established in a machine learning manner, the solved alpha is input into the relation model, if the mapping of the alpha and the color of the pixel point is a linear combination relation, the pixel point is set to belong to the foreground F, otherwise, the pixel point is set to belong to the background B, and therefore the region to be detected is accurately scratched from the image to be synthesized.
Referring back to fig. 1, step 120, rotating, zooming and translating the image to be synthesized according to the size and the position of the region to be detected;
in the embodiment of the application, after the region to be detected is extracted from the plurality of images to be synthesized, the problem that the size or the position of the region to be detected is not matched may exist, so that the whole image is zoomed and/or rotated before image splicing is carried out, and the size and the position of the region to be detected in each image are matched;
referring to fig. 3, the scaling of the image specifically includes the following sub-steps:
step 310, judging whether rotation operation needs to be carried out on the images to be synthesized according to the edge feature points of the areas to be detected in each image to be synthesized, if so, rotating each image to be synthesized into the state that the vector directions of the edge feature points are consistent, and then executing step 320, otherwise, directly executing step 320;
specifically, the edge feature points are obtained from each to-be-detected area, then whether all color pixels of the edge feature points at each corresponding position in each to-be-detected area are consistent or not is judged, if yes, image rotation is not needed, otherwise, the whole original image of the to-be-detected area with inconsistent color pixels is rotated, and the color pixels of the edge feature points in each to-be-synthesized image are consistent.
Step 320, calculating the size of the area to be detected in each image to be synthesized, and scaling the vector of each pixel point of each image to be synthesized in the two-dimensional plane;
specifically, a topmost feature point and a bottommost feature point of a region to be detected in each image to be synthesized are obtained, a first difference value between the topmost feature point and the bottommost feature point is calculated, or a leftmost feature point and a rightmost feature point of the region to be detected in each image to be synthesized are obtained, a second difference value between the leftmost feature point and the rightmost feature point is calculated, whether the image needs to be zoomed is determined according to the first difference value or the second difference value calculated by the region to be detected in each image to be synthesized, wherein the zooming is to perform equal-proportion zooming on the transverse dimension and the longitudinal dimension;
carrying out zooming operation on each image to be synthesized to convert the zooming operation into the zooming of each pixel point in each image to be synthesized, regarding the image as a vector on a two-dimensional plane, and zooming the vector (comprising coordinate values in the vector X direction and the vector Y direction) of each pixel point to obtain the zooming of the image; specifically, each pixel point of the original image is firstly multiplied by a matrix
Figure BDA0002667527670000081
Forward mapping to corresponding pixel points of new image
Figure BDA0002667527670000082
Figure BDA0002667527670000083
Then multiplying both sides of the matrix expression by the inverse of the amplification matrix
Figure BDA0002667527670000084
Figure BDA0002667527670000085
And mapping the zoomed pixel points to the corresponding pixel points of the new image.
Step 330, translating each zoomed image to be synthesized to the same splicing direction according to the position of the area to be detected in the zoomed image to be synthesized;
the image to be synthesized after the translation and the zoom is specifically that all pixel coordinates of the image to be synthesized after the zoom are respectively added with a specified horizontal offset and a specified vertical offset, so that the area to be detected in the image to be synthesized after the zoom is in the same horizontal plane, that is, only the position of the image to be synthesized after the zoom is moved, and the relative position of the area to be detected in the image is not changed;
taking an image of a region to be detected closest to the upper left of the image as a reference image, if the region to be detected in some zoomed images to be synthesized and the reference image need to be spliced along the x-axis direction, translating the images to be synthesized to the x-axis direction of the reference image, and if the region to be detected in some zoomed images to be synthesized and the reference image need to be spliced along the y-axis direction, translating the images to be synthesized to the y-axis direction of the reference image.
It should be noted that the preprocessing of rotating, scaling and translating the image to be synthesized in step 120 is performed based on the different size positions of the plurality of images to be synthesized, and this step may be skipped directly when the size position of the region to be detected of the image to be synthesized is detected to be appropriate.
Referring back to fig. 1, step 130, obtaining edge feature points of the to-be-detected area in each zoomed image to be synthesized, calculating a distance between the edge feature point of the to-be-detected area in each zoomed image to be synthesized and the edge of the to-be-synthesized image, and determining an image synthesis sequence of the zoomed images to be synthesized according to the distance;
in the embodiment of the application, because the positions of the areas to be detected of the zoomed images to be synthesized are different, the images to be synthesized after all zoomed images to be synthesized are sequenced according to the positions of the areas to be detected and then spliced can be spliced more conveniently; specifically, the edge feature points of the area to be detected in each zoomed image to be synthesized are obtained, and the left edge of the image to be synthesized and the upper edge of the image to be synthesized, which are matched with the left edge and the upper edge of the reference image set in step 330, are searched for in each zoomed image to be synthesized:
for other zoomed images to be synthesized in the x-axis direction of the reference image, calculating the vertical distance between the leftmost characteristic point of the region to be detected in the other zoomed images to be synthesized translated in the x-axis direction of the reference image and the left edge of the image on the image, and sorting the images from small to large according to the vertical distance;
and for other zoomed images to be synthesized in the y-axis direction of the reference image, calculating the vertical distance between the uppermost characteristic point of the region to be detected in the other zoomed images to be synthesized translated in the y-axis direction of the reference image and the upper edge of the image on the image, and sorting the images from small to large according to the vertical distance.
Step 140, extracting feature points of each zoomed image to be synthesized, and registering and splicing the zoomed images to be synthesized according to the extracted feature points and the image synthesis sequence;
in the embodiment of the present application, obtaining matching feature points from each zoomed image to be synthesized specifically includes, as shown in fig. 4, the following sub-steps:
step 410, converting the zoomed image to be synthesized into a gray image, and obtaining the positions of feature points with different scales by using a non-maximum suppression method;
step 420, calculating the wavelet response of each feature point in the horizontal direction and the vertical direction in a circular area with a first preset number as the radius by taking each point in the zoomed image to be synthesized as the center;
step 430, taking each feature point as a center, and taking a second preset number as a radius in a sector area, calculating a wavelet response accumulated value in the sector area, and taking the maximum accumulated value as the main direction of the feature point;
specifically, the modulus m (x, y) and the direction θ (x, y) of the gradient of each point L (x, y) are calculated using the following formula:
Figure BDA0002667527670000101
Figure BDA0002667527670000102
after the gradient direction is obtained through calculation, counting the gradient direction and the amplitude corresponding to the pixels in the neighborhood of the feature point by using a histogram, specifically, the horizontal axis of the histogram of the gradient direction is the angle of the gradient direction, and the vertical axis of the histogram of the gradient direction is the accumulation of the gradient amplitude corresponding to the gradient direction, so that the main direction of the feature point is obtained;
and step 440, calculating the absolute value of the sum of the main direction of each feature point and the response value perpendicular to the main direction, and taking the absolute value as a feature value to form a feature vector set.
The image registration and stitching of the zoomed two images to be synthesized is performed according to the extracted feature points and the image synthesis sequence, as shown in fig. 5, the method specifically comprises the following substeps:
step 510, obtaining a plurality of optimal matching points which are not influenced by a scale space from the extracted feature points;
step 520, obtaining projection mapping matrixes of two zoomed images to be synthesized, selecting a plurality of groups of optimal matching points from the optimal matching points, and calculating the degree of freedom parameters of the projection mapping matrixes of the first zoomed image to be synthesized and the second zoomed image to be synthesized, wherein the degree of freedom parameters are used as initial values;
step 530, repeatedly iterating and using the transformation matrix to search and determine the correspondence of the feature points in the area near the epipolar line until the number of the feature points is stable, so as to obtain a spliced image;
it should be noted that, the operation of the image synthesis order of the images to be synthesized in step 130 is performed based on the situation that the order of the images to be synthesized is not arranged according to the optimal order adjusted in step 130, and this step can be directly skipped when the order of the images to be synthesized is the optimal order; alternatively, step 140 may be performed directly without adjusting the order of the images to be synthesized;
correspondingly, step 140 is to perform feature point extraction on each zoomed image to be synthesized, and perform image registration and stitching on the zoomed images to be synthesized according to the extracted feature points.
Referring back to fig. 1, step 150, performing illumination transition processing on the junction of the spliced images to obtain a final composite image.
In the embodiment of the application, after the spliced image is obtained by splicing, calculating the weighted average value of each color pixel at the junction of the spliced image, and taking the weighted average value as the pixel after fusion and overlapping; assume that each pixel in the image is: i isi(x,y)＝(αiR,αiG,αiB,αjAnd, where (R, G, B) is the color value of the pixel, the pixel value calculated (x, y) in the stitched output image is:
(α1R,α1G,α1B,α1)+(α2R,α2G,α2B,α2)]/(α1+α2)。
example two
The second embodiment of the present application provides an image synthesis apparatus, as shown in fig. 6, including an image segmentation module 61, an image preprocessing module 62, an image synthesis order confirmation module 63, an image registration and stitching module 64, and an image synthesis module 65;
the image segmentation module 61 is used for respectively matting the to-be-detected areas from the plurality of images to be synthesized; the image preprocessing module 62 performs rotation, scaling and translation of the image to be synthesized according to the size and position of the region to be detected; the image synthesis sequence confirmation module 63 obtains the edge feature point of the area to be detected in each image to be synthesized, calculates the distance between the edge feature point of the area to be detected in each image to be synthesized and the edge of the image to be synthesized, and determines the image synthesis sequence of the image to be synthesized according to the distance; the image registration and stitching module 64 extracts the feature points of each image to be synthesized, and performs image registration and stitching according to the extracted feature points and the image synthesis sequence; the image synthesis module 65 performs illumination transition processing on the spliced image boundary to obtain a final synthesized image.
Specifically, the image segmentation module 61 is specifically configured to extract the regions to be detected from the multiple images to be synthesized, specifically, extract a region having a common feature point in each image to be synthesized from all the images to be synthesized, and use the region as the region to be detected; or taking any image to be synthesized as an initial image, then obtaining a region of which the other images to be synthesized and the initial image have the least common characteristic points, and taking the region as a region to be detected; or the same object is obtained from each image to be synthesized, and the region where the object is located is used as the region to be detected.
The image segmentation module 61 specifically includes a selection sub-module 611, a search sub-module 612, and an area division sub-module 613;
the selecting submodule 611 acquires a known region and an unknown region in the image to be synthesized, and each point in the unknown region is used as a central point; the search submodule 612 calculates a distance between a pixel color in a neighborhood with a preset length as a radius and a pixel color at a position of the center point, with each center point as a center, and divides a pixel point with the pixel distance larger than a set maximum threshold and smaller than a set minimum threshold into a known region to narrow the range of an unknown region; the region division submodule 613 calculates the probability distribution of the known region type to which the pixel point belongs for the pixel point whose pixel distance is between the set maximum threshold and the set minimum threshold, and divides the unknown region into the corresponding known region type according to the probability.
The area division submodule 613 is specifically configured to calculate a pixel color distance with a certain pixel point in the unknown area as a center and a pixel in a neighborhood with a radius of a preset length, divide the pixel point into a foreground area of the known area if the pixel color distance is greater than a given maximum threshold, and divide the pixel point into a background area of the known area if the pixel color distance is less than a given minimum threshold.
In addition, the region division sub-module 613 is further specifically configured to set an unknown region in the image to be a foreground and a background which are superimposed according to a weight, where the unknown region I is an unknown regioni(x, y) ═ α Fi + (1- α) Bi, where α denotes transparency, F is a foreground pixel, B is a background pixel, if the transparency α is 1, the unknown regions all belong to the foreground, if the transparency α is 0, the unknown regions all belong to the background, if the transparency α is between 0 and 1, the unknown regions are; and establishing a relation model of alpha and a color characteristic vector in a machine learning mode, inputting the solved alpha into the relation model, if the mapping of the alpha and the color of the pixel point is a linear combination relation, setting the pixel point to belong to a foreground F, and otherwise, setting the pixel point to belong to a background B, thereby accurately scratching the area to be detected from the image to be synthesized.
Further, the image synthesis apparatus further includes an image preprocessing module 62, configured to perform rotation, scaling, and translation of the image to be synthesized according to the size and the position of the region to be detected; the image preprocessing module 62 specifically includes a rotation determination sub-module 621, a scaling sub-module 622, and a translation sub-module 623;
the rotation determination submodule 621 determines whether to perform rotation operation on the image to be synthesized according to the edge feature point of the area to be detected in each image to be synthesized, and if so, rotates each image to be synthesized to make the vector directions of the edge feature points consistent, and triggers the scaling submodule 622, and if not, directly triggers the scaling submodule 622; the scaling submodule 622 is configured to calculate a size of a region to be detected in each image to be synthesized, and scale a vector of each pixel point of each image to be synthesized in the two-dimensional plane; the translation sub-module 623 is configured to translate each zoomed image to be synthesized to the same stitching direction according to the position of the region to be detected in the zoomed image to be synthesized.
Specifically, the rotation determination submodule 621 is specifically configured to acquire the edge feature point from each to-be-detected region, determine whether all color pixels of the edge feature point at each corresponding position in each to-be-detected region are consistent, if so, do not need to perform a rotation operation, otherwise rotate the entire original image of the to-be-detected region where the color pixels are inconsistent, so that the color pixels of the edge feature point are consistent.
The scaling submodule 622 is specifically configured to obtain a topmost feature point and a bottommost feature point of a region to be detected in each image to be synthesized, calculate a first difference between the topmost feature point and the bottommost feature point, or obtain a leftmost feature point and a rightmost feature point of the region to be detected in each image to be synthesized, calculate a second difference between the leftmost feature point and the rightmost feature point, and determine whether the image needs scaling according to the first difference or the second difference calculated in the region to be detected in each image to be synthesized, where scaling is to perform equal-proportion scaling on the lateral and longitudinal dimensions.
The translation sub-module 623 is specifically configured to add specified horizontal offset and vertical offset to all pixel coordinates of the zoomed image to be synthesized, so that the zoomed region to be detected in the image to be synthesized is on the same horizontal plane, that is, only the position of the zoomed image to be synthesized is moved, and the relative position of the region to be detected in the image is not changed; taking an image of a region to be detected closest to the upper left of the image as a reference image, if the region to be detected in some zoomed images to be synthesized and the reference image need to be spliced along the x-axis direction, translating the images to be synthesized to the x-axis direction of the reference image, and if the region to be detected in some zoomed images to be synthesized and the reference image need to be spliced along the y-axis direction, translating the images to be synthesized to the y-axis direction of the reference image.
After the image preprocessing module finishes the preprocessing operation, the image synthesis device further comprises an image synthesis sequence determining module 63, configured to obtain edge feature points of the to-be-detected area in each zoomed image to be synthesized, calculate a distance between the edge feature point of the to-be-detected area in each zoomed image to be synthesized and the edge of the to-be-synthesized image, and determine an image synthesis sequence of the zoomed images to be synthesized according to the distance;
further, the image synthesis sequence confirming module 63 is specifically configured to obtain edge feature points of the area to be detected in each of the zoomed images to be synthesized, and search, in each zoomed image to be synthesized, a left edge of the image to be synthesized and an upper edge of the image to be synthesized, which are matched with a left edge and an upper edge of the area to be detected in the image to be synthesized, which are closest to the upper left reference image in the image; for other zoomed images to be synthesized in the x-axis direction of the reference image, calculating the vertical distance between the leftmost characteristic point of the region to be detected in the other zoomed images to be synthesized translated in the x-axis direction of the reference image and the left edge of the image on the image, and sorting the images from small to large according to the vertical distance; and for other zoomed images to be synthesized in the y-axis direction of the reference image, calculating the vertical distance between the uppermost characteristic point of the region to be detected in the other zoomed images to be synthesized translated in the y-axis direction of the reference image and the upper edge of the image on the image, and sorting the images from small to large according to the vertical distance.
In the embodiment of the present application, in the image registration and stitching module 64, feature point extraction is performed on each image to be synthesized, and specifically includes a feature point position determining submodule 641, a feature point calculating submodule 642, a feature point principal direction determining submodule 643, and a feature vector set calculating submodule 644;
the feature point position determining submodule 641 converts the zoomed image to be synthesized into a grayscale image, and obtains feature point positions of different scales by using a non-maximum suppression method; the feature point calculation submodule 642 calculates wavelet responses of each feature point in the horizontal direction and the vertical direction in a circular area with a first preset number as a radius by taking each point in the zoomed image to be synthesized as a center; the feature point main direction determining sub-module 643 calculates the wavelet response accumulated value in a sector area with each feature point as the center and a second predetermined number as the radius, and takes the maximum accumulated value as the main direction of the feature point; the feature vector set calculation sub-module 644 calculates the absolute value of the sum of the principal direction of each feature point and the response value perpendicular to the principal direction, and uses this as a feature value to form a feature vector set.
The feature point main direction determining submodule 643 is specifically configured to calculate a modulus and a direction of a gradient of each point; after the gradient direction is obtained through calculation, the histogram is used for counting the gradient direction and the amplitude corresponding to the pixels in the neighborhood of the feature point, specifically, the horizontal axis of the histogram of the gradient direction is the angle of the gradient direction, and the vertical axis of the histogram of the gradient direction is the accumulation of the gradient amplitude corresponding to the gradient direction, so that the main direction of the feature point is obtained.
In addition, in the image registration and stitching module 64, every two images of the zoomed images to be synthesized are registered and stitched according to the extracted feature points according to the image synthesis sequence, and the image registration and stitching module is specifically used for acquiring a plurality of optimal matching points which are not influenced by the scale space from the extracted feature points; acquiring projection mapping matrixes of two zoomed images to be synthesized, selecting a plurality of groups of optimal matching points from a plurality of optimal matching points, and calculating the degree of freedom parameters of the projection mapping matrixes of a first zoomed image to be synthesized and a second zoomed image to be synthesized, wherein the degree of freedom parameters are used as initial values; and repeatedly iterating and searching the area near the epipolar line by using the transformation matrix to determine the correspondence of the characteristic points until the number of the corresponding characteristic points is stable, thereby obtaining a spliced image.
The image synthesis module 65 is specifically configured to perform illumination transition processing on the spliced image boundary to obtain a final synthesized image, and specifically includes: and after the spliced image is obtained by splicing, calculating the weighted average value of each color pixel at the boundary of the spliced image, and taking the weighted average value as the pixel after fusion and overlapping.
The above-mentioned embodiments are only specific embodiments of the present application, and are used for illustrating the technical solutions of the present application, but not limiting the same, and the scope of the present application is not limited thereto, and although the present application is described in detail with reference to the foregoing embodiments, those skilled in the art should understand that: any person skilled in the art can modify or easily conceive the technical solutions described in the foregoing embodiments or equivalent substitutes for some technical features within the technical scope disclosed in the present application; such modifications, changes or substitutions do not depart from the spirit and scope of the present disclosure, which should be construed in light of the above teachings. Are intended to be covered by the scope of the present application. Therefore, the protection scope of the present application shall be subject to the protection scope of the claims.

Claims (10)

1. An image synthesis method, comprising:
respectively digging a region to be detected from a plurality of images to be synthesized;
acquiring edge feature points of the areas to be detected in each image to be synthesized, calculating the distance between the edge feature points of the areas to be detected in each image to be synthesized and the edge of the image to be synthesized, and determining the image synthesis sequence of the images to be synthesized according to the distance;
and extracting characteristic points of each image to be synthesized, and registering and splicing the images according to the extracted characteristic points and the image synthesis sequence to obtain a final synthesized image.
2. The image synthesis method according to claim 1, wherein the region to be detected is extracted from the plurality of images to be synthesized, specifically, a region having a common feature point in each image to be synthesized is extracted from all the images to be synthesized, and the region is taken as the region to be detected; or taking any image to be synthesized as an initial image, then obtaining a region of which the other images to be synthesized and the initial image have the least common characteristic points, and taking the region as a region to be detected; or the same object is obtained from each image to be synthesized, and the region where the object is located is used as the region to be detected.
3. The image synthesis method according to claim 2, wherein the same object is obtained from each image to be synthesized, and the region where the object is located is taken as the region to be detected, specifically comprising the following substeps:
acquiring a known region and an unknown region in an image to be synthesized, and taking each point in the unknown region as a central point;
calculating the distance between the pixel color in the neighborhood taking each central point position as the center and taking the preset length as the radius and the pixel color at the position of the central point, and dividing the pixel point with the pixel distance larger than a set maximum threshold value and smaller than a set minimum threshold value into known regions so as to reduce the range of unknown regions;
and calculating the probability distribution of the known region types of the pixel points for the pixel points with the pixel distance between the set maximum threshold and the set minimum threshold, and dividing the unknown region into the corresponding known region types according to the probability.
4. The image synthesis method according to claim 3, wherein a pixel color distance is calculated with a certain pixel point in the unknown region as a center and a pixel in a neighborhood with a radius of a preset length, and if the pixel color distance is greater than a given maximum threshold, the pixel point is classified into a foreground region of the known region, and if the pixel color distance is less than a given minimum threshold, the pixel point is classified into a background region of the known region.
5. An image synthesis method according to claim 3, wherein the unknown region in the image is set to be composed of a foreground and a background superimposed by a weight, and the unknown region I is set to bei(x, y) ═ α Fi + (1- α) Bi, where α denotes transparency, F is a foreground pixel, B is a background pixel, if the transparency α is 1, the unknown regions all belong to the foreground, if the transparency α is 0, the unknown regions all belong to the background, if the transparency α is between 0 and 1, the unknown regions are;
and establishing a relation model of alpha and a color characteristic vector in a machine learning mode, inputting the solved alpha into the relation model, if the mapping of the alpha and the color of the pixel point is a linear combination relation, setting the pixel point to belong to a foreground F, and otherwise, setting the pixel point to belong to a background B, thereby accurately scratching the area to be detected from the image to be synthesized.
6. An image synthesizing apparatus, comprising: the image registration and splicing system comprises an image segmentation module, an image synthesis sequence confirmation module and an image registration and splicing module;
the image segmentation module is used for respectively scratching the to-be-detected areas from the plurality of images to be synthesized;
the image synthesis sequence confirmation module is used for acquiring edge feature points of the areas to be detected in each image to be synthesized, calculating the distance between the edge feature points of the areas to be detected in each image to be synthesized and the edge of the image to be synthesized, and determining the image synthesis sequence of the images to be synthesized according to the distance;
and the image registration and splicing module is used for extracting the characteristic points of each image to be synthesized and performing image registration and splicing according to the extracted characteristic points and the image synthesis sequence to obtain a final synthesized image.
7. The image synthesis apparatus according to claim 6, wherein the image segmentation module is specifically configured to extract the region to be detected from the plurality of images to be synthesized, specifically, extract a region having a common feature point in each image to be synthesized from all the images to be synthesized, and use the region as the region to be detected; or taking any image to be synthesized as an initial image, then obtaining a region of which the other images to be synthesized and the initial image have the least common characteristic points, and taking the region as a region to be detected; or the same object is obtained from each image to be synthesized, and the region where the object is located is used as the region to be detected.
8. The image synthesis apparatus according to claim 7, wherein the image segmentation module specifically includes a selection sub-module, a search sub-module, and a region division sub-module;
the method comprises the steps that a submodule is selected to obtain a known area and an unknown area in an image to be synthesized, and each point in the unknown area is used as a central point; the searching submodule calculates the distance between the pixel color in the neighborhood with the preset length as the radius and the pixel color at the position of the central point by taking each central point as the center, and divides the pixel point with the pixel distance larger than a set maximum threshold value and smaller than a set minimum threshold value into known areas so as to narrow the range of unknown areas; and the region division submodule calculates the probability distribution of the known region type to which the pixel point belongs for the pixel point of which the pixel distance is between the set maximum threshold and the set minimum threshold, and divides the unknown region into the corresponding known region type according to the probability.
9. The image synthesis apparatus according to claim 7, wherein the region division submodule is specifically configured to calculate a pixel color distance with a pixel in a neighborhood of a predetermined length and a radius of a certain pixel in the unknown region as a center, divide the pixel into a foreground region of the known region if the pixel color distance is greater than a given maximum threshold, and divide the pixel into a background region of the known region if the pixel color distance is less than a given minimum threshold.
10. An image synthesis apparatus as claimed in claim 7, wherein the region division sub-module is further configured to set the unknown region in the image to be a weighted overlap composition of foreground and background, the unknown region Ii(x, y) ═ α Fi + (1- α) Bi, where α denotes transparency, F is a foreground pixel, B is a background pixel, if the transparency α is 1, the unknown regions all belong to the foreground, if the transparency α is 0, the unknown regions all belong to the background, if the transparency α is between 0 and 1, the unknown regions are;
and establishing a relation model of alpha and a color characteristic vector in a machine learning mode, inputting the solved alpha into the relation model, if the mapping of the alpha and the color of the pixel point is a linear combination relation, setting the pixel point to belong to a foreground F, and otherwise, setting the pixel point to belong to a background B, thereby accurately scratching the area to be detected from the image to be synthesized.
CN202010923535.1A 2020-09-04 2020-09-04 Image synthesis method and device Withdrawn CN112017120A (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202010923535.1A CN112017120A (en) 2020-09-04 2020-09-04 Image synthesis method and device

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202010923535.1A CN112017120A (en) 2020-09-04 2020-09-04 Image synthesis method and device

Publications (1)

Publication Number Publication Date
CN112017120A true CN112017120A (en) 2020-12-01

Family

ID=73516856

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202010923535.1A Withdrawn CN112017120A (en) 2020-09-04 2020-09-04 Image synthesis method and device

Country Status (1)

Country Link
CN (1) CN112017120A (en)

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN112506228A (en) * 2020-12-28 2021-03-16 广东电网有限责任公司中山供电局 Substation unmanned aerial vehicle optimal emergency hedge path selection method

Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN105205781A (en) * 2015-08-24 2015-12-30 电子科技大学 Power transmission line aerial photographing image splicing method
US20160133027A1 (en) * 2014-11-12 2016-05-12 Ricoh Company, Ltd. Method and apparatus for separating foreground image, and non-transitory computer-readable recording medium
CN106683043A (en) * 2015-11-10 2017-05-17 中国航天科工集团第四研究院指挥自动化技术研发与应用中心 Parallel image stitching method and device for multi-channel optical detection system
CN110197455A (en) * 2019-06-03 2019-09-03 北京石油化工学院 Acquisition methods, device, equipment and the storage medium of two-dimensional panoramic image

Patent Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20160133027A1 (en) * 2014-11-12 2016-05-12 Ricoh Company, Ltd. Method and apparatus for separating foreground image, and non-transitory computer-readable recording medium
CN105205781A (en) * 2015-08-24 2015-12-30 电子科技大学 Power transmission line aerial photographing image splicing method
CN106683043A (en) * 2015-11-10 2017-05-17 中国航天科工集团第四研究院指挥自动化技术研发与应用中心 Parallel image stitching method and device for multi-channel optical detection system
CN110197455A (en) * 2019-06-03 2019-09-03 北京石油化工学院 Acquisition methods, device, equipment and the storage medium of two-dimensional panoramic image

Cited By (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN112506228A (en) * 2020-12-28 2021-03-16 广东电网有限责任公司中山供电局 Substation unmanned aerial vehicle optimal emergency hedge path selection method
CN112506228B (en) * 2020-12-28 2023-11-07 广东电网有限责任公司中山供电局 Optimal emergency risk avoiding path selection method for unmanned aerial vehicle of transformer substation

Similar Documents

Publication Publication Date Title
JP7149692B2 (en) Image processing device, image processing method
US10334168B2 (en) Threshold determination in a RANSAC algorithm
CN109903331B (en) Convolutional neural network target detection method based on RGB-D camera
US11348267B2 (en) Method and apparatus for generating a three-dimensional model
CN107273895B (en) Method for recognizing and translating real-time text of video stream of head-mounted intelligent device
Mistry et al. Image stitching using Harris feature detection
CN111160291B (en) Human eye detection method based on depth information and CNN
CN110909724B (en) Thumbnail generation method of multi-target image
CN111553845B (en) Quick image stitching method based on optimized three-dimensional reconstruction
CN106548494A (en) A kind of video image depth extraction method based on scene Sample Storehouse
CN112634125B (en) Automatic face replacement method based on off-line face database
CN111709317B (en) Pedestrian re-identification method based on multi-scale features under saliency model
CN115393519A (en) Three-dimensional reconstruction method based on infrared and visible light fusion image
JP2013101423A (en) Image matching device and image matching program
CN112017120A (en) Image synthesis method and device
JP7312026B2 (en) Image processing device, image processing method and program
CN112070674B (en) Image synthesis method and device
CN112070672B (en) Image synthesis method and device
CN111783497A (en) Method, device and computer-readable storage medium for determining characteristics of target in video
US10430971B2 (en) Parallax calculating apparatus
Tola DAISY: A Fast Descriptor for Dense Wide Baseline Stereo and Multiview Reconstruction.
KR102637913B1 (en) Method and apparatus for composing images
CN112258561A (en) Matching point acquisition method for image stitching
Yao et al. Robust perspective invariant quasidense matching across large oblique images
CN110602479A (en) Video conversion method and system

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
WW01 Invention patent application withdrawn after publication
WW01 Invention patent application withdrawn after publication

Application publication date: 20201201