CN114049260A - Image splicing method, device and equipment - Google Patents
Image splicing method, device and equipment Download PDFInfo
- Publication number
- CN114049260A CN114049260A CN202210029091.6A CN202210029091A CN114049260A CN 114049260 A CN114049260 A CN 114049260A CN 202210029091 A CN202210029091 A CN 202210029091A CN 114049260 A CN114049260 A CN 114049260A
- Authority
- CN
- China
- Prior art keywords
- image
- sub
- images
- position information
- module
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
- 238000000034 method Methods 0.000 title claims abstract description 57
- 238000006073 displacement reaction Methods 0.000 claims abstract description 51
- 238000012937 correction Methods 0.000 claims abstract description 50
- 238000009434 installation Methods 0.000 claims abstract description 40
- 230000015572 biosynthetic process Effects 0.000 claims description 20
- 238000003786 synthesis reaction Methods 0.000 claims description 20
- 238000012549 training Methods 0.000 claims description 15
- 238000004590 computer program Methods 0.000 claims description 10
- 230000008569 process Effects 0.000 claims description 7
- 238000012986 modification Methods 0.000 claims description 4
- 230000004048 modification Effects 0.000 claims description 4
- 230000002194 synthesizing effect Effects 0.000 claims description 4
- 239000000126 substance Substances 0.000 claims description 3
- 238000010586 diagram Methods 0.000 description 8
- 238000003384 imaging method Methods 0.000 description 8
- 239000011159 matrix material Substances 0.000 description 6
- 230000000007 visual effect Effects 0.000 description 4
- 230000008878 coupling Effects 0.000 description 3
- 238000010168 coupling process Methods 0.000 description 3
- 238000005859 coupling reaction Methods 0.000 description 3
- 230000006870 function Effects 0.000 description 3
- 238000005259 measurement Methods 0.000 description 3
- 238000004422 calculation algorithm Methods 0.000 description 2
- 238000004364 calculation method Methods 0.000 description 2
- 238000004891 communication Methods 0.000 description 2
- 230000000694 effects Effects 0.000 description 2
- 238000004377 microelectronic Methods 0.000 description 2
- 238000012545 processing Methods 0.000 description 2
- 239000004065 semiconductor Substances 0.000 description 2
- 230000009466 transformation Effects 0.000 description 2
- 230000009286 beneficial effect Effects 0.000 description 1
- 238000006243 chemical reaction Methods 0.000 description 1
- 238000013145 classification model Methods 0.000 description 1
- 239000003086 colorant Substances 0.000 description 1
- 239000002131 composite material Substances 0.000 description 1
- 238000013500 data storage Methods 0.000 description 1
- 238000000354 decomposition reaction Methods 0.000 description 1
- 239000000835 fiber Substances 0.000 description 1
- 230000010365 information processing Effects 0.000 description 1
- 238000007689 inspection Methods 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
- 238000005192 partition Methods 0.000 description 1
- 230000002085 persistent effect Effects 0.000 description 1
- 239000007787 solid Substances 0.000 description 1
- 230000001052 transient effect Effects 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T3/00—Geometric image transformations in the plane of the image
- G06T3/40—Scaling of whole images or parts thereof, e.g. expanding or contracting
- G06T3/4038—Image mosaicing, e.g. composing plane images from plane sub-images
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
- G06N3/045—Combinations of networks
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/08—Learning methods
Landscapes
- Engineering & Computer Science (AREA)
- Physics & Mathematics (AREA)
- Theoretical Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Data Mining & Analysis (AREA)
- Molecular Biology (AREA)
- Biophysics (AREA)
- Computational Linguistics (AREA)
- Artificial Intelligence (AREA)
- Evolutionary Computation (AREA)
- General Health & Medical Sciences (AREA)
- Biomedical Technology (AREA)
- Computing Systems (AREA)
- General Engineering & Computer Science (AREA)
- Life Sciences & Earth Sciences (AREA)
- Mathematical Physics (AREA)
- Software Systems (AREA)
- Health & Medical Sciences (AREA)
- Image Processing (AREA)
Abstract
The application provides an image splicing method, an image splicing device and image splicing equipment, which comprise the following steps: acquiring at least two sub-images acquired by an image acquisition device, speed information and preset displacement of the image acquisition device when acquiring each sub-image, and installation correction parameters corresponding to the image acquisition device; and inputting the at least two sub-images, the speed information, the preset displacement and the installation correction parameters into a pre-trained image splicing model to obtain a spliced image. The method and the device solve the problems that a large error exists between the spliced image and the workpiece to be detected, and the whole appearance of the workpiece to be detected cannot be accurately obtained.
Description
Technical Field
The application relates to the technical field of image splicing, in particular to an image splicing method, device and equipment.
Background
In the microelectronics industry, it is necessary to acquire a complete picture of a workpiece with an industrial camera for work path planning, quality inspection, and the like. But due to the imaging size limitation of the camera, the size of a single imaging is not enough to cover the whole workpiece to be detected, and images imaged by a plurality of visual angles are required to be spliced together to obtain the full appearance of the workpiece. In the semiconductor industry, the precision and the requirement for image splicing are extremely high, the error is generally controlled within 10 micrometers (the error is converted into the pixel error of a high-resolution camera and is within 1 pixel), and the speed is not lower than 30FPS (30 images are processed per second).
In a general image stitching method, a homography matrix is usually calculated according to feature information between images, and then image pose transformation is performed according to all the homography matrices to obtain a stitched panoramic image. The method has the disadvantages that firstly, the characteristic information and the corresponding relation of all images need to be calculated every time, the calculation amount is large, and the efficiency is slow; secondly, if the image features are few or unstable, the algorithm will fail. And the other method is that a high-precision moving platform is used for driving a camera to move, so that the camera and the workpiece to be measured move relatively to a plane, and the pixel position of the image of each visual angle in the spliced image is converted by using the relationship between the motion coordinate of the moving platform and the pixel coordinate of the camera, so that the splicing is completed. The method can splice any workpiece only by calibrating once in advance without depending on image characteristics, but depends on the positioning precision of the motion platform. Even if the platform moves with high precision, the small vibration during operation can also cause the error of pixel level, thereby causing the large error between the spliced image and the workpiece to be detected, and the complete picture of the workpiece to be detected can not be accurately obtained.
Disclosure of Invention
The application provides an image splicing method, an image splicing device and image splicing equipment, which are used for solving the problems that in the existing image splicing method, due to the fact that a motion platform vibrates slightly, pixel-level errors are caused, so that large errors exist between spliced images and a workpiece to be tested, and the full view of the workpiece to be tested cannot be obtained accurately.
In a first aspect, the present application provides an image stitching method, including:
acquiring at least two sub-images acquired by an image acquisition device, speed information and preset displacement of the image acquisition device when acquiring each sub-image, and installation correction parameters corresponding to the image acquisition device;
and inputting the at least two sub-images, the speed information, the preset displacement and the installation correction parameters into a pre-trained image splicing model to obtain a spliced image.
Optionally, the image stitching model includes a pixel coordinate prediction module, an image synthesis module, and a correction module;
the pixel coordinate prediction module is used for obtaining initial pixel position information of each sub-image, wherein the pixel position information is position information of each sub-image in a pixel coordinate system corresponding to the spliced image;
the image synthesis module is used for predicting the size of the spliced panoramic image, splicing at least two input sub-images and outputting the initial pixel position information;
the correction module is used for eliminating double images of overlapping areas in spliced images output by the image synthesis module, modifying errors of a splicing structure of the image synthesis module and outputting final pixel position information and the spliced images.
Optionally, the pre-trained image stitching model is generated by the following training method:
acquiring a sample set comprising a plurality of samples, wherein each sample comprises at least two sub-images acquired by an image acquisition device, and the speed information, the preset displacement and the actual displacement of the image acquisition device when acquiring each sub-image, the pixel position information of each sub-image and the installation correction parameters corresponding to the image acquisition device are acquired;
training the initial image mosaic model for multiple times based on the sample set to obtain the trained image mosaic model, wherein each training process comprises the following steps:
obtaining a sample from the sample set;
inputting the sample into an initial image splicing model, and performing parameter adjustment on the initial image splicing model based on the pixel position information of each sub-image output by the initial image splicing model and the pixel position information of each corresponding input sub-image.
Optionally, the sub-image includes at least one block, the pixel value of each block is preset, and a part of the blocks in the at least one block is selected to carry position information of the block in the sub-image;
and determining the pixel position information of two adjacent sub-images according to the position information of the overlapped sub-blocks between the two adjacent sub-images.
Optionally, determining the installation correction parameters corresponding to the image collector according to the following method includes:
respectively acquiring any sub-image acquired by an image acquisition device and a calibrated reference image corresponding to the sub-image;
acquiring a plurality of feature points from any sub-image, and determining the position information of the feature points in the sub-image;
determining the position information of the plurality of characteristic points in the reference image;
determining the installation correction parameters if the following formula meets preset conditions;
wherein the content of the first and second substances,M 11,M 12,M 13,M 21,M 22,M 23,M 31,M 32,M 33in order to correct the parameters for the installation,xandythe position information of each characteristic point in the image is shot for the image collector,x i ’andy i ’is the position information of each characteristic point in the reference image.
Optionally, the image stitching model includes a pixel coordinate prediction module, an image synthesis module, and a modification module, and includes:
the pixel coordinate prediction module is a CNN network model;
the image synthesis module comprises a space transformer layer and a weighting device, wherein the space transformer layer is used for predicting the size of the spliced panoramic image and synthesizing the subimages, and the weighting device is used for weighting and fusing the overlapping parts between two adjacent images;
the correction module comprises a cascade device and a coding-decoding device, wherein the cascade device is used for cascading the synthetic images input by the space transformer, the coding device is used for decomposing the image data output by the cascade device into basic data, and the decoding device is used for recombining the basic data to obtain spliced images.
Optionally, acquiring at least two sub-images acquired by the image acquirer includes:
the image collector slides in the horizontal direction and/or the vertical direction to collect images of the workpiece to be detected; and when the image collector moves to preset displacement, the image collector respectively collects images of the workpiece to be detected.
In a second aspect, the present application provides an image stitching method and apparatus, where the apparatus includes:
the data acquisition module is used for acquiring at least two sub-images acquired by the image acquisition device, speed information and preset displacement of the image acquisition device when acquiring each sub-image, and installation correction parameters corresponding to the image acquisition device;
and the splicing module is used for inputting the at least two sub-images, the speed information, the preset displacement and the installation correction parameter into a pre-trained image splicing model to obtain a spliced image.
In a third aspect, the present application provides an image stitching apparatus, including: at least one processor and at least one memory;
wherein the memory stores program code that, when executed by the processor, causes the processor to perform the following:
acquiring at least two sub-images acquired by an image acquisition device, speed information and preset displacement of the image acquisition device when acquiring each sub-image, and installation correction parameters corresponding to the image acquisition device;
and inputting the at least two sub-images, the speed information, the preset displacement and the installation correction parameters into a pre-trained image splicing model to obtain a spliced image.
In a fourth aspect, the present application also provides a computer storage medium having a computer program stored thereon, which when executed by a processor, performs the steps of the method of the first aspect.
In a fifth aspect, the present application further provides a computer program product comprising a computer program, the computer program comprising program instructions, which when executed by an electronic device, cause the electronic device to execute any one of the above-mentioned classification model training methods for intention classification recognition.
In addition, for technical effects brought by any one implementation manner of the second aspect to the fifth aspect, reference may be made to technical effects brought by different implementation manners of the first aspect, and details are not described here.
The image splicing method, the image splicing device and the image splicing equipment have the following beneficial effects that:
according to the image splicing method, the image splicing device and the image splicing equipment, image splicing is carried out through a trained image splicing model, when data are input, besides sub-images collected by an image collector, preset displacement of the image collector when each image is collected and installation correction parameters of the image collector, speed information of the image collector when each sub-image is collected is input, so that errors caused by image splicing due to vibration of the image collector when the image collector moves can be greatly reduced, the image splicing speed can be greatly increased through the image splicing model, and the image splicing precision is improved.
Drawings
In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings needed to be used in the description of the embodiments will be briefly introduced below, and it is obvious that the drawings in the following description are only some embodiments of the present application, and it is obvious for those skilled in the art to obtain other drawings based on these drawings without inventive exercise.
Fig. 1 is a schematic diagram of an image stitching method according to an embodiment of the present application;
fig. 2 is a schematic view of an image stitching system platform according to an embodiment of the present disclosure;
fig. 3 is a schematic diagram of an ideal pose and an actual pose of an image collector provided in the embodiment of the present application;
fig. 4 is a schematic diagram of an image collector motion coordinate system and an image pixel coordinate system according to an embodiment of the present disclosure;
FIG. 5 is a schematic diagram of two adjacent calibration plates with overlapping areas according to an embodiment of the present application;
fig. 6 is a schematic structural diagram of an image stitching model provided in an embodiment of the present application;
fig. 7 is a schematic structural diagram of an image stitching apparatus according to an embodiment of the present application;
fig. 8 is a schematic structural diagram of an image stitching device according to an embodiment of the present application.
Detailed Description
In order to make the objects, technical solutions and advantages of the present invention clearer, the present invention will be described in further detail with reference to the accompanying drawings, and it is apparent that the described embodiments are only a part of the embodiments of the present invention, not all of the embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.
The term "and/or" in the embodiments of the present invention describes an association relationship of associated objects, and indicates that three relationships may exist, for example, a and/or B may indicate: a exists alone, A and B exist simultaneously, and B exists alone. The character "/" generally indicates that the former and latter associated objects are in an "or" relationship.
The application scenario described in the embodiment of the present invention is for more clearly illustrating the technical solution of the embodiment of the present invention, and does not form a limitation on the technical solution provided in the embodiment of the present invention, and it can be known by a person skilled in the art that with the occurrence of a new application scenario, the technical solution provided in the embodiment of the present invention is also applicable to similar technical problems. In the description of the present invention, the term "plurality" means two or more unless otherwise specified.
In a general image stitching method, a homography matrix is usually calculated according to feature information between images, and then image pose transformation is performed according to all the homography matrices to obtain a stitched panoramic image. The method has the disadvantages that firstly, the characteristic information and the corresponding relation of all images need to be calculated every time, the calculation amount is large, and the efficiency is slow; secondly, if the image features are few or unstable, the algorithm will fail. And the other method is that a high-precision moving platform is used for driving a camera to move, so that the camera and the workpiece to be measured move relatively to a plane, and the pixel position of the image of each visual angle in the spliced image is converted by using the relationship between the motion coordinate of the moving platform and the pixel coordinate of the camera, so that the splicing is completed. The method can splice any workpiece only by calibrating once in advance without depending on image characteristics, but depends on the positioning precision of the motion platform. Even if the platform moves with high precision, the small vibration during operation can also cause the error of pixel level, thereby causing the large error between the spliced image and the workpiece to be detected, and the complete picture of the workpiece to be detected can not be accurately obtained.
In order to solve the problems, the application provides an image splicing method, which can splice images after obtaining a plurality of sub-images collected by an image collector, speed information and preset displacement of the image collector when collecting each sub-image and installation correction parameters corresponding to the image collector, and can greatly reduce errors during splicing and accelerate the speed and efficiency of image splicing by splicing the images spliced by a pre-trained image splicing model.
As shown in fig. 1, an image stitching method provided in the embodiment of the present application includes:
step S101, acquiring at least two sub-images acquired by an image acquisition device, speed information and preset displacement of the image acquisition device when acquiring each sub-image, and installation correction parameters corresponding to the image acquisition device;
when the whole appearance of the measured object is obtained, due to the imaging size limitation of the image collector, the size of single imaging is not enough to cover the whole measured object, and at the moment, images imaged by a plurality of visual angles are spliced together to obtain the whole appearance of the measured object. The object to be measured may be, but is not limited to, a micro device in the microelectronics industry, and may also be any object larger than the view angle of the image collector in other industries, which is not limited herein.
In order to completely obtain the complete appearance of a measured object, the image collector can slide in the horizontal direction to collect images when collecting the images, and the images are collected once when reaching the preset displacement through the preset displacement; after the first line of image acquisition, the second line of image acquisition is carried out until the complete picture of the measured object is obtained; the image acquisition can also be carried out by sliding in the vertical direction, and similarly, the image acquisition is carried out once when the image reaches the preset displacement through the preset displacement; of course, the method can also include sliding the collected images in the horizontal direction and the vertical direction at the same time, and those skilled in the art can set the images according to actual requirements, which is not described herein again. In addition, the image collector can be an image collecting structure such as a camera, as long as image collection can be achieved, a specific implementation structure is set by a person skilled in the art according to actual needs, one possible structure for collecting images in the application is shown in fig. 2, and the structure can be based onxyzThe axis moves in three degrees of freedom, wherein the image collector in the embodiment of the present application is the camera system in fig. 2, the camera system shoots perpendicular to the measurement plane, the shot image is the content in the camera view, and the measurement plane is the plane where the object to be measured is located.
In order to reduce the large error caused by the splicing of images due to the small shaking of the image collector, the speed information of the image collector at the moment is recorded when the image collector collects the images.
The ideal image collector mounting position is that the ideal imaging plane of the image collector is parallel to the measured object and the measuring plane. Under the condition, when the height is unchanged, the precise image splicing work can be completed only by calculating the two-dimensional movement parameters. However, in practice, the camera has a slight installation error, which causes the actual imaging plane to be inclined to the measurement plane, as shown in fig. 3, the ideal imaging plane of the same camera view in the ideal pose of the camera is horizontal, but the actual imaging plane of the actual pose of the camera is inclined because the actual pose of the camera has a certain angle with the horizontal plane, and if no correction is performed, the continuously shot images have obvious errors in the overlapped gaps. The method and the device for correcting the image collected by the image collector are used for correcting the image collected by the image collector based on the corresponding installation correction parameters of the image collector. Firstly, any sub-image shot by an image collector is obtained, and a calibrated reference image corresponding to the sub-image is obtained. And acquiring a plurality of characteristic points from the arbitrary sub-image, and determining the position information, namely the pixel coordinates, of the characteristic points in the sub-image. The parameters of the image collector are known (camera internal parameters are known), the actual distance of each block on the calibration plate is also known, and the reference image is calibrated by the calibration plate, so when a measured object is placed on a plane, the pixel coordinates of each characteristic point on the measured object on an ideal horizontal projection can be obtained through internal parameter conversion. Therefore, the position information, i.e. the pixel coordinates, of the plurality of feature points in the reference image can also be determined. The mounting correction parameters are found by making the results of the following equations satisfy preset conditions.
Specifically, when the preset condition is that the value satisfying the formula is minimum, the installation correction parameter is determined, and in the present application, the installation correction parameter isM 11,M 12,M 13,M 21,M 22,M 23,M 31,M 32,M 33The installation correction parameters form a correction matrix:
correcting the matrix:
xandythe position information of each characteristic point in the image is shot for the image collector,x i ’andy i ’is the position information of each characteristic point in the reference image. And after the image collector collects the image, the corrected image is obtained through the calibration matrix.
And S102, inputting at least two sub-images, speed information, preset displacement and installation correction parameters into a pre-trained image splicing model to obtain a spliced image.
The pre-trained image pre-stitching model is generated by the following training mode:
acquiring a sample set comprising a plurality of samples, wherein each sample comprises at least two sub-images acquired by an image acquisition device, speed information, preset displacement and actual displacement of the image acquisition device when acquiring each sub-image, pixel position information of each sub-image and installation correction parameters corresponding to the image acquisition device, wherein the pixel position information is position information of each sub-image in a pixel coordinate system corresponding to a spliced panoramic image, namely pixel coordinates;
training the initial image mosaic model for multiple times based on a sample set to obtain the trained image mosaic model, wherein each training process comprises the following steps:
obtaining a sample from a sample set;
and inputting the sample into an initial image splicing model, and performing parameter adjustment on the initial image splicing model based on the pixel position information of each sub-image output by the initial image splicing model and the pixel position information of each corresponding input sub-image.
Therefore, when the model is trained, the speed information of the image collector when collecting the subimages needs to be input to preset displacement and actual displacement, even if the image collector collects the images once after preset displacement is set, but the speed of the image collector is different, the vibration amplitude is different, the image collector possibly has certain angles with horizontal lines and vertical lines when sliding, and the actual displacement and the preset displacement are different when collecting the images each time, so that the pixel position information of each subimage is different, and if the pixel position of each subimage is determined only according to the preset displacement, a large error can be caused. However, if the actual displacement needs to be recalculated each time, the efficiency and speed of image stitching are greatly reduced. Therefore, in the implementation of the present application, the samples during the training of the image mosaic model include at least two sub-images acquired by the image acquisition device, the speed information, the preset displacement and the actual displacement of the image acquisition device during the acquisition of each sub-image, the pixel position information of each sub-image, and the installation correction parameters corresponding to the image acquisition device.
In the application, the actual displacement of the image collector is determined by the grating ruler on the motion platform, and the grating ruler of the motion platform comprises three grating rulers which respectively correspond to the directions of a coordinate system (shown in figure 4) of the motion platform and are respectively used for recording the actual displacement of the image collector when the motion platform horizontally and vertically moves or moves up and down.
Next, a method for determining pixel position information of a corresponding sub-image according to a preset displacement and an actual displacement of an image collector will be described. Specifically, first, a corresponding relationship between an image collector motion coordinate system and an image pixel coordinate system is determined, as shown in fig. 4, the image collector motion coordinate system is formed by physical coordinate positions of the camera system motion, and the image pixel coordinate system is formed by pixel positions of images collected by different camera views. According to the corresponding relation between the image collector motion coordinate system and the image pixel coordinate system, the coordinates of each point of the sub-image in the pixel coordinate system can be respectively determined. In the present application, each sub-image is divided into a plurality of blocks, and the pixel value of each block is set in advance, so that the pixel coordinates of the four corners of each block are known. In order to accurately calculate the pixel coordinates of each sub-image, a part of blocks in a plurality of blocks in each sub-image are selected, and the part of blocks carry the position information of the blocks corresponding to the sub-image. Therefore, according to the position information of the overlapped sub-blocks between two adjacent sub-images, the pixel position information, namely the pixel coordinates, of the two adjacent sub-images can be obtained. As an alternative embodiment, a specific calibration board may be used to partition each sub-image and carry the position information. As an alternative embodiment, as shown in fig. 5, the colors of two adjacent blocks of each sub-image are different, and the sizes of the blocks may be the same or different, and are set by those skilled in the art according to actual requirements. The position information of the block in the sub-image is carried, but not limited to directly displaying the pixel coordinates of the block on the block, and a two-dimensional code corresponding to the pixel coordinates of the block may also be set on the block. The way of selecting the partial block to carry the position information is set by the person skilled in the art according to the actual requirements. Fig. 5 shows a specific embodiment, where none of the outermost circle blocks carries position information, and the inner black block obtains a partial block carrying position information by selecting every other row.
Specifically, each sub-image corresponds to a respective coordinate system, and then the pixel coordinates of each sub-image in the pixel coordinate system corresponding to the spliced image are determined according to the position information of the overlapped sub-blocks of the two adjacent sub-images. In FIG. 5, the gray frames 1 and 2 are the images acquired by the image acquisition device twice, and the coordinates of the O point are
And calculating the coordinates of the P point. Due to the presence of the specific calibration plate, the same two-dimensional code can be found in the two acquired images. In the embodiment of the application, the coordinates of the left vertex coordinates of the two-dimensional codes in the overlapping area in the acquisition frame 1 are
The coordinates in the acquisition frame 2 are
Therefore, it is
. I.e. sub-image 2 is shifted in pixel coordinates with respect to sub-image 1Movable part
. The coordinates of the sub-images in the corresponding pixel coordinate system of the spliced images are calculated as follows, the coordinate of the upper left corner of the sub-image 1 is set as
The relative pixel shift between sub-image 2 and sub-image 1 is
So that the coordinates of the sub-image 2 in the corresponding pixel coordinate system of the stitched image are
The sub-image 3 is shifted with respect to the sub-image 2
Then the coordinates of the sub-image 3 in the corresponding pixel coordinate system of the stitched image are
. So that the coordinate of the sub-image s in the corresponding pixel coordinate system of the spliced image is
。
The image splicing model comprises three modules, namely a pixel coordinate prediction module, an image synthesis module and a correction module; optionally, in this embodiment of the present invention, the pixel coordinate prediction module may be, but not limited to, a CNN network model, and the pixel coordinate prediction model obtains the initial pixel position information of each sub-image by using an installation correction parameter, speed information of the image collector, a preset displacement, and an actual displacement of the image collector.
Image synthesisThe module is used for predicting the size of the spliced panoramic image, splicing at least two input sub-images and outputting the initial pixel position information; the image synthesis module comprises a space transformer layer and a weighter, wherein the space transformer layer is used for predicting the size of the spliced panoramic image and synthesizing the subimages, and the weighter is used for weighting and fusing the overlapping part between two adjacent images; specifically, the spatial transformer layer may be, but is not limited to, an STL module, since the pixel coordinate prediction module outputs numerical data, which is two different data types from the image. The role of the STL module is to fuse different types of data. The STL module predicts the size of the panorama image from the coarse pixel coordinate values and generateskA blank image of the size is laid out,kis the number of input images. Then, the STL module combines the image data with the rough pixel coordinate value, predicts the pixel position of each image on the blank image and the overlapping area between the images, and outputsk+1 image data. WhereinkThe sheet image being inputkA composite image of the original images to be stitched placed in the corresponding pixel coordinates, the other image beingkThe sheet images are combined images which are all overlapped together according to pixel coordinates. The specific function of the weighter is tokWhen images are combined, the overlapping regions between the images are weighted and fused, so that image data in the overlapping regions are generated by multiplying adjacent images by different weights.
The correction module comprises a cascade device and a coding-decoding device, wherein the cascade device is used for cascading the synthetic images input by the space transformer, the coding device is used for decomposing the image data output by the cascade device into basic data, and the decoding device is used for recombining according to the basic data to obtain spliced images. In particular, the module is used for correcting tiny flaws in the output images of the image registration module, so as to eliminate double images in the overlapping area and modify splicing structure errors. The cascade device has the function ofkThe +1 output image data are concatenated together and input to the codec. Encoder in encoder-decoder decomposes input image data from three-channel RGB image into more channelsThe features of the trace are represented until the final decomposition into the most basic features that cannot be decomposed. The decoder, whose structure is a mirror image of the encoder, but with three channels in the last layer, aims to reorganize the feature basis into a more complex feature representation and to recombine the basic features into the required concatenation result. The specific structure of the image stitching model is shown in fig. 6, where the real pixel coordinates are also the pixel position information of each sub-image in the embodiment of the present application, and the predicted pixel coordinates are also the final pixel position information output by the image stitching model in the embodiment of the present application. The final output of the image stitching model in FIG. 6 is the panoramic image and the predicted pixel coordinates
Calculating predicted pixel coordinates during model training
With the input true pixel coordinates
For adjusting the model parameters.
In the embodiment of the application, shooting is carried out according to the set trackkOpening sub-images to be spliced, and inputting the sub-images into the image splicing modelkArranging correction matrix for sub-images to be spliced and installing cameraHAnd finally, obtaining the complete spliced panoramic image by setting the speed and displacement of the motion platform. By the method, even under the conditions that the positioning error of the motion platform is large and the vibration influence is large, the most suitable splicing parameters can be calculated, so that the splicing result with the minimum error compared with the actual workpiece can be obtained.
An image stitching method according to the present invention is explained above, and an apparatus for performing the image stitching method is explained below.
Referring to fig. 7, an image stitching apparatus according to an embodiment of the present invention includes:
the data acquisition module 701 is used for acquiring at least two sub-images acquired by an image acquisition device, speed information and preset displacement of the image acquisition device during acquisition of the sub-images, and installation correction parameters corresponding to the image acquisition device;
a stitching module 702, configured to input the at least two sub-images, the speed information, the preset displacement, and the installation correction parameter into a pre-trained image stitching model to obtain a stitched image.
Optionally, the image stitching model includes a pixel coordinate prediction module, an image synthesis module, and a correction module;
the pixel coordinate prediction module is used for obtaining initial pixel position information of each sub-image, wherein the pixel position information is position information of each sub-image in a pixel coordinate system corresponding to the spliced image;
the image synthesis module is used for predicting the size of the spliced panoramic image, splicing at least two input sub-images and outputting the initial pixel position information;
the correction module is used for eliminating double images of overlapping areas in spliced images output by the image synthesis module, modifying errors of a splicing structure of the image synthesis module and outputting final pixel position information and the spliced images.
Optionally, the pre-trained image stitching model is generated by the following training method:
acquiring a sample set comprising a plurality of samples, wherein each sample comprises at least two sub-images acquired by an image acquisition device, and the speed information, the preset displacement and the actual displacement of the image acquisition device when acquiring each sub-image, the pixel position information of each sub-image and the installation correction parameters corresponding to the image acquisition device are acquired;
training the initial image mosaic model for multiple times based on the sample set to obtain the trained image mosaic model, wherein each training process comprises the following steps:
obtaining a sample from the sample set;
inputting the sample into an initial image splicing model, and performing parameter adjustment on the initial image splicing model based on the pixel position information of each sub-image output by the initial image splicing model and the pixel position information of each corresponding input sub-image.
Optionally, the sub-image includes at least one block, the pixel value of each block is preset, and a part of the blocks in the at least one block is selected to carry position information of the block in the sub-image;
and determining the pixel position information of two adjacent sub-images according to the position information of the overlapped sub-blocks between the two adjacent sub-images.
Optionally, determining the installation correction parameters corresponding to the image collector according to the following method includes:
respectively acquiring any sub-image acquired by an image acquisition device and a calibrated reference image corresponding to the sub-image;
acquiring a plurality of feature points from any sub-image, and determining the position information of the feature points in the sub-image;
determining the position information of the plurality of characteristic points in the reference image;
determining the installation correction parameters if the following formula meets preset conditions;
wherein the content of the first and second substances,M 11,M 12,M 13,M 21,M 22,M 23,M 31,M 32,M 33in order to correct the parameters for the installation,xandythe position information of each characteristic point in the image is shot for the image collector,x i ’andy i ’is the position information of each characteristic point in the reference image.
Optionally, the image stitching model includes a pixel coordinate prediction module, an image synthesis module, and a modification module, and includes:
the pixel coordinate prediction module is a CNN network model;
the image synthesis module comprises a space transformer layer and a weighting device, wherein the space transformer layer is used for predicting the size of the spliced panoramic image and synthesizing the subimages, and the weighting device is used for weighting and fusing the overlapping parts between two adjacent images;
the correction module comprises a cascade device and a coding-decoding device, wherein the cascade device is used for cascading the synthetic images input by the space transformer, the coding device is used for decomposing the image data output by the cascade device into basic data, and the decoding device is used for recombining the basic data to obtain spliced images.
Optionally, acquiring at least two sub-images acquired by the image acquirer includes:
the image collector slides in the horizontal direction and/or the vertical direction to collect images of the workpiece to be detected; and when the image collector moves to preset displacement, the image collector respectively collects images of the workpiece to be detected. An image stitching device in the embodiment of the present application is described above from the perspective of a modular functional entity, and an image stitching device in the embodiment of the present application is described below from the perspective of hardware processing.
Referring to fig. 8, an image stitching apparatus in an embodiment of the present application includes at least one processor 801 and at least one memory 802, and a bus system 809;
wherein the memory stores program code that, when executed by the processor, causes the processor to perform the following:
acquiring at least two sub-images acquired by an image acquisition device, speed information and preset displacement of the image acquisition device when acquiring each sub-image, and installation correction parameters corresponding to the image acquisition device;
and inputting the at least two sub-images, the speed information, the preset displacement and the installation correction parameters into a pre-trained image splicing model to obtain a spliced image.
Fig. 8 is a schematic diagram of an image stitching apparatus 800 according to an embodiment of the present disclosure, where the apparatus 800 may have a relatively large difference due to different configurations or performances, and may include one or more processors (CPU) 801 (e.g., one or more processors) and a memory 802, and one or more storage media 803 (e.g., one or more mass storage devices) for storing applications 804 or data 805. Memory 802 and storage medium 803 may be, among other things, transient storage or persistent storage. The program stored in the storage medium 803 may include one or more modules (not shown), and each module may include a series of instruction operations for the information processing apparatus. Still further, the processor 801 may be configured to communicate with the storage medium 803 to execute a series of instruction operations in the storage medium 803 on the device 800.
The device 800 may also include one or more wired or wireless network interfaces 807, one or more input-output interfaces 808, and/or one or more operating systems 806, such as Windows Server, Mac OS X, Unix, Linux, FreeBSD, etc.
An embodiment of the present invention further provides a computer-readable storage medium, which includes instructions, and when the computer-readable storage medium runs on a computer, the computer is caused to execute an image stitching method provided in the foregoing embodiment.
The embodiment of the present application further provides a computer program product, which includes a computer program, where the computer program includes program instructions, and when the program instructions are executed by an electronic device, the electronic device is caused to execute the image stitching method provided in the foregoing embodiment.
It can be clearly understood by those skilled in the art that, for convenience and brevity of description, the specific working processes of the above-described apparatuses and modules may refer to the corresponding processes in the foregoing method embodiments, and are not described herein again.
In the several embodiments provided in the present application, it should be understood that the disclosed system, apparatus and method may be implemented in other manners. For example, the above-described apparatus embodiments are merely illustrative, and for example, the division of the modules is merely a logical division, and in actual implementation, there may be other divisions, for example, multiple modules or components may be combined or integrated into another system, or some features may be omitted, or not implemented. In addition, the shown or discussed mutual coupling or direct coupling or communication connection may be an indirect coupling or communication connection through some interfaces, devices or modules, and may be in an electrical, mechanical or other form.
The modules described as separate parts may or may not be physically separate, and parts displayed as modules may or may not be physical modules, may be located in one place, or may be distributed on a plurality of network modules. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of the present embodiment.
In addition, functional modules in the embodiments of the present application may be integrated into one processing module, or each of the modules may exist alone physically, or two or more modules are integrated into one module. The integrated module can be realized in a hardware mode, and can also be realized in a software functional module mode. The integrated module, if implemented in the form of a software functional module and sold or used as a stand-alone product, may be stored in a computer readable storage medium.
In the above embodiments, the implementation may be wholly or partially realized by software, hardware, firmware, or any combination thereof. When implemented in software, may be implemented in whole or in part in the form of a computer program product.
The computer program product includes one or more computer instructions. When loaded and executed on a computer, cause the processes or functions described in accordance with the embodiments of the application to occur, in whole or in part. The computer may be a general purpose computer, a special purpose computer, a network of computers, or other programmable device. The computer instructions may be stored in a computer readable storage medium or transmitted from one computer readable storage medium to another, for example, from one website site, computer, server, or data center to another website site, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, Digital Subscriber Line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.). The computer-readable storage medium can be any available medium that a computer can store or a data storage device, such as a server, a data center, etc., that is integrated with one or more available media. The usable medium may be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., Solid State Disk (SSD)), among others.
The technical solutions provided by the present application are introduced in detail, and the present application applies specific examples to explain the principles and embodiments of the present application, and the descriptions of the above examples are only used to help understand the method and the core ideas of the present application; meanwhile, for a person skilled in the art, according to the idea of the present application, there may be variations in the specific embodiments and the application scope, and in summary, the content of the present specification should not be construed as a limitation to the present application.
Claims (10)
1. An image stitching method, comprising:
acquiring at least two sub-images acquired by an image acquisition device, speed information and preset displacement of the image acquisition device when acquiring each sub-image, and installation correction parameters corresponding to the image acquisition device;
and inputting the at least two sub-images, the speed information, the preset displacement and the installation correction parameters into a pre-trained image splicing model to obtain a spliced image.
2. The method of claim 1, wherein the image stitching model comprises a pixel coordinate prediction module, an image synthesis module, and a modification module;
the pixel coordinate prediction module is used for obtaining initial pixel position information of each sub-image, wherein the pixel position information is position information of each sub-image in a pixel coordinate system corresponding to the spliced panoramic image;
the image synthesis module is used for predicting the size of the spliced panoramic image, splicing at least two input sub-images and outputting the initial pixel position information;
the correction module is used for eliminating double images of overlapping areas in spliced images output by the image synthesis module, modifying errors of a splicing structure of the image synthesis module and outputting final pixel position information and the spliced images.
3. The method according to any one of claims 1 or 2, wherein the pre-trained image stitching model is generated by training:
acquiring a sample set comprising a plurality of samples, wherein each sample comprises at least two sub-images acquired by an image acquisition device, and the speed information, the preset displacement and the actual displacement of the image acquisition device when acquiring each sub-image, the pixel position information of each sub-image and the installation correction parameters corresponding to the image acquisition device are acquired;
training the initial image mosaic model for multiple times based on the sample set to obtain the trained image mosaic model, wherein each training process comprises the following steps:
obtaining a sample from the sample set;
inputting the sample into an initial image splicing model, and performing parameter adjustment on the initial image splicing model based on the pixel position information of each sub-image output by the initial image splicing model and the pixel position information of each corresponding input sub-image.
4. The method according to claim 3, wherein the sub-image comprises at least one block, the pixel value of each block is preset, and a part of the at least one block is selected to carry the position information of the block corresponding to the sub-image;
and determining the pixel position information of two adjacent sub-images according to the position information of the overlapped sub-blocks between the two adjacent sub-images.
5. The method of claim 3, wherein determining the corresponding installation correction parameters of the image collector according to the following method comprises:
respectively acquiring any sub-image acquired by an image acquisition device and a calibrated reference image corresponding to the sub-image;
acquiring a plurality of feature points from any sub-image, and determining the position information of the feature points in the sub-image;
determining the position information of the plurality of characteristic points in the reference image;
determining the installation correction parameters if the following formula meets preset conditions;
wherein the content of the first and second substances,M 11,M 12,M 13,M 21,M 22,M 23,M 31,M 32,M 33in order to correct the parameters for the installation,xandythe position information of each characteristic point in the image is shot for the image collector,x i ’andy i ’is the position information of each characteristic point in the reference image.
6. The method of claim 2, wherein the image stitching model comprises a pixel coordinate prediction module, an image synthesis module, and a modification module, comprising:
the pixel coordinate prediction module is a CNN network model;
the image synthesis module comprises a space transformer layer and a weighting device, wherein the space transformer layer is used for predicting the size of the spliced panoramic image and synthesizing the subimages, and the weighting device is used for weighting and fusing the overlapping parts between two adjacent images;
the correction module comprises a cascade device and a coding-decoding device, wherein the cascade device is used for cascading the synthetic images input by the space transformer, the coding device is used for decomposing the image data output by the cascade device into basic data, and the decoding device is used for recombining the basic data to obtain spliced images.
7. The method of claim 1, wherein acquiring at least two sub-images acquired by an image acquirer comprises:
the image collector slides in the horizontal direction and/or the vertical direction to collect images of the workpiece to be detected; and when the image collector moves to preset displacement, the image collector respectively collects images of the workpiece to be detected.
8. An image stitching method device is characterized by comprising the following steps:
the data acquisition module is used for acquiring at least two sub-images acquired by the image acquisition device, speed information and preset displacement of the image acquisition device when acquiring each sub-image, and installation correction parameters corresponding to the image acquisition device;
and the splicing module is used for inputting the at least two sub-images, the speed information, the preset displacement and the installation correction parameter into a pre-trained image splicing model to obtain a spliced image.
9. An image stitching device, characterized by comprising: a processor and a memory, wherein the memory is configured to store a program;
the processor is configured to execute the program in the memory to cause the computer to perform the method of any of claims 1 to 7.
10. A computer-readable storage medium comprising computer program instructions which, when run on a computer, cause the computer to perform the method of any one of claims 1 to 7.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210029091.6A CN114049260B (en) | 2022-01-12 | 2022-01-12 | Image splicing method, device and equipment |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210029091.6A CN114049260B (en) | 2022-01-12 | 2022-01-12 | Image splicing method, device and equipment |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN114049260A true CN114049260A (en) | 2022-02-15 |
| CN114049260B CN114049260B (en) | 2022-03-22 |
Family
ID=80196237
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202210029091.6A Active CN114049260B (en) | 2022-01-12 | 2022-01-12 | Image splicing method, device and equipment |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN114049260B (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN116612168A (en) * | 2023-04-20 | 2023-08-18 | 北京百度网讯科技有限公司 | Image processing method, device, electronic equipment, image processing system and medium |
Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20170221244A1 (en) * | 2016-02-02 | 2017-08-03 | Morpho, Inc. | Image processing device, image processing method, non-transitory computer readable recording medium and photographing assist equipment |
| CN111161153A (en) * | 2019-12-31 | 2020-05-15 | 无锡祥生医疗科技股份有限公司 | Wide scene splicing method and device and storage medium |
| CN112712037A (en) * | 2020-12-31 | 2021-04-27 | 苏州清研微视电子科技有限公司 | Vehicle-mounted environment sensing method and system based on panoramic image and target detection |
| US20210166477A1 (en) * | 2019-12-03 | 2021-06-03 | Augustus Intelligence Inc. | Synthesizing images from 3d models |
-
2022
- 2022-01-12 CN CN202210029091.6A patent/CN114049260B/en active Active
Patent Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20170221244A1 (en) * | 2016-02-02 | 2017-08-03 | Morpho, Inc. | Image processing device, image processing method, non-transitory computer readable recording medium and photographing assist equipment |
| US20210166477A1 (en) * | 2019-12-03 | 2021-06-03 | Augustus Intelligence Inc. | Synthesizing images from 3d models |
| CN111161153A (en) * | 2019-12-31 | 2020-05-15 | 无锡祥生医疗科技股份有限公司 | Wide scene splicing method and device and storage medium |
| CN112712037A (en) * | 2020-12-31 | 2021-04-27 | 苏州清研微视电子科技有限公司 | Vehicle-mounted environment sensing method and system based on panoramic image and target detection |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN116612168A (en) * | 2023-04-20 | 2023-08-18 | 北京百度网讯科技有限公司 | Image processing method, device, electronic equipment, image processing system and medium |
Also Published As
| Publication number | Publication date |
|---|---|
| CN114049260B (en) | 2022-03-22 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US20230345135A1 (en) | Method, apparatus, and device for processing images, and storage medium | |
| CN111445533B (en) | Binocular camera calibration method, device, equipment and medium | |
| CN100428782C (en) | Information processing method and apparatus | |
| US20120257086A1 (en) | Imaging system, imaging method, and storage medium storing imaging program | |
| US7158690B2 (en) | Enhancing the resolution of measurement systems employing image capturing systems to measure lengths | |
| JP2009134509A (en) | Device for and method of generating mosaic image | |
| JPH11118425A (en) | Calibration method and device and calibration data production | |
| CN111263142A (en) | Method, device, equipment and medium for testing optical anti-shake of camera module | |
| WO2013069555A1 (en) | Image processing device, method, and program | |
| WO2020188799A1 (en) | Camera calibration device, camera calibration method, and non-transitory computer-readable medium having program stored thereon | |
| CN114049260B (en) | Image splicing method, device and equipment | |
| JP2003050107A (en) | Camera calibration device | |
| CN110612428A (en) | Three-dimensional measurement method using characteristic quantity and apparatus therefor | |
| CN111445537A (en) | Calibration method and system of camera | |
| CN114792345A (en) | Calibration method based on monocular structured light system | |
| JP3282516B2 (en) | 3D image measurement method | |
| JP2000121358A (en) | Method and device for obtaining inspection point position | |
| CN1510391A (en) | Image measuring system and method | |
| US5986694A (en) | Image processing apparatus for moving camera | |
| JP4776983B2 (en) | Image composition apparatus and image composition method | |
| JPH1096606A (en) | Shape measuring method and device | |
| JP3754401B2 (en) | Image distortion correction method, image distortion correction apparatus and image distortion correction program based on representative point measurement | |
| JPH06195472A (en) | Image-processing system | |
| CN113790711A (en) | Unmanned aerial vehicle low-altitude flight pose uncontrolled multi-view measurement method and storage medium | |
| JP3924576B2 (en) | Three-dimensional measurement method and apparatus by photogrammetry |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |