CN112770041A - Image processing method for switching multiple zoom lenses and camera - Google Patents
Image processing method for switching multiple zoom lenses and camera Download PDFInfo
- Publication number
- CN112770041A CN112770041A CN201911069778.7A CN201911069778A CN112770041A CN 112770041 A CN112770041 A CN 112770041A CN 201911069778 A CN201911069778 A CN 201911069778A CN 112770041 A CN112770041 A CN 112770041A
- Authority
- CN
- China
- Prior art keywords
- image
- matrix
- focal length
- lens
- offset
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000003672 processing method Methods 0.000 title claims abstract description 9
- 238000000034 method Methods 0.000 claims abstract description 53
- 238000012545 processing Methods 0.000 claims abstract description 47
- 230000008569 process Effects 0.000 claims abstract description 29
- 239000011159 matrix material Substances 0.000 claims description 186
- 238000013519 translation Methods 0.000 claims description 48
- PXFBZOLANLWPMH-UHFFFAOYSA-N 16-Epiaffinine Natural products C1C(C2=CC=CC=C2N2)=C2C(=O)CC2C(=CC)CN(C)C1C2CO PXFBZOLANLWPMH-UHFFFAOYSA-N 0.000 claims description 42
- 230000002159 abnormal effect Effects 0.000 claims description 30
- 238000013507 mapping Methods 0.000 claims description 21
- 238000005520 cutting process Methods 0.000 claims description 18
- 230000009466 transformation Effects 0.000 claims description 11
- 239000000126 substance Substances 0.000 claims description 9
- 230000001960 triggered effect Effects 0.000 claims description 2
- 230000003287 optical effect Effects 0.000 claims 1
- 238000010586 diagram Methods 0.000 description 11
- 230000008859 change Effects 0.000 description 4
- 238000004590 computer program Methods 0.000 description 4
- 230000009471 action Effects 0.000 description 3
- 230000006870 function Effects 0.000 description 3
- 238000003860 storage Methods 0.000 description 3
- 230000004044 response Effects 0.000 description 2
- 238000003491 array Methods 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 230000000694 effects Effects 0.000 description 1
- 238000009434 installation Methods 0.000 description 1
- 238000004519 manufacturing process Methods 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
Images
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
- H04N23/60—Control of cameras or camera modules
- H04N23/67—Focus control based on electronic image sensor signals
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
- H04N23/60—Control of cameras or camera modules
Landscapes
- Engineering & Computer Science (AREA)
- Multimedia (AREA)
- Signal Processing (AREA)
- Studio Devices (AREA)
Abstract
The application discloses an image processing method for switching a plurality of zoom lenses, which comprises the steps that when a first lens is switched from a first focal length area to a second focal length area of a second lens, when the first lens and the second lens are both at switching positions in a focal length overlapping area, rotation offset and/or field angle offset processing is carried out on a first image from a first sensor according to a pre-stored calibration parameter for switching processing, so that the first image before switching is displayed and a second image after switching is displayed, wherein the second image from a second sensor has the same field angle. The method and the device solve the problem that the field angle of the image changes suddenly when the zooming process is switched from a larger focal length to a smaller focal length.
Description
Technical Field
The present invention relates to the field of image processing, and in particular, to an image processing method for switching a plurality of zoom lenses.
Background
The zoom lens is an image acquisition lens which can change focal length within a certain range, thereby obtaining different wide and narrow field angles, images with different sizes and different scene ranges. The zoom lens can change the shooting range by changing the focal length under the condition of not changing the shooting distance, thereby being very beneficial to image composition.
In order to widen the focal length range of the lens, and to make image capturing devices such as cameras and video cameras have larger lens magnification, zoom splicing apparatuses based on multiple (at least two) zoom lenses are introduced in the industry, and lenses with different focal lengths are used for splicing to expand the focal length range.
Referring to fig. 1, fig. 1 is a schematic diagram of a three focal length lens. The focal length ranges of the a lens are fa1-fa2, the second lens is fb1-fb2, and the third lens is fc1-fc2, the focal length ranges of the three lenses are sequentially connected, an overlapping area exists between two adjacent focal lengths, any position in the overlapping area can be used as a switching position between two focal lengths, and as shown in the figure, fab and fbc are fixed for a finished product. In the process of zooming from fa1 to fc2, a first image is displayed in the fa1-fab section range, the image is switched to a second image after reaching fab, and the image is switched to a third image after reaching fbc, so that a telephoto lens in the focal length range fa1-fc2 can be obtained, the focal length range of the lens is greatly widened, a camera has larger lens magnification, but various special effect images are usually displayed at the switching position fab or fbc, and the user experience is poor.
Because the axes of the lenses are usually not on the same straight line when the lenses are installed, for example, two lenses are installed side by side, one image is deviated from the left and the other image is deviated from the right, see fig. 2, fig. 2 shows a schematic diagram of an image captured by the first lens and an image captured by the second lens, a sudden change of the field angle occurs in the zooming process, and the user experience is poor.
Disclosure of Invention
The invention provides an image processing method for switching a plurality of zoom lenses, which ensures that the field angle of an image is not changed when switching of different focal length ranges occurs in the zooming process.
The invention provides an image processing method for switching a plurality of zoom lenses, which comprises the following steps: the zoom lenses at least comprise a first lens module with a first sensor and a first lens and a second lens module with a second sensor and a second lens; the first focal length area of the first lens is overlapped with the adjacent area of the second focal length area of the second lens to form a focal length overlapping area;
when the first lens is switched from the first focal length area to the second focal length area,
performing rotation offset and/or view angle offset processing on a first image from a first sensor according to a pre-stored calibration parameter for switching processing at a switching position where the first lens and the second lens are both in the focal length overlapping region, so that the first image before switching is displayed and a second image after switching is displayed, wherein the first image before switching and the second image after switching have the same view angle;
wherein the content of the first and second substances,
the calibration parameters for the switching process are calibrated based on the first image from the first sensor and the second image from the second sensor, wherein the calibration parameters comprise a rotation offset and/or a field angle offset;
the field angle corresponding to the first focal length is larger than the field angle corresponding to the second focal length;
the focal length overlapping area is a focal length area between the upper limit position of the second focal length area and the lower limit position of the first focal length area, and the switching position is a preset focal length in the focal length overlapping area.
Alternatively,
the calibration parameters for the switching process include a rotation offset calibration parameter and/or a field angle offset calibration parameter,
the calibration parameters for the switching process are triggered at the switching location based on a calibration of a first image from a first sensor and a second image from a second sensor, including,
estimating a homography matrix according to matching feature points of the current first image and the current second image at the switching position, performing affine transformation at least including rotation offset and field angle offset on the first image to obtain an affine matrix corresponding to the affine transformation,
according to the relation between the estimated homography matrix and the affine matrix, the rotation offset and/or the field angle offset in the affine matrix are/is obtained, and the obtained rotation offset and/or the obtained field angle offset are/is used as the calibrated parameters and are stored;
wherein the content of the first and second substances,
the rotational offset includes rotating the first image about its image center to a rotational angle parallel to the second image,
the field angle offset includes an upper offset between upper boundaries, a lower offset between lower boundaries, a left offset between left boundaries, and a right offset between right boundaries of the first image and the second image rotated to be parallel to the second image.
Optionally, said estimating a homography matrix from matching feature points of said first and second images comprises,
searching for matched characteristic point pairs based on the first image and the second image to obtain characteristic point pair samples,
and estimating a homography matrix corresponding to the optimal point pair set selected from the characteristic point pair samples by a random consistency algorithm, wherein the homography matrix is a 3 x 3 matrix.
Optionally, the performing affine transformation at least including rotation offset and field angle offset on the first image to obtain an affine matrix corresponding to the affine transformation includes,
constructing a translation matrix such that: translating the first image by taking the translation of the image center of the first image from the original position to the origin of the pixel coordinate system as a target,
constructing a rotation matrix such that: rotating the first image with an image center according to the rotational offset, wherein a rotation matrix is related to the rotational offset,
constructing an inverse translation matrix such that: taking the image center from the origin of the pixel coordinate system to the original position as a target, reversely translating the rotated first image,
constructing a clipping matrix such that: based on the first image after the inverse translation, clipping according to an area formed by the field angle offset; wherein the clipping matrix is related to the field angle offset,
and multiplying the inverse matrix of the translation matrix, the inverse matrix of the rotation matrix, the inverse matrix of the inverse translation matrix and the inverse matrix of the cutting matrix in sequence to obtain the affine matrix.
Optionally, after the constructing a rotation matrix to rotate the first image by the image center according to the rotation offset, further comprises,
determining a scaling relation according to the abnormal pixel proportion in the rotated first image; wherein the abnormal pixel proportion is related to the rotation offset,
constructing a scaling matrix based on the scaling relation so as to scale the rotated first image;
multiplying the inverse matrix of the translation matrix, the inverse matrix of the rotation matrix, the inverse matrix of the inverse translation matrix and the inverse matrix of the cutting matrix in sequence to obtain an affine matrix,
and multiplying the inverse matrix of the translation matrix, the inverse matrix of the rotation matrix, the inverse matrix of the scaling matrix, the inverse matrix of the inverse translation matrix and the inverse matrix of the cutting matrix in sequence to obtain the affine matrix.
Optionally, said determining a scaling according to the abnormal pixel proportion in the rotated first image comprises,
a relation of a first height of an abnormal pixel in a height direction of the image panel is determined,
removing the first height of the abnormal pixels from the breadth height to obtain a second height of the residual pixels,
taking the ratio of the breadth height to the second height as a relation of scaling;
alternatively, the first and second electrodes may be,
calculating a relation of a first width of the abnormal pixel in the image width direction,
removing the first width of the abnormal pixels from the breadth width to obtain a second width of the residual pixels,
and taking the ratio of the breadth width to the second width as a relation of scaling.
Optionally, the relation for calculating the first height of the abnormal pixel in the height direction of the image breadth comprises,
according to an affine mapping relation rotating by the image center, mapping between a first image pixel coordinate and a first image pixel coordinate after rotation is established, wherein the affine mapping comprises a translation matrix, a rotation matrix and an inverse translation matrix;
selecting any two known pixel coordinates from the rotated first image, substituting the known pixel coordinates into the affine mapping relation to obtain two corresponding pixel coordinates in the first image,
calculating the absolute value of the difference between the vertical coordinates of the two pixel coordinates to obtain the first height;
the calculating a relation of a first width of the abnormal pixel in the width direction of the image web according to the rotational offset includes,
and calculating the absolute value of the difference between the abscissas of the two obtained pixel coordinates to obtain the first width.
Optionally, the calculating the rotation offset and the field angle offset in the affine matrix according to the relationship between the estimated homography matrix and the affine matrix comprises,
solving the rotation offset and the field angle offset in the affine matrix according to the equal relation between the affine matrix and the inverse matrix of the homography matrix, and storing the result obtained by the solution;
the processing of the first image according to the calibrated parameters comprises,
obtaining a rotation offset according to the solution, determining a rotation matrix and a scaling matrix,
determining a cutting matrix according to the solved field angle offset,
sequentially multiplying the translation matrix, the determined rotation matrix, the determined scaling matrix, the inverse translation matrix and the determined cutting matrix to obtain a multiplication result;
and multiplying the multiplied result by each pixel coordinate vector in the first image to obtain each pixel coordinate vector of the processed image.
Optionally, the homography matrix is:
wherein F, G, K, P, Q, V is an element in the matrix;
the translation matrix is:
wherein, W is the width of the image breadth, and H is the height of the image breadth;
the rotation matrix is:
wherein θ is the rotational offset;
the scaling matrix is:
or the following steps:
the inverse translation matrix is:
the cutting matrix is as follows:
wherein U is an upper offset, D is a lower offset, L is a left offset, and R is a right offset;
the solution yields a rotational offset of:
and is
The solving results in the field angle offsets of respectively,
left offset:
right offset:
upper offset:
offset below:
wherein the content of the first and second substances,
the processing of rotating offset and/or angle of view offset is carried out on the first image from the first sensor according to the pre-stored calibration parameters for switching processing, which comprises,
and multiplying each pixel vector in the first image by a translation matrix A, a rotation matrix C, a scaling matrix R, an inverse translation matrix B and a cutting matrix E in sequence to obtain a pixel vector processed by the first image.
The invention provides a camera, which comprises,
the first lens module comprises a first lens with a first focal length area and a first sensor for sensing light rays incident from the first lens and generating a first image;
the second lens module comprises a second lens with a second focal length area and a second sensor which senses light rays incident from the second lens and generates a second image;
the first focal length area of the first lens is overlapped with the adjacent area of the second focal length area of the second lens to form a focal length overlapping area; the field angle of the first image is greater than the field angle of the second image;
a processor for receiving a first image generated by the first sensor and receiving a second image generated by the second sensor, and outputting for display the received first image generated by the first sensor or the received second image generated by the second sensor;
in response to a focus switching instruction, wherein the focus switching instruction is used to instruct the camera to switch from a first focus area to a second focus area, the processor is configured to perform:
when the fact that the first lens is driven so that the focal length of the first lens is located at a switching position is detected, acquiring a first image of the first lens located at the switching position; wherein the switching position is a preset focal length in the focal length overlapping region;
according to pre-stored calibration parameters, performing image processing of rotation offset and/or field angle offset on a first image of the first lens at the switching position to generate a processed first image, so that the field angle of the processed first image and a second image generated by the second sensor have the same field angle; the prestored calibration parameters are obtained by performing calibration processing according to the image of the first lens of the camera at the switching position and the image of the second lens of the camera at the switching position, wherein the calibration processing comprises rotation offset and/or field angle offset processing;
outputting the processed first image for display until detecting that the second lens is driven such that the second lens focal length is at the switch position, the processor stopping outputting the first image from the first sensor and starting outputting the image from the second sensor.
The invention provides a calibration method of zoom splicing equipment of a plurality of zoom lenses,
the zoom lenses at least comprise a first lens module with a first sensor and a first lens and a second lens module with a second sensor and a second lens; the first focal length area of the first lens is overlapped with the adjacent area of the second focal length area of the second lens, and a focal length overlapping area exists; the method comprises the steps of (1) carrying out,
when the first lens is switched from the first focal length area to the second focal length area of the second lens, when the first lens and the second lens are at the switching position in the focal length overlapping area, triggering the calibration of the rotation offset and/or the field angle offset based on the current first image from the first sensor and the current second image from the second sensor, and obtaining the calibration parameters for the switching process,
wherein the content of the first and second substances,
the focal length overlapping area is a focal length area between the upper limit position of the second focal length area and the lower limit position of the first focal length area, and the switching position is a preset focal length in the focal length overlapping area;
the field angle corresponding to the first focal length is larger than the field angle corresponding to the second focal length.
The present invention also provides a computer readable storage medium having stored therein a computer program which, when executed by a processor, implements the steps of the image processing method for switching a plurality of zoom lenses as described above, and/or implements the steps of the calibration method for a zoom stitching device for a plurality of zoom lenses as described above.
According to the invention, when the first focal length is switched to the second focal length, the image from the first focal length lens is processed through the calibration parameters for switching processing, so that the problem of abrupt change of the field angle of the image when the first focal length is switched to the second focal length in the zooming process is solved, even under the condition of no calibration parameters for switching processing, the calibration can be automatically carried out in real time on the basis of the current first image from the first focal length lens and the current second image from the second focal length, the calibration parameters are obtained, no complex external hardware assistance is needed, the image from the larger focal length lens is processed through the calibration parameters, so that the field angle is not different during switching, the user experience is improved, and the image quality when the axes of the lenses are not on the same straight line when the lenses are installed is improved.
Drawings
FIG. 1 is a schematic diagram with three focal lengths;
fig. 2 is a schematic diagram of an image captured by a first lens and an image captured by a second lens.
Fig. 3 is a schematic diagram illustrating that a first image captured by the first lens and a second image captured by the second lens have an angular field offset.
FIG. 4 is a flowchart illustrating image processing for switching a plurality of zoom lenses according to an embodiment of the present invention.
FIG. 5 is a flowchart illustrating a process of estimating a homography matrix using a random consensus algorithm according to an embodiment of the present invention.
Fig. 6 is a schematic diagram of the whole process of processing the first image.
Fig. 7 is a schematic diagram of a control module of a zoom stitching apparatus including two zoom lenses according to an embodiment of the present invention.
Detailed Description
For the purpose of making the objects, technical means and advantages of the present application more apparent, the present application will be described in further detail with reference to the accompanying drawings.
In one embodiment of the invention, when switching from a first focal length area to a second focal length area is carried out at a switching position in an adjacent focal length overlapping area, whether a calibration parameter for switching processing is stored is judged, and if the calibration parameter is stored, an image from a first focal length lens is processed according to the calibration parameter; otherwise, triggering a current calibration based on a current first image from the first focal length lens (first sensor) and a current second image from the second focal length lens (second sensor), wherein the first focal length is greater than the second focal length. According to the zoom lens, the current images from the multiple zoom lenses are processed through the calibration parameters, so that the field angle of the images is unchanged when switching of different focal length ranges occurs in the zooming process.
In the following, the description is given by taking as an example the switching from the first focal length region of the first lens to the second focal length region of the second lens in fig. 2, where the first focal length is larger than the second focal length, i.e. the field angle corresponding to the first focal length is larger than the field angle corresponding to the second focal length, and it should be understood that the switching from the second focal length of the second lens to the third focal length of the third lens, even from the ith focal length of the ith lens to the (i + 1) th focal length of the (i + 1) th lens, and so on, as long as the switching is from the larger focal length to the smaller focal length.
Due to the installation deviation of the plurality of zoom lenses, during zooming, two images before and after the switching of the field angles of the images at the time of switching of different focal length ranges have a certain rotational offset θ and a certain field angle offset, as shown in fig. 2, a first image captured by the first lens and a second image captured by the second lens have a certain rotational offset θ, as shown in fig. 3, and as shown in fig. 3, a schematic diagram of the field angle offset of the first image captured by the first lens and the second image captured by the second lens exists, wherein the field angle offset of the two images includes an upper offset U between upper boundaries of the images, a lower offset D between lower boundaries, a left offset L between left boundaries, and a right offset R between right boundaries.
In this application, the switching occurring in two adjacent focal length ranges refers to a switching process from a larger focal length to a smaller focal length, wherein the angle of view of one image is larger than that of the other image, for example, in fig. 2, the first image includes all images of the second lens. When switching from the first focal length (source focal length) of the first lens to the second focal length (target focal length) of the second lens, in order to ensure that the two images are switched from the first image (source image) to the second image (target image) in the switching process to be consistent, the following two operations need to be completed:
the first step is as follows: the first image (source image) is rotated by an angle theta to make the image parallel to the second image (target image).
The second step is that: and cutting the first image according to the offset of the field angle, and cutting out pixels in an area with the width of W-R-L and the height of H-U-D from the positions of U, D, L and R away from the upper boundary and the left boundary of the first image respectively to make the first image consistent with the second image, wherein W is the width of the image breadth, H is the height of the image breadth, and W is greater than H.
Thus, if the 5 parameters θ and U, D, L, R are obtained as the calibration parameters for switching between the first focal length (source focal length) of the first lens and the second focal length (target focal length) of the second lens, any first image at the time of zoom switching can be processed so as to be consistent with the second image by the calibration parameters, and there is no jump in the angle of view at the time of switching.
When the second focal length (source focal length) of the second lens is switched to the first focal length (target focal length) of the first lens, namely, the smaller focal length is switched to the larger focal length, because the image information of the smaller focal length is less than that of the larger focal length, calibration is not needed, and the image processing according to the smaller focal length or the larger focal length is only needed.
Referring to fig. 4, fig. 4 is a schematic flow chart of image processing for switching among multiple zoom lenses according to an embodiment of the present invention. When switching from a first focal length to a second focal length is carried out at a switching position in an adjacent focal length overlapping area, judging whether a calibration parameter is stored, and if the calibration parameter is stored, processing an image from a first focal length lens according to the calibration parameter; otherwise, triggering a current calibration based on a current first image from the first focal length lens and a current second image from a second focal length, wherein the first focal length is greater than the second focal length.
The description of the calibration process will be made below in the case where calibration parameters have not been stored.
To find the rotation offset θ and the field angle offset U, D, L, R, a homography matrix needs to be estimated. Referring to fig. 5, fig. 5 is a schematic flow chart illustrating a process of estimating a homography matrix by using a random consistency algorithm according to an embodiment of the present invention.
The homography transformation is used to describe the position mapping relationship of the object between the world coordinate system and the pixel coordinate system. The corresponding transformation matrix is called as a homography matrix, is a bridge of the same target with different view angles, and is a 3 multiplied by 3 matrix. Based on the first image and the second image matching the feature points, a homography matrix may be estimated. In this embodiment, a random consensus algorithm (RANSAC) may be employed to estimate the homography matrix. Specifically, the method comprises the following steps of,
505, judging that the number of elements in the current interior point set is greater than the number of the optimal interior point set, and if the number of elements in the current interior point set is greater than the number of the optimal interior point set, updating the optimal interior point set by using the current interior point set and updating the iteration times; otherwise, returning to the step of taking the next characteristic point pair as the current characteristic point pair,
and 506, judging that the iteration number reaches a set iteration threshold, ending if the iteration number reaches the set iteration threshold, otherwise, returning to 503 until an optimal inner point set with the maximum number of inner points is obtained, and taking the current homography matrix as a final estimation result.
The estimated homography matrix can be mathematically expressed as:
and after the homography matrix is obtained, solving the rotation offset and the field angle offset included in the affine matrix obtained by affine mapping of the first image according to the estimated homography matrix. The affine mapping comprises a translation mapping of an image center translating from an original position to a pixel coordinate system origin, a mapping of a first image center rotating to a rotation offset theta, an inverse translation mapping of an image center restoring to the original position, and a mapping of a clipped region according to a field angle offset. Preferably, the method further comprises removing the scaling map of the abnormal pixel after the rotation.
The following description will be given taking, as an example, a process of sequentially performing translation, rotation, scaling, inverse translation, and cropping of an image.
Since the rotation needs to be performed with the image center, the image center needs to be translated from the original position to the origin of the pixel coordinate system, correspondingly, all pixels in the image are translated, wherein the translation matrix of the translation map can be mathematically expressed as:
similarly, translating the center of the image from the origin of the pixel coordinate system to the original position, correspondingly translating all pixels in the image, as an inverse translation with respect to the translation, the inverse translation matrix of the inverse translation map can be mathematically expressed as:
a rotation mapping from the image center rotation to the rotational offset θ, whose rotation matrix can be mathematically expressed as:
the affine mapping relation of rotation by the image center is as follows:
wherein A is-1Is the inverse of the A matrix, B-1Is the inverse of the B matrix, C-1Is the inverse of the C matrix, (x ', y') is the mapped coordinates of point (x, y). Substituting the origin of coordinates s (0,0) into equation 1 yields the s' coordinate as:
similarly, substituting t (W,0) into equation 1 yields the t' coordinate as:
since a part of abnormal pixels exist in the image after the image is rotated, as shown in fig. 4, fig. 4 is a schematic diagram of the abnormal pixels existing after the image is rotated, wherein the shaded part is the abnormal pixels, and the part is usually filled with black pixels, and in order to improve the definition and the integrity of the image, scaling needs to be performed in a certain proportion so as to replace the pixels with scaled pixel values, thereby removing the abnormal pixels. The method specifically comprises the following steps:
obtaining a first height | Wsin theta | of the abnormal pixel in the height direction (longitudinal direction) according to the difference between formula 5 and formula 3, dividing the height H of the pixel in the height direction by a second height H | Wsin theta | of the remaining pixels after the abnormal pixel is removed in the height direction, and obtaining a scaling ratio in the height direction as follows:
to ensure that the image scale is constant, the vertical and horizontal scales are the same, i.e., the (horizontal) scale in the width direction coincides with the vertical direction. The scaling matrix for rejecting abnormal pixels is:
similarly, the difference between equation 4 and equation 2 may be used to obtain a first width | Wcos θ | of the abnormal pixel in the width direction (lateral direction), and the scaling in the width direction may be obtained by dividing the width W of the pixel in the width direction by the second width of the remaining pixels after the abnormal pixel is removed in the width direction:
in order to ensure that the image scale is unchanged, and therefore the scaling ratios in the longitudinal direction and the transverse direction are the same, the scaling matrix for eliminating the abnormal pixels is as follows:
according to the mapping of the clipping area of the field angle offset, the clipping matrix can be expressed mathematically as:
referring to fig. 6, fig. 6 is a schematic diagram of the whole process of processing the first image. The processing process comprises the steps of translating the center of a first image from an original position to the origin of a pixel coordinate system, rotating according to the rotation offset theta, removing abnormal pixel parts, translating the image to the original coordinate position, and finally cutting according to the angle of view offset. The affine mapping of this process can be expressed as:
A-1C-1R-1B-1E-1
since the estimated homography matrix is a mapping matrix that maps the first image to the second image and the processing also maps the first image to the second image, the affine mapping of the processing is the same as the estimated homography matrix, so that:
A-1C-1R-1B-1E-1＝J-1
substituting the inverse of A, C, B, R, E, J into the above equation yields:
solving to obtain:
order to
Then, the following steps are obtained:
in the same way, order
Then, the following steps are obtained:
the 5 parameters θ and U, D, L, R obtained are stored as calibration parameters.
For the zoom stitching device stored with the calibration parameters, in the process of capturing the image, when the first focal length is switched to the second focal length, the current first image is processed according to fig. 6, so that an equivalent result of homography transformation can be obtained, and thus, the field angle is not different when the focal lengths are switched. In particular implementation, namely: multiplying each pixel vector in the first image by a translation matrix A, a rotation matrix C, a scaling matrix R, an inverse translation matrix B and a cutting matrix E in sequence to obtain a processed pixel vector, wherein the pixel vector can be expressed mathematically as:
where (x, y) is the pixel coordinate after processing, and (x ', y') is the pixel coordinate before processing.
The embodiment of the invention can realize real-time automatic calibration through any current source image and target image, and greatly reduces the production cost without the assistance of complex external hardware resources compared with the common calibration of the auxiliary calibration lens switching through a standard tool; the zoom splicing equipment is switched and calibrated to obtain parameters required by image processing, so that images with more information can be compensated for images with less information, image offset caused by different field angles is processed, the field angles are not changed suddenly during switching, and user experience is good.
The present invention also provides a camera, characterized by comprising:
the first lens module comprises a first lens with a first focal length area and a first sensor for sensing light rays incident from the first lens and generating a first image;
the second lens module comprises a second lens with a second focal length area and a second sensor which senses light rays incident from the second lens and generates a second image;
the first focal length area of the first lens is overlapped with the adjacent area of the second focal length area of the second lens to form a focal length overlapping area; the field angle of the first image is greater than the field angle of the second image;
a processor for receiving a first image generated by the first sensor and receiving a second image generated by the second sensor, and outputting for display the received first image generated by the first sensor or the received second image generated by the second sensor;
in response to a focus switching instruction, wherein the focus switching instruction is used to instruct the camera to switch from a first focus area to a second focus area, the processor is configured to perform:
when the fact that the first lens is driven so that the focal length of the first lens is located at a switching position is detected, acquiring a first image of the first lens located at the switching position; wherein the switching position is a preset focal length in the focal length overlapping region;
according to pre-stored calibration parameters, performing image processing of rotation offset and/or field angle offset on a first image of the first lens at the switching position to generate a processed first image, so that the field angle of the processed first image and a second image generated by the second sensor have the same field angle; the prestored calibration parameters are obtained by performing calibration processing according to the image of the first lens of the camera at the switching position and the image of the second lens of the camera at the switching position, wherein the calibration processing comprises rotation offset and/or field angle offset processing;
outputting the processed first image for display until detecting that the second lens is driven such that the second lens focal length is at the switch position, the processor stopping outputting the first image from the first sensor and starting outputting the image from the second sensor.
Referring to fig. 7, fig. 7 is a schematic diagram of a control module of a camera including two zoom lenses according to an embodiment of the present invention. The camera comprises a main chip for processing images and a secondary chip for controlling lenses, wherein the secondary chip is respectively connected with each lens motor control chip, and each lens motor control chip respectively controls a focusing motor and a zoom magnification (zoom) motor of each lens so as to control the focusing of the lens through the focusing motor to make the images clear and control the zooming of the lens through the zoom motor; the auxiliary chip is also respectively connected with the diaphragm control chip, the infrared chip and the motor initial position control chip of each lens; the main chip is respectively connected with the sensors of the lenses and receives data from the sensors, and the main chip is connected with the auxiliary chip so as to interact control information with the auxiliary chip. Wherein, the main chip judges whether the calibration parameters are stored or not when the switching position in the focal length overlapping region at least comprising the first focal length and the second focal length is switched from the first focal length to the second focal length,
if the calibration parameters are stored, processing the image from the first focal length lens according to the calibration parameters;
otherwise, calibrating based on a first image from the first focal length lens and a second image from the second focal length, wherein the first focal length is larger than the second focal length;
and processing the image from the first focal length lens according to the calibrated parameters.
The main chip includes a Memory and a processor, where the Memory may include a Random Access Memory (RAM) or a Non-Volatile Memory (NVM), such as at least one disk Memory. Optionally, the memory may also be at least one memory device located remotely from the processor.
The Processor may be a general-purpose Processor, including a Central Processing Unit (CPU), a Network Processor (NP), and the like; but also Digital Signal Processors (DSPs), Application Specific Integrated Circuits (ASICs), Field Programmable Gate Arrays (FPGAs) or other Programmable logic devices, discrete Gate or transistor logic devices, discrete hardware components.
An embodiment of the present invention further provides a computer-readable storage medium, in which a computer program is stored, and when the computer program is executed by a processor, the computer program implements the following steps:
performing rotation offset and/or field angle offset processing on a first image from a first sensor at a switching position where the first lens and the second lens are both in a focal length overlapping region according to a pre-stored calibration parameter for switching processing, so that the first image before switching is displayed and a second image after switching is displayed, wherein the second image from a second sensor has the same field angle;
wherein the content of the first and second substances,
the calibration parameters for the switching process are calibrated based on a first image from a first sensor and a second image from a second sensor;
the field angle corresponding to the first focal length is larger than the field angle corresponding to the second focal length;
the focal length overlapping region is a focal length region between the upper limit position of the second focal length and the lower limit position of the first focal length.
For the device/network side device/storage medium embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and for the relevant points, refer to the partial description of the method embodiment.
In this document, relational terms such as first and second, and the like may be used solely to distinguish one entity or action from another entity or action without necessarily requiring or implying any actual such relationship or order between such entities or actions. Also, the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising an … …" does not exclude the presence of other identical elements in a process, method, article, or apparatus that comprises the element.
The above description is only for the purpose of illustrating the preferred embodiments of the present invention and is not to be construed as limiting the invention, and any modifications, equivalents, improvements and the like made within the spirit and principle of the present invention should be included in the scope of the present invention.
Claims (13)
1. The image processing method for switching the multiple zoom lenses is characterized in that the multiple zoom lenses at least comprise a first lens module with a first sensor and a first lens and a second lens module with a second sensor and a second lens; the first focal length area of the first lens is overlapped with the adjacent area of the second focal length area of the second lens to form a focal length overlapping area;
the method comprises the steps of (1) carrying out,
when the first lens is switched from the first focal length area to the second focal length area of the second lens,
performing rotation offset and/or view angle offset processing on a first image from a first sensor according to a pre-stored calibration parameter for switching processing at a switching position where the first lens and the second lens are both in the focal length overlapping region, so that the first image before switching is displayed and a second image after switching is displayed, wherein the first image before switching and the second image after switching have the same view angle;
wherein the content of the first and second substances,
the calibration parameters for the switching process are calibrated based on the first image from the first sensor and the second image from the second sensor, wherein the calibration parameters comprise a rotation offset and/or a field angle offset;
the field angle corresponding to the first focal length is larger than the field angle corresponding to the second focal length;
the focal length overlapping area is a focal length area between the upper limit position of the second focal length area and the lower limit position of the first focal length area, and the switching position is a preset focal length in the focal length overlapping area.
2. The method according to claim 1, wherein the calibration parameters for the switching process comprise rotation offset calibration parameters and/or field angle offset calibration parameters,
the calibration parameters for the switching process are triggered at the switching location based on a calibration of a first image from a first sensor and a second image from a second sensor, including,
estimating a homography matrix according to matching feature points of the current first image and the current second image at the switching position,
performing affine transformation including at least rotation offset and/or field angle offset on the first image to obtain an affine matrix corresponding to the affine transformation,
according to the relation between the estimated homography matrix and the affine matrix, the rotation offset and/or the field angle offset in the affine matrix are/is obtained, and the obtained rotation offset and/or the obtained field angle offset are/is used as the calibrated parameters and are stored;
wherein the content of the first and second substances,
the rotational offset includes rotating the first image about its image center to a rotational angle parallel to the second image,
the field angle offset includes an upper offset between upper boundaries, a lower offset between lower boundaries, a left offset between left boundaries, and a right offset between right boundaries of the first image and the second image rotated to be parallel to the second image.
3. The method of claim 2, wherein said estimating a homography matrix based on matching feature points of said first and second images comprises,
searching for matched characteristic point pairs based on the first image and the second image to obtain characteristic point pair samples,
and estimating a homography matrix corresponding to the optimal point pair set selected from the characteristic point pair samples by a random consistency algorithm, wherein the homography matrix is a 3 x 3 matrix.
4. The method of claim 3, wherein performing an affine transformation comprising at least a rotation offset and a field angle offset on the first image results in an affine matrix corresponding to the affine transformation, comprising,
constructing a translation matrix such that: translating the first image by taking the translation of the image center of the first image from the original position to the origin of the pixel coordinate system as a target,
constructing a rotation matrix such that: rotating the first image with an image center according to the rotational offset, wherein a rotation matrix is related to the rotational offset,
constructing an inverse translation matrix such that: taking the image center from the origin of the pixel coordinate system to the original position as a target, reversely translating the rotated first image,
constructing a clipping matrix such that: based on the first image after the inverse translation, clipping according to an area formed by the field angle offset; wherein the clipping matrix is related to the field angle offset,
and multiplying the inverse matrix of the translation matrix, the inverse matrix of the rotation matrix, the inverse matrix of the inverse translation matrix and the inverse matrix of the cutting matrix in sequence to obtain the affine matrix.
5. The method of claim 4, wherein after constructing a rotation matrix to rotate the first image about an image center according to the rotational offset, further comprising,
determining a scaling relation according to the abnormal pixel proportion in the rotated first image; wherein the abnormal pixel proportion is related to the rotation offset,
constructing a scaling matrix based on the scaling relation so as to scale the rotated first image;
multiplying the inverse matrix of the translation matrix, the inverse matrix of the rotation matrix, the inverse matrix of the inverse translation matrix and the inverse matrix of the cutting matrix in sequence to obtain an affine matrix,
and multiplying the inverse matrix of the translation matrix, the inverse matrix of the rotation matrix, the inverse matrix of the scaling matrix, the inverse matrix of the inverse translation matrix and the inverse matrix of the cutting matrix in sequence to obtain the affine matrix.
6. The method of claim 5, wherein determining the scaling based on the abnormal pixel scale in the rotated first image comprises,
a relation of a first height of an abnormal pixel in a height direction of the image panel is determined,
removing the first height of the abnormal pixels from the breadth height to obtain a second height of the residual pixels,
taking the ratio of the breadth height to the second height as a relation of scaling;
alternatively, the first and second electrodes may be,
calculating a relation of a first width of the abnormal pixel in the image width direction,
removing the first width of the abnormal pixels from the breadth width to obtain a second width of the residual pixels,
and taking the ratio of the breadth width to the second width as a relation of scaling.
7. The method of claim 6, wherein the calculating the relationship for the first height of the abnormal pixel in the height direction of the image panel comprises,
according to an affine mapping relation rotating by the image center, mapping between a first image pixel coordinate and a first image pixel coordinate after rotation is established, wherein the affine mapping comprises a translation matrix, a rotation matrix and an inverse translation matrix;
selecting any two known pixel coordinates from the rotated first image, substituting the known pixel coordinates into the affine mapping relation to obtain two corresponding pixel coordinates in the first image,
calculating the absolute value of the difference between the vertical coordinates of the two pixel coordinates to obtain the first height;
the calculating a relation of a first width of the abnormal pixel in the width direction of the image web according to the rotational offset includes,
and calculating the absolute value of the difference between the abscissas of the two obtained pixel coordinates to obtain the first width.
8. The method of claim 7, wherein the deriving the rotation offset and the field angle offset in the affine matrix based on the relationship between the estimated homography matrix and the affine matrix comprises,
solving the rotation offset and the field angle offset in the affine matrix according to the equal relation between the affine matrix and the inverse matrix of the homography matrix, and storing the result obtained by the solution;
the processing of the first image according to the calibrated parameters comprises,
obtaining a rotation offset according to the solution, determining a rotation matrix and a scaling matrix,
determining a cutting matrix according to the solved field angle offset,
sequentially multiplying the translation matrix, the determined rotation matrix, the determined scaling matrix, the inverse translation matrix and the determined cutting matrix to obtain a multiplication result;
and multiplying the multiplied result by each pixel coordinate vector in the first image to obtain each pixel coordinate vector of the processed image.
9. The method of claim 8, wherein the homography matrix is:
wherein F, G, K, P, Q, V is an element in the matrix;
the translation matrix is:
wherein, W is the width of the image breadth, and H is the height of the image breadth;
the rotation matrix is:
wherein θ is the rotational offset;
the scaling matrix is:
or the following steps:
the inverse translation matrix is:
the cutting matrix is as follows:
wherein U is an upper offset, D is a lower offset, L is a left offset, and R is a right offset;
the solution yields a rotational offset of:
and is
The solving results in the field angle offsets of respectively,
left offset:
right offset:
upper offset:
offset below:
wherein the content of the first and second substances,
10. the method according to claim 9, wherein the performing of the rotation offset and/or the viewing angle offset processing on the first image from the first sensor according to the pre-stored calibration parameters for the switching processing comprises,
and multiplying each pixel vector in the first image by a translation matrix A, a rotation matrix C, a scaling matrix R, an inverse translation matrix B and a cutting matrix E in sequence to obtain a pixel vector processed by the first image.
11. A camera, comprising:
the first lens module comprises a first lens with a first focal length area and a first sensor for sensing light rays incident from the first lens and generating a first image;
the second lens module comprises a second lens with a second focal length area and a second sensor which senses light rays incident from the second lens and generates a second image;
the first focal length area of the first lens is overlapped with the adjacent area of the second focal length area of the second lens to form a focal length overlapping area; the field angle of the first image is greater than the field angle of the second image;
a processor for receiving a first image generated by the first sensor and receiving a second image generated by the second sensor, and outputting for display the received first image generated by the first sensor or the received second image generated by the second sensor;
the processor configured to perform:
when the fact that the first lens is driven so that the focal length of the first lens is located at a switching position is detected, acquiring a first image of the first lens located at the switching position; wherein the switching position is a preset focal length in the focal length overlapping region;
according to pre-stored calibration parameters, performing image processing of rotation offset and/or field angle offset on a first image of the first lens at the switching position to generate a processed first image, so that the field angle of the processed first image and a second image generated by the second sensor have the same field angle; the prestored calibration parameters are obtained by performing calibration processing according to the image of the first lens of the camera at the switching position and the image of the second lens of the camera at the switching position, wherein the calibration processing comprises rotation offset and/or field angle offset processing;
outputting the processed first image for display until detecting that the second lens is driven such that the second lens focal length is at the switch position, the processor stopping outputting the first image from the first sensor and starting outputting the image from the second sensor.
12. The camera of claim 11, wherein optical axes of the first lens and the second lens are not collinear;
the processor performs image processing on the first image according to any one of the image processing methods described in claims 2 to 10.
13. The calibration method of the zooming splicing equipment of the multiple zoom lenses is characterized in that the multiple zoom lenses at least comprise a first lens module with a first sensor and a first lens and a second lens module with a second sensor and a second lens; the first focal length area of the first lens is overlapped with the adjacent area of the second focal length area of the second lens, and a focal length overlapping area exists; the method comprises the steps of (1) carrying out,
when the first lens is switched from the first focal length area to the second focal length area of the second lens, when the first lens and the second lens are at the switching position in the focal length overlapping area, triggering the calibration of the rotation offset and/or the field angle offset based on the current first image from the first sensor and the current second image from the second sensor, and obtaining the calibration parameters for the switching process,
wherein the content of the first and second substances,
the focal length overlapping area is a focal length area between the upper limit position of the second focal length area and the lower limit position of the first focal length area, and the switching position is a preset focal length in the focal length overlapping area;
the field angle corresponding to the first focal length is larger than the field angle corresponding to the second focal length.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201911069778.7A CN112770041A (en) | 2019-11-05 | 2019-11-05 | Image processing method for switching multiple zoom lenses and camera |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201911069778.7A CN112770041A (en) | 2019-11-05 | 2019-11-05 | Image processing method for switching multiple zoom lenses and camera |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN112770041A true CN112770041A (en) | 2021-05-07 |
Family
ID=75692630
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201911069778.7A Pending CN112770041A (en) | 2019-11-05 | 2019-11-05 | Image processing method for switching multiple zoom lenses and camera |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN112770041A (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113688824A (en) * | 2021-09-10 | 2021-11-23 | 福建汇川物联网技术科技股份有限公司 | Information acquisition method and device for construction node and storage medium |
Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| JP2014165517A (en) * | 2013-02-21 | 2014-09-08 | Canon Inc | Image pickup device and control method therefor |
| CN106056560A (en) * | 2015-04-03 | 2016-10-26 | 康耐视公司 | Homography rectification |
| CN107852467A (en) * | 2015-08-13 | 2018-03-27 | 核心光电有限公司 | Supported with video and switch the/Based on Dual-Aperture zoom camera without switching dynamic control |
| CN209017154U (en) * | 2018-10-31 | 2019-06-21 | 杭州海康威视数字技术股份有限公司 | A kind of video camera |
| CN110300267A (en) * | 2019-07-19 | 2019-10-01 | 维沃移动通信有限公司 | Photographic method and terminal device |
-
2019
- 2019-11-05 CN CN201911069778.7A patent/CN112770041A/en active Pending
Patent Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| JP2014165517A (en) * | 2013-02-21 | 2014-09-08 | Canon Inc | Image pickup device and control method therefor |
| CN106056560A (en) * | 2015-04-03 | 2016-10-26 | 康耐视公司 | Homography rectification |
| CN107852467A (en) * | 2015-08-13 | 2018-03-27 | 核心光电有限公司 | Supported with video and switch the/Based on Dual-Aperture zoom camera without switching dynamic control |
| CN209017154U (en) * | 2018-10-31 | 2019-06-21 | 杭州海康威视数字技术股份有限公司 | A kind of video camera |
| CN110300267A (en) * | 2019-07-19 | 2019-10-01 | 维沃移动通信有限公司 | Photographic method and terminal device |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113688824A (en) * | 2021-09-10 | 2021-11-23 | 福建汇川物联网技术科技股份有限公司 | Information acquisition method and device for construction node and storage medium |
| CN113688824B (en) * | 2021-09-10 | 2024-02-27 | 福建汇川物联网技术科技股份有限公司 | Information acquisition method, device and storage medium for construction node |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN108833785B (en) | Fusion method and device of multi-view images, computer equipment and storage medium | |
| US10694101B2 (en) | Contrast-enhanced combined image generation systems and methods | |
| US20230059657A1 (en) | Multi-camera zoom control method and apparatus, and electronic system and storage medium | |
| CN110536057B (en) | Image processing method and device, electronic equipment and computer readable storage medium | |
| US9946955B2 (en) | Image registration method | |
| US11107246B2 (en) | Method and device for capturing target object and video monitoring device | |
| US20150125070A1 (en) | Method and optical system for determining a depth map of an image | |
| KR101784787B1 (en) | Imaging device and method for automatic focus in an imaging device as well as a corresponding computer program | |
| WO2021184302A1 (en) | Image processing method and apparatus, imaging device, movable carrier, and storage medium | |
| US10346709B2 (en) | Object detecting method and object detecting apparatus | |
| US20210120194A1 (en) | Temperature measurement processing method and apparatus, and thermal imaging device | |
| CN112753217B (en) | Hybrid depth processing | |
| CN109559353B (en) | Camera module calibration method and device, electronic equipment and computer readable storage medium | |
| CN113875219B (en) | Image processing method and device, electronic equipment and computer readable storage medium | |
| US20150317770A1 (en) | Camera defocus direction estimation | |
| TW201616447A (en) | Method of quickly building up depth map and image processing device | |
| CN112087571A (en) | Image acquisition method and device, electronic equipment and computer readable storage medium | |
| US20230269475A1 (en) | Image reconstruction method and device | |
| CN112770041A (en) | Image processing method for switching multiple zoom lenses and camera | |
| CN111917941A (en) | Camera picture processing method and camera | |
| CN111369513A (en) | Abnormity detection method, abnormity detection device, terminal equipment and storage medium | |
| CN112866547B (en) | Focusing method and device, electronic equipment and computer readable storage medium | |
| CN110796084A (en) | Lane line recognition method, lane line recognition device, lane line recognition equipment and computer-readable storage medium | |
| Tai et al. | A fully automatic approach for fisheye camera calibration | |
| JP6381212B2 (en) | Imaging apparatus and control method thereof |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20210507 |