WO2020010945A1

WO2020010945A1 - Image processing method and apparatus, electronic device and computer-readable storage medium

Info

Publication number: WO2020010945A1
Application number: PCT/CN2019/088882
Authority: WO
Inventors: 陈岩
Original assignee: Oppo广东移动通信有限公司
Priority date: 2018-07-11
Filing date: 2019-05-28
Publication date: 2020-01-16
Also published as: CN110717942B; CN110717942A

Abstract

Disclosed is an image processing method, comprising: acquiring a calibrated image; testing the calibrated image to obtain a bevel edge area; acquiring a first modulus transfer function of the bevel edge area; reading a second modulus transfer function of a central area of the calibrated image; acquiring the ratio of the first modulus transfer function to the second modulus transfer function; and when the ratio exceeds a threshold value, determining that the definition of the calibrated image meets a preset condition.

Description

Image processing method and device, electronic device, computer-readable storage medium

Cross-reference to related applications

This application claims the priority of a Chinese patent application filed with the Chinese Patent Office on July 11, 2018, with application number 201810758592.1, and the invention name is "image processing method and device, electronic equipment, computer-readable storage medium" Incorporated by reference in this application.

Technical field

The present application relates to the field of imaging, and in particular, to an image processing method and device, an electronic device, and a computer-readable storage medium.

Background technique

With the development of electronic devices and imaging technology, more and more users use the cameras of electronic devices to collect images. The camera needs parameter calibration before leaving the factory, and calibration images need to be collected during the calibration operation.

Summary of the invention

The embodiments of the present application provide an image processing method and device, an electronic device, and a computer-readable storage medium, which can detect images that meet requirements and improve the accuracy of subsequent calibrations.

An image processing method includes:

Obtaining a calibration image;

Detecting the calibration image to obtain a hypotenuse region;

Obtaining a first modulus transfer function of the hypotenuse region;

Reading a second modulus transfer function of a central region of the calibration image;

Obtaining a ratio of the first modulus transfer function to a second modulus transfer function; and

When the ratio exceeds a threshold, it is determined that the sharpness of the calibration image meets a preset condition.

A calibration board including

The carrier; and

A preset pattern is provided on the carrier;

The preset pattern includes a calibration pattern and a hypotenuse pattern, the hypotenuse pattern is located on four sides of the calibration pattern, and a gap exists between the hypotenuse pattern and the calibration pattern.

An image processing device includes:

An image acquisition module for acquiring a calibration image;

A detection module, configured to detect the calibration image to obtain a hypotenuse region;

A parameter acquisition module, configured to acquire a first modulus transfer function of the hypotenuse region;

A reading module, configured to read a second modulus transfer function of a central region of the calibration image;

A ratio obtaining module, configured to obtain a ratio of the first modulus transfer function to a second modulus transfer function; and

A determining module, configured to determine that the sharpness of the calibration image meets a preset condition when the ratio exceeds a threshold.

An electronic device includes a memory and a processor. The memory stores a computer program. When the computer program is executed by the processor, the processor causes the processor to perform operations of the image processing method.

A computer-readable storage medium stores a computer program thereon, and when the computer program is executed by a processor, the operations of the image processing method are implemented.

The image processing method and device, electronic device, and computer-readable storage medium of the embodiments of the present application obtain a hypotenuse region by detecting the calibration image, obtain a first modulus transfer function of the hypotenuse region, and then obtain the center of the calibration image The second modulus transfer function of the region. When the ratio of the first modulus transfer function to the second modulus transfer function exceeds a threshold value, it means that the sharpness of the calibrated image meets the preset conditions, so a calibration that meets the criteria is selected Image, subsequent calibration, can improve the calibration accuracy.

BRIEF DESCRIPTION OF THE DRAWINGS

In order to explain the technical solutions in the embodiments of the present application or the prior art more clearly, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings in the following description are merely These are some embodiments of the present application. For those of ordinary skill in the art, other drawings can be obtained based on these drawings without paying creative work.

FIG. 1 is a schematic diagram of an application environment of dual camera calibration in an embodiment.

FIG. 2 is a schematic diagram of a chart of a conventional calibration board in an embodiment.

FIG. 3 is a flowchart of all blurred calibration images captured in an embodiment.

FIG. 4 is a schematic diagram of a partially blurred calibration image taken in an embodiment.

FIG. 5 is a feature point obtained by detecting the graph in FIG. 2.

FIG. 6 is a feature point obtained by detecting the blur image in FIG. 3.

FIG. 7 is a diagram illustrating pixel differences of feature points in FIGS. 5 and 6.

FIG. 8 is a schematic diagram of a preset pattern of a calibration plate in an embodiment.

FIG. 9 is a flowchart of an image processing method according to an embodiment.

FIG. 10 is a schematic diagram of dividing a hypotenuse region into multiple sub-regions according to an embodiment.

FIG. 11 is a flowchart of an image processing method in another embodiment.

FIG. 12 is a structural block diagram of an image processing apparatus in an embodiment.

FIG. 13 is a structural block diagram of an image processing apparatus in another embodiment.

FIG. 14 is a schematic diagram of an internal structure of an electronic device in an embodiment.

FIG. 15 is a schematic diagram of an image processing circuit in one embodiment.

detailed description

In order to make the purpose, technical solution, and advantages of the present application clearer, the present application is described in further detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the application, and are not used to limit the application.

It can be understood that the terms “first”, “second”, and the like used in this application can be used herein to describe various elements, but these elements are not limited by these terms. These terms are only used to distinguish the first element from another element. For example, without departing from the scope of the present application, the first calibration image may be referred to as a second calibration image, and similarly, the second calibration image may be referred to as a first calibration image. Both the first calibration image and the second calibration image are calibration images, but they are not the same calibration image.

FIG. 1 is a schematic diagram of an application environment of dual camera calibration in an embodiment. As shown in FIG. 1, the application environment includes a dual camera 110 and a calibration plate 120. The dual camera fixture 110 is used to place an electronic device with a dual camera module or a dual camera module. The calibration plate 120 (chart) has a chart pattern. The calibration plate 120 can be rotated to maintain different postures. The dual camera module on the dual camera 110 or the electronic device with the dual camera module takes the chart pattern on the calibration board 120 at different distances and different angles, usually at least 3 angles, as shown in the dual camera module in Figure 1. The optical axis of the group is perpendicular to the rotation axis of the calibration plate. The calibration plate 120 rotates three angles around the Y axis, one of which is 0 degrees, and the other two rotation angles are ± θ degrees, and θ is greater than 15 to ensure decoupling between attitudes. The calibration images of different angles are taken by the dual camera module to obtain calibration images of different angles. The hypotenuse region in the calibration image is obtained by detection. The first modulus transfer function of the hypotenuse region is obtained, and then the center region of the calibration image is obtained. The second modulus transfer function. When the ratio of the first modulus transfer function to the second modulus transfer function is within a preset range, it is determined that the calibration image meets the preset conditions, and the internal and external parameters of the single camera are determined according to the calibration image. Perform calibration, and obtain the external parameters of the dual camera module according to the internal and external parameters of the single camera. In this way, the calibration accuracy of internal and external parameters of a single camera is improved, and the calibration accuracy of external parameters of a dual camera module is also improved.

FIG. 2 is a chart of a conventional calibration board in one embodiment. As shown in Figure 2, the chart is a checkerboard diagram, which consists of a black grid and a white grid staggered. Among them, the length of the checkerboard may be equal or different, and the actual physical distance may be 5 to 30 cm. In other embodiments, the chart may be a circular chart. Due to the poor focus of the camera or damage to the lens, the image will be blurred. As shown in Figure 3, the calibration images taken are all blurred, and Figure 4 shows that the captured calibration images are partially blurred.

FIG. 5 is a feature point obtained by detecting the map in FIG. 2, FIG. 6 is a feature point obtained by detecting a fuzzy map in FIG. 3, and FIG. 7 is a pixel difference between the feature points in FIG. 5 and FIG. 6. In FIG. 7, the white dot-shaped area is a line connecting the feature points in FIG. 5 and the feature points in FIG. 6, and the point knowledge that appears is one pixel difference.

FIG. 8 is a schematic diagram of a preset pattern of a calibration plate in an embodiment of the present application. As shown in FIG. 8, the preset pattern includes a calibration pattern 810 and a hypotenuse pattern 820. The calibration pattern 810 is composed of black and white squares. The hypotenuse pattern 820 is located on four sides of the calibration pattern 810. The hypotenuse pattern 820 includes a left hypotenuse pattern, a right hypotenuse pattern, an upper hypotenuse pattern, and a bottom hypotenuse pattern. The upper, lower, left, and right are centered on the calibration pattern 810, and the hypotenuse pattern 820 is divided into four oblique sides, namely, upper, lower, left, and right, relative to the calibration pattern 810. There is a gap between the hypotenuse pattern 820 and the calibration pattern 810. The closest distance of the gap between the hypotenuse pattern 820 and the calibration pattern 810 is between 0.1 and 1 times the distance between two adjacent feature points in the calibration pattern 810. The inclined angle of the hypotenuse is controlled within 2 to 10 degrees.

FIG. 9 is a flowchart of an image processing method according to an embodiment. As shown in FIG. 9, in one embodiment, an image processing method includes operations 902 to 912.

Operation 902: Obtain a calibration image.

A calibration image containing a preset pattern is captured by a camera to obtain a calibration image.

In operation 904, the calibrated image is detected to obtain a hypotenuse region.

The beveled area is obtained by detecting the calibration image through edge and contour detection. Edge and contour detection can be implemented by filtering functions, including Laplacian (), Sobel (), and Scharr (). In other embodiments, a Canny edge detection algorithm can be used to implement edge detection to obtain a hypotenuse region or a connected domain algorithm to obtain a hypotenuse region.

Operation 906: Obtain a first modulus transfer function of the hypotenuse region.

MTF (Modulation, Transfer, Function) is an important indicator for measuring the performance of a lens. The loyalty of the lens to reproduce the contrast of the subject on the image surface is represented by the spatial frequency characteristics, and then the MTF curve is drawn. The horizontal axis of the graph represents the image height (the distance from the imaging center, in millimeters), and the vertical axis represents the contrast value, and the maximum contrast value is 1.

The first modulus transfer function of the hypotenuse region can be performed by the Slanted Edge Method (SEM) or Spatial Frequency Response (SFR).

The oblique edge method for detecting the oblique edge region includes: obtaining an edge spread function (ESF) of the oblique edge, and then deriving a corresponding line spread function (Linear Spread Function, LSF), and finally obtaining a MTF by Fourier transform.

The response function for a beveled edge can be represented by an impulse function:

When the edge response function is imaged by a perfect (no aberration) optical system, the imaging quality of the system is not degraded. Therefore, when the edge function is imaged by a linear invariant optical system, the output O (x) of the system is equal to the convolution of the line transfer function LSF and the response function S (x) of the system:

When x-α <0, the step function S (x) = 0, otherwise S (x) = 1, so ESF (x) can be expressed as:

Therefore, the derivative of ESF (x) can be written as:

So MTF can be written as the following function of LSF:

In general, the MTF normalizes the amplitude of zero frequency. At the same time, the MTF of the cascade system can be derived from the convolution definition and Fourier transform theory:

MTF _{opticalsystem} = MTF _lens × MTF _camera × MTF _display formula (6)

Obtaining the MTF by using the STR curve includes: obtaining the Edge Spread Function (ESF) of the oblique edge, then deriving the corresponding Line Spread Function (LSF), and finally obtaining the MTF by Fourier transform.

MTF can be the ratio of the difference between the maximum brightness and the minimum brightness and the sum of the maximum brightness and the minimum brightness, that is, MTF = (maximum brightness-minimum brightness) / (maximum brightness + minimum brightness).

Operation 908: Read the second modulus transfer function of the central region of the calibration image.

The center area of the calibration image may be an area that occupies the entire calibration image area to a preset ratio with the center point of the calibration image as the center. The preset ratio can be set as required, and the preset ratio can be 30% to 50%. The second modulus transfer function of the middle region of the calibration image can be statistically obtained through the actual module specification data.

Operation 910: Obtain a ratio of the first modulus transfer function and the second modulus transfer function.

Calculate the ratio of the first modulus transfer function to the second modulus transfer function. The formula is ratio = MTF _border / MTF _center , where ratio is the ratio of the first modulus transfer function to the second modulus transfer function, MTF _border. Is the first modulus transfer function, and MTF _center is the second modulus transfer function.

In operation 912, when the ratio exceeds the threshold, it is determined that the sharpness of the calibration image meets a preset condition.

The threshold can be set according to the specifications of the camera module. For example, the threshold can be a value within [0.3, 0.5].

When the ratio is in the range of [0, r], it is considered that the sharpness of the calibration image has not reached a preset condition. Among them, r is a threshold. The preset condition means that the sharpness reaches a preset standard.

The image processing method in the above embodiment obtains the hypotenuse region by detecting the calibration image, obtains the first modulus transfer function of the hypotenuse region, and then obtains the second modulus transfer function of the central region of the calibration image. The ratio of the first modulus transfer function to the second modulus transfer function exceeds the threshold value, which means that the sharpness of the calibration image meets a preset condition, so that a calibration image that meets the sharpness condition is selected, and subsequent calibration can improve the calibration accuracy.

In one embodiment, obtaining a calibration image includes: obtaining a calibration image by photographing a calibration board including a preset pattern, wherein the preset pattern includes a calibration pattern and a hypotenuse pattern, and the hypotenuse pattern is located on four sides of the calibration pattern And there is a gap between the hypotenuse pattern and the calibration pattern. The closest distance between the hypotenuse pattern and the calibration pattern is 0.1 to 1 times the distance between two adjacent feature points in the calibration pattern.

The hypotenuse pattern is located on the four sides of the calibration pattern. Then, the obtained calibration image is taken. The calibration image is detected to obtain four hypotenuse regions. The first modulus transfer functions of each of the four hypotenuse regions are obtained, and then calculated separately. Ratios of the four first modulus transfer functions to the second modulus transfer functions. When all four ratios exceed a threshold, it is determined that the sharpness of the calibration image meets a preset condition.

Two adjacent feature points in the calibration pattern refer to two adjacent feature points on the same row or column of the calibration pattern.

In one embodiment, the hypotenuse region is obtained through detection by a connected domain algorithm. Connected domain refers to an image area composed of pixels with the same pixel value and adjacent positions in the image. The connected domain algorithm refers to finding and labeling each connected area in the image. The connected domain algorithm can be the algorithm in the connected area labeling function bwlabel in Matlab. It traverses the image once, records the equivalent pairs of each line, and then relabels the original image through the equivalent pairs. You can also use the labeling algorithm used in the open source library cvBlob to label the entire image by locating the inner and outer contours of the connected area.

The specific operations of the connected area labeling function algorithm include: scanning the calibration image line by line, grouping consecutive white pixels in each line into a sequence called a clique, and recording its start and end points and its line number. For all the cliques in a row except one line, if it does not overlap with all the cliques in the previous line, give it a new label; if it only has a coincident area with a clique in the previous line, the previous line's The group's label is assigned to it; if it has an overlapping area with more than 2 groups in the previous line, the current group is assigned a minimum label of the connected group, and the marks of the several groups in the previous line are written into the equivalent Yes, it shows that they belong to one category. To convert equivalent pairs into equivalent sequences, each sequence needs to be given the same label. Starting from 1, give each equivalent sequence a label. Iterate through the tags of the starting group, find equivalent sequences, and give them new tags. The label of each group is filled into the calibration image.

In one embodiment, obtaining the first modulus transfer function of the hypotenuse region includes: obtaining the hypotenuse region, dividing the hypotenuse region into a first number of subregions; and obtaining the modulus transfer of the first number of subregions. Function; obtaining the first modulus transfer function of the hypotenuse region according to the modulus transfer function of the first number of sub-regions.

The processor may divide the hypotenuse region into a first number of sub-regions, and the first number may be set as required, such as 1, 2, 3, 5, 10, and the like. The hypotenuse of the hypotenuse region can be divided into a first number of line segments of the same size, or the hypotenuse of the hypotenuse region can be divided into a first number of line segments of different sizes.

As shown in FIG. 10, the hypotenuse region is the left hypotenuse region. Vertex A, vertex B, and vertex C of the hypotenuse region are identified. The edges of vertex A and vertex C are selected to calculate the first model of the hypotenuse region. The volume transfer function MTF divides the edge selection level of vertex A and vertex C into N parts, finds the modulus transfer function of each part, and then calculates the average value of the modulus transfer function of N parts to obtain the slope. The first modulus transfer function of the edge region. It is also possible to select a part of the sub-region's modulus transfer function from the first number of sub-areas, and obtain the first modulus transfer function of the hypotenuse region by obtaining a weighted average.

By dividing the hypotenuse region into multiple subregions, and then obtaining the modulus transfer function of the subregion, the first modulus transfer function of the hypotenuse region is obtained according to the modulus transfer function of the subregion, and the calculation is more accurate.

In an embodiment, obtaining the first modulus transfer function of the hypotenuse region according to the modulus transfer function of the first number of sub-regions includes: obtaining the modulus transfer function of the second number of sub-regions, the second The number of sub-regions is selected from the first number of sub-regions; the modulus transfer function of the second number of sub-regions is averaged to obtain the first modulus transfer function of the hypotenuse region.

The second number is less than the first number. The second quantity can be set as required. Sort the first number of subregions according to the division order to obtain the subregion sequence, select the second number of subregions in the middle position of the subregion sequence, calculate the modulus transfer function of the second number of subregions, and then obtain the average The first modulus transfer function of the hypotenuse region.

By selecting the second number of subregions in the middle position, the calculation result is more accurate.

FIG. 11 is a flowchart of an image processing method in another embodiment. As shown in FIG. 11, the image processing method includes:

Operation 1102: Obtain calibration images respectively captured by the first camera and the second camera in the dual camera module.

The calibration image is obtained by shooting the calibration plate through the first camera and the second camera in the dual camera module.

Operation 1104: Detect each calibration image to obtain a corresponding hypotenuse region.

The beveled area is obtained by detecting the calibration image through edge and contour detection. Or, the hypotenuse region can be obtained by detecting the connected domain algorithm.

Operation 1106: Obtain a first modulus transfer function of the hypotenuse region.

Operation 1108: Read the second modulus transfer function of the central region of the calibration image.

In operation 1110, a ratio of the first modulus transfer function to the second modulus transfer function is obtained.

In operation 1112, when the ratio exceeds the threshold, it is determined that the sharpness of the calibration image meets a preset condition.

Operation 1114: Obtain the internal and external parameters of the first camera and the internal and external parameters of the second camera according to the calibration image that meets the preset conditions.

The internal parameters of a single camera can include f _x , f _y , c _x , and c _y , where f _x represents the unit size of the focal length in the image coordinate system x-axis direction, and f _y represents the unit of the focal length in the image coordinate system y-axis direction. Pixel size, c _x , c _y represent the coordinates of the principal point of the image plane, and the principal point is the intersection of the optical axis and the image plane. f _x = f / d _x , f _y = f / d _y , where f is the focal length of a single camera, d _x is the width of a pixel in the x-axis direction of the image coordinate system, and d _y is the y-axis direction in the image coordinate system The width of one pixel. The image coordinate system is a coordinate system established on the basis of a two-dimensional image captured by a camera, and is used to specify the position of an object in the captured image. The origin of the (x, y) coordinate system in the image coordinate system is located on the focal point (c _x , c _y ) of the optical axis of the camera and the imaging plane. The unit is the length unit, that is, meters, (u, v) in the pixel coordinate system. The origin of the coordinate system is in the upper left corner of the image, and the unit is a quantity unit, that is, a unit. (x, y) is used to represent the perspective projection relationship of the object from the camera coordinate system to the image coordinate system, and (u, v) is used to represent the pixel coordinates. The conversion relationship between (x, y) and (u, v) is as shown in formula (1):

Perspective projection refers to a single-sided projection image that is closer to visual effects by projecting a shape onto a projection surface using the center projection method.

The external parameters of a single camera include a rotation matrix and a translation matrix that transform the coordinates in the world coordinate system to the coordinates in the camera coordinate system. The world coordinate system reaches the camera coordinate system through rigid body transformation, and the camera coordinate system reaches the image coordinate system through perspective projection transformation. Rigid body transformation refers to the movement of rotation and translation of a geometric object when the object is not deformed in three-dimensional space, which is the rigid body transformation. The rigid body transformation is as shown in formula (8).

Among them, X _c represents the camera coordinate system, X represents the world coordinate system, R represents the rotation matrix from the world coordinate system to the camera coordinate system, and T represents the translation matrix from the world coordinate system to the camera coordinate system. The distance between the origin of the world coordinate system and the origin of the camera coordinate system is controlled by the components in the three axis directions x, y, and z. It has three degrees of freedom, and R is the sum of the effects of rotation around the X, Y, and Z axes, respectively . t _x represents the translation amount in the x-axis direction, t _y represents the translation amount in the y-axis direction, and t _z represents the translation amount in the z-axis direction.

The world coordinate system is an absolute coordinate system in objective three-dimensional space and can be established at any position. For example, for each calibration image, the world coordinate system can be established with the upper left corner point of the calibration plate as the origin, the calibration plate plane as the XY plane, and the Z axis perpendicular to the calibration plate plane upwards. The camera coordinate system is based on the optical center of the camera as the origin, the optical axis of the camera is used as the Z axis, and the X and Y axes are respectively parallel to the X and Y axes of the image coordinate system. The principal point of the image coordinate system is the intersection of the optical axis and the image plane. The image coordinate system uses the principal point as the origin. The pixel coordinate system means that the origin is defined in the upper left corner of the image plane.

Using a single camera to take calibration plates with different angles to obtain a calibration image, extract feature points from the calibration image, and calculate the distortion-free case for the five internal and two external parameters of a single camera. Apply the least square method to calculate the distortion coefficient. The maximum likelihood method is optimized to obtain the final internal and external parameters of a single camera.

First, a camera model is established, and formula (9) is obtained.

among them,

Homogeneous coordinates represent the pixel coordinates (u, v, 1) of the image plane,

Homogeneous coordinates represent the coordinate points of the world coordinate system (X, Y, Z, 1), A represents the internal parameter matrix, R represents the rotation matrix converted from the world coordinate system to the camera coordinate system, and T represents the world coordinate system converted to the camera coordinate system Translation matrix.

Among them, α = f / d _x , β = f / d _y , f is the focal length of a single camera, d _x represents the width of one pixel in the x-axis direction of the image coordinate system, and d _y represents one pixel in the y-axis direction of the image coordinate system The width. γ represents the deviation of the pixel point in the x, y direction. u ₀ and v ₀ represent the coordinates of the principal point of the image plane, and the principal point is the intersection of the optical axis and the image plane.

The world coordinate system is constructed on the plane of Z = 0, and then the homography calculation is performed. If Z = 0, the above is converted into formula (11).

Homogeneity is defined in computer vision as a projection mapping from one plane to another. Let H = A [r ₁ r ₂ t], where H is a homography matrix. H is a 3 * 3 matrix and has one element as a homogeneous coordinate. Therefore, H has 8 unknowns to be solved. The homography matrix is written in the form of three column vectors, that is, H = [h ₁ h ₂ h ₃ ], thereby obtaining formula (12).

[h ₁ h ₂ h ₃ ] = λA [r ₁ r ₂ t] Formula (12)

For formula (14), two constraints are used. First, r ₁ , r _{2 are} orthogonal, and r ₁ r ₂ = 0, and r ₁ , r ₂ are rotated around the x and y axes, respectively. Second, the modulus of the rotation vector is 1, that is, | r ₁ | = | r ₂ | = 1. Through two constraints, r ₁ , r _{2 is} replaced by h ₁ , h ₂ and A. That is, r ₁ = h ₁ A ^-1 and r ₂ = h ₂ A ^-1 . According to two constraints, formula (15) can be obtained:

make

B is a symmetric matrix, so the effective elements of B are 6, and the 6 elements constitute the vector b.

b = [B ₁₁ , B ₁₂ , B ₂₂ , B ₁₃ , B ₂₃ , B ₃₃ ] ^T

It can be calculated that V _ij = [h _i1 h _j1 , h _i1 h _j2 + h _i2 h _j1 , h _i2 h _j2 , h _i3 h _j1 + h _i1 h _j3 , h _i3 h _j2 + h _i2 h _j3 , h _i3 h _j3 ] ^T

Use constraints to get the equations:

Using at least three images, apply formula (16) to estimate B, and decompose B to obtain the initial value of the internal parameter matrix A of the camera.

The external parameter matrix is calculated based on the internal parameter matrix to obtain the initial value of the external parameter matrix.

Where λ = 1 / || A ^-1 h ₁ || = 1 / || A ^-1 h ₂ ||.

The complete geometric model of the camera uses the formula (16)

Among them, formula (16) is a geometric model obtained by constructing the world coordinate system on the plane Z is 0, X, Y are the world coordinates of the feature points on the calibration plate, and x, y, and z are the feature points on the calibration plate on the camera The physical coordinates of the coordinate system.

R is the rotation matrix from the world coordinate system of the calibration plate to the camera coordinate system, and T is the translation matrix from the world coordinate system of the calibration plate to the camera coordinate system.

The physical coordinates [x, y, z] of the feature points on the calibration board in the camera coordinate system are normalized to obtain the target coordinate points (x ', y').

Distortion processing is performed on the camera coordinate system image points using the distortion model.

Use internal parameters to convert physical coordinates to image coordinates.

The initial values of the internal parameter matrix and the external parameter matrix are imported into the maximum likelihood formula to obtain the final internal parameter matrix and external parameter matrix. The maximum likelihood formula is

Find the minimum.

The dual camera module includes a first camera and a second camera. The first camera and the second camera may both be color cameras, or one is a black and white camera, one is a color camera, or two black and white cameras.

In operation 1116, external parameters of the dual camera module are obtained according to the internal and external parameters of the first camera and the internal and external parameters of the second camera.

The dual camera calibration refers to determining the external parameter value of the dual camera module. The external parameters of the dual camera module include a rotation matrix between the dual cameras and a translation matrix between the dual cameras. The rotation matrix and translation matrix between the two cameras can be obtained by formula (20).

Among them, R ′ is the rotation matrix between the two cameras, T ′ is the translation matrix between the two cameras, and R _r is the rotation matrix of the first calibration camera relative to the calibration object (that is, the coordinates of the calibration object in the world coordinate system are converted to The rotation matrix of the coordinates of the camera coordinate system of the first camera), T _r is the translation matrix of the relative calibration object obtained by the calibration of the first camera (that is, the coordinates of the calibration object in the world coordinate system are converted to the camera coordinate system of the first camera Coordinate translation matrix). R _l is the rotation matrix of the second camera relative to the calibration object (that is, the coordinates of the calibration object in the world coordinate system are converted to the coordinates of the camera camera coordinate system of the second camera), T _l is the second camera calibration The obtained translation matrix of the relative calibration object (ie, the translation matrix of the coordinates of the calibration object in the world coordinate system and the coordinates of the camera coordinate system of the second camera).

In this embodiment, the first modulus transfer function of the hypotenuse region in the calibration image and the second modulus transfer function of the center region are calculated from the calibration images collected by the dual camera module, and the ratio between the two is calculated. According to the ratio and the threshold value, Compare to determine whether the sharpness of the calibration image meets the preset conditions and meet the preset conditions, then use the calibration image with sharpness that meets the preset conditions for single-camera and dual-camera module calibration to improve the calibration accuracy.

The embodiment of the present application further provides a calibration board. The calibration plate includes a carrier; a preset pattern is disposed on the carrier; the preset pattern includes a calibration pattern and a hypotenuse pattern, the hypotenuse pattern is located on four sides of the calibration pattern, and the hypotenuse pattern and the There are gaps between the calibration patterns. The closest distance between the hypotenuse pattern and the calibration pattern is 0.1 to 1 times the distance between two adjacent feature points in the calibration pattern.

The hypotenuse pattern is located on the four sides of the calibration pattern. Then, the obtained calibration image is taken. The calibration image is detected to obtain four hypotenuse regions. The first modulus transfer functions of each of the four hypotenuse regions are obtained, and then calculated separately. Ratios of the four first modulus transfer functions to the second modulus transfer functions. When all four ratios exceed a threshold, it is determined that the sharpness of the calibration image meets a preset condition. Two adjacent feature points in the calibration pattern refer to two adjacent feature points on the same row or column of the calibration pattern.

In one embodiment, the inclined angle of the hypotenuse in the hypotenuse pattern is controlled within 2 to 10 degrees.

FIG. 12 is a structural block diagram of an image processing apparatus in an embodiment. As shown in FIG. 12, the image processing apparatus includes an image acquisition module 1202, a detection module 1204, a parameter acquisition module 1206, a reading module 1208, a ratio acquisition module 1210, and a determination module 1212. among them:

The image acquisition module 1202 is configured to acquire a calibration image.

The detection module 1204 is configured to detect the calibration image to obtain a hypotenuse region.

The parameter obtaining module 1206 is configured to obtain a first modulus transfer function of the hypotenuse region.

The reading module 1208 is configured to read a second modulus transfer function of a central area of the calibration image.

The ratio obtaining module 1210 is configured to obtain a ratio of the first modulus transfer function and the second modulus transfer function.

The determining module 1212 is configured to determine that the sharpness of the calibration image meets a preset condition when the ratio exceeds a threshold.

The image processing apparatus in the above embodiment obtains the hypotenuse region by detecting the calibration image, obtains the first modulus transfer function of the hypotenuse region, and then obtains the second modulus transfer function of the central region of the calibration image. The ratio of the first modulus transfer function to the second modulus transfer function exceeds the threshold value, which means that the sharpness of the calibration image meets a preset condition, so that a calibration image that meets the sharpness condition is selected, and subsequent calibration can improve the calibration accuracy.

In one embodiment, the image acquisition module 1202 is further configured to obtain a calibration image by shooting a calibration plate that includes a preset pattern, where the preset pattern includes a calibration pattern and a hypotenuse pattern, and the hypotenuse pattern is located at four points of the calibration pattern. The closest distance between the hypotenuse pattern and the calibration pattern is between 0.1 and 1 times the distance between two adjacent feature points in the calibration pattern.

In one embodiment, the parameter obtaining module 1206 is further configured to obtain the hypotenuse region, and divide the hypotenuse region into a first number of subregions; acquire a modulus transfer function of the first number of subregions; and according to the first The modulus transfer function of the number of sub-regions obtains the first modulus transfer function of the hypotenuse region.

In one embodiment, the parameter obtaining module 1206 is further configured to obtain a modulus transfer function of a second number of sub-regions, the second number of sub-regions being selected from the first number of sub-regions; The modulus transfer function of the number of subregions is averaged to obtain the first modulus transfer function of the hypotenuse region.

In one embodiment, the parameter obtaining module 1206 is further configured to obtain the first modulus transfer function of the hypotenuse region by using a sloped edge method or a spatial frequency domain response curve.

FIG. 13 is a structural block diagram of an image processing apparatus in another embodiment. As shown in FIG. 13, the image processing apparatus includes an image acquisition module 1202, a detection module 1204, a parameter acquisition module 1206, a reading module 1208, a ratio acquisition module 1210, and a determination module 1212, and further includes a calibration module 1214. The image acquisition module 1202 is further configured to acquire calibration images respectively captured by the first camera and the second camera in the dual camera module. The detection module 1204 is further configured to detect each calibration image to obtain a corresponding hypotenuse region. The parameter acquisition module 1206 is further configured to acquire a first modulus transfer function of the hypotenuse region. The reading module 1208 is further configured to read a second modulus transfer function of a central region of the calibration image. The ratio obtaining module 1210 is further configured to obtain a ratio of the first modulus transfer function and the second modulus transfer function. The determining module 1212 is further configured to determine that the sharpness of the calibration image meets a preset condition when the ratio exceeds a threshold. The calibration module 1214 is configured to obtain the internal and external parameters of the first camera and the internal and external parameters of the second camera according to the calibration image that meets the preset conditions; according to the internal and external parameters of the first camera and the second The internal and external parameters of the camera obtain the external parameters of the dual camera module.

An embodiment of the present application further provides an electronic device. The electronic device includes a memory and a processor. A computer program is stored in the memory. When the computer program is executed by the processor, the processor causes the processor to perform operations in the image processing method.

An embodiment of the present application provides a non-volatile computer-readable storage medium. A non-transitory computer-readable storage medium having stored thereon a computer program that, when executed by a processor, implements operations in the following image processing methods.

FIG. 14 is a schematic diagram of an internal structure of an electronic device in an embodiment. As shown in FIG. 14, the electronic device includes a processor, a memory, and a network interface connected through a system bus. The processor is used to provide computing and control capabilities to support the operation of the entire electronic device. The memory is used to store data, programs, and the like. At least one computer program is stored on the memory, and the computer program can be executed by a processor to implement the wireless network communication method applicable to the electronic device provided in the embodiments of the present application. The memory may include a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The computer program can be executed by a processor to implement an image processing method provided by each of the following embodiments. The internal memory provides a cached operating environment for operating system computer programs in a non-volatile storage medium. The network interface may be an Ethernet card or a wireless network card, and is used to communicate with external electronic devices. The electronic device may be a mobile phone, a tablet computer, or a personal digital assistant or a wearable device.

The implementation of each module in the image processing apparatus provided in the embodiments of the present application may be in the form of a computer program. The computer program can be run on a terminal or a server. The program module constituted by the computer program can be stored in the memory of the terminal or server. When the computer program is executed by a processor, the operations of the method described in the embodiments of the present application are implemented.

A computer program product containing instructions that, when run on a computer, causes the computer to perform an image processing method.

An embodiment of the present application further provides an electronic device. The above electronic device includes an image processing circuit. The image processing circuit may be implemented by hardware and / or software components, and may include various processing units that define an ISP (Image Signal Processing) pipeline. FIG. 15 is a schematic diagram of an image processing circuit in one embodiment. As shown in FIG. 15, for ease of description, only aspects of the image processing technology related to the embodiments of the present application are shown.

As shown in FIG. 15, the image processing circuit includes a first ISP processor 1530, a second ISP processor 1540, and a control logic 1550. The first camera 1510 includes one or more first lenses 1512 and a first image sensor 1514. The first image sensor 1514 may include a color filter array (such as a Bayer filter). The first image sensor 1514 may obtain light intensity and wavelength information captured by each imaging pixel of the first image sensor 1514, and provide information that can be captured by the first ISP. A set of image data processed by the processor 1530. The second camera 1520 includes one or more second lenses 1522 and a second image sensor 1524. The second image sensor 1524 may include a color filter array (such as a Bayer filter). The second image sensor 1524 may obtain light intensity and wavelength information captured by each imaging pixel of the second image sensor 1524, and provide information that may be captured by the second ISP. A set of image data processed by the processor 1540.

The first image collected by the first camera 1510 is transmitted to the first ISP processor 1530 for processing. After the first ISP processor 1530 processes the first image, the first image statistical data (such as the brightness of the image and the contrast value of the image) can be processed. , Image color, etc.) to the control logic 1550. The control logic 1550 can determine the control parameters of the first camera 1510 according to the statistical data, so that the first camera 1515 can perform operations such as autofocus and automatic exposure according to the control parameters. The first image may be stored in the image memory 1560 after being processed by the first ISP processor 1530, and the first ISP processor 1530 may also read the image stored in the image memory 1560 for processing. In addition, the first image can be directly sent to the display 1570 for display after being processed by the ISP processor 1530. The display 1570 can also read the image in the image memory 1560 for display.

Among them, the first ISP processor 1530 processes image data pixel by pixel in a variety of formats. For example, each image pixel may have a bit depth of 8, 10, 12, or 14 bits, and the first ISP processor 1530 may perform one or more image processing operations on the image data and collect statistical information about the image data. The image processing operation may be performed with the same or different bit depth calculation accuracy.

The image memory 1560 may be a part of a memory device, a storage device, or a separate dedicated memory in an electronic device, and may include a DMA (Direct Memory Access) feature.

When receiving the interface from the first image sensor 1514, the first ISP processor 1530 may perform one or more image processing operations, such as time-domain filtering. The processed image data may be sent to the image memory 1560 for further processing before being displayed. The first ISP processor 1530 receives processed data from the image memory 1560, and performs image data processing in the RGB and YCbCr color spaces on the processed data. The image data processed by the first ISP processor 1530 may be output to the display 1570 for viewing by a user and / or further processed by a graphics engine or a GPU (Graphics Processing Unit). In addition, the output of the first ISP processor 1530 can also be sent to the image memory 1560, and the display 1570 can read image data from the image memory 1560. In one embodiment, the image memory 1560 may be configured to implement one or more frame buffers.

The statistical data determined by the first ISP processor 1530 may be sent to the control logic 1550. For example, the statistical data may include the first image sensor 1514 statistical information such as auto exposure, auto white balance, auto focus, flicker detection, black level compensation, and first lens 1512 shading correction. The control logic 1550 may include a processor and / or a microcontroller that executes one or more routines (such as firmware). The one or more routines may determine the control parameters and the first parameters of the first camera 1510 according to the received statistical data. Control parameters of an ISP processor 1530. For example, the control parameters of the first camera 1510 may include gain, integration time of exposure control, image stabilization parameters, flash control parameters, control parameters of the first lens 1512 (for example, focal length for focusing or zooming), or a combination of these parameters. The ISP control parameters may include a gain level and a color correction matrix for automatic white balance and color adjustment (eg, during RGB processing), and a first lens 1512 shading correction parameter.

Similarly, the second image collected by the second camera 1520 is transmitted to the second ISP processor 1540 for processing. After the second ISP processor 1540 processes the first image, statistical data of the second image (such as image brightness, image (Contrast value, image color, etc.) are sent to the control logic 1550. The control logic 1550 can determine the control parameters of the second camera 1520 according to statistical data, so that the second camera 1520 can perform operations such as autofocus and automatic exposure according to the control parameters . The second image may be stored in the image memory 1560 after being processed by the second ISP processor 1540, and the second ISP processor 1540 may also read the image stored in the image memory 1560 for processing. In addition, the second image may be directly sent to the display 1570 for display after being processed by the ISP processor 1540, and the display 1570 may also read the image in the image memory 1560 for display. The second camera 1520 and the second ISP processor 1540 may also implement processing operations as described by the first camera 1510 and the first ISP processor 1530.

The following is the operation of implementing the image processing method using the image processing technology in FIG. 15.

Any reference to memory, storage, database, or other media used in this application may include non-volatile and / or volatile memory. Suitable non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory can include random access memory (RAM), which is used as external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDR, SDRAM), enhanced SDRAM (ESDRAM), synchronous Link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

The above-mentioned embodiments only express several implementation manners of the present application, and their descriptions are more specific and detailed, but they should not be construed as limiting the patent scope of the present application. It should be noted that, for those of ordinary skill in the art, without departing from the concept of the present application, several modifications and improvements can be made, and these all belong to the protection scope of the present application. Therefore, the protection scope of this application patent shall be subject to the appended claims.

Claims

An image processing method includes:

Obtaining a calibration image;

Detecting the calibration image to obtain a hypotenuse region;

Obtaining a first modulus transfer function of the hypotenuse region;

Reading a second modulus transfer function of a central region of the calibration image;

Obtaining a ratio of the first modulus transfer function to a second modulus transfer function; and

When the ratio exceeds a threshold, it is determined that the sharpness of the calibration image meets a preset condition.
The method according to claim 1, wherein the acquiring a calibration image comprises:

A calibration image is obtained by shooting a calibration plate containing a preset pattern, wherein the preset pattern includes a calibration pattern and a hypotenuse pattern, the hypotenuse pattern is located on four sides of the calibration pattern, and the hypotenuse pattern There is a gap with the calibration pattern.
The method according to claim 1, wherein the acquiring a first modulus transfer function of the hypotenuse region comprises:

Acquiring the hypotenuse region and dividing the hypotenuse region into a first number of subregions;

Obtaining a modulus transfer function of the first number of sub-regions; and

A first modulus transfer function of the hypotenuse region is obtained according to a modulus transfer function of the first number of sub-regions.
The method according to claim 3, wherein the obtaining the first modulus transfer function of the hypotenuse region according to the modulus transfer function of the first number of sub-regions comprises:

Obtaining a modulus transfer function for a second number of sub-regions, the second number of sub-regions being selected from the first number of sub-regions; and

The modulus transfer function of the second number of sub-regions is averaged to obtain the first modulus transfer function of the hypotenuse region.
The method according to any one of claims 1 to 4, further comprising:

The oblique edge method or the spatial frequency domain response curve is used to obtain the first modulus transfer function of the hypotenuse region.
The method according to claim 1, further comprising:

Obtaining calibration images taken by the first camera and the second camera in the dual camera module;

Detecting each calibration image to obtain a corresponding hypotenuse region;

Obtaining a first modulus transfer function of the hypotenuse region;

Reading a second modulus transfer function of a central region of the calibration image;

Obtaining a ratio of the first modulus transfer function to a second modulus transfer function;

When the ratio exceeds a threshold, determining that the sharpness of the calibration image meets a preset condition;

Obtaining internal and external parameters of a first camera and a internal and external parameter of a second camera in a dual camera module according to a calibration image that meets preset conditions; and

Obtaining external parameters of the dual camera module according to the internal and external parameters of the first camera and the internal and external parameters of the second camera.
The method according to claim 1, further comprising:

When the hypotenuse pattern is located on the four sides of the calibration image, the calibration image is detected to obtain four hypotenuse regions, and the first modulus transfer function of each of the four hypotenuse regions is obtained. The ratio of the first modulus transfer function to the second modulus transfer function. When all four ratios exceed the threshold, it is determined that the sharpness of the calibration image meets a preset condition.
A calibration board characterized by comprising:

The carrier; and

A preset pattern is provided on the carrier;

The preset pattern includes a calibration pattern and a hypotenuse pattern, the hypotenuse pattern is located on four sides of the calibration pattern, and a gap exists between the hypotenuse pattern and the calibration pattern.
The calibration plate according to claim 8, wherein a closest distance between a gap between the hypotenuse pattern and the calibration pattern is 0.1 to 1 of a distance between two adjacent feature points in the calibration pattern Times.
An image processing device, comprising:

An image acquisition module for acquiring a calibration image;

A detection module, configured to detect the calibration image to obtain a hypotenuse region;

A parameter acquisition module, configured to acquire a first modulus transfer function of the hypotenuse region;

A reading module, configured to read a second modulus transfer function of a central region of the calibration image;

A ratio obtaining module, configured to obtain a ratio of the first modulus transfer function to a second modulus transfer function; and

A determining module, configured to determine that the sharpness of the calibration image meets a preset condition when the ratio exceeds a threshold.
An electronic device includes a memory and a processor. The memory stores a computer program. When the computer program is executed by the processor, the processor causes the processor to perform the following operations:

Obtaining a calibration image;

Detecting the calibration image to obtain a hypotenuse region;

Obtaining a first modulus transfer function of the hypotenuse region;

Reading a second modulus transfer function of a central region of the calibration image;

Obtaining a ratio of the first modulus transfer function to a second modulus transfer function; and

When the ratio exceeds a threshold, it is determined that the sharpness of the calibration image meets a preset condition.
The electronic device according to claim 11, wherein the processor is further configured to execute:

A calibration image is obtained by shooting a calibration plate containing a preset pattern, wherein the preset pattern includes a calibration pattern and a hypotenuse pattern, the hypotenuse pattern is located on four sides of the calibration pattern, and the hypotenuse pattern There is a gap with the calibration pattern.
The electronic device according to claim 11, wherein the processor is further configured to execute:

Acquiring the hypotenuse region and dividing the hypotenuse region into a first number of subregions;

Obtaining a modulus transfer function of the first number of sub-regions; and

A first modulus transfer function of the hypotenuse region is obtained according to a modulus transfer function of the first number of sub-regions.
The electronic device according to claim 13, wherein the processor is further configured to execute:

Obtaining a modulus transfer function for a second number of sub-regions, the second number of sub-regions being selected from the first number of sub-regions; and

The modulus transfer function of the second number of sub-regions is averaged to obtain the first modulus transfer function of the hypotenuse region.
The electronic device according to any one of claims 11 to 14, wherein the processor is further configured to execute:

The oblique edge method or the spatial frequency domain response curve is used to obtain the first modulus transfer function of the hypotenuse region.
The electronic device according to claim 11, wherein the processor is further configured to execute:

Obtaining calibration images taken by the first camera and the second camera in the dual camera module;

Detecting each calibration image to obtain a corresponding hypotenuse region;

Obtaining a first modulus transfer function of the hypotenuse region;

Reading a second modulus transfer function of a central region of the calibration image;

Obtaining a ratio of the first modulus transfer function to a second modulus transfer function;

When the ratio exceeds a threshold, determining that the sharpness of the calibration image meets a preset condition;

Obtaining internal and external parameters of a first camera and a internal and external parameter of a second camera in a dual camera module according to a calibration image that meets preset conditions; and

Obtaining external parameters of the dual camera module according to the internal and external parameters of the first camera and the internal and external parameters of the second camera.
The electronic device according to claim 11, wherein the processor is further configured to execute:

When the hypotenuse pattern is located on the four sides of the calibration image, the calibration image is detected to obtain four hypotenuse regions, and the first modulus transfer function of each of the four hypotenuse regions is obtained. The ratio of the first modulus transfer function to the second modulus transfer function. When all four ratios exceed the threshold, it is determined that the sharpness of the calibration image meets a preset condition.
A computer-readable storage medium having stored thereon a computer program, wherein when the computer program is executed by a processor, the operations of the image processing method according to any one of claims 1 to 7 are realized.