WO2023279849A1

WO2023279849A1 - Image processing method and apparatus, electronic device, and storage medium

Info

Publication number: WO2023279849A1
Application number: PCT/CN2022/092755
Authority: WO
Inventors: 周阳
Original assignee: Oppo广东移动通信有限公司
Priority date: 2021-07-09
Filing date: 2022-05-13
Publication date: 2023-01-12
Also published as: CN113572929B; CN113572929A

Abstract

The present application provides an image processing method, comprising a lens, an image sensor, an optical image stabilization assembly, and an electronic image stabilization assembly. Optical image stabilization of the lens and the image sensor is implemented by the optical image stabilization assembly to obtain an optically stabilized image; the attitude of the optically stabilized image is restored and corrected according to position and attitude data of the lens as well as position and attitude data and rotation attitude data of the image sensor to obtain a corrected image; and the corrected image is subjected to electronic image stabilization by the electronic image stabilization assembly.

Description

Image processing method, device, electronic device and storage medium

This application claims the priority of the Chinese patent application with the application number 202110777734.0 and the application name "image processing method, device, electronic equipment and storage medium" submitted to the China Patent Office on July 09, 2021, the entire contents of which are incorporated by reference in this application.

technical field

The present application relates to the technical field of image processing, and in particular to an image processing method, device, electronic equipment and storage medium.

Background technique

At present, electronic devices such as mobile phones and tablet computers are usually equipped with cameras, so as to provide users with a camera function, so that users can record what is happening around them, the scenery they see, etc. anytime and anywhere through these electronic devices. However, because the user usually holds the electronic device for shooting, and the electronic device held by the user will introduce different degrees of shaking and affect the stability of the shooting of the electronic device, resulting in poor quality of the captured image.

Contents of the invention

Embodiments of the present application provide an image processing method, device, electronic equipment, and storage medium.

In a first aspect, the present application discloses an image processing method applied to an electronic device, the electronic device includes a lens, an image sensor, an optical anti-shake component, and an electronic anti-shake component, and the image processing method includes:

Acquire motion data of the electronic device, and according to the motion data, control the lens to perform compensation translation in the X-axis direction and the Y-axis direction of the lens through the optical anti-shake component, and control the image sensor in the performing compensation translation in the X-axis direction and the Y-axis direction of the image sensor, and controlling the compensation rotation of the image sensor around the Z-axis of the image sensor;

Acquiring the optical image stabilization image corresponding to the motion data collected by the image sensor, and acquiring the first position and posture data of the lens, the second position and posture data of the image sensor and the first rotation attitude data;

Restoring and correcting the posture of the optical image stabilization image according to the first position and posture data, the second position and posture data, and the first rotation posture data to obtain a corrected image;

According to the motion data, electronic anti-shake processing is performed on the corrected image through the electronic anti-shake component to obtain an electronic anti-shake image.

In a second aspect, the present application also discloses an image processing device, which is applied to an electronic device, and the electronic device includes a lens, an image sensor, an optical anti-shake component, and an electronic anti-shake component, and the image processing device includes:

an optical anti-shake module, configured to acquire motion data of the electronic device, and control the lens to perform compensation translation in the X-axis direction and the Y-axis direction of the lens through the optical anti-shake component according to the motion data, controlling the image sensor to perform compensation translation in the X-axis direction and the Y-axis direction of the image sensor, and controlling the image sensor to perform compensation rotation around the Z-axis of the image sensor;

an attitude acquisition module, configured to acquire the optical image stabilization image corresponding to the movement data collected by the image sensor, and acquire the first position attitude data of the lens, the second position and attitude data of the image sensor from the optical image stabilization component Position attitude data and first rotation attitude data;

A pose correction module, configured to restore and correct the pose of the optical image stabilization image according to the first position and pose data, the second position and pose data, and the first rotation pose data, to obtain a corrected image;

The electronic anti-shake module is configured to perform electronic anti-shake processing on the corrected image through the electronic anti-shake component according to the motion data, so as to obtain an electronic anti-shake image.

In the third aspect, the present application also discloses an electronic device, including a lens, an image sensor, an optical anti-shake component, an electronic anti-shake component, a processor, and a memory. Steps in the provided image processing method.

In a fourth aspect, the present application also discloses a storage medium on which a computer program is stored, and when the computer program is loaded by a processor, the steps in the image processing method provided in the present application are executed.

Description of drawings

In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following briefly introduces the drawings that need to be used in the description of the embodiments.

FIG. 1 is a schematic diagram of a hardware structure of an electronic device provided by an embodiment of the present application.

FIG. 2 is a schematic flowchart of an image processing method provided by an embodiment of the present application.

FIG. 3 is an example diagram of the degrees of freedom of movement of the camera module in the embodiment of the present application.

FIG. 4 is an example diagram of correcting an optical image stabilization image in an embodiment of the present application to obtain a corrected image.

FIG. 5 is a partial schematic diagram of a preset calibration tool provided by an embodiment of the present application.

Fig. 6 is an example diagram of a rectangular calibration plate provided in the embodiment of the present application.

FIG. 7 is a schematic structural diagram of an image processing device provided by an embodiment of the present application.

FIG. 8 is a schematic diagram of another hardware structure of an electronic device provided by an embodiment of the present application.

detailed description

It should be noted that the terms "first", "second" and "third" in this application are used to distinguish different objects, rather than to describe a specific order. Furthermore, the terms "include" and "have", as well as any variations thereof, are intended to cover a non-exclusive inclusion. For example, a process, method, system, product, or device that includes a series of steps or modules is not limited to the listed steps or modules, but some embodiments also include steps or modules that are not listed, or some embodiments Other steps or modules inherent to these processes, methods, products or devices are also included.

Reference herein to an "embodiment" means that a particular feature, structure, or characteristic described in connection with the embodiment can be included in at least one embodiment of the present application. The occurrences of this phrase in various places in the specification are not necessarily all referring to the same embodiment, nor are separate or alternative embodiments mutually exclusive of other embodiments. It is understood explicitly and implicitly by those skilled in the art that the embodiments described herein can be combined with other embodiments.

An embodiment of the present application provides an image processing method, an image processing device, a computer-readable storage medium, and an electronic device, wherein the execution subject of the image processing method may be the image processing device provided in the embodiment of the present application, or integrate the image processing An electronic device of a device, wherein the image processing device may be implemented in a hardware or software manner. Wherein, the electronic device may be a device equipped with a processor and capable of data processing, such as a smart phone, a tablet computer, a palmtop computer, and a notebook computer.

The technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the accompanying drawings. Apparently, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in this application, all other embodiments obtained by those skilled in the art without making creative efforts belong to the scope of protection of this application.

This embodiment provides an image processing method, which is applied to an electronic device. The electronic device includes a lens, an image sensor, an optical anti-shake component, and an electronic anti-shake component. The image processing method includes:

Obtain the motion data of the electronic device, and according to the motion data, control the lens to perform compensation translation in the X-axis direction and Y-axis direction of the lens through the optical anti-shake component, and control the image sensor to perform compensation translation in the X-axis direction and Y-axis direction of the image sensor , and controlling the image sensor to perform compensation rotation around the Z axis of the image sensor;

Obtaining the optical image stabilization image corresponding to the motion data collected by the image sensor, and acquiring the first position and attitude data of the lens, the second position and attitude data and the first rotation attitude data of the image sensor from the optical anti-shake component;

Restoring and correcting the posture of the optical image stabilization image according to the first position and posture data, the second position and posture data and the first rotation posture data to obtain a corrected image;

In some embodiments, the posture of the optical image stabilization image is restored and corrected according to the first position and posture data, the second position and posture data, and the first rotation posture data to obtain a corrected image, including:

According to the first position and attitude data and the second position and attitude data, a first correction translation vector for performing translation correction on the optical image stabilization image is obtained;

Acquiring a first correction rotation matrix for performing rotation correction on the optical image stabilization image according to the first rotation attitude data;

According to the first correction translation vector and the first correction rotation matrix, translation correction and rotation correction are respectively performed on the pose of the optical image stabilization image to obtain a correction image.

In some embodiments, according to the first position and attitude data and the second position and attitude data, obtaining a first correction translation vector for translation correction of the optical image stabilization image includes:

Acquiring a first translation vector corresponding to the first position and attitude data according to the first functional relationship between the position and attitude data and the translation vector;

Acquiring a second translation vector corresponding to the second position and attitude data according to the second functional relationship between the position and attitude data and the translation vector;

According to the first translation vector, the first fusion parameter corresponding to the first translation vector, the second translation vector and its corresponding second fusion parameter, fused to obtain a fusion translation vector of the first translation vector and the second translation vector;

The fused translation vector is used as the first corrected translation vector.

In some embodiments, the electronic device is fixed to the preset calibration tool, so that the position of the electronic device and the rectangular calibration plate in the preset calibration tool is relatively fixed, the rectangular calibration plate includes a center mark, and the first functional relationship is pre-generated according to the following steps :

Turn on the lens translation anti-shake function of the optical image stabilization component, and turn off the image sensor translation anti-shake function and the image sensor rotation anti-shake function;

sending the first translation control command to the preset calibration tool, so that the preset calibration tool drives the electronic device to translate in the X-axis direction and the Y-axis direction of the electronic device;

During the translation process of the electronic device, optical anti-shake processing is performed through the optical anti-shake component, and the first image sequence of the rectangular calibration plate collected by the image sensor during the translation process is obtained;

Acquiring the third position and attitude data of the lens corresponding to each first image in the first image sequence from the optical anti-shake component, and acquiring the first position coordinates of the center mark in each first image;

According to the third position and posture data corresponding to each first image and the first position coordinates, the first functional relationship is obtained by fitting.

In some embodiments, the electronic device is fixed to the preset calibration tool, so that the position of the electronic device and the rectangular calibration plate in the preset calibration tool is relatively fixed, the rectangular calibration plate includes a center mark, and the second functional relationship is pre-generated according to the following steps :

Turn on the image sensor translation anti-shake function of the optical image stabilization component, and turn off the lens translation anti-shake function and the image sensor rotation anti-shake function;

sending the second translation control command to the preset calibration tool, so that the preset calibration tool drives the electronic device to translate in the X-axis direction and the Y-axis direction of the electronic device;

During the translation process of the electronic device, optical anti-shake processing is performed through the optical anti-shake component, and a second image sequence of the rectangular calibration plate collected by the image sensor during the translation process is acquired;

Obtaining the fourth position and posture data when the image sensor captures each second image in the second image sequence from the optical anti-shake component, and acquiring the second position coordinates of the center mark in each second image;

According to the fourth position and posture data corresponding to each second image and the second position coordinates, a second functional relationship is obtained by fitting.

In some embodiments, the electronic device is fixed to the preset calibration tool, so that the position of the electronic device and the rectangular calibration plate in the preset calibration tool is relatively fixed, the rectangular calibration plate includes a center mark, a first fusion parameter and a second fusion parameter Follow the steps below to pre-determine:

Turn on the lens pan anti-shake function and the image sensor pan anti-shake function of the optical image stabilization component, and turn off the image sensor rotation anti-shake function;

sending the third translation control command to the preset calibration tool, so that the preset calibration tool drives the electronic device to translate in the X-axis direction and the Y-axis direction of the electronic device;

During the translation process of the electronic device, optical anti-shake processing is performed through the optical anti-shake component, and a third image sequence of the rectangular calibration plate collected by the image sensor during the translation process is acquired;

Acquire the fifth position and attitude data of the lens corresponding to each third image in the third image sequence from the optical anti-shake component, and obtain the sixth position when the image sensor captures each third image in the third image sequence from the optical anti-shake component attitude data;

Obtaining a third translation vector corresponding to each fifth position and attitude data according to the first functional relationship, and obtaining a fourth translation vector corresponding to each sixth position and attitude data according to the second functional relationship;

Obtain the third position coordinates of the center mark in each third image, and determine the first fusion parameter and the second fusion parameter according to the third position coordinates, the third translation vector and the fourth translation vector corresponding to each third image .

In some embodiments, according to the first rotation pose data, obtaining a first correction rotation matrix for performing rotation correction on the optical image stabilization image includes:

According to the third functional relationship between the rotation attitude data and the rotation angle, the rotation angle corresponding to the first rotation attitude data is obtained;

Obtain the rotation center coordinates of the optical image stabilization image, and obtain a first correction rotation matrix according to the rotation center coordinates and the rotation angle.

In some embodiments, obtaining the first corrected rotation matrix according to the coordinates of the rotation center and the rotation angle includes:

According to the coordinates of the rotation center and the rotation angle, the first correction rotation matrix is obtained according to the following formula:

Wherein, R represents the first correction rotation matrix, C represents cos(γ), γ represents the rotation angle, S represents sin(γ), Cx represents the abscissa in the rotation center coordinates, and Cy represents the ordinate in the rotation center coordinates.

In some embodiments, the electronic device is fixed to the preset calibration tool, so that the position of the electronic device and the rectangular calibration plate in the preset calibration tool is relatively fixed, the rectangular calibration plate includes four corner marks, and the coordinates of the rotation center and the third functional relationship are according to Generated as follows:

Turn on the image sensor rotation anti-shake function of the optical image stabilization component, and turn off the lens translation anti-shake function and the image sensor translation anti-shake function;

sending a rotation control command to the preset calibration tool, so that the preset calibration tool drives the electronic device to rotate around the Z axis of the image sensor;

During the rotation process of the electronic device, optical anti-shake processing is performed through the optical anti-shake component, and a fourth image sequence of the rectangular calibration plate collected by the image sensor during the rotation process is acquired;

Obtaining the second rotation attitude data when the image sensor captures each fourth image in the fourth image sequence from the optical anti-shake component, and obtaining the fourth position coordinates of the four corner marks in each fourth image;

According to the second rotation attitude data and the fourth position coordinates corresponding to each fourth image, the coordinates of the rotation center and the third functional relationship are obtained by fitting.

In some embodiments, the third functional relationship includes a polynomial function, and according to the second rotation attitude data and the fourth position coordinates corresponding to each fourth image, the coordinates of the rotation center and the third functional relationship are obtained by fitting, including:

According to the second rotation attitude data and the fourth position coordinates corresponding to each fourth image, the relationship between the rotation center coordinates and the third function is obtained by fitting according to the following formula:

θ = [Cx, Cy, α];

Among them, P _i represents the fourth position coordinates of the i-th four-corner mark, h ₂ ' represents the second rotation attitude data, f() represents a polynomial function, r represents a preset search radius, α represents the polynomial coefficient of the polynomial function, and imW represents The width of the fourth image, imH represents the height of the fourth image.

In some embodiments, also include:

Turn on the lens pan anti-shake function, the image sensor pan anti-shake function and the image sensor rotation anti-shake function of the optical image stabilization component;

sending a rotation and translation control command to the preset calibration tool, so that the preset calibration tool drives the electronic device to rotate around the Z-axis of the image sensor, and drives the electronic device to translate in the X-axis direction and the Y-axis direction of the electronic device;

During the rotation and translation process of the electronic device, optical anti-shake processing is performed through the optical anti-shake component, and the fifth image sequence of the rectangular calibration plate collected by the image sensor during the rotation and translation process is acquired;

Obtaining the seventh position and attitude data of the lens corresponding to each fifth image in the fifth image sequence, the eighth position and attitude data and the third rotation attitude data when the image sensor collects each fifth image from the optical anti-shake component;

According to the seventh position and attitude data, the eighth position and attitude data and the first functional relationship and the second functional relationship corresponding to each fifth image, a second correction translation vector for translation correction of each fifth image is obtained;

Obtaining a second correction rotation matrix for performing rotation correction on each fifth image according to the third rotation posture data corresponding to each fifth image, the third functional relationship, and the coordinates of the rotation center;

respectively performing translation correction and rotation correction on the center pixel of each fifth image according to the second correction translation vector and the second correction rotation matrix, to obtain the correction coordinates of each center pixel;

Acquiring the fifth position coordinates of the center mark in each fifth image, and according to the difference between the correction coordinates corresponding to each fifth image and the fifth position coordinates, the first functional relationship, the second functional relationship and the third functional relationship Make joint corrections.

In some embodiments, acquiring an optical image stabilization image corresponding to motion data collected by an image sensor includes:

Correcting the first acquired time stamp of the motion data according to the first time delay parameter corresponding to the motion data to obtain the first corrected time stamp;

According to the first corrected time stamp, the optical image stabilization image corresponding to the motion data collected by the image sensor is acquired.

In some embodiments, the first delay parameter is pre-generated according to the following steps:

Get a sequence of sample dithered images;

Acquiring sample motion data corresponding to each sample shake image according to the second acquisition time stamp of each sample shake image in the sample shake image sequence to obtain a sample motion data sequence;

According to the preset time delay parameter, the preset search step size, the sample shaking image sequence and the corresponding sample motion data sequence, the first time delay parameter is obtained by searching.

In some embodiments, according to the preset time delay parameter, the preset search step size, the sample shaking image sequence and its corresponding sample motion data sequence, the first time delay parameter is obtained by searching, including

Acquiring a preset number of candidate delay parameters according to a preset delay parameter and a preset search step;

Correcting the sample motion data sequence according to each candidate delay parameter to obtain a preset number of corrected sample motion data sequences;

According to each corrected sample motion data sequence, electronic anti-shake processing is performed on the sample shake image sequence through the electronic anti-shake component to obtain a preset number of anti-shake image sequences;

Quantify and score the anti-shake quality of each anti-shake image sequence to obtain the anti-shake quality score of each anti-shake image sequence;

Updating the preset delay parameter to the candidate delay parameter corresponding to the anti-shake image sequence with the highest anti-shake quality score, and updating the preset search step according to the preset update parameter;

The above steps are repeated until the preset search condition is met, and the preset delay parameter when the preset search condition is met is used as the first delay parameter.

In some embodiments, the preset search conditions include:

The stabilization quality score of any stabilized image sequence is greater than or equal to the scoring threshold; or

The updated preset search step size is smaller than the step size threshold.

In some embodiments, acquiring the first position and posture data of the lens, the second position and posture data and the first rotation posture data of the image sensor from the optical anti-shake component includes:

Correcting the first modified time stamp according to the second delay parameter corresponding to the optical anti-shake component to obtain a second modified time stamp;

According to the second modified time stamp, the first position and posture data of the lens, the second position and posture data and the first rotation posture data of the image sensor are acquired from the optical anti-shake component.

In some embodiments, the motion data includes acceleration data and angular velocity data.

Please refer to FIG. 1 , which shows a schematic diagram of a hardware structure of an electronic device 100 . As shown in FIG. 1 , the electronic device 100 includes a motion sensor 110 , a camera module 120 , an optical anti-shake component 130 and an electronic anti-shake component 140 . Here, there is no specific limitation on the installation positions of the motion sensor 110, the camera module 120, the optical anti-shake component 130 and the electronic anti-shake component 140 in the electronic device, which can be set by those skilled in the art according to actual needs. In addition, those skilled in the art can understand that the structure shown in FIG. 1 does not constitute a limitation on the electronic device 100, and the electronic device 100 may include more or less components than those shown in the illustration, or combine certain components, Or a different arrangement of components, etc.

Wherein, the motion sensor 110 is configured to sense the motion state of the electronic device 100 in real time, and correspondingly obtain motion data used to describe the motion state of the electronic device 100 . The type and quantity of the motion sensors 110 are not specifically limited here, and can be set by those skilled in the art according to actual needs. Exemplarily, the motion sensor 110 can only sense the motion state of the electronic device 100 in a single dimension, and correspondingly obtain motion data in a single dimension (for example, the motion sensor 110 can only sense the acceleration of the electronic device 100), or can sense the electronic device 100 In the multi-dimensional motion state, correspondingly obtain multi-dimensional motion data (for example, the motion sensor 110 can sense the acceleration and angular velocity of the electronic device 100 ). It should be noted that the motion sensor 110 can be arranged at any position of the electronic device 100 as constrained by the rigid connection between the motion sensor 110 and the body of the electronic device 100 .

The camera module 120 is configured to collect images, including at least a lens 1210 and an image sensor 1220, wherein the lens 1210 is used to project external light signals to the image sensor 1220, and the image sensor 1220 is used to perform photoelectric conversion on the light signals projected by the lens 1210 , convert the optical signal into a usable electrical signal, and obtain a digitized image. Here, there is no specific limitation on how the lens 1210 and the image sensor 1220 are arranged. For example, the lens 1210 and the image sensor 1220 may be arranged in parallel or non-parallel.

After the camera module 120 is enabled, it can shoot the shooting scene in real time. The shooting scene can be understood as the real area where the camera module 120 is aimed after being enabled, that is, the area where the camera module 120 can convert light signals into corresponding images. For example, after the electronic device 100 enables the camera module 120 according to the user operation, if the user controls the camera module 120 of the electronic device 100 to aim at an area including a certain object, then the area including the object is the area of the camera module 120. Shoot the scene. Wherein, the camera module 120 is configured to be movable relative to the electronic device 100, that is, the camera module 120 has a certain degree of freedom of movement relative to the electronic device 100 (the lens 1210 and the image sensor 1220 both have a degree of freedom of movement, or One of the lens 1210 and the image sensor 1220 has a degree of freedom of movement), so that when the electronic device 100 moves, the camera module 120 can be driven to perform compensatory movement, so as to offset the movement of the electronic device 100 as much as possible, so that the imaging The light path is stable.

It should be noted that this embodiment does not specifically limit the installation position of the camera module 120 in the electronic device 100 , and can be configured by those skilled in the art according to actual needs. Taking a mobile phone as an example, the camera module 120 can be installed on the side where the screen of the mobile phone is located, or the camera module 120 can be installed on the opposite side of the screen of the mobile phone, and the camera module 120 can also be installed on both the side where the screen of the mobile phone is located and the opposite surface. .

The optical anti-shake component 130 is configured to perform optical anti-shake processing on the camera module 120 according to the movement of the electronic device. For example, when the lens 1210 has a translation degree of freedom in its X-axis direction and Y-axis direction, the optical anti-shake component can control the lens 1210 to perform compensation translation in its X-axis direction and Y-axis direction according to the movement of the electronic device 100 . To counteract the movement of the electronic device 100, so that the imaging optical path of the camera module 120 is stabilized.

The electronic anti-shake component 140 is configured to stabilize the image through electronic anti-shake. Electronic anti-shake is an algorithm operation, such as calculating the motion between the image and other images through the motion data corresponding to the image. And the movement inside the image, and after aligning the image according to the movement and performing appropriate cropping, stretching and deformation processing, a relatively stable image is obtained.

Please refer to FIG. 2 , which is a schematic flowchart of an image processing method provided by an embodiment of the present application. The image processing method is applied to the electronic device 100 shown in FIG. 1. The electronic device 100 includes a lens 1210, an image sensor 1220, an optical anti-shake component 130, and an electronic anti-shake component 140. As shown in FIG. 2, the embodiment of the present application provides The flow of the image processing method can be as follows:

In 210, the motion data of the electronic device 100 is obtained, and according to the motion data, the optical image stabilization component 130 controls the lens 1210 to perform compensation translation in the X-axis direction and the Y-axis direction of the lens 1210, and controls the image sensor 1220 to move in the direction of the image sensor 1220. The compensation translation is performed in the X-axis direction and the Y-axis direction, and the compensation rotation of the image sensor 1220 is controlled around the Z-axis of the image sensor 1220 .

Please refer to FIG. 3 , in this embodiment, the camera module 120 has 5-dimensional freedom of movement, including the 2-dimensional translation freedom of the lens 1210 in its X-axis direction and Y-axis direction, and the image sensor 1220 in its X-axis direction and Y-axis direction. A 2-dimensional translational degree of freedom in the Y-axis direction, and a 1-dimensional rotational degree of freedom for the image sensor 1220 in its Z-axis. Correspondingly, the optical anti-shake component 130 is configured as an optical anti-shake in the aforementioned 5-dimensional freedom of movement. It should be noted that in this embodiment, the structural layout of the electronic device 100, the lens 1210 and the image sensor 1220 is not limited, and can be configured by those skilled in the art according to actual needs, for example, the electronic device 100, the lens 1210 and the The X axes of the three image sensors 1220 are parallel to each other, the Y axes are parallel to each other, and the Z axes are parallel to each other.

Based on the above optical anti-shake component 130 , in this embodiment, firstly, motion data of the electronic device 100 is acquired through the motion sensor 110 provided on the electronic device 100 . The type and quantity of the motion sensors 110 are not specifically limited here, and can be set by those skilled in the art according to actual needs. For example, the three-axis acceleration of the electronic device 100 may be obtained through a three-axis acceleration sensor, and the three-axis angular velocity of the electronic device 100 may also be obtained through a three-axis gyroscope.

As above, after the motion data of the electronic device 100 is acquired, the motion data of the electronic device 100 is input to the optical image stabilization component 130, and the optical image stabilization component 130 drives the lens 1210 and the image sensor 1220 according to the motion data of the electronic device 100 to perform optical image stabilization. shake. Among them, according to the motion data of the electronic device 100, the optical anti-shake component 130 controls the lens 1210 to perform compensation translation in the X-axis direction and the Y-axis direction of the lens 1210, and controls the image sensor 1220 to perform compensation translation in the X-axis direction and the Y-axis direction of the image sensor 1220. The translation is compensated, and the image sensor 1220 is controlled to perform compensation rotation around the Z-axis of the image sensor 1220 , so as to offset the movement of the electronic device 100 and stabilize the imaging optical path.

It should be noted that, according to the actually acquired motion data of the electronic device 100, the lens 1210 can be controlled by the optical anti-shake component 130 to produce actual displacement in its X-axis direction and Y-axis direction at the same time (for example, the lens 1210 in its X-axis direction and Y-axis direction produce the same magnitude of displacement, then the displacement produced by the lens 1210 can be regarded as a displacement along the 45-degree direction), and can also be controlled by the optical anti-shake component 130 to only occur in one of the X-axis direction and the Y-axis direction Actual displacement (such as lens 1210 only produces actual displacement in the X-axis direction, or only produces actual displacement in its Y-axis direction), similarly, the image sensor 1220 can be controlled by the optical anti-shake component 130 while simultaneously in its X-axis direction and Y-axis direction Axis direction produces actual displacement, also can be controlled by optical anti-shake assembly 130 and only produce actual displacement in one of its X-axis direction and Y-axis direction, in addition, image sensor 1220 is controlled by optical anti-shake assembly 130 and rotates around its Z-axis The angle of can be zero (ie, the image sensor 1220 is not actually rotated around its Z-axis), or non-zero (ie, the image sensor 1220 is actually rotated around its Z-axis).

In 220, obtain the optical image stabilization image corresponding to the motion data collected by the image sensor 1220, and obtain the first position and posture data of the lens 1210, the second position and posture data and the first rotation posture of the image sensor 1220 from the optical stabilization component 130 data.

According to the above related descriptions, it can be seen that the present application controls the lens 1210 and the image sensor 1220 through the optical anti-shake component 130 to perform optical anti-shake, so that the imaging optical path of the image sensor 1220 is stabilized. 1220 When the translation compensation and rotation compensation are completed, the image captured by the image sensor 1220 at this time is obtained, and recorded as the optical image stabilization image corresponding to the aforementioned motion data.

It should be noted that the optical anti-shake component 130 is also provided with a first position and attitude sensor for sensing the position and attitude of the lens 1210 in real time, a second position and attitude sensor for sensing the position and attitude of the image sensor 1220 in real time, and a second position and attitude sensor for real-time sensing of the position and attitude of the image sensor 1220. A rotational attitude sensor that senses the rotational attitude of the image sensor 1220 . The types of the first position and attitude sensor, the second position and attitude sensor and the rotation attitude sensor are not specifically limited here, and can be selected by those skilled in the art according to needs. For example, in this embodiment, the first position and attitude sensor, the second position and attitude sensor, and the rotation and attitude sensor are all realized by Hall sensors.

In this embodiment, in addition to obtaining the optical image stabilization image corresponding to the motion data collected by the image sensor 1220, the first position and posture data of the lens 1210 and the second position and posture data of the image sensor 1220 and The first rotation pose data. For example, the optical anti-shake component 130 senses the position and posture of the lens 1210 through the first position and posture sensor to obtain the first position and posture data of the lens 1210, and senses the position and posture of the image sensor 1220 through the second position and posture sensor to obtain the image The second position and posture data of the sensor 1220 is sensed by the rotation posture sensor to the rotation posture of the image sensor 1220 to obtain the first rotation posture data of the image sensor 1220, and the first position and posture data, the second position and posture data and the first The combination of rotation attitude data is output as attitude data set. Correspondingly, the aforementioned first position and attitude data, the aforementioned second position and attitude data, and the aforementioned first rotation attitude data can be obtained from the attitude data set output by the optical anti-shake component 130 .

Exemplarily, the posture data set can be expressed as H=[h ₀ |h ₁ |h ₂ ], wherein h ₀ represents the first position and posture data, h ₁ represents the second position and posture data, and h ₂ represents the first rotation posture data.

In 230, restore and correct the posture of the optical image stabilization image according to the first position and posture data, the second position and posture data, and the first rotation posture data to obtain a corrected image.

In this embodiment, after the first position and attitude data, the second position and attitude data, and the first rotation attitude data are acquired, according to the configured correction strategy, according to the first position and attitude data, the second position and attitude data and the first rotation attitude data, Restoring and correcting the attitude of the optical image stabilization image based on the rotation attitude data to obtain the corrected image. Here, the restoration and correction of posture can be commonly understood as "reverting" the optical image stabilization image to the posture that the optical image stabilization component 130 does not perform optical stabilization, that is, "reverting" the optical image stabilization image to the point where the lens 1210 does not move, and The corresponding attitude of the image sensor 1220 when it does not move and rotate.

In 240, according to the motion data, the electronic anti-shake component 140 performs electronic anti-shake processing on the corrected image to obtain an electronic anti-shake image.

In this embodiment, after the correction of the optical image stabilization image is completed and the corrected image is obtained, the aforementioned motion data and the corrected image are further input into the electronic anti-shake component 140, and the electronic anti-shake component 140 processes the corrected image according to the aforementioned motion data. Electronic anti-shake processing to obtain electronic anti-shake images.

As can be seen from the above, in this application, the electronic device 100 includes a lens 1210, an image sensor 1220, an optical anti-shake component 130, and an electronic anti-shake component 140. The image sensor 1220 performs five-dimensional optical image stabilization to obtain an optical image stabilization image; then, based on the optical image stabilization image, according to the first position and posture data of the lens 1210, the second position and posture data of the image sensor 1220 and The first rotation attitude data is to restore and correct the attitude of the optical image stabilization image to obtain the corrected image after the attitude restoration; finally, according to the motion data, the electronic anti-shake component 140 is used to perform electronic anti-shake processing on the corrected image to obtain the electronic anti-shake image. In this way, the present application uses the optical image stabilization component 130 to provide an optical image stabilization image, and after restoring the posture change caused by the optical image stabilization component, the electronic image stabilization component 140 performs electronic image stabilization processing to obtain a high-quality electronic image stabilization image.

Optionally, in an embodiment, the posture of the optical image stabilization image is restored and corrected according to the first position and posture data, the second position and posture data, and the first rotation posture data to obtain a corrected image, including:

(1) According to the first position and attitude data and the second position and attitude data, obtain a first correction translation vector for performing translation correction to the optical image stabilization image;

(2) Acquiring a first correction rotation matrix for performing rotation correction on the optical image stabilization image according to the first rotation attitude data;

(3) Perform translation correction and rotation correction on the optical image stabilization image according to the first correction translation vector and the first correction rotation matrix, respectively, to obtain a correction image.

According to the relevant description above, it can be seen that the optical image stabilization in this embodiment includes 5-dimensional degrees of freedom, which are respectively 2-dimensional degrees of freedom of the lens 1210 on the X-axis and Y-axis, and 2-dimensional degrees of freedom of the image sensor 1220 on the X-axis, Y-axis and Z-axis. 3D degrees of freedom. It can be understood that the anti-shake compensation of the aforementioned 5-dimensional freedom will lead to the translation and rotation of the image content. Correspondingly, the restorative correction of the posture of the optical anti-shake image in this embodiment will include two parts, respectively translation correction and rotation correction .

It can be understood that the translation of the image content is caused by the respective translations of the lens 1210 and the image sensor 1220 , while the rotation of the image content is caused by the rotation of the image sensor 1220 . Therefore, in this embodiment, on the one hand, according to the first position and posture data of the lens 1210 and the second position and posture data of the image sensor 1220, the first correction translation vector used for translation correction of the optical image stabilization image is obtained; on the other hand, according to the image The first rotation posture data of the sensor 1220 is used to obtain a first correction rotation matrix for performing rotation correction on the optical image stabilization image. Afterwards, according to the acquired first correction translation vector and first correction rotation matrix, translation correction and rotation correction are respectively performed on the posture of the optical image stabilization image to obtain a correction image, as shown in FIG. 4 . Wherein, according to the acquired first correction translation vector and first correction rotation matrix, respectively perform translation correction and rotation correction on the attitude of the optical image stabilization image, which can be expressed as:

P'=R·P+t;

Wherein, P represents the position of a pixel in the OIS image, R represents the first corrected rotation matrix, t represents the first corrected translation vector, and P' represents the position of the pixel in the corrected image after correction.

Optionally, in an embodiment, according to the first position and attitude data and the second position and attitude data, obtaining a first correction translation vector for translation correction of the optical image stabilization image includes:

(1) According to the first functional relationship between the position and attitude data and the translation vector, obtain the first translation vector corresponding to the first position and attitude data;

(2) According to the second functional relationship between the position and attitude data and the translation vector, obtain the second translation vector corresponding to the second position and attitude data;

(3) According to the first translation vector, the first fusion parameter corresponding to the first translation vector, the second translation vector and its corresponding second fusion parameter, fused to obtain a fusion translation vector of the first translation vector and the second translation vector;

(4) Take the fused translation vector as the first corrected translation vector.

It should be noted that this embodiment presets a first functional relationship between the position and attitude data and the translation vector, and the first functional relationship describes the corresponding relationship between the position and attitude changes of the image content caused by the changes in the position and attitude of the lens 1210 . In addition, this embodiment also presets a second functional relationship between the position and attitude data and the translation vector, and the second functional relationship describes the corresponding relationship between the position and attitude changes of the image content caused by the position and attitude changes of the image sensor 1220 .

Based on the first functional relationship and the second functional relationship above, in this embodiment, when obtaining the first correction translation vector for translation correction of the optical image stabilization image according to the first position and attitude data and the second position and attitude data, according to the position and attitude According to the first functional relationship between the data and the translation vector, the first translation vector corresponding to the first position and attitude data is obtained, and the second translation vector corresponding to the second position and attitude data is obtained according to the second functional relationship between the position and attitude data and the translation vector.

As mentioned above, the rotation of the image content is caused by the respective translations of the lens 1210 and the image sensor 1220 , and correspondingly, the aforementioned first translation vector and the second translation vector need to be fused. To this end, in this embodiment, a first fusion parameter corresponding to the first translation vector and a second fusion parameter corresponding to the second translation vector are pre-configured. Correspondingly, in this embodiment, after the first translation vector corresponding to the first position and attitude data and the second translation vector corresponding to the second position and attitude data are acquired, that is, according to the first translation vector and the first translation vector corresponding to the first translation vector, A fusion parameter, a second translation vector and its corresponding second fusion parameter are fused to obtain a fusion translation vector of the first translation vector and the second translation vector, expressed as:

t'=a·t ₁ +b·t ₂ ;

Among them, t' denotes the fused translation vector, t ₁ denotes the first translation vector, t ₂ denotes the second translation vector, a denotes the first fusion parameter, and b denotes the second fusion parameter.

It should be noted that the sum of the first fusion parameter and the second fusion parameter is constrained to be 1, and those skilled in the art can select the value according to actual needs.

As above, after the fused translation vector of the first translation vector and the second translation vector is obtained through fusion, this embodiment will use the fused translation vector as the first corrected translation vector.

Optionally, in an embodiment, the electronic device 100 is fixed to the preset calibration tool, so that the positions of the electronic device 100 and the rectangular calibration plate in the preset calibration tool are relatively fixed, the rectangular calibration plate includes a center mark, and the first function Relationships are pre-generated as follows:

(1) Turn on the translation anti-shake function of the lens 1210 of the optical anti-shake component 130, and turn off the translation anti-shake function of the image sensor 1220 and the rotation anti-shake function of the image sensor 1220;

(2) sending the first translation control command to the preset calibration tool, so that the preset calibration tool drives the electronic device 100 to translate in the X-axis direction and the Y-axis direction of the electronic device 100;

(3) During the translation process of the electronic device 100, perform optical anti-shake processing through the optical anti-shake component 130, and acquire the first image sequence of the rectangular calibration plate collected by the image sensor 1220 during the translation process;

(4) Obtain the third position and attitude data of the lens 1210 corresponding to each first image in the first image sequence from the optical anti-shake component 130, and acquire the first position coordinates of the center mark in each first image;

(5) According to the third position and posture data corresponding to each first image and the first position coordinates, the first functional relationship is obtained by fitting.

In this embodiment, the first functional relationship is obtained through pre-calibration by using a preset calibration tool.

Wherein, firstly, the electronic device 100 is fixed on the preset calibration tool, so that the positions of the electronic device 100 and the rectangular calibration plate in the preset calibration tool are relatively fixed. It should be noted that there is no specific limitation on the structure of the preset calibration tool here, and it can be set by those skilled in the art according to actual needs.

Exemplarily, referring to FIG. 5, the preset calibration tool is composed of four parts: a target backplane, an equipment carrier, a slide rail and a driving mechanism (not shown in FIG. 5), wherein the target backplane is used to fix a rectangular calibration plate , the device carrier is used to fix the electronic device 100, the target backplane and the device carrier can slide along the tracks of their respective slide rails, and the driving mechanism is used to drive the target backplane, the device carrier, and the slide rail to move synchronously.

Based on the preset calibration tool shown in FIG. 5 , in this embodiment, the electronic device 100 is fixed on the device carrier, and the positions of the device carrier and the target backplane are adjusted along their respective tracks, so that the rectangular calibration plate is positioned on the image sensor 1220 The imaging size of is consistent with the size of the imaging area of the image sensor 1220. In layman's terms, the imaging image of the rectangular calibration plate covers the imaging area of the image sensor 1220.

After the position adjustment of the device carrier and the target backboard is completed, the device carrier and the target backboard are locked in their respective current positions, so that the positions of the electronic device 100 and the rectangular calibration plate are relatively fixed.

It should be noted that in order to ensure that the positions of the electronic device 100 and the rectangular calibration plate are relatively fixed, thereby ensuring the parallel relationship between the rectangular calibration plate and the camera module 120 (lens 1210 and image sensor 1220) of the electronic device 100, the target backplane Use a rigid sheet, such as a rigid metal sheet. In addition, it is also necessary to ensure that the equipment carrier and the target backplane are rigidly connected to the slide rails where they are located, so as to avoid disturbance errors.

As above, after the electronic device 100 is fixed on the preset calibration tool, and the positions of the electronic device 100 and the rectangular calibration plate in the preset calibration tool are relatively fixed, the first functional relationship can be calibrated by using the preset calibration tool.

Since only the first functional relationship is calibrated at this time, the translation anti-shake function of the lens 1210 of the optical anti-shake assembly 130 is correspondingly turned on, and the translation anti-shake function of the image sensor 1220 and the rotation anti-shake function of the image sensor 1220 are turned off, so that the optical anti-shake When the electronic device 100 is in motion, the anti-shake component 130 only compensates for the translation of the lens 1210 according to the motion of the electronic device 100 . Correspondingly, the first translation control command is sent to the preset calibration tool, so that the preset calibration tool drives the electronic device 100 to translate in the X-axis direction and the Y-axis direction of the electronic device 100 , as shown in FIG. 5 . For example, the first translation control instruction enables the preset calibration tool to drive the electronic device 100 to alternately translate in the X-axis direction and the Y-axis direction, wherein the distance of translation and the frequency of the alternation can be determined by those skilled in the art according to actual needs. There are no specific restrictions.

During the translation process of the electronic device 100 , the motion data of the electronic device 100 is acquired in real time and provided to the optical anti-shake component 130 , so that the optical anti-shake process is performed by the optical anti-shake component 130 . It can be understood that, since this embodiment only enables the lens 1210 translation anti-shake function of the optical anti-shake assembly 130 , the optical anti-shake assembly 130 will only compensate for the translation of the lens 1210 during the movement of the electronic device 100 . In addition, the image sequence of the rectangular calibration plate captured by the image sensor 1220 during the translation process is acquired, which is denoted as the first image sequence. For example, the image sensor 1220 collects a one-minute image sequence of the rectangular calibration plate during the translation process to obtain the first image sequence.

Please refer to FIG. 6 , the rectangular calibration plate includes a center mark and four corner marks (respectively upper left corner mark, right upper corner mark, right lower corner mark and left lower corner mark), due to the position of the rectangular calibration plate in the electronic device 100 and the preset calibration tool Relatively fixed, the optical anti-shake processing performed by the optical anti-shake component 130 due to the translation of the electronic device 100 will cause the position of the center mark in the first image obtained by imaging to deviate from the center of the first image.

Correspondingly, in this embodiment, the position and posture data of the lens 1210 corresponding to each first image in the first image sequence is further obtained from the optical anti-shake component 130, that is, the position of the lens 1210 when the image sensor 1220 captures each first image The attitude data is recorded as the third position attitude data. In addition, the position coordinates of the center mark in each first image are also obtained, which are recorded as the first position coordinates.

It can be understood that, for a first image, the first position coordinates of the aforementioned center mark in the first image are equal to the translation vector from the center of the first image to the center mark. Therefore, the first functional relationship can be obtained by performing function fitting according to the third position posture data corresponding to each first image and the first position coordinates. There is no specific limitation on the method of function fitting here, including but not limited to least square method, random sampling consistent method, and polynomial fitting method. For example, in this embodiment, the least square method is used to fit to obtain the first functional relationship.

In other embodiments, after the first functional relationship is obtained by fitting, the fitting error of the first functional relationship is also evaluated (the evaluation method can be selected by those skilled in the art according to actual needs, and no specific limitation is made here, for example, it can be used mean square error) to obtain the fitting error of the first functional relationship, if the fitting error of the first functional relationship is greater than the first error threshold (experienced values can be obtained by those skilled in the art according to actual needs), then identify and eliminate obviously abnormal After the third position attitude data and the first position coordinates are fitted again, if the fitting error of the newly fitted first functional relationship is still greater than the first error threshold, a new first image sequence and a new third image sequence are reacquired. The position and attitude data are fitted until a first functional relationship with a fitting error less than or equal to a first error threshold is obtained.

Optionally, in an embodiment, the second functional relationship is pre-generated according to the following steps:

(1) Turn on the translational anti-shake function of the image sensor 1220 of the optical anti-shake assembly 130, and turn off the translational anti-shake function of the lens 1210 and the rotation anti-shake function of the image sensor 1220;

(2) sending the second translation control command to the preset calibration tool, so that the preset calibration tool drives the electronic device 100 to translate in the X-axis direction and the Y-axis direction of the electronic device 100;

(3) During the translation process of the electronic device 100, perform optical anti-shake processing through the optical anti-shake component 130, and acquire a second image sequence of the rectangular calibration plate collected by the image sensor 1220 during the translation process;

(4) Obtain the fourth position and posture data when the image sensor 1220 acquires each second image in the second image sequence from the optical anti-shake component 130, and acquire the second position coordinates of the center mark in each second image;

(5) According to the fourth position and posture data corresponding to each second image and the second position coordinates, a second functional relationship is obtained by fitting.

In this embodiment, the second functional relationship is obtained by pre-calibrating with a preset calibration tool. Similarly, firstly, the electronic device 100 is fixed to the preset calibration tool according to the above method, so that the positions of the electronic device 100 and the rectangular calibration plate in the preset calibration tool are relatively fixed. For details, reference may be made to relevant descriptions in the foregoing embodiments, and details are not repeated here.

After the electronic device 100 is fixed on the preset calibration tool, and the positions of the electronic device 100 and the rectangular calibration plate in the preset calibration tool are relatively fixed, the second functional relationship can be calibrated by using the preset calibration tool.

Since only the second functional relationship is calibrated at this time, the image sensor 1220 translation anti-shake function of the optical image stabilization component 130 is correspondingly turned on, and the lens 1210 translation anti-shake function and the image sensor 1220 rotation anti-shake function are turned off, so that the optical image stabilization When the electronic device 100 is in motion, the shaking component 130 only compensates for the translation of the image sensor 1220 according to the motion of the electronic device 100 . Correspondingly, the second translation control command is sent to the preset calibration tool, so that the preset calibration tool drives the electronic device 100 to translate in the X-axis direction and the Y-axis direction of the electronic device 100 , as shown in FIG. 5 . For example, the second translation control instruction enables the preset calibration tool to drive the electronic device 100 to alternately translate in the X-axis direction and the Y-axis direction, wherein the distance of translation and the frequency of the alternation can be determined by those skilled in the art according to actual needs. There are no specific restrictions.

During the translation process of the electronic device 100 , the motion data of the electronic device 100 is acquired in real time and provided to the optical anti-shake component 130 , so that the optical anti-shake process is performed by the optical anti-shake component 130 . It can be understood that, since this embodiment only enables the translational anti-shake function of the image sensor 1220 of the optical anti-shake component 130 , the optical anti-shake component 130 will only compensate for the translation of the image sensor 1220 during the movement of the electronic device 100 . In addition, an image sequence of the rectangular calibration plate captured by the image sensor 1220 during the translation process is acquired, which is denoted as a second image sequence. For example, the image sensor 1220 collects a one-minute image sequence of the rectangular calibration plate during the translation process to obtain the second image sequence.

In this embodiment, the image sensor 1220 also acquires the position and attitude data of each second image in the second image sequence from the optical anti-shake component 130, which is recorded as the fourth position and attitude data. In addition, the position coordinates of the center mark in each second image are also obtained, which are recorded as the second position coordinates.

It can be understood that, for a second image, the second position coordinates of the aforementioned center mark in the second image are equal to the translation vector from the center of the second image to the center mark. Therefore, the second functional relationship can be obtained by performing function fitting according to the fourth position and posture data corresponding to each second image and the second position coordinates. There is no specific limitation on the method of function fitting here, including but not limited to least square method, random sampling consistent method, and polynomial fitting method. For example, in this embodiment, the least square method is used to fit to obtain the second functional relationship.

In other embodiments, after the second functional relationship is obtained by fitting, the fitting error of the second functional relationship is also evaluated (the evaluation method can be selected by those skilled in the art according to actual needs, and no specific limitation is made here, for example, mean square error) to obtain the fitting error of the second functional relationship, if the fitting error of the second functional relationship is greater than the second error threshold (experienced values can be obtained by those skilled in the art according to actual needs), then identify and remove obviously abnormal After the fourth position attitude data and the second position coordinates are fitted again, if the fitting error of the newly fitted second functional relationship is still greater than the second error threshold, a new second image sequence and a new fourth image sequence are reacquired. The position and attitude data are fitted with the second position coordinates until a second functional relationship is obtained in which the fitting error is less than or equal to the second error threshold.

Optionally, in an embodiment, the first fusion parameter and the second fusion parameter are predetermined according to the following steps:

(1) Turn on the lens 1210 translation anti-shake function and the image sensor 1220 translation anti-shake function of the optical anti-shake component 130, and turn off the image sensor 1220 rotation anti-shake function;

(2) sending the third translation control command to the preset calibration tool, so that the preset calibration tool drives the electronic device 100 to translate in the X-axis direction and the Y-axis direction of the electronic device 100;

(3) During the translation process of the electronic device 100, perform optical anti-shake processing through the optical anti-shake component 130, and acquire a third image sequence of the rectangular calibration plate collected by the image sensor 1220 during the translation process;

(4) Acquire from the optical anti-shake component 130 the fifth position and attitude data of the lens 1210 corresponding to each third image in the third image sequence, and acquire the image sensor 1220 from the optical anti-shake component 130 to collect each of the third image sequences in the third image sequence The sixth position and posture data of three images;

(5) Acquiring a third translation vector corresponding to each fifth position and attitude data according to the first functional relationship, and obtaining a fourth translation vector corresponding to each sixth position and attitude data according to the second functional relationship;

(6) Obtain the third position coordinates of the center mark in each third image, and obtain the first fusion parameter and the first fusion parameter according to the third position coordinates, the third translation vector and the fourth translation vector corresponding to each third image Two fusion parameters.

In this embodiment, the first fusion parameter and the second fusion parameter for fusing the first translation vector and the second translation vector are obtained through pre-calibration by using a preset calibration tool. Similarly, firstly, the electronic device 100 is fixed to the preset calibration tool according to the above method, so that the positions of the electronic device 100 and the rectangular calibration plate in the preset calibration tool are relatively fixed. For details, reference may be made to relevant descriptions in the foregoing embodiments, and details are not repeated here.

After the electronic device 100 is fixed on the preset calibration tool, and the positions of the electronic device 100 and the rectangular calibration plate in the preset calibration tool are relatively fixed, the first fusion parameter and the second fusion parameter can be obtained by calibration with the preset calibration tool. parameter.

Since at this time, for the first fusion parameter and the second fusion parameter used to fuse the first translation vector and the second translation vector, the translation anti-shake function of the lens 1210 and the translation anti-shake function of the image sensor 1220 of the optical anti-shake component 130 are correspondingly turned on, And turn off the rotation anti-shake function of the image sensor 1220 , so that when the electronic device 100 moves, the optical anti-shake component 130 compensates and shifts the lens 1210 and the image sensor 1220 simultaneously according to the movement of the electronic device 100 . Correspondingly, the third translation control command is sent to the preset calibration tool, so that the preset calibration tool drives the electronic device 100 to translate in the X-axis direction and the Y-axis direction of the electronic device 100 , as shown in FIG. 5 . For example, the third translation control command enables the preset calibration tool to drive the electronic device 100 to alternately translate in the X-axis direction and the Y-axis direction, wherein the distance of translation and the frequency of the alternation can be determined by those skilled in the art according to actual needs. There are no specific restrictions.

During the translation process of the electronic device 100 , the motion data of the electronic device 100 is acquired in real time and provided to the optical anti-shake component 130 , so that the optical anti-shake process is performed by the optical anti-shake component 130 . It can be understood that, since this embodiment enables both the lens 1210 translation anti-shake function and the image sensor 1220 translation anti-shake function of the optical anti-shake component 130, the optical anti-shake component 130 will simultaneously scan the lens during the movement of the electronic device 100. 1210 and image sensor 1220 for compensating translation. In addition, an image sequence of the rectangular calibration plate captured by the image sensor 1220 during the translation process is obtained, which is denoted as a third image sequence. For example, the image sensor 1220 captures a one-minute image sequence of the rectangular calibration plate during the translation process to obtain a third image sequence.

In this embodiment, the position and posture data of the lens 1210 corresponding to each third image in the third image sequence is also obtained from the optical anti-shake component 130, that is, the position and posture data of the lens 1210 when the image sensor 1220 captures each third image, Recorded as the fifth position and attitude data. In addition, the position and posture data when the image sensor 1220 captures each third image is also obtained from the optical anti-shake component 130, which is recorded as the sixth position and posture data. In addition, the position coordinates of the center mark in each third image are also obtained, which are denoted as the third position coordinates.

In addition, according to the first functional relationship obtained by fitting, the translation vector corresponding to each fifth position and attitude data is obtained, which is recorded as the third translation vector, and according to the second functional relationship obtained by fitting, the translation vector corresponding to each The translation vector of the sixth position and attitude data is recorded as the fourth translation vector.

It can be understood that, for a third image, the third position coordinates of the aforementioned center mark in the third image are equivalent to the translation vector from the center to the center mark of the third image, and the translation vector should be determined by the third image The corresponding third translation vector and the fourth translation vector are fused to obtain. Therefore, fitting is performed according to the third position coordinates, the third translation vector and the fourth translation vector corresponding to each third image to obtain the first fusion parameter and the second fusion parameter. Here, there is no specific limitation on the manner of obtaining the first fusion parameter and the second fusion parameter through fitting, which can be selected by those skilled in the art according to actual needs. For example, in this embodiment, a linear regression method is used for fitting to obtain the first fusion parameter and the second fusion parameter.

In other embodiments, after fitting the first fusion parameter and the second fusion parameter, the fitting error of the first fusion parameter and the second fusion parameter is also evaluated (the evaluation method can be selected by those skilled in the art according to actual needs , not specifically limited here, for example, the mean square error can be used to obtain the fitting error of the first fusion parameter and the second fusion parameter, if the fitting error of the first fusion parameter and the second fusion parameter is greater than the third error threshold ( According to actual needs, those skilled in the art can take empirical values), then identify and eliminate the obviously abnormal fifth position and attitude data, the sixth position and attitude data and the third position coordinates and then perform fitting again, if the first fusion of the new fit If the fitting error of the parameter and the second fusion parameter is still greater than the third error threshold, then reacquire the new third image sequence and the new fifth position and attitude data, the sixth position and attitude data and the third position coordinates for fitting until The first fusion parameter and the second fusion parameter whose fitting error is less than or equal to the third error threshold are obtained.

Optionally, in an embodiment, according to the first rotation posture data, obtaining a first correction rotation matrix for performing rotation correction on the optical image stabilization image includes:

(1) According to the third functional relationship between the rotation attitude data and the rotation angle, obtain the rotation angle corresponding to the first rotation attitude data;

(2) Obtain the rotation center coordinates of the optical image stabilization image, and obtain a first correction rotation matrix according to the rotation center coordinates and the rotation angle.

It should be noted that this embodiment presets a third functional relationship between the rotation attitude data and the rotation angle, and the third functional relationship describes the correspondence between the changes in the rotation attitude of the image content caused by the change in the rotation attitude of the image sensor 1220 . In addition, this embodiment also pre-fits the rotation center coordinates of the images collected by the image sensor 1220 . There is no limitation to the specific fitting method here, and a suitable fitting method can be selected by those skilled in the art according to actual needs.

Based on the third functional relationship above, in this embodiment, when obtaining the first correction rotation matrix for performing rotation correction on the optical image stabilization image according to the first rotation posture data, firstly, according to the third functional relationship, the corresponding first rotation matrix is obtained. The rotation angle of the posture data, in addition, the rotation center coordinates of the optical image stabilization image, that is, the rotation center coordinates of the images collected by the image sensor 1220 obtained through pre-fitting.

As above, after obtaining the rotation angle corresponding to the first rotation attitude data and the rotation center coordinates of the optical image stabilization image, the first correction rotation matrix is further obtained according to the rotation angle and the rotation center coordinates.

Optionally, in an embodiment, obtaining the first corrected rotation matrix according to the coordinates of the rotation center and the rotation angle includes:

Optionally, in an embodiment, the rectangular calibration plate further includes four-corner marks, and the coordinates of the rotation center and the third functional relationship are generated according to the following steps:

(1) Turn on the image sensor 1220 rotation anti-shake function of the optical anti-shake assembly 130, and turn off the lens 1210 translation anti-shake function and the image sensor 1220 translation anti-shake function;

(2) sending a rotation control command to the preset calibration tool, so that the preset calibration tool drives the electronic device 100 to rotate around the Z axis of the image sensor 1220;

(3) During the rotation process of the electronic device 100, perform optical anti-shake processing through the optical anti-shake component 130, and acquire a fourth image sequence of the rectangular calibration plate collected by the image sensor 1220 during the rotation process;

(4) Obtain from the optical anti-shake component 130 the second rotation attitude data when the image sensor 1220 captures each fourth image in the fourth image sequence, and acquire the fourth position coordinates of the four corner marks in each fourth image;

(5) According to the second rotation attitude data and the fourth position coordinates corresponding to each fourth image, the coordinates of the rotation center and the third functional relationship are obtained by fitting.

In this embodiment, the coordinates of the rotation center and the third functional relationship are obtained through pre-calibration using a preset calibration tool. Similarly, firstly, the electronic device 100 is fixed to the preset calibration tool according to the above method, so that the positions of the electronic device 100 and the rectangular calibration plate in the preset calibration tool are relatively fixed. For details, reference may be made to relevant descriptions in the foregoing embodiments, and details are not repeated here.

After the electronic device 100 is fixed to the preset calibration tool, and the positions of the electronic device 100 and the rectangular calibration plate in the preset calibration tool are relatively fixed, the coordinates of the center of rotation and the third functional relationship can be obtained by calibration with the preset calibration tool .

Since the coordinates of the rotation center and the third functional relationship are calibrated at this time, the image sensor 1220 rotation anti-shake function of the optical anti-shake component 130 is correspondingly turned on, and the translation anti-shake function of the lens 1210 and the translation anti-shake function of the image sensor 1220 are turned off, thereby , when the electronic device 100 moves, the optical anti-shake component 130 only performs compensation rotation on the image sensor 1220 according to the movement of the electronic device 100 . Correspondingly, a rotation control command is sent to the preset calibration tool, so that the preset calibration tool drives the electronic device 100 to rotate around the Z-axis of the image sensor 1220 . For example, the rotation control command enables the preset calibration tool to drive the electronic device 100 to alternately rotate around the Z-axis of the image sensor 1220, wherein the rotation speed and the alternating frequency can be determined by those skilled in the art according to actual needs, and no specific limitation is made here. .

During the rotation of the electronic device 100 , the motion data of the electronic device 100 is acquired in real time and provided to the optical anti-shake component 130 , so that the optical anti-shake process is performed by the optical anti-shake component 130 . It can be understood that, since this embodiment only enables the anti-shake function of the image sensor 1220 of the optical anti-shake component 130 , the optical anti-shake component 130 will only compensate for the rotation of the image sensor 1220 during the movement of the electronic device 100 . In addition, an image sequence of the rectangular calibration plate captured by the image sensor 1220 during the rotation process is obtained, which is denoted as a fourth image sequence. For example, the image sensor 1220 captures a one-minute image sequence of the rectangular calibration plate during the rotation process to obtain a fourth image sequence.

In this embodiment, the image sensor 1220 also acquires the rotation attitude data of each fourth image in the fourth image sequence collected from the optical anti-shake component 130, which is recorded as the second rotation attitude data. In addition, the position coordinates of the four-corner markers (upper-left marker, upper-right marker, lower-right marker, and lower-left marker) in each fourth image are also obtained, which are recorded as fourth position coordinates.

It can be understood that, for a fourth image, if the first correction rotation matrix is used to correct the rotation of the aforementioned four-corner marks, the corrected four-corner marks should be located at the corresponding four-corner positions of the corrected fourth image. Therefore, by performing fitting according to the second rotation posture data and the fourth position coordinates corresponding to each fourth image, the rotation center coordinates and the third functional relationship can be obtained by fitting.

In other embodiments, after the rotation center coordinates and the third functional relationship are obtained through fitting, the fitting error between the rotation center coordinates and the third functional relationship is also evaluated (evaluation methods can be selected by those skilled in the art according to actual needs, here There is no specific limitation, for example, reprojection error can be used to obtain the fitting error between the rotation center coordinates and the third function relationship, if the fitting error between the rotation center coordinates and the third function relationship is greater than the fourth error threshold (which can be determined by those skilled in the art personnel according to actual needs), then identify and eliminate the obviously abnormal second rotation attitude data and fourth position coordinates, and then perform fitting again. If the fitting error between the newly fitted rotation center coordinates and the third function relationship is still If it is greater than the fourth error threshold, re-acquire a new fourth image sequence, new second rotation attitude data, and fourth position coordinates for fitting until a third functional relationship with a fitting error less than or equal to the fourth error threshold is obtained.

Optionally, in an embodiment, the third functional relationship includes a polynomial function, and according to the second rotation posture data and the fourth position coordinates corresponding to each fourth image, the coordinates of the rotation center and the third functional relationship are obtained by fitting, including :

θ = [Cx, Cy, α];

Among them, Pi represents the fourth position coordinates of the i-th four-corner marker, h ₂ ' represents the second rotation attitude data, f() represents a polynomial function, r represents a preset search radius, α represents the polynomial coefficient of the polynomial function, imW represents the first The width of the four images, imH represents the height of the fourth image.

It should be noted that when fitting f(), a suitable loss function can be constructed by those skilled in the art according to actual needs, for example, it can be constructed by those skilled in the art according to the actual geometric relationship, including but not limited to the chord length relationship or rotation relations etc.

In addition, there is no specific limitation on the value of the preset search radius r in this embodiment, and it can be selected by those skilled in the art according to actual needs.

In addition, the order of the polynomial function is not specifically limited in this embodiment, and can be selected by those skilled in the art according to actual needs. For example, in this embodiment, a polynomial function of degree 3 is used.

Optionally, in one embodiment, the image processing method provided by the present application further includes:

(1) Turn on the lens 1210 translation anti-shake function of the optical anti-shake component 130, the image sensor 1220 translation anti-shake function and the image sensor 1220 rotation anti-shake function;

(2) Send a rotation and translation control command to the preset calibration tool, so that the preset calibration tool drives the electronic device 100 to rotate around the Z-axis of the image sensor 1220, and drives the electronic device 100 to translate in the X-axis direction and the Y-axis direction of the electronic device 100 ;

(3) During the rotation and translation process of the electronic device 100, perform optical anti-shake processing through the optical anti-shake component 130, and acquire the fifth image sequence of the rectangular calibration plate collected by the image sensor 1220 during the rotation and translation process;

(4) Obtain from the optical anti-shake component 130 the seventh position and attitude data of the lens 1210 corresponding to each fifth image in the fifth image sequence, the eighth position and attitude data and the third rotation when the image sensor 1220 captures each fifth image attitude data;

(5) According to the seventh position and attitude data, the eighth position and attitude data and the first functional relationship and the second functional relationship corresponding to each fifth image, obtain the second corrected translation for translation correction of each fifth image vector;

(6) According to the third rotation posture data corresponding to each fifth image, the third functional relationship and the coordinates of the rotation center, obtain a second correction rotation matrix for performing rotation correction on each fifth image;

(7) Perform translation correction and rotation correction on the center pixel of each fifth image according to the second correction translation vector and the second correction rotation matrix, to obtain the correction coordinates of each center pixel;

(8) Obtain the fifth position coordinates of the center mark in each fifth image, and according to the difference between the correction coordinates corresponding to each fifth image and the fifth position coordinates, the first functional relationship, the second functional relationship and the first functional relationship The three-function relationship is jointly corrected.

In this embodiment, the first functional relationship, the second functional relationship and the third functional relationship are jointly corrected by using a preset calibration tool. Similarly, firstly, the electronic device 100 is fixed to the preset calibration tool according to the above method, so that the positions of the electronic device 100 and the rectangular calibration plate in the preset calibration tool are relatively fixed. For details, reference may be made to relevant descriptions in the foregoing embodiments, and details are not repeated here.

After the electronic device 100 is fixed to the preset calibration tool, and the positions of the electronic device 100 and the rectangular calibration plate in the preset calibration tool are relatively fixed, the first functional relationship and the second functional relationship can be determined using the preset calibration tool. and the third functional relationship for joint correction.

Since the first functional relationship, the second functional relationship, and the third functional relationship are jointly corrected at this time, all the anti-shake functions of the optical anti-shake component 130 are correspondingly turned on, that is, the translational anti-shake function of the image sensor 1220 and the translational anti-shake function of the lens 1210 and the image sensor 1220 rotation anti-shake function, so that when the electronic device 100 moves, the optical anti-shake component 130 performs compensation translation for the lens 1210 according to the movement of the electronic device 100 , and performs compensation translation and compensation rotation for the image sensor 1220 . Correspondingly, the rotation and translation control command is sent to the preset calibration tool, so that the preset calibration tool drives the electronic device 100 to rotate around the Z-axis of the image sensor 1220, and at the same time drives the electronic device 100 to translate in the X-axis direction and the Y-axis direction of the electronic device 100 . For example, the rotation and translation control command makes the preset calibration tool drive the electronic device 100 to alternately translate in its X-axis direction and Y-axis direction, and at the same time alternately rotate around the Z-axis of the image sensor 1220, wherein the distance of translation, the frequency of alternate translation, the rotation The speed and the alternating rotation frequency can be selected by those skilled in the art according to actual needs, and no specific limitation is made here.

During the rotation and translation process of the electronic device 100 , the motion data of the electronic device 100 is acquired in real time and provided to the optical anti-shake component 130 , so that the optical anti-shake process is performed by the optical anti-shake component 130 . It can be understood that, since all the anti-shake functions of the optical anti-shake component 130 are turned on in this embodiment, the optical anti-shake component 130 will compensate the translation of the lens 1210 and compensate the image sensor 1220 during the movement of the electronic device 100 . Translation and compensated rotation. In addition, the image sequence of the rectangular calibration plate captured by the image sensor 1220 during the rotation and translation is acquired, which is denoted as the fifth image sequence. For example, the image sensor 1220 collects a one-minute image sequence of the rectangular calibration plate during the rotation and translation process to obtain a fifth image sequence.

In this embodiment, the position and posture data of the lens 1210 corresponding to each fifth image in the fifth image sequence is obtained from the optical anti-shake component 130, that is, the position and posture data of the lens 1210 when the image sensor 1220 captures each fifth image , recorded as the seventh position and posture data, in addition, the position and posture data when the image sensor 1220 captures every fifth image is also obtained, which is recorded as the eighth position and posture data, and the rotation when the image sensor 1220 captures every fifth image Attitude data, denoted as the third rotation attitude data.

As above, after obtaining the seventh position and posture data, the eighth position and posture data and the third rotation posture data corresponding to each fifth image, further according to the seventh position and posture data, the eighth position and posture data corresponding to each fifth image, The data and the first functional relationship and the second functional relationship are used to obtain a correction translation vector for translation correction of each fifth image, which is denoted as a second correction translation vector. Specifically, it may be implemented correspondingly with reference to the manner of obtaining the first correction translation vector in the above embodiments, which will not be repeated here.

In addition, according to the third functional relationship, the coordinates of the rotation center, and the third rotation posture data corresponding to each fifth image, a corrected rotation matrix for performing rotation correction on each fifth image is obtained, which is denoted as the second corrected rotation matrix . Specifically, it can be implemented correspondingly with reference to the manner of obtaining the first correction rotation matrix in the above embodiments, and details are not repeated here.

As above, after obtaining the second correction translation vector for translation correction of each fifth image and the second correction rotation matrix for rotation correction of each fifth image, that is, according to the second correction translation vector and The second correction rotation matrix respectively performs translation correction and rotation correction on the central pixel of each fifth image to obtain corrected coordinates of the central pixel of each fifth image. In addition, the position coordinates of the center mark in each fifth image are also obtained, which are denoted as fifth position coordinates. It can be understood that, for a fifth image, the closer the corrected coordinates of the fifth image pair are to the fifth position coordinates, the more general simulations of the first functional relationship, the second functional relationship and the third functional relationship in this embodiment are described. The smaller the combined error. Therefore, in this embodiment, according to the difference between the correction coordinates corresponding to each fifth image and the fifth position coordinates, with the goal of eliminating the difference as much as possible, the first functional relationship, the second functional relationship, and the third functional relationship are calculated. Joint revision.

For example, in this embodiment, according to the size of the fitting errors of the first functional relationship, the second functional relationship and the third functional relationship, corrections are made in descending order.

Optionally, in an embodiment, acquiring an optical image stabilization image corresponding to motion data collected by the image sensor 1220 includes:

(1) Correcting the first acquisition time stamp of the motion data according to the first time delay parameter corresponding to the motion data to obtain the first corrected time stamp;

(3) According to the first corrected time stamp, acquire the optical image stabilization image corresponding to the motion data collected by the image sensor 1220 .

In this embodiment, when acquiring the optical image stabilization image corresponding to the movement data collected by the image sensor 1220, the acquisition time stamp of the movement data is first acquired, which is recorded as the first acquisition time stamp, and the first acquisition time stamp is used to describe the acquired The acquisition time of motion data.

It can be understood that there is a certain time delay between the motion sensor 110 collecting motion data and outputting the motion data. There is a first time delay parameter corresponding to the motion data, and the first time delay parameter is used to describe the time delay from the motion sensor 110 collecting the motion data to outputting the motion data.

Correspondingly, in this embodiment, the first acquisition timestamp is corrected according to the first delay parameter of the corresponding motion data, and the corrected first acquisition timestamp is marked as the first modified timestamp, which describes the The collection time of the aforementioned motion data. At this point, according to the first corrected time stamp, the optical image stabilization image collected by the image sensor 1220 and corresponding to the aforementioned motion data can be acquired.

Optionally, in an embodiment, the first delay parameter is pre-generated according to the following steps:

(1) Obtain a sample dithering image sequence;

(2) Acquiring sample motion data corresponding to each sample shake image according to the second acquisition time stamp of each sample shake image in the sample shake image sequence to obtain a sample motion data sequence;

(3) According to the preset time delay parameter, the preset search step size, the sample shaking image sequence and the corresponding sample motion data sequence, search to obtain the first time delay parameter.

This embodiment provides an optional calibration manner of the first delay parameter.

Among them, the image sequence without electronic anti-shake processing is first obtained as a sample, which is recorded as a sample shake image sequence. For example, the anti-shake functions of the optical anti-shake component 130 and the electronic anti-shake component 140 are turned off, and an image sequence of a certain duration is captured by the electronic device 100 as a sample shake image sequence.

In addition, the sample motion data corresponding to each sample shake image is acquired according to the second acquisition time stamp (used to describe the output moment when the image sensor 1220 outputs the corresponding sample shake image) of each sample shake image in the sample shake image sequence, A sample motion data sequence corresponding to the sample shaken image sequence is obtained.

As above, after obtaining the sample shaking image sequence and its corresponding sample motion data sequence, according to the preset search strategy, according to the preset time delay parameter, preset search step size, sample shaking image sequence and its corresponding sample motion data sequence , to obtain the first delay parameter. Here, there is no specific limitation on the configuration of the preset search strategy, the preset delay parameter, and the preset search step size, which can be configured by those skilled in the art according to actual needs.

Optionally, in an embodiment, according to a preset delay parameter, a preset search step size, a sample shaking image sequence and its corresponding sample motion data sequence, the first delay parameter is obtained by searching, including

(1) Acquiring a preset number of candidate time delay parameters according to a preset time delay parameter and a preset search step;

(2) Correcting the sample motion data sequence according to each candidate delay parameter to obtain a preset number of corrected sample motion data sequences;

(3) According to each corrected sample motion data sequence, electronic anti-shake processing is performed on the sample shake image sequence through the electronic anti-shake component 140, to obtain a preset number of anti-shake image sequences;

(4) Quantify and score the anti-shake quality of each anti-shake image sequence, obtain the anti-shake quality score of each anti-shake image sequence;

(5) updating the preset time delay parameter to the candidate time delay parameter corresponding to the anti-shake image sequence with the highest anti-shake quality score, and updating the preset search step according to the preset update parameter;

(6) The above steps are repeated until the preset search condition is met, and the preset delay parameter when the preset search condition is met is used as the first delay parameter.

This embodiment provides a search strategy for optionally searching the first delay parameter.

Wherein, firstly, a preset number of candidate delay parameters are acquired according to a preset delay parameter and a preset search step. Here, there is no specific limitation on the value of the preset number. For example, when the value is 3, the preset delay parameter D ₀ is the center, and the preset search step size Δd is used to generate [D ₀ -Δd,D ₀ , D ₀ +Δd] 3 candidate delay parameters.

Then, the sample motion data sequence is respectively corrected according to each candidate time delay parameter, that is, the corresponding relationship between the sample motion data in the sample motion data sequence and the sample shake image in the sample shake image sequence is updated according to each candidate time delay parameter , correspondingly obtain a preset number of corrected sample motion data sequences;

Then, according to each corrected sample motion data sequence, electronic anti-shake processing is performed on the sample shake image sequence through the electronic anti-shake component 140 to obtain a preset number of anti-shake image sequences, wherein, for a corrected sample motion data sequence, the In the corrected sample motion data sequence, each corrected sample motion data and sample dithered image sequence corresponding to the sample dithered image is input to the sample dithered image for anti-shake processing, and an anti-shake image sequence corresponding to the corrected sample motion data sequence is correspondingly obtained.

As above, after the preset number of anti-shake image sequences are obtained, the anti-shake quality of each anti-shake image sequence is quantified and scored according to the preset quantitative scoring strategy, and the anti-shake quality score of each anti-shake image sequence is obtained. The configuration of the quantitative scoring strategy is not specifically limited this time, and can be configured by those skilled in the art according to actual needs.

For example, in this embodiment, the method of frame-by-frame feature matching and average feature distance statistics is used to quantify and score the anti-shake quality of the anti-shake image sequence, which can be expressed as:

Among them, s represents the anti-shake quality score (the larger the value, the higher the anti-shake quality), p represents the feature matching point set, i represents the frame number of the anti-shake image sequence, and N represents the number of anti-shake images in the anti-shake image sequence total frames. It should be noted that the feature matching algorithm and its descriptor can be selected by those skilled in the art according to actual needs, including but not limited to SIFT, FAST, Harris, ORB and so on.

As above, after the quantitative scoring of each anti-shake image sequence is completed, and the anti-shake quality score of each anti-shake image sequence is correspondingly obtained, the preset delay parameter is updated to correspond to the anti-shake image sequence with the highest anti-shake quality score The candidate time delay parameters, and according to the preset update parameters, the preset search step is updated according to the preset update strategy. Here, there is no specific restriction on the configuration of the preset update strategy, and the preset update strategy can be configured by those skilled in the art according to actual needs with the restriction of reducing the preset search step size.

For example, the default update policy configured in this embodiment can be expressed as:

Δd'=Δd/M; M>1;

Wherein, Δd' represents a preset search step after update, Δd represents a preset search step before update, and M represents a preset update parameter.

After updating the preset delay parameters and preset search steps, repeat the above steps to continue searching until the preset search conditions are met, and use the preset delay parameters when the preset search conditions are met as the first delay parameter. The configuration of the preset search conditions is not specifically limited here, and can be configured by those skilled in the art according to actual needs.

Optionally, in an embodiment, the preset search conditions include:

The updated preset search step size is smaller than the step size threshold; or

The number of searches has reached the number threshold.

It should be noted that this embodiment does not specifically limit the scoring threshold, the step threshold, and the number of times threshold, and those skilled in the art can obtain empirical values according to actual needs.

Optionally, in an embodiment, acquiring the first position and posture data of the lens 1210, the second position and posture data and the first rotation posture data of the image sensor 1220 from the optical anti-shake component 130 includes:

(1) Correcting the first corrected timestamp according to the second time delay parameter corresponding to the optical anti-shake component 130 to obtain a second corrected timestamp;

(2) Obtain the first position and posture data of the lens 1210 , the second position and posture data and the first rotation posture data of the image sensor 1220 from the optical anti-shake component 130 according to the second corrected time stamp.

It can be understood that there is a certain delay in the optical anti-shake component 130 from collecting the position and attitude data of the lens 1210, the position and attitude data and the rotation attitude data of the image sensor 1220 to outputting these attitude data. In this embodiment, in order to eliminate this Time delay, the first position and attitude data, the second position and attitude data and the first rotation attitude data accurately corresponding to the optical image stabilization image are obtained, and the second time delay parameter corresponding to the optical anti-shake component 130 is pre-calibrated. The delay parameter is used to describe the delay from when the optical image stabilization component 130 collects the aforementioned attitude data until it outputs the aforementioned attitude data.

Correspondingly, in this embodiment, when acquiring the first position and posture data of the lens 1210 , the second position and posture data and the first rotation posture data of the image sensor 1220 from the optical anti-shake component 130 , firstly, according to the first position and posture data of the corresponding optical anti-shake component 130 The second delay parameter corrects the first corrected time stamp to obtain the second corrected time stamp; then according to the second corrected time stamp, the first position and attitude data of the lens 1210 and the second position and attitude data of the image sensor 1220 are obtained from the optical anti-shake component 130. The position and attitude data and the first rotation attitude data, that is, the position and attitude data of the lens 1210 whose output time stamp matches the second modified time stamp are obtained from the optical anti-shake component 130 as the first position and attitude data, obtained from the optical anti-shake component 130 Output the position and attitude data of the image sensor 1220 whose time stamp matches the second modified time stamp as the second position and attitude data, and acquire the rotation attitude of the image sensor 1220 whose output time stamp matches the second modified time stamp from the optical image stabilization component 130 data as the first rotation attitude data.

It should be noted that, this embodiment does not specifically limit the calibration manner of the second delay parameter, which can be selected by those skilled in the art according to actual needs. For example, this embodiment adopts the same calibration strategy as that of the first delay parameter (for details, please refer to the relevant description in the above embodiments, which will not be repeated here), and after the calibration of the first delay parameter is completed, the The second delay parameter is calibrated.

Optionally, in an embodiment, the motion data includes acceleration data and angular velocity data.

An embodiment of the present application also provides an image processing device, which is applied to an electronic device, and the electronic device includes a lens, an image sensor, an optical anti-shake component, and an electronic anti-shake component. The image processing device includes:

The optical anti-shake module is used to obtain the motion data of the electronic device, and according to the motion data, control the lens to perform compensation translation in the X-axis direction and the Y-axis direction of the lens through the optical anti-shake component, and control the image sensor to move in the X-axis direction of the image sensor Perform compensation translation with the Y axis direction, and control the image sensor to perform compensation rotation around the Z axis of the image sensor;

The attitude acquisition module is used to acquire the optical image stabilization image corresponding to the motion data collected by the image sensor, and acquire the first position attitude data of the lens, the second position attitude data and the first rotation attitude data of the image sensor from the optical anti-shake component;

The attitude correction module is used to restore and correct the attitude of the optical image stabilization image according to the first position attitude data, the second position attitude data and the first rotation attitude data, so as to obtain the corrected image;

The electronic anti-shake module is used to perform electronic anti-shake processing on the corrected image through the electronic anti-shake component according to the motion data to obtain an electronic anti-shake image.

Please refer to FIG. 7. In order to better implement the image processing method provided by the present application, this embodiment also provides an image processing device 300. As shown in FIG. 7, the image processing device 300 includes an optical image stabilization module 310, a gesture Acquisition module 320, posture correction module 330 and electronic anti-shake module 340, the image processing device 300 is configured to execute the steps in any image processing method provided in this application, such as:

The optical anti-shake module 310 is used to obtain motion data of the electronic device, and according to the motion data, control the lens to perform compensation translation in the X-axis direction and the Y-axis direction of the lens through the optical anti-shake component, and control the image sensor to move in the X-axis direction of the image sensor direction and Y-axis direction to perform compensation translation, and control the image sensor to perform compensation rotation around the Z-axis of the image sensor;

The attitude acquisition module 320 is configured to acquire the optical image stabilization image corresponding to the motion data collected by the image sensor, and acquire the first position attitude data of the lens, the second position attitude data and the first rotation attitude data of the image sensor from the optical image stabilization component ;

A pose correction module 330, configured to restore and correct the pose of the optical image stabilization image according to the first position and pose data, the second position and pose data, and the first rotation pose data, to obtain a corrected image;

The electronic anti-shake module 340 is configured to perform electronic anti-shake processing on the corrected image through the electronic anti-shake component according to the motion data to obtain an electronic anti-shake image.

It should be noted that the image processing device 300 provided in the embodiment of the present application belongs to the same idea as the image processing method in the above embodiment, and its specific implementation process refers to the above related embodiments, and details are not repeated here.

The embodiment of the present application also provides an electronic device, the electronic device includes a lens, an image sensor, an optical anti-shake component, an electronic anti-shake component, a processor and a memory, the memory stores a computer program, and the processor executes the above-mentioned The method mentioned in the examples.

Please refer to FIG. 1 and FIG. 8 in combination, the electronic device 100 provided in this application may further include a memory 150 and a processor 160 . Those skilled in the art can understand that the structure of the electronic device 100 shown in FIG. Part placement.

Among them, the memory 150 can be used to store computer programs and data. The computer programs stored in the memory 150 include executable codes. A computer program can be divided into various functional modules.

The processor 160 is the control center of the electronic device 100. It uses various interfaces and lines to connect various parts of the entire electronic device 100. By running or executing the computer program stored in the memory 150 and calling the data stored in the memory 150, the processor 160 executes Various functions and processing data of the electronic device 100 , so as to control the electronic device 100 as a whole.

In this embodiment of the application, the processor 160 in the electronic device 100 loads the executable code corresponding to one or more computer programs into the memory 150, and is executed by the processor 160 to execute any image provided by the application. Steps in the processing method, such as:

Obtain the motion data of the electronic device 100, and according to the motion data, control the lens 1210 to perform compensation translation in the X-axis direction and the Y-axis direction of the lens 1210 through the optical anti-shake component 130, and control the image sensor 1220 in the X-axis direction and the Y-axis direction of the image sensor 1220. Y-axis analysis for compensating translation, and controlling the image sensor 1220 to perform compensating rotation around the Z-axis of the image sensor 1220;

Acquiring the optical image stabilization image corresponding to the motion data collected by the image sensor 1220, and obtaining the first position and posture data of the lens 1210, the second position and posture data and the first rotation posture data of the image sensor 1220 from the optical anti-shake component 130;

According to the motion data, electronic anti-shake processing is performed on the corrected image through the electronic anti-shake component 140 to obtain an electronic anti-shake image.

It should be noted that the electronic device 100 provided in the embodiment of the present application is based on the same idea as the image processing method in the above embodiment, and its specific implementation process may refer to the above related embodiments, which will not be repeated here.

The embodiment of the present application also provides an electronic device, including a lens, an image sensor, an optical anti-shake component, an electronic anti-shake component, a processor, and a memory. Steps in an image processing method. .

The embodiment of the present application also provides a computer-readable storage medium, in which a computer program is stored, and the computer program can be loaded by a processor to execute the steps in any image processing method provided in the embodiment of the present application.

Wherein, the storage medium may include: a read-only memory (ROM, Read Only Memory), a random access memory (RAM, Random Access Memory), a magnetic disk or an optical disk, and the like.

Since the computer program stored in the computer-readable storage medium can execute the steps in any image processing method provided by the embodiment of the application, therefore, any image processing method provided by the embodiment of the application can be realized For the beneficial effects that can be achieved by the method, refer to the previous embodiments for details, and will not be repeated here.

The image processing method, device, electronic device, and storage medium provided in the embodiments of the present application have been introduced in detail above. In this paper, specific examples are used to illustrate the principles and implementation methods of the present application, and the descriptions of the above embodiments are only used to help understand the present application. At the same time, for those skilled in the art, based on the idea of this application, there will be changes in the specific implementation and application scope. In summary, the content of this specification should not be construed as limiting the application.

Claims

An image processing method applied to an electronic device, wherein the electronic device includes a lens, an image sensor, an optical anti-shake component, and an electronic anti-shake component, and the image processing method includes:

Acquire motion data of the electronic device, and according to the motion data, control the lens to perform compensation translation in the X-axis direction and the Y-axis direction of the lens through the optical anti-shake component, and control the image sensor in the performing compensation translation in the X-axis direction and the Y-axis direction of the image sensor, and controlling the compensation rotation of the image sensor around the Z-axis of the image sensor;

Acquiring the optical image stabilization image corresponding to the motion data collected by the image sensor, and acquiring the first position and posture data of the lens, the second position and posture data of the image sensor and the first rotation attitude data;

Restoring and correcting the posture of the optical image stabilization image according to the first position and posture data, the second position and posture data, and the first rotation posture data to obtain a corrected image;

According to the motion data, electronic anti-shake processing is performed on the corrected image through the electronic anti-shake component to obtain an electronic anti-shake image.
The method according to claim 1, wherein the restoration and correction of the posture of the optical image stabilization image is performed according to the first position and posture data, the second position and posture data, and the first rotation posture data, Get the corrected image, including:

Acquiring a first correction translation vector for performing translation correction on the optical image stabilization image according to the first position and attitude data and the second position and attitude data;

Acquiring a first correction rotation matrix for performing rotation correction on the optical image stabilization image according to the first rotation attitude data;

Perform translation correction and rotation correction on the pose of the optical image stabilization image according to the first correction translation vector and the first correction rotation matrix to obtain the correction image.
The method according to claim 2, wherein, according to the first position and attitude data and the second position and attitude data, a first correction translation vector for translation correction of the optical image stabilization image is obtained, include:

Acquiring a first translation vector corresponding to the first position and attitude data according to the first functional relationship between the position and attitude data and the translation vector;

Acquiring a second translation vector corresponding to the second position and attitude data according to a second functional relationship between the position and attitude data and the translation vector;

According to the first translation vector, the first fusion parameter corresponding to the first translation vector, the second translation vector and its corresponding second fusion parameter, fused to obtain the first translation vector and the second translation vector of fused translation vectors;

The fused translation vector is used as the first corrected translation vector.
The method according to claim 3, wherein the electronic device is fixed to a preset calibration tool such that the position of the electronic device and the rectangular calibration plate in the preset calibration tool are relatively fixed, and the rectangular calibration plate Including the central mark, the first functional relationship is pre-generated according to the following steps:

Turn on the lens translation anti-shake function of the optical anti-shake component, and turn off the image sensor translation anti-shake function and the image sensor rotation anti-shake function;

sending a first translation control command to the preset calibration tool, so that the preset calibration tool drives the electronic device to translate in the X-axis direction and the Y-axis direction of the electronic device;

During the translation process of the electronic device, performing optical anti-shake processing through the optical anti-shake component, and acquiring a first image sequence of the rectangular calibration plate collected by the image sensor during the translation process;

Acquiring third position and posture data of the lens corresponding to each first image in the first image sequence from the optical anti-shake component, and acquiring first position coordinates of the center mark in each first image;

According to the third position and posture data corresponding to each first image and the first position coordinates, the first functional relationship is obtained by fitting.
The method according to claim 3, wherein the electronic device is fixed to a preset calibration tool such that the position of the electronic device and the rectangular calibration plate in the preset calibration tool are relatively fixed, and the rectangular calibration plate Including the central mark, the second functional relationship is pre-generated according to the following steps:

Turn on the image sensor translation anti-shake function of the optical anti-shake component, and turn off the lens translation anti-shake function and the image sensor rotation anti-shake function;

sending a second translation control command to the preset calibration tool, so that the preset calibration tool drives the electronic device to translate in the X-axis direction and the Y-axis direction of the electronic device;

During the translation process of the electronic device, performing optical anti-shake processing through the optical anti-shake component, and acquiring a second image sequence of the rectangular calibration plate collected by the image sensor during the translation process;

Acquiring fourth position and attitude data from the optical anti-shake component when the image sensor captures each second image in the second image sequence, and acquiring a second position of the center mark in each second image coordinate;

According to the fourth position and posture data corresponding to each second image and the second position coordinates, the second functional relationship is obtained by fitting.
The method according to claim 3, wherein the electronic device is fixed to a preset calibration tool such that the position of the electronic device and the rectangular calibration plate in the preset calibration tool are relatively fixed, and the rectangular calibration plate Including the center mark, the first fusion parameter and the second fusion parameter are predetermined according to the following steps:

Turn on the lens translation anti-shake function and the image sensor translation anti-shake function of the optical anti-shake component, and turn off the image sensor rotation anti-shake function;

sending a third translation control command to the preset calibration tool, so that the preset calibration tool drives the electronic device to translate in the X-axis direction and the Y-axis direction of the electronic device;

During the translation process of the electronic device, performing optical anti-shake processing through the optical anti-shake component, and acquiring a third image sequence of the rectangular calibration plate collected by the image sensor during the translation process;

Obtaining the fifth position and posture data of the lens corresponding to each third image in the third image sequence from the optical anti-shake component, and obtaining the third image captured by the image sensor from the optical anti-shake component The sixth position and attitude data of each third image in the sequence;

Obtaining a third translation vector corresponding to each fifth position and attitude data according to the first functional relationship, and obtaining a fourth translation vector corresponding to each sixth position and attitude data according to the second functional relationship;

Acquiring the third position coordinates of the center mark in each third image, and determining the third position coordinates, the third translation vector, and the fourth translation vector corresponding to each third image. The first fusion parameter and the second fusion parameter.
The method according to claim 3, wherein said acquiring a first correction rotation matrix for performing rotation correction on said optical image stabilization image according to said first rotation attitude data comprises:

Acquiring a rotation angle corresponding to the first rotation attitude data according to a third functional relationship between the rotation attitude data and the rotation angle;

Obtain the rotation center coordinates of the optical image stabilization image, and obtain the first correction rotation matrix according to the rotation center coordinates and the rotation angle.
The method according to claim 7, wherein said obtaining said first correction rotation matrix according to said rotation center coordinates and said rotation angle comprises:

According to the rotation center coordinates and the rotation angle, the first corrected rotation matrix is obtained according to the following formula:

Wherein, R represents the first correction rotation matrix, C represents cos(γ), γ represents the rotation angle, S represents sin(γ), Cx represents the abscissa in the rotation center coordinates, and Cy represents the rotation Ordinate in center coordinates.
The method according to claim 7, wherein the electronic device is fixed to a preset calibration tool such that the positions of the electronic device and the rectangular calibration plate in the preset calibration tool are relatively fixed, and the rectangular calibration plate Including four-corner marks, the coordinates of the rotation center and the third functional relationship are generated according to the following steps:

Turn on the image sensor rotation anti-shake function of the optical anti-shake component, and turn off the lens translation anti-shake function and the image sensor translation anti-shake function;

sending a rotation control command to the preset calibration tool, so that the preset calibration tool drives the electronic device to rotate around the Z axis of the image sensor;

During the rotation process of the electronic device, perform optical anti-shake processing through the optical anti-shake component, and acquire a fourth image sequence of the rectangular calibration plate collected by the image sensor during the rotation process;

Acquiring the second rotation posture data when the image sensor captures each fourth image in the fourth image sequence from the optical anti-shake component, and acquiring the fourth position of the four-corner mark in each fourth image coordinate;

According to the second rotation attitude data and the fourth position coordinates corresponding to each fourth image, the coordinates of the rotation center and the third functional relationship are obtained by fitting.
The method according to claim 9, wherein the third functional relationship includes a polynomial function, and according to the second rotation attitude data corresponding to each fourth image and the fourth position coordinates, the fitting is obtained The coordinates of the center of rotation and the third functional relationship include:

According to the second rotation attitude data and the fourth position coordinates corresponding to each fourth image, the relationship between the rotation center coordinates and the third function is obtained by fitting according to the following formula:

θ = [Cx, Cy, α];

Wherein, P i represents the fourth position coordinates of the i-th four-corner mark, h 2 ' represents the second rotation attitude data, f() represents the polynomial function, r represents the preset search radius, and α represents the polynomial of the polynomial function coefficient, imW represents the width of the fourth image, and imH represents the height of the fourth image.
The method according to claim 10, further comprising:

Turn on the lens translation anti-shake function, the image sensor translation anti-shake function and the image sensor rotation anti-shake function of the optical anti-shake component;

Sending a rotation and translation control instruction to the preset calibration tool, so that the preset calibration tool drives the electronic device to rotate around the Z-axis of the image sensor, and drives the electronic device in the direction of the X-axis of the electronic device and Y-axis direction translation;

During the rotation and translation process of the electronic device, performing optical anti-shake processing through the optical anti-shake component, and acquiring a fifth image sequence of the rectangular calibration plate collected by the image sensor during the rotation and translation process;

The seventh position and attitude data of the lens corresponding to each fifth image in the fifth image sequence, the eighth position and attitude data and the eighth position and attitude data when the image sensor captures each fifth image are obtained from the optical anti-shake component. Three rotation attitude data;

According to the seventh position and attitude data, the eighth position and attitude data and the first functional relationship and the second functional relationship corresponding to each fifth image, the translation correction for each fifth image is obtained. The second correction translation vector of ;

Obtaining a second correction rotation matrix for performing rotation correction on each fifth image according to the third rotation posture data corresponding to each fifth image, the third functional relationship, and the rotation center coordinates;

respectively performing translation correction and rotation correction on the center pixel of each fifth image according to the second correction translation vector and the second correction rotation matrix, to obtain the correction coordinates of each center pixel;

Acquiring the fifth position coordinates of the center mark in each fifth image, and according to the difference between the correction coordinates corresponding to each fifth image and the fifth position coordinates, the first functional relationship, the The second functional relationship and the third functional relationship are jointly corrected.
The method according to any one of claims 1-11, wherein the acquiring the optical image stabilization image corresponding to the motion data collected by the image sensor comprises:

Correcting the first acquisition timestamp of the motion data according to the first delay parameter corresponding to the motion data to obtain a first modified timestamp;

According to the first corrected time stamp, an optical image stabilization image collected by the image sensor and corresponding to the motion data is acquired.
The method according to claim 12, wherein the first delay parameter is pre-generated according to the following steps:

Get a sequence of sample dithered images;

Acquiring sample motion data corresponding to each sample shake image according to the second acquisition time stamp of each sample shake image in the sample shake image sequence to obtain a sample motion data sequence;

The first time delay parameter is obtained by searching according to a preset time delay parameter, a preset search step size, the sample shaking image sequence and its corresponding sample motion data sequence.
The method according to claim 13, wherein the first time delay parameter is obtained by searching according to the preset time delay parameter, the preset search step size, the sample shaking image sequence and its corresponding sample motion data sequence ,include

Acquiring a preset number of candidate delay parameters according to the preset delay parameter and the preset search step;

Correcting the sample motion data sequence according to each candidate delay parameter to obtain a preset number of corrected sample motion data sequences;

According to each corrected sample motion data sequence, the electronic anti-shake component is used to perform electronic anti-shake processing on the sample shake image sequence to obtain a preset number of anti-shake image sequences;

Quantify and score the anti-shake quality of each anti-shake image sequence to obtain the anti-shake quality score of each anti-shake image sequence;

Updating the preset delay parameter to a candidate delay parameter corresponding to the anti-shake image sequence with the highest anti-shake quality score, and updating the preset search step according to the preset update parameter;

The above steps are repeated until the preset search condition is met, and the preset delay parameter when the preset search condition is met is used as the first delay parameter.
The method according to claim 14, wherein the preset search conditions include:

The stabilization quality score of any stabilized image sequence is greater than or equal to the scoring threshold; or

The updated preset search step size is smaller than the step size threshold.
The method according to claim 12, wherein said acquiring the first position and attitude data of the lens, the second position and attitude data and the first rotation attitude data of the image sensor from the optical anti-shake component comprises:

Correcting the first corrected timestamp according to a second delay parameter corresponding to the optical anti-shake component to obtain a second corrected timestamp;

According to the second corrected timestamp, the first position and posture data of the lens, the second position and posture data and the first rotation posture data of the image sensor are obtained from the optical anti-shake component.
The method according to any one of claims 1-11, wherein the motion data includes acceleration data and angular velocity data.
An image processing device applied to electronic equipment, wherein the electronic equipment includes a lens, an image sensor, an optical anti-shake component, and an electronic anti-shake component, and the image processing device includes:

an optical anti-shake module, configured to acquire motion data of the electronic device, and control the lens to perform compensation translation in the X-axis direction and the Y-axis direction of the lens through the optical anti-shake component according to the motion data, controlling the image sensor to perform compensation translation in the X-axis direction and the Y-axis direction of the image sensor, and controlling the image sensor to perform compensation rotation around the Z-axis of the image sensor;

an attitude acquisition module, configured to acquire the optical image stabilization image corresponding to the movement data collected by the image sensor, and acquire the first position attitude data of the lens, the second position and attitude data of the image sensor from the optical image stabilization component Position attitude data and first rotation attitude data;

A pose correction module, configured to restore and correct the pose of the optical image stabilization image according to the first position and pose data, the second position and pose data, and the first rotation pose data, to obtain a corrected image;

The electronic anti-shake module is configured to perform electronic anti-shake processing on the corrected image through the electronic anti-shake component according to the motion data, so as to obtain an electronic anti-shake image.
An electronic device, the electronic device includes a lens, an image sensor, an optical anti-shake component, an electronic anti-shake component, a processor and a memory, the memory stores a computer program, wherein the processor loads the computer program Execute the method as described in any one of claims 1-17.
A computer-readable storage medium on which a computer program is stored, wherein the method according to any one of claims 1-17 is executed when the computer program is loaded by a processor.