CN113596322B

CN113596322B - Image processing method, device, electronic equipment and storage medium

Info

Publication number: CN113596322B
Application number: CN202110778715.XA
Authority: CN
Inventors: 周阳
Original assignee: Guangdong Oppo Mobile Telecommunications Corp Ltd
Current assignee: Guangdong Oppo Mobile Telecommunications Corp Ltd
Priority date: 2021-07-09
Filing date: 2021-07-09
Publication date: 2023-06-30
Anticipated expiration: 2041-07-09
Also published as: CN113596322A

Abstract

The application provides an image processing method, an image processing device, electronic equipment and a storage medium, wherein the electronic equipment comprises a lens, an image sensor, an optical anti-shake component and an electronic anti-shake component, wherein the optical anti-shake component is used for carrying out optical anti-shake of five-dimensional degrees of freedom on the lens and the image sensor according to motion data of the electronic equipment to obtain an optical anti-shake image; then, based on the optical anti-shake image, restoring and correcting the posture of the optical anti-shake image according to the first deflection posture data of the lens, the first position posture data and the first rotation posture data of the image sensor to obtain a posture corrected image after posture restoration; and finally, carrying out electronic anti-shake processing on the posture correction image through an electronic anti-shake component according to the motion data to obtain an electronic anti-shake image. Therefore, the optical anti-shake image is provided through the optical anti-shake assembly, and the electronic anti-shake assembly is used for carrying out electronic anti-shake processing after the posture change caused by the optical anti-shake is reduced, so that a high-quality electronic anti-shake image is obtained.

Description

Image processing method, device, electronic equipment and storage medium

Technical Field

The present disclosure relates to the field of image processing technologies, and in particular, to an image processing method, an image processing device, an electronic device, and a storage medium.

Background

Currently, electronic devices such as mobile phones and tablet computers are generally configured with cameras, so as to provide a photographing function for users, so that the users can record things happening around, scenes seen, and the like through the electronic devices at any time and any place. However, since the user usually holds the electronic device to take a photograph, the user holds the electronic device with a shake of different degrees to affect the stability of the photographing of the electronic device, resulting in poor quality of the photographed image.

Disclosure of Invention

The embodiment of the application provides an image processing method, an image processing device, electronic equipment and a storage medium, which can improve the image shooting quality of the electronic equipment.

The application discloses an image processing method, which is applied to electronic equipment, wherein the electronic equipment comprises a lens, an image sensor, an optical anti-shake component and an electronic anti-shake component, and the image processing method comprises the following steps:

acquiring motion data of electronic equipment, controlling a lens to perform compensation deflection in the X-axis direction and the Y-axis direction of the lens through an optical anti-shake assembly according to the motion data, controlling an image sensor to perform compensation translation in the X-axis direction and the Y-axis direction of the image sensor, and controlling the image sensor to perform compensation rotation around the Z-axis of the image sensor;

Acquiring an optical anti-shake image of corresponding motion data acquired by an image sensor, and acquiring first deflection attitude data of a lens, first position attitude data and first rotation attitude data of the image sensor from an optical anti-shake assembly;

restoring and correcting the posture of the optical anti-shake image according to the first deflection posture data, the first position posture data and the first rotation posture data to obtain a posture correction image;

and carrying out electronic anti-shake processing on the posture correction image through the electronic anti-shake component according to the motion data to obtain an electronic anti-shake image.

The application also discloses image processing device is applied to electronic equipment, and electronic equipment includes camera lens, image sensor, optics anti-shake subassembly and electron anti-shake subassembly, and image processing device includes:

the optical anti-shake module is used for acquiring motion data of the electronic equipment, controlling the lens to perform compensation deflection in the X-axis direction and the Y-axis direction of the lens through the optical anti-shake component, controlling the image sensor to perform compensation translation in the X-axis direction and the Y-axis direction of the image sensor, and controlling the image sensor to perform compensation rotation around the Z-axis of the image sensor;

the gesture acquisition module is used for acquiring an optical anti-shake image of corresponding motion data acquired by the image sensor, and acquiring first deflection gesture data of the lens, first position gesture data and first rotation gesture data of the image sensor from the optical anti-shake assembly;

The gesture correction module is used for restoring and correcting the gesture of the optical anti-shake image according to the first deflection gesture data, the first position gesture data and the first rotation gesture data to obtain a gesture correction image;

and the electronic anti-shake module is used for carrying out electronic anti-shake processing on the posture correction image through the electronic anti-shake component according to the motion data to obtain an electronic anti-shake image.

The application also discloses an electronic device comprising a lens, an image sensor, an optical anti-shake component, an electronic anti-shake component, a processor and a memory, wherein the memory stores a computer program, and the processor executes steps in the image processing method provided by the application by loading the computer program.

The present application also discloses a storage medium having stored thereon a computer program which, when loaded by a processor, performs steps in an image processing method as provided herein.

In the application, the electronic equipment comprises a lens, an image sensor, an optical anti-shake assembly and an electronic anti-shake assembly, wherein the optical anti-shake assembly is used for carrying out optical anti-shake of five-dimensional degrees of freedom on the lens and the image sensor according to motion data of the electronic equipment to obtain an optical anti-shake image; then, based on the optical anti-shake image, restoring and correcting the posture of the optical anti-shake image according to the first deflection posture data of the lens, the first position posture data and the first rotation posture data of the image sensor to obtain a posture corrected image after posture restoration; and finally, carrying out electronic anti-shake processing on the posture correction image through an electronic anti-shake component according to the motion data to obtain an electronic anti-shake image. Therefore, the optical anti-shake image is provided through the optical anti-shake assembly, and the electronic anti-shake assembly is used for carrying out electronic anti-shake processing after the posture change caused by the optical anti-shake is reduced, so that a high-quality electronic anti-shake image is obtained.

Drawings

In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings that are required to be used in the description of the embodiments will be briefly described below.

Fig. 1 is a schematic hardware structure of an electronic device according to an embodiment of the present application.

Fig. 2 is a flow chart of an image processing method according to an embodiment of the present application.

Fig. 3 is an exemplary diagram of a degree of freedom of movement of the camera module in the embodiment of the present application.

Fig. 4 is an exemplary diagram of compensating deflection of a lens in an embodiment of the present application.

Fig. 5 is an exemplary diagram of correcting an optical anti-shake image to obtain an attitude correction image in the embodiment of the present application.

Fig. 6 is a schematic partial view of a preset calibration tool according to an embodiment of the present application.

Fig. 7 is an example diagram of a rectangular calibration plate provided in an embodiment of the present application.

Fig. 8 is a schematic structural diagram of an image processing apparatus according to an embodiment of the present application.

Fig. 9 is a schematic diagram of another hardware structure of an electronic device according to an embodiment of the present application.

Detailed Description

It should be noted that the terms "first," "second," and "third," etc. in this application are used to distinguish between different objects and are not used to describe a particular order. Furthermore, the terms "comprise" and "have," as well as any variations thereof, are intended to cover a non-exclusive inclusion. For example, a process, method, system, article, or apparatus that comprises a list of steps or modules is not limited to the particular steps or modules listed and certain embodiments may include additional steps or modules not listed or inherent to such process, method, article, or apparatus.

Reference herein to "an embodiment" means that a particular feature, structure, or characteristic described in connection with the embodiment may be included in at least one embodiment of the present application. The appearances of such phrases in various places in the specification are not necessarily all referring to the same embodiment, nor are separate or alternative embodiments mutually exclusive of other embodiments. Those of skill in the art will explicitly and implicitly appreciate that the embodiments described herein may be combined with other embodiments.

The embodiment of the application provides an image processing method, an image processing apparatus, a computer-readable storage medium, and an electronic device, where an execution subject of the image processing method may be the image processing apparatus provided in the embodiment of the application, or the electronic device integrated with the image processing apparatus, where the image processing apparatus may be implemented in a hardware or software manner. The electronic device may be a device with a data processing capability, such as a smart phone, a tablet computer, a palm computer, a notebook computer, etc. configured with a processor.

The following description of the embodiments of the present application will be made clearly and fully with reference to the accompanying drawings, in which it is evident that the embodiments described are only some, but not all, of the embodiments of the present application. All other embodiments, which can be made by those skilled in the art based on the embodiments herein without making any inventive effort, are intended to be within the scope of the present application.

Referring to fig. 1, a hardware structure of an electronic device 100 is shown, and as shown in fig. 1, the electronic device 100 includes a motion sensor 110, a camera module 120, an optical anti-shake assembly 130, and an electronic anti-shake assembly 140. The positions of the motion sensor 110, the camera module 120, the optical anti-shake assembly 130, and the electronic anti-shake assembly 140 in the electronic device 100 are not particularly limited, and may be set by those skilled in the art according to actual needs. In addition, those skilled in the art will appreciate that the structure shown in FIG. 1 does not constitute a limitation of the electronic device 100, and the electronic device 100 may include more or fewer components than shown, or may combine certain components, or a different arrangement of components, etc.

The motion sensor 110 is configured to sense a motion state of the electronic device 100 in real time, and accordingly obtain motion data describing the motion state of the electronic device 100. The type and number of the motion sensors 110 are not particularly limited herein, and may be set by those skilled in the art according to actual needs. For example, the motion sensor 110 may sense only a motion state of the electronic device 100 in a single dimension, and accordingly obtain motion data in a single dimension (for example, the motion sensor 110 may sense only an acceleration of the electronic device 100), or may sense a motion state of the electronic device 100 in multiple dimensions, and accordingly obtain motion data in multiple dimensions (for example, the motion sensor 110 may sense an acceleration and an angular velocity of the electronic device 100). It should be noted that, with the motion sensor 110 rigidly connected to the body of the electronic device 100 as a constraint, the motion sensor 110 may be disposed at any position of the electronic device 100.

The camera module 120 is configured to collect an image, and at least includes a lens 1210 and an image sensor 1220, wherein the lens 1210 is configured to project an external optical signal to the image sensor 1220, and the image sensor 1220 is configured to photoelectrically convert the optical signal projected by the lens 1210 into a usable electrical signal, so as to obtain a digitized image. The arrangement of the lens 1210 and the image sensor 1220 is not particularly limited, and for example, the lens 1210 and the image sensor 1220 may be arranged in parallel or may be arranged in non-parallel.

After the camera module 120 is enabled, the shooting scene can be shot in real time. The shooting scene may be understood as a real area where the camera module 120 is aligned after being enabled, i.e. an area where the camera module 120 can convert an optical signal into a corresponding image. For example, after the image capturing module 120 is enabled according to the user operation, if the user controls the image capturing module 120 of the electronic device 100 to aim at an area including a certain object, the area including the object is the shooting scene of the image capturing module 120. The image capturing module 120 is configured to be movable relative to the electronic device 100, that is, the image capturing module 120 has a certain degree of freedom of movement relative to the electronic device 100 (may be that the lens 1210 and the image sensor 1220 both have the degree of freedom of movement, or that one of the lens 1210 and the image sensor 1220 both have the degree of freedom of movement), so that when the electronic device 100 moves, the image capturing module 120 can be driven to perform compensation movement to offset the movement of the electronic device 100 as much as possible, so that the imaging light path is stable.

It should be noted that, the setting position of the camera module 120 in the electronic device 100 is not limited in this embodiment, and may be configured by those skilled in the art according to actual needs. Taking a mobile phone as an example, the camera module 120 can be arranged on one surface of the mobile phone screen, the camera module 120 can be arranged on the opposite surface of the mobile phone screen, and the camera module 120 can be arranged on both the opposite surface and the opposite surface of the mobile phone screen.

The optical anti-shake assembly 130 is configured to perform an optical anti-shake process on the camera module 120 according to the movement of the electronic device 100. For example, when the lens 1210 has a degree of freedom of deflection in the X-axis direction and the Y-axis direction, the optical anti-shake assembly 130 can correspondingly control the lens 1210 to perform compensation deflection in the X-axis direction and the Y-axis direction according to the movement of the electronic device 100, so as to offset the movement of the electronic device 100, and stabilize the imaging light path of the imaging module 120.

The electronic anti-shake component 140 is configured to perform anti-shake on an image in an electronic anti-shake manner, where the electronic anti-shake is an algorithm operation, for example, motion conditions between the image and other images and motion conditions inside the image are calculated according to motion data corresponding to the image, and after the images are aligned according to the motion conditions, appropriate clipping, stretching, deformation and other processes are performed to obtain a relatively stable image.

Referring to fig. 2, fig. 2 is a flowchart of an image processing method according to an embodiment of the present application. The image processing method is applied to the electronic device 100 shown in fig. 1, where the electronic device 100 includes a lens 1210, an image sensor 1220, an optical anti-shake component 130 and an electronic anti-shake component 140, as shown in fig. 2, the flow of the image processing method provided in the embodiment of the present application may be as follows:

at 210, motion data of the electronic device 100 is acquired, and based on the motion data, the optical anti-shake assembly 130 is used to control the lens 1210 to perform compensation deflection in an X-axis direction and a Y-axis direction of the lens 1210, to control the image sensor 1220 to perform compensation translation in the X-axis direction and the Y-axis direction of the image sensor 1220, and to control the image sensor 1220 to perform compensation rotation around a Z-axis of the image sensor 1220.

Referring to fig. 3, in the present embodiment, the image capturing module 120 has 5 degrees of freedom, including 2 degrees of freedom of deflection (or Tilt) of the lens 1210 in the X-axis direction and the Y-axis direction, 2 degrees of freedom of translation of the image sensor 1220 in the X-axis direction and the Y-axis direction, and 1 degree of freedom of rotation of the image sensor 1220 in the Z-axis direction. Accordingly, the optical anti-shake assembly 130 is configured to optically anti-shake in the aforementioned 5-dimensional freedom of movement. It should be noted that, in the present embodiment, the structural layout of the electronic device 100, the lens 1210 and the image sensor 1220 is not limited, and may be configured by those skilled in the art according to actual needs, for example, the X-axis of the electronic device 100, the lens 1210 and the image sensor 1220 may be parallel to each other, the Y-axis may be parallel to each other, and the Z-axis may be parallel to each other.

Based on the above optical anti-shake component 130, in the present embodiment, the motion data of the electronic device 100 is first acquired by the motion sensor 110 of the electronic device 100. The type and number of the motion sensors 110 are not particularly limited herein, and may be set by those skilled in the art according to actual needs. For example, the three-axis acceleration of the electronic apparatus 100 may be obtained by a three-axis acceleration sensor, the three-axis angular velocity of the electronic apparatus 100 may be obtained by a three-axis gyroscope, and the like.

As described above, after the motion data of the electronic device 100 is acquired, the motion data of the electronic device 100 is input into the optical anti-shake component 130, and the lens 1210 and the image sensor 1220 are driven by the optical anti-shake component 130 to perform optical anti-shake according to the motion data of the electronic device 100. According to the motion data of the electronic device 100, the optical anti-shake assembly 130 controls the lens 1210 to perform compensation deflection in the X-axis direction and the Y-axis direction of the lens 1210, controls the image sensor 1220 to perform compensation translation in the X-axis direction and the Y-axis direction of the image sensor 1220, and controls the image sensor 1220 to perform compensation rotation around the Z-axis of the image sensor 1220, so as to offset the motion of the electronic device 100, so that the imaging light path is stable. For example, referring to fig. 4, in the embodiment of the present application, during the deflection of the lens 1210, the plane of the lens 1210 is kept parallel to the sensor plane of the image sensor 1220 based on the control of the optical anti-shake component 130.

At 220, an optical anti-shake image of corresponding motion data acquired by the image sensor 1220 is acquired, and first yaw pose data of the lens 1210, first position pose data of the image sensor 1220, and first rotation pose data are acquired from the optical anti-shake assembly 130.

As can be seen from the above related description, the present application performs optical anti-shake on the lens 1210 and the image sensor 1220 through the optical anti-shake component 130, so that the imaging light path of the image sensor 1220 is stable, and accordingly, when the lens 1210 completes compensating deflection and the image sensor 1220 completes compensating translation and compensating rotation, the present embodiment obtains the image collected by the image sensor 1220 at this time, and records the image as an optical anti-shake image corresponding to the motion data.

It should be noted that a deflection posture sensor for sensing a deflection posture of the lens 1210 in real time, a position posture sensor for sensing a position posture of the image sensor 1220 in real time, and a rotation posture sensor for sensing a rotation posture of the image sensor 1220 in real time are also provided in the optical anti-shake assembly 130. The types of the yaw attitude sensor, the position attitude sensor, and the rotation attitude sensor are not particularly limited herein, and may be selected as required by those skilled in the art. For example, in this embodiment, the yaw attitude sensor, the position attitude sensor, and the rotation attitude sensor are all implemented by hall sensors.

In this embodiment, in addition to the optical anti-shake image of the corresponding motion data acquired by the image sensor 1220, the first deflection posture data of the lens 1210, the first position posture data of the image sensor 1220, and the first rotation posture data are acquired from the optical anti-shake assembly 130. For example, the optical anti-shake component 130 senses the deflection gesture of the lens 1210 through a deflection gesture sensor to obtain first deflection gesture data of the lens 1210, senses the position gesture of the image sensor 1220 through a position gesture sensor to obtain first position gesture data of the image sensor 1220, senses the rotation gesture of the image sensor 1220 through a rotation gesture sensor to obtain first rotation gesture data of the image sensor 1220, and combines the first deflection gesture data, the first position gesture data and the first rotation gesture data into a gesture data set to be output. Accordingly, the first yaw attitude data, the first position attitude data, and the first rotation attitude data may be obtained from the attitude data set output from the optical anti-shake assembly 130.

For example, the gesture data set may be represented as h= [ H ] ₀ |h ₁ |h ₂ ]Wherein h is ₀ Representing first deflection attitude data, h ₁ Representing first position and orientation data, h ₂ Representing first rotational gesture data.

At 230, the pose of the optical anti-shake image is restored and corrected according to the first deflection pose data, the first position pose data, and the first rotation pose data, to obtain a pose corrected image.

In this embodiment, after the first yaw attitude data, the first position attitude data, and the first rotation attitude data are obtained, according to a configured correction policy, the attitude of the optical anti-shake image is restored and corrected according to the first yaw attitude data, the first position attitude data, and the first rotation attitude data, so as to obtain an attitude correction image. The restoration correction of the posture is generally understood as "restoring" the optical anti-shake image to the posture where the optical anti-shake assembly 130 does not perform optical anti-shake, that is, to the posture corresponding to the situation where the optical anti-shake image is "restored" to the lens 1210 without deflection and the image sensor 1220 does not translate or rotate.

In 240, the gesture correction image is subjected to an electronic anti-shake process by the electronic anti-shake component 140 according to the motion data, so as to obtain an electronic anti-shake image.

In this embodiment, after the correction of the optical anti-shake image is completed and the posture correction image is obtained, the motion data and the posture correction image are further input into the electronic anti-shake component 140, and the electronic anti-shake component 140 performs the electronic anti-shake processing on the posture correction image according to the motion data, so as to obtain the electronic anti-shake image.

As can be seen from the above, in the present application, the electronic device 100 includes a lens 1210, an image sensor 1220, an optical anti-shake component 130 and an electronic anti-shake component 140, wherein the optical anti-shake component 130 performs optical anti-shake with five-dimensional degrees of freedom on the lens 1210 and the image sensor 1220 according to the motion data of the electronic device 100 to obtain an optical anti-shake image; then, based on the optical anti-shake image, the posture of the optical anti-shake image is restored and corrected according to the first deflection posture data of the lens 1210, the first position posture data and the first rotation posture data of the image sensor 1220, to obtain a posture-corrected image after posture restoration; finally, according to the motion data, the electronic anti-shake component 140 performs electronic anti-shake processing on the posture correction image to obtain an electronic anti-shake image. In this way, the present application provides an optical anti-shake image through the optical anti-shake assembly 130, and the electronic anti-shake assembly 140 performs the electronic anti-shake processing after the posture change caused by the optical anti-shake is reduced, so as to obtain a high-quality electronic anti-shake image.

Optionally, in an embodiment, restoring and correcting the posture of the optical anti-shake image according to the first deflection posture data, the first position posture data and the first rotation posture data to obtain a corrected image includes:

(1) Acquiring a first correction rotation matrix for performing rotation correction on the optical anti-shake image according to the first deflection posture data and the first rotation posture data;

(2) Performing rotation correction on the posture of the optical anti-shake image according to the first correction rotation matrix to obtain a rotation correction image;

(3) Acquiring a first correction translation vector for carrying out translation correction on the rotation correction image according to the first position and posture data;

(4) And carrying out translational correction on the posture of the rotation correction image according to the first correction translational vector to obtain a posture correction image.

As can be seen from the above description, the image capturing module 120 in the present embodiment has 5 degrees of freedom of movement, namely, 2 degrees of freedom of deflection of the lens 1210 in the X-axis direction and the Y-axis direction, 2 degrees of freedom of translation of the image sensor 1220 in the X-axis direction and the Y-axis direction, and 1 degree of freedom of rotation of the image sensor 1220 about the Z-axis. Wherein a 2-yaw-dimension degree of freedom of the lens 1210 in the X-axis direction and the Y-axis direction, and a 1-dimensional rotational degree of freedom of the image sensor 1220 about the Z-axis will result in rotation of the image content, and a 2-dimensional translational degree of freedom of the image sensor 1220 in the X-axis direction and the Y-axis direction will result in translation of the image content. Accordingly, the restoring correction of the optical anti-shake image posture in this embodiment will include two parts, namely, translational correction and rotational correction.

As above, rotation of the image content is caused by a combination of deflection of the lens 1210 and rotation of the image sensor 1220, and translation of the image content is caused by translation of the image sensor 1220. Referring to fig. 5, in the present embodiment, a first correction rotation matrix for rotationally correcting an optical anti-shake image is obtained according to first deflection gesture data of a lens 1210 and first rotation gesture data of an image sensor 1220, and the gesture of the optical anti-shake image is rotationally corrected according to the first correction rotation matrix, and the rotationally corrected optical anti-shake image is recorded as a rotationally corrected image. After the rotation correction image is obtained, a first correction translation vector for performing translation correction on the rotation correction image is further obtained according to the first position and posture data of the image sensor 1220, and the rotation correction image after the translation correction is the posture correction image according to the first correction translation vector.

Optionally, in an embodiment, acquiring a first correction rotation matrix for rotationally correcting the optical anti-shake image according to the first deflection gesture data and the first rotation gesture data includes:

(1) Acquiring a first deflection angle corresponding to the first deflection attitude data according to a first functional relation between the first deflection attitude data and the deflection angle;

(2) Acquiring a second deflection angle corresponding to the first deflection attitude data according to a second functional relation between the first deflection attitude data and the deflection angle;

(3) Acquiring a first rotation angle corresponding to the first rotation gesture data according to a third functional relation between the first rotation gesture data and the rotation angle;

(4) And acquiring a first correction rotation matrix according to the first deflection angle, the second deflection angle and the first rotation angle.

It should be noted that the present embodiment presets a first functional relationship of the first deflection posture data and the deflection angle, which describes a correspondence relationship between the first deflection posture data of the lens 1210 and the deflection angle thereof in the X-axis. In addition, the second functional relationship between the first deflection posture data and the deflection angle of the lens 1210 is also preset in this embodiment, and the second functional relationship describes the correspondence relationship between the first deflection posture data and the deflection angle of the lens 1210 in the Y axis. In addition, the present embodiment is also preset with a third functional relationship of the first rotation posture data and the rotation angle, which describes a correspondence relationship between the first rotation posture data of the image sensor 1220 and the rotation angle thereof around the Z axis.

Based on the first functional relationship, the second functional relationship and the third functional relationship, when the first correction rotation matrix for performing rotation correction on the optical anti-shake image is obtained according to the first deflection posture data and the first rotation posture data, the first deflection angle corresponding to the first deflection posture data is obtained according to the first functional relationship of the first deflection posture data and the deflection angle, the second deflection angle corresponding to the first deflection posture data is obtained according to the second functional relationship of the first deflection posture data and the deflection angle, and the first rotation angle corresponding to the first rotation posture data is obtained according to the third functional relationship of the first rotation posture data and the rotation angle.

As described above, after the first deflection angle, the second deflection angle, and the first rotation angle are acquired, a first correction rotation matrix is further acquired based on the first deflection angle, the second deflection angle, and the first rotation angle.

Optionally, in an embodiment, obtaining the first correction rotation matrix according to the first deflection angle, the second deflection angle, and the first rotation angle includes:

According to the first deflection angle, the second deflection angle and the first rotation angle, a first correction rotation matrix is obtained according to the following formula:

R＝R _X ·R _Y ·R _Z ；

wherein R represents a first correction rotation matrix, C represents cos (), S represents sin (), α represents a first deflection angle, β represents a second deflection angle, and γ represents a first rotation angle.

It should be noted that this embodiment is exemplified by the outward rotation of X-Y-Z, in other embodiments R _X 、R _Y And R is _Z Can be determined by one skilled in the art based on the specific driving mode of the deflection of the lens 1210 and the rotation of the image sensor 1220.

Optionally, in an embodiment, performing rotation correction on the pose of the optical anti-shake image according to the first correction rotation matrix to obtain a rotation corrected image, including:

(1) Acquiring the deflection center coordinates of the deflection center of the compensating deflection of the lens 1210 under a pixel coordinate system;

(2) Converting the optical anti-shake image from a pixel coordinate system to an image coordinate system according to the deflection center coordinate;

(3) And carrying out rotation correction on the posture of the optical anti-shake image under an image coordinate system according to the first correction rotation matrix to obtain a rotation correction image.

The coordinates of the deflection center of the lens 1210 for compensating deflection in the pixel coordinate system can be understood as: when the lens 1210 compensates for the deflection, the resulting rotation center of the image content rotates coordinates in the pixel coordinate system. It should be noted that the deflection center coordinates are obtained by fitting in advance, and the specific fitting manner is not limited herein, and a suitable fitting manner may be selected by those skilled in the art according to actual needs.

In this embodiment, when the rotation correction is performed on the posture of the optical anti-shake image according to the first correction rotation matrix to obtain the rotation correction image, the deflection center coordinates of the deflection center of the lens 1210 for compensating deflection in the pixel coordinate system are first obtained, and then the optical anti-shake image is converted from the pixel coordinate system to the image coordinate system according to the deflection center coordinates, which may be expressed as:

(u, v) represents the position coordinates of a pixel point in the optical anti-shake image in the pixel coordinate system, (c) _u ，c _v ) The deflection center coordinates are represented, and p' represents the position coordinates of the pixel point under the conversion to the image coordinate system.

As described above, after the optical anti-shake image is converted from the pixel coordinate system to the image coordinate system, the posture of the optical anti-shake image may be rotationally corrected according to the first correction rotation matrix in the image coordinate system, so as to obtain a rotation correction image, which may be expressed as:

P' _p ＝R·P _p ；

wherein P is _p Representing the position of a pixel point in the optical anti-shake image under the image coordinate system, R represents a first correction rotation matrix, and P' _p The position (in the image coordinate system) of the pixel point in the rotation-corrected image after the rotation correction is performed is indicated.

Optionally, in an embodiment, performing translational correction on the posture of the rotation correction image according to the first correction translational vector to obtain a posture correction image, including:

(1) Acquiring rotation center coordinates of the rotation center of the image sensor 1220 for compensation rotation in a pixel coordinate system;

(2) Acquiring a correction vector for correcting the first correction translation vector according to the deflection center coordinate and the rotation center coordinate;

(3) Correcting the first correction translation vector according to the correction vector to obtain a corrected first correction translation vector;

(4) And carrying out translational correction on the posture of the rotation correction image according to the corrected first correction translational vector to obtain a posture correction image.

The rotation center coordinates of the rotation center of the image sensor 1220 for the compensation rotation in the pixel coordinate system can be understood as: when the lens 1210 performs the compensation rotation, the rotation center of the image content rotation is caused to be the coordinates in the pixel coordinate system. It should be noted that, due to the assembly of the lens 1210 and the image sensor 1220, the rotation center of the image sensor 1220 and the deflection center of the lens 1210 are not consistent, so the coordinates of the rotation center are also obtained by fitting in advance in this embodiment, the specific fitting method is not limited, and a suitable fitting method can be selected by a person skilled in the art according to actual needs.

In the embodiment, when performing translational correction on the posture of the rotation correction image according to the first correction translational vector to obtain the posture correction image, the rotation center coordinate of the rotation center of the image sensor 1220 for compensation rotation in the pixel coordinate system is first obtained; then, a vector between the deflection center coordinates and the rotation center coordinates is obtained and used as a correction vector for correcting the first correction translation vector; then, the first correction translation vector is corrected according to the correction vector, so that the translation error caused by the inconsistency between the rotation center of the image sensor 1220 and the deflection center of the lens 1210 is eliminated, and a corrected first correction translation vector is obtained; and finally, carrying out translational correction on the posture of the rotation correction image according to the corrected first correction translational vector to obtain a posture correction image.

The present embodiment can improve the accuracy of the translational correction by eliminating the translational error caused by the inconsistent rotation center of the image sensor 1220 and the deflection center of the lens 1210.

Optionally, in an embodiment, the electronic device 100 is fixed to the preset calibration tool, so that the electronic device 100 is relatively fixed to a position of a rectangular calibration plate in the preset calibration tool, the rectangular calibration plate includes a center mark, and the deflection center coordinates, the first functional relationship, and the second functional relationship are determined according to the following steps:

(1) The lens 1210 of the optical anti-shake assembly 130 is turned on to deflect the anti-shake function, and the image sensor 1220 is turned off to translate the anti-shake function and the image sensor 1220 is turned on to rotate the anti-shake function;

(2) Transmitting a first driving control instruction to a preset calibration tool, so that the preset calibration tool drives the electronic device 100 to translate in the X-axis direction and the Y-axis direction of the electronic device 100;

(3) During the translation process of the electronic device 100, performing optical anti-shake processing through the optical anti-shake component 130, and acquiring a first image sequence of the rectangular calibration plate acquired by the image sensor 1220 during the translation process;

(4) Sending a stop instruction to a preset calibration tool, so that the preset calibration tool stops driving the electronic device 100 to translate, and acquiring a second image sequence of the rectangular calibration plate acquired by the image sensor 1220 in a static process;

(5) Acquiring a first position coordinate of a center mark in each first image in the first image sequence and a second position coordinate of the center mark in each second image in the second image sequence under a pixel coordinate system;

(6) Acquiring second deflection pose data of lens 1210 at the time of acquisition of each first image of the first image sequence by image sensor 1220 from optical anti-shake assembly 130, and determining an average coordinate of all second position coordinates;

(7) And fitting according to the average coordinates, the first position coordinates corresponding to each first image and the second deflection gesture data to obtain a deflection center coordinate, a first functional relation and a second functional relation.

In this embodiment, a preset calibration tool is used to obtain a first functional relationship and a second functional relationship through pre-calibration.

The electronic device 100 is first fixed to a preset calibration tool, so that the positions of the electronic device 100 and a rectangular calibration plate in the preset calibration tool are relatively fixed. It should be noted that the structure of the preset calibration tool is not particularly limited herein, and may be set by those skilled in the art according to actual needs.

For example, referring to fig. 6, the preset calibration tool is composed of a target backboard, an equipment carrier, a slide rail and a driving mechanism (not shown in fig. 6), wherein the target backboard is used for fixing the rectangular calibration board, the equipment carrier is used for fixing the electronic equipment 100, and the target backboard and the equipment carrier can slide along the track of the slide rail.

Based on the preset calibration tool shown in fig. 6, the present embodiment fixes the electronic device 100 to the device carrier, and adjusts the positions of the device carrier and the target backboard along the respective tracks, so that the imaging size of the rectangular calibration board on the image sensor 1220 is consistent with the imaging area of the image sensor 1220, and in colloquial terms, the imaging image of the rectangular calibration board is fully paved on the imaging area of the image sensor 1220.

After the position adjustment of the device carrier and the target backboard is completed, the device carrier and the target backboard are locked at their respective current positions, so that the positions of the electronic device 100 and the rectangular calibration board are relatively fixed.

It should be noted that, in order to ensure that the positions of the electronic device 100 and the rectangular calibration plate are relatively fixed, so as to ensure a parallel relationship between the rectangular calibration plate and the camera module 120 (the lens 1210 and the image sensor 1220) of the electronic device 100, the target back plate is made of a hard plate material, such as a hard metal plate material. In addition, the equipment carrier and the target backboard are required to be rigidly connected with the slide rails where the equipment carrier and the target backboard are located, so that disturbance errors are avoided.

As described above, after the electronic device 100 is fixed to the preset calibration tool and the positions of the electronic device 100 and the rectangular calibration plate in the preset calibration tool are relatively fixed, the deflection center coordinates, the first functional relationship, and the second functional relationship can be obtained by using the preset calibration tool, that is, the deflection anti-shake function of the lens 1210 of the optical anti-shake assembly 130 is calibrated.

The lens 1210 of the optical anti-shake assembly 130 is correspondingly turned on to deflect the anti-shake function, and the image sensor 1220 is turned off to translate the anti-shake function and the image sensor 1220 is turned on to rotate the anti-shake function, so that the optical anti-shake assembly 130 only compensates for deflection of the lens 1210 according to the movement of the electronic device 100 when the electronic device 100 moves. Correspondingly, a first driving control instruction is sent to the preset calibration tool, so that the preset calibration tool drives the electronic device 100 to translate in the X-axis direction and the Y-axis direction of the electronic device 100. For example, the first driving control command causes the preset calibration tool to drive the electronic device 100 to alternately translate in the X-axis direction and the Y-axis direction, as shown in fig. 6, where the distance of translation and the frequency of alternation can be determined by those skilled in the art according to actual needs, and the present invention is not limited thereto.

During the translation process of the electronic device 100, the motion data of the electronic device 100 is acquired in real time and provided to the optical anti-shake component 130, so that the optical anti-shake processing is performed through the optical anti-shake component 130. It can be appreciated that, since the present embodiment only turns on the lens 1210 of the optical anti-shake assembly 130 to deflect the anti-shake function, the optical anti-shake assembly 130 only compensates for the deflection of the lens 1210 during the movement of the electronic device 100. Wherein, a sequence of images of the rectangular calibration plate acquired by the image sensor 1220 during translation is acquired and denoted as a first sequence of images. For example, a first image sequence is obtained by image sensor 1220 acquiring an image sequence of a rectangular calibration plate for a period of one minute during translation.

In addition, a stop command is sent to the preset calibration tool, so that the preset calibration tool stops driving the electronic device 100 to translate, and an image sequence of the rectangular calibration plate acquired by the image sensor 1220 in the static process is acquired and recorded as a second image sequence. It should be noted that, in other embodiments, the second image sequence of the rectangular calibration plate acquired by the image sensor 1220 during the rest process may be acquired first, and then the first image sequence of the rectangular calibration plate acquired by the image sensor 1220 during the translation process may be acquired through the translation of the driving electronic device 100 according to the preset calibration.

Referring to fig. 7, the rectangular calibration plate includes a center mark and four corner marks (an upper left corner mark, an upper right corner mark, a lower right corner mark and a lower left corner mark, respectively), and since the positions of the electronic device 100 and the rectangular calibration plate in the predetermined calibration tool are relatively fixed, the optical anti-shake processing performed by the optical anti-shake component 130 due to the translation of the electronic device 100 will cause the position of the center mark in the first image obtained by imaging to deviate from the center of the first image.

Accordingly, in this embodiment, further under the pixel coordinate system, the first position coordinate of the center mark in each first image in the first image sequence is acquired, the second position coordinate of the center mark in each second image in the second image sequence is acquired, the average coordinates of all the second position coordinates are determined, and in addition, the deflection gesture data of the lens 1210 when the image sensor 1220 acquires each image in the first image sequence is acquired from the optical anti-shake component 130 and is recorded as the second deflection gesture data. And then, performing function fitting according to the average coordinates, the first position coordinates corresponding to each first image and the second deflection posture data, and fitting to obtain the deflection center coordinates, the first function relation and the second function relation. The manner of function fitting is not particularly limited herein, and includes, but is not limited to, least squares, random sample consensus, polynomial fitting, and the like. For example, the present embodiment uses a least square method for function fitting.

In other embodiments, after the above function fitting is completed, the fitting error is further evaluated (the evaluation mode may be selected by a person skilled in the art according to actual needs, and no specific limitation is made here, for example, a mean square error may be adopted), if the fitting error is greater than a first error threshold (an empirical value may be selected by a person skilled in the art according to actual needs), the second deflection gesture data and the first position coordinates with obvious anomalies are identified and removed, and then fitting is performed again, if the new fitting error is still greater than the first error threshold, then fitting is performed again by acquiring the new first image sequence, the second image sequence and the new second deflection gesture data until the fitting error is less than or equal to the first error threshold.

Optionally, in an embodiment, fitting to obtain the deflection center coordinate, the first functional relationship, and the second functional relationship according to the average coordinate and the first position coordinate and the second deflection posture data corresponding to each first image includes:

according to the average coordinates, the first position coordinates and the second deflection posture data corresponding to each first image, a deflection center coordinate, a first function relation and a second function relation are obtained through fitting according to the following formula:

min‖P ₁ -f(p ₀ ,h ₀ '；θ)‖ ₂ ；

θ＝[C _x1 ,C _y1 ,θ ₀ ,θ ₁ ]；

Wherein f () represents the first polynomial function or the second polynomial function, P ₁ Representing the first position coordinates, p ₀ Represents the average coordinates, h ₀ ' represents second deflection attitude data, C _x1 Represents the abscissa in the center of deflection coordinates, C _y1 Representing the ordinate, θ, in the center of deflection coordinates ₀ Polynomial coefficients, θ, representing a first polynomial function ₁ Polynomial coefficients representing the second polynomial function.

It should be noted that f () can be regarded as a projection function, which will p ₀ Projected to h ₀ ' corresponding gesture P ₁ In fitting f (), a suitable loss function can be constructed by those skilled in the art according to actual needs, for example, the loss function is constructed according to the reprojection error in the present embodiment.

In addition, in this embodiment, the degree of the first polynomial function and the second polynomial function is not particularly limited, and may be selected by those skilled in the art according to actual needs, for example, in this embodiment, the first polynomial function and the second polynomial function both use 3 th degree polynomial functions.

Optionally, in an embodiment, the electronic device 100 is fixed to the preset calibration tool, so that the electronic device 100 is relatively fixed to a position of a rectangular calibration plate in the preset calibration tool, the rectangular calibration plate includes four corner marks, and the rotation center coordinate and the third functional relationship are determined according to the following steps:

(1) Turning on the image sensor 1220 of the optical anti-shake assembly 130 to rotate the anti-shake function, and turning off the deflection anti-shake function of the lens 1210 and the translation anti-shake function of the image sensor 1220;

(2) Sending a second driving control instruction to the preset calibration tool, so that the preset calibration tool drives the electronic device 100 to rotate around the Z axis of the image sensor 1220;

(3) During the rotation process of the electronic device 100, performing optical anti-shake processing through the optical anti-shake component 130, and acquiring a third image sequence of the rectangular calibration plate acquired by the image sensor 1220 during the rotation process;

(4) Acquiring second rotational pose data from the optical anti-shake assembly 130 when the image sensor 1220 acquires each third image in the third image sequence, and acquiring third position coordinates of the four corner markers in each third image in the pixel coordinate system

(5) And fitting according to the second rotation gesture data and the third position coordinates corresponding to each third image to obtain a rotation center coordinate and a third functional relation.

In this embodiment, the rotation center coordinate and the third functional relationship are also obtained by pre-calibrating with a pre-set calibration tool, that is, the rotation anti-shake function of the image sensor 1220 of the optical anti-shake assembly 130 is calibrated. Also, the electronic device 100 is fixed to the preset calibration tool in the above manner such that the positions of the electronic device 100 and the rectangular calibration plate in the preset calibration tool are relatively fixed. Reference may be made specifically to the relevant descriptions in the above embodiments, and details are not repeated here.

After the electronic device 100 is fixed on the preset calibration tool and the positions of the electronic device 100 and the rectangular calibration plate in the preset calibration tool are relatively fixed, the rotation center coordinate and the third functional relationship can be obtained by using the preset calibration tool.

Since the rotation center coordinate and the third functional relationship are calibrated at this time, the rotation anti-shake function of the image sensor 1220 of the optical anti-shake assembly 130 is correspondingly turned on, and the deflection anti-shake function of the lens 1210 and the translation anti-shake function of the image sensor 1220 are turned off, so that the optical anti-shake assembly 130 only performs the compensation rotation on the image sensor 1220 according to the movement of the electronic device 100 when the electronic device 100 moves. Accordingly, a second driving control command is sent to the preset calibration tool, so that the preset calibration tool drives the electronic device 100 to rotate around the Z axis of the image sensor 1220. For example, the second driving control command causes the preset calibration tool to drive the electronic device 100 to alternately rotate around the Z-axis of the image sensor 1220, wherein the angular speed of rotation and the frequency of alternation can be determined by those skilled in the art according to actual needs, and the present invention is not limited thereto.

During the rotation of the electronic device 100, motion data of the electronic device 100 is acquired in real time and provided to the optical anti-shake assembly 130, so that an optical anti-shake process is performed through the optical anti-shake assembly 130. It can be appreciated that, since the present embodiment only turns on the rotation anti-shake function of the image sensor 1220 of the optical anti-shake assembly 130, the optical anti-shake assembly 130 will only perform the compensation rotation of the image sensor 1220 during the movement of the electronic apparatus 100. In addition, a sequence of images of the rectangular calibration plate acquired by the image sensor 1220 during rotation is acquired and noted as a third sequence of images. For example, a third image sequence is acquired by image sensor 1220 during rotation of the rectangular calibration plate for a period of one minute.

In this embodiment, the image sensor 1220 also acquires rotational pose data of each third image in the third image sequence from the optical anti-shake assembly 130, and records the rotational pose data as second rotational pose data. Further, the position coordinates of the four corner marks (upper left corner mark, upper right corner mark, lower right corner mark, and lower left corner mark) in each third image are also acquired under the pixel coordinate system, and are noted as third position coordinates.

It will be appreciated that for a third image, if the rotation correction is performed on the four-corner mark using the first correction rotation matrix, the corrected four-corner mark should be located at the corresponding four-corner position of the corrected third image. Therefore, fitting is performed according to the second rotation gesture data and the third position coordinates corresponding to each third image, and then the rotation center coordinates and the third functional relation can be obtained through fitting.

In other embodiments, after the fitting to obtain the rotation center coordinate and the third functional relationship, the fitting error of the rotation center coordinate and the third functional relationship is further evaluated (the evaluation mode may be selected by a person skilled in the art according to actual needs, for example, a reprojection error may be used to obtain the fitting error of the rotation center coordinate and the third functional relationship, if the fitting error of the rotation center coordinate and the third functional relationship is greater than the second error threshold (an empirical value may be taken by a person skilled in the art according to actual needs), the second rotation gesture data and the third position coordinate with obvious anomalies are identified and removed, and then fitting is performed again, if the fitting error of the newly fitted rotation center coordinate and the third functional relationship is still greater than the second error threshold, fitting is performed again to obtain a new third image sequence and new second rotation gesture data and third position coordinate until the fitting error is less than or equal to the third functional relationship of the second error threshold.

Optionally, in an embodiment, the third functional relationship includes a third polynomial function, and fitting to obtain a rotation center coordinate and a third functional relationship according to the second rotation posture data and the third position coordinate corresponding to each third image includes:

and according to the second rotation gesture data and the third position coordinates corresponding to each third image, fitting according to the following formula to obtain a rotation center coordinate and a third functional relation:

θ'＝[C _x2 ,C _y2 ,θ ₂ ]；

imW/2-r<c _x <imW/2+r

imH/2-r<c _y <imH/2+r；

wherein P is _i A third position coordinate indicating the ith four-corner mark, h ₂ 'represents the second rotational gesture data, f' () represents the third polynomial function, r represents the preset search radius, θ ₂ Polynomial coefficients representing a third polynomial function, C _x2 Represents the abscissa in the rotation center coordinates, C _y2 Representing the ordinate in the rotation center coordinates, imW representing the width of the third image and imH representing the height of the third image.

It should be noted that when fitting f' (), a suitable loss function can be constructed by those skilled in the art according to actual needs, for example, by those skilled in the art according to actual geometric relationships, including but not limited to chord length relationships or rotational relationships, and the like.

In addition, the value of the preset search radius r is not particularly limited in this embodiment, and may be determined by those skilled in the art according to actual needs.

In addition, the degree of the third polynomial function is not particularly limited in this embodiment, and may be selected by those skilled in the art according to actual needs, for example, in this embodiment, the third polynomial function uses a 3-degree polynomial function.

Optionally, in an embodiment, acquiring a first correction translation vector for performing translation correction on the rotation correction image according to the first position and orientation data includes:

and taking the translation vector corresponding to the first position and posture data as a first correction translation vector according to a fourth functional relation of the first position and posture data and the translation vector.

It should be noted that the present embodiment presets a fourth functional relationship of the first position posture data and the translation vector, which describes a correspondence relationship between the position posture change of the image content caused by the position posture change of the lens 1210.

Based on the above fourth functional relationship, in the present embodiment, when a first correction translation vector for performing translation correction on a rotation correction image is obtained from first position and orientation data, the translation vector corresponding to the first position and orientation data is used as the first correction translation vector according to the fourth functional relationship of the first position and orientation data and the translation vector.

Optionally, in an embodiment, the electronic device 100 is fixed to the preset calibration tool, so that the electronic device 100 is relatively fixed to a position of a rectangular calibration plate in the preset calibration tool, the rectangular calibration plate includes a center mark, and the fourth functional relation is determined according to the following steps:

(1) Turning on the image sensor 1220 of the optical anti-shake unit 130 to translate the anti-shake function, and turning off the deflection anti-shake function of the lens 1210 and the rotation anti-shake function of the image sensor 1220;

(2) Transmitting a third driving control instruction to a preset calibration tool, so that the preset calibration tool drives the electronic device 100 to translate in the X-axis direction and the Y-axis direction of the electronic device 100;

(3) During the translation process of the electronic device 100, performing optical anti-shake processing through the optical anti-shake component 130, and acquiring a fourth image sequence of the rectangular calibration plate acquired by the image sensor 1220 during the translation process;

(4) Acquiring second position and orientation data from the optical anti-shake component 130 when the image sensor 1220 acquires each fourth image in the fourth image sequence, and acquiring fourth position coordinates of the center mark in each fourth image in the pixel coordinate system;

(5) And fitting according to the second position and posture data and the fourth position coordinates corresponding to each first image to obtain a fourth functional relation.

In this embodiment, a fourth functional relationship is obtained by pre-calibrating with a preset calibration tool, that is, calibrating the translational anti-shake function of the image sensor 1220 of the optical anti-shake assembly 130. Also, the electronic device 100 is fixed to the preset calibration tool in the above manner such that the positions of the electronic device 100 and the rectangular calibration plate in the preset calibration tool are relatively fixed. Reference may be made specifically to the relevant descriptions in the above embodiments, and details are not repeated here.

After the electronic device 100 is fixed to the preset calibration tool and the positions of the electronic device 100 and the rectangular calibration plate in the preset calibration tool are relatively fixed, the fourth functional relationship can be obtained by using the preset calibration tool for calibration.

Since only the fourth functional relationship is calibrated at this time, the image sensor 1220 of the optical anti-shake assembly 130 is correspondingly turned on to perform the translational anti-shake function, and the lens 1210 is turned off to deflect the anti-shake function and the image sensor 1220 is turned on to perform the rotational anti-shake function, so that the optical anti-shake assembly 130 performs the translational compensation on the image sensor 1220 according to the movement of the electronic device 100 when the electronic device 100 moves. Correspondingly, a third driving control instruction is sent to the preset calibration tool, so that the preset calibration tool drives the electronic device 100 to translate in the X-axis direction and the Y-axis direction of the electronic device 100. For example, the third driving control command causes the preset calibration tool to drive the electronic device 100 to translate alternately in the X-axis direction and the Y-axis direction, as shown in fig. 6, where the distance of translation and the frequency of alternation can be determined by those skilled in the art according to actual needs, and the present invention is not limited thereto.

During the translation process of the electronic device 100, the motion data of the electronic device 100 is acquired in real time and provided to the optical anti-shake component 130, so that the optical anti-shake processing is performed through the optical anti-shake component 130. It is understood that, since the present embodiment only turns on the image sensor 1220 of the optical anti-shake assembly 130 to perform the translational anti-shake function, the optical anti-shake assembly 130 will only perform the compensating translational motion on the image sensor 1220 during the movement of the electronic device 100. In addition, a sequence of images of the rectangular calibration plate acquired by the image sensor 1220 during translation is acquired, denoted as a fourth sequence of images. For example, a fourth image sequence is acquired by image sensor 1220 during translation of the rectangular calibration plate for a one minute period.

Because the electronic device 100 is relatively fixed to the rectangular calibration plate in the preset calibration tool, the optical anti-shake assembly 130 performs the optical anti-shake process due to the translation of the electronic device 100, which results in the position of the center mark in the fourth image being imaged to deviate from the center of the fourth image. Accordingly, in the present embodiment, the position and posture data of the lens 1210 when each fourth image is collected by the image sensor 1220 is further acquired from the optical anti-shake component 130 and recorded as the second position and posture data. Further, the position coordinates of the center mark in each fourth image are also acquired under the pixel coordinate system, and are noted as fourth position coordinates.

It will be appreciated that for a fourth image, the fourth position coordinates of the center mark in the fourth image are equivalent to the center-to-center mark translation vector of the fourth image. Therefore, the fourth functional relation can be obtained by performing functional fitting according to the second position posture data and the fourth position coordinates corresponding to each fourth image. The manner of function fitting is not particularly limited herein, and includes, but is not limited to, least squares, random sample consensus, polynomial fitting, and the like. For example, the fourth functional relationship is obtained by using least square fitting in the present embodiment.

In other embodiments, after the fourth functional relationship is obtained by fitting, the fitting error of the fourth functional relationship is further evaluated (the evaluation mode may be selected by those skilled in the art according to actual needs, and is not particularly limited herein, for example, a mean square error may be adopted), so as to obtain the fitting error of the fourth functional relationship, if the fitting error of the fourth functional relationship is greater than the third error threshold (the skilled person may take an experience value according to actual needs), the second position gesture data and the fourth position coordinate with obvious anomalies are identified and removed, and then fitting is performed again, and if the fitting error of the newly fitted fourth functional relationship is still greater than the third error threshold, a new fourth image sequence and new second position gesture data are obtained again for fitting until a fourth functional relationship with the fitting error less than or equal to the third error threshold is obtained.

Optionally, in an embodiment, the image processing method provided in the present application further includes:

(1) A lens 1210 of the optical anti-shake assembly 130 is turned on to deflect the anti-shake function, the image sensor 1220 is shifted to the anti-shake function, and the image sensor 1220 is rotated to the anti-shake function;

(2) Transmitting a fourth driving control instruction to the preset calibration tool, so that the preset calibration tool drives the electronic device 100 to rotate around the Z axis of the image sensor 1220 and drives the electronic device 100 to translate in the X axis direction and the Y axis direction of the electronic device 100;

(3) During the rotation and translation process of the electronic device 100, performing optical anti-shake processing through the optical anti-shake component 130, and acquiring a fifth image sequence of the rectangular calibration plate acquired by the image sensor 1220 during the rotation and translation process;

(4) Acquiring third deflection attitude data of the lens 1210 when the image sensor 1220 acquires each fifth image in the sequence of fifth images from the optical anti-shake assembly 130, and acquiring third position attitude data and third rotation attitude data when the image sensor 1220 acquires each fifth image;

(5) Acquiring a second correction rotation matrix for carrying out rotation correction on each fifth image according to third deflection gesture data, third rotation gesture data, a first functional relation, a second functional relation and a third functional relation corresponding to each fifth image;

(6) Acquiring a second correction translation vector for carrying out rotation correction on each fifth image according to third position and posture data corresponding to each fifth image and a fourth functional relation;

(7) Correcting the central pixel of each fifth image according to the second correction translation vector and the second correction rotation matrix, and carrying out joint correction on the first functional relation, the second functional relation, the third functional relation and the fourth functional relation according to the position difference of the central pixel of each fifth image before and after correction.

In this embodiment, the first functional relationship, the second functional relationship, the third functional relationship and the fourth functional relationship are further jointly modified by using a preset calibration tool. Also, the electronic device 100 is fixed to the preset calibration tool in the above manner such that the positions of the electronic device 100 and the rectangular calibration plate in the preset calibration tool are relatively fixed. Reference may be made specifically to the relevant descriptions in the above embodiments, and details are not repeated here.

After the electronic device 100 is fixed on the preset calibration tool and the positions of the electronic device 100 and the rectangular calibration plate in the preset calibration tool are relatively fixed, the first functional relationship, the second functional relationship, the third functional relationship and the fourth functional relationship can be jointly corrected by using the preset calibration tool.

Since the first functional relationship, the second functional relationship, the third functional relationship and the fourth functional relationship are jointly modified at this time, all the anti-shake functions of the optical anti-shake assembly 130, that is, the image sensor 1220 translates the anti-shake function, the lens 1210 deflects the anti-shake function and the image sensor 1220 rotates the anti-shake function, are correspondingly turned on, so that the optical anti-shake assembly 130 performs compensation deflection on the lens 1210 according to the motion of the electronic device 100 and performs compensation translation and compensation rotation on the image sensor 1220 when the electronic device 100 moves. Accordingly, a fourth driving control command is sent to the preset calibration tool, so that the preset calibration tool drives the electronic device 100 to translate in the X-axis direction and the Y-axis direction of the electronic device 100 while rotating around the Z-axis of the image sensor 1220. For example, the fourth driving control command causes the preset calibration tool to drive the electronic device 100 to alternately translate in the X-axis direction and the Y-axis direction, as shown in fig. 6, and simultaneously alternately rotate around the Z-axis of the image sensor 1220, where the translation distance, the frequency of alternate translation, the angular speed of rotation, and the frequency of alternate rotation can be all taken by those skilled in the art according to actual needs, and the present invention is not limited thereto.

During the rotation and translation process of the electronic device 100, the motion data of the electronic device 100 is acquired in real time and provided to the optical anti-shake component 130, so that the optical anti-shake processing is performed through the optical anti-shake component 130. It will be appreciated that since the present embodiment turns on the entire anti-shake function of the optical anti-shake assembly 130, the optical anti-shake assembly 130 will perform compensation deflection on the lens 1210 during the movement of the electronic device 100, and at the same time perform compensation translation and compensation rotation on the image sensor 1220. In addition, a sequence of images of the rectangular calibration plate acquired by the image sensor 1220 during the rotational translation is acquired, denoted as a fifth sequence of images. For example, a fifth image sequence is obtained by image sensor 1220 acquiring an image sequence of a rectangular calibration plate for a period of one minute during the rotational translation.

In the present embodiment, the third deflection posture data of the lens 1210 when the image sensor 1220 collects each fifth image in the fifth image sequence is acquired from the optical anti-shake component 130, and the third position posture data and the third rotation posture data when the image sensor 1220 collects each fifth image are acquired.

Further, according to the third deflection gesture data, the third rotation gesture data, the first functional relationship, the second functional relationship, and the third functional relationship corresponding to each fifth image, a second correction rotation matrix for performing rotation correction on each fifth image is obtained, and specifically, the method of obtaining the first correction rotation matrix for performing rotation correction on the optical anti-shake image in the above embodiment may be correspondingly obtained, which is not described herein.

In addition, the second correction translation vector for performing rotation correction on each fifth image is obtained according to the third position and posture data and the fourth functional relationship corresponding to each fifth image, and specifically, the method for obtaining the first correction translation vector in the above embodiment may be correspondingly obtained, which is not described herein.

As described above, after the second correction translation vector for performing the translation correction on each fifth image and the second correction rotation matrix for performing the rotation correction on each fifth image are acquired, the center pixel of each fifth image is further corrected according to the second correction translation vector and the second correction rotation matrix. It will be appreciated that, for a fifth image, the smaller the difference between the positions of the center pixels of the fifth image before and after correction, the smaller the total fitting error of the first, second, third and fourth functional relationships will be. Therefore, the present embodiment further performs joint correction on the first functional relationship, the second functional relationship, the third functional relationship, and the fourth functional relationship according to the position difference of the center pixel of each fifth image before and after correction, with the objective of eliminating the position difference as much as possible.

For example, in this embodiment, the correction is performed in order from the largest to the smallest according to the fitting errors corresponding to the first functional relationship, the second functional relationship, the third functional relationship, and the fourth functional relationship.

Optionally, in an embodiment, acquiring the optical anti-shake image of the corresponding motion data acquired by the image sensor 1220 includes:

(1) Correcting the first acquisition time stamp of the motion data according to the first delay parameter of the corresponding motion data to obtain a first correction time stamp;

(2) According to the first corrected time stamp, an optical anti-shake image of the corresponding motion data acquired by the image sensor 1220 is acquired.

In this embodiment, when acquiring the optical anti-shake image of the corresponding motion data acquired by the image sensor 1220, the acquisition time stamp of the motion data is first acquired and is denoted as the first acquisition time stamp, where the first acquisition time stamp is used to describe the acquisition time when the motion data is acquired.

It can be understood that there is a certain time delay from the motion data acquisition to the motion data output of the motion sensor 110, in this embodiment, in order to eliminate the time delay, an optical anti-shake image accurately corresponding to the motion data is acquired, and a first time delay parameter corresponding to the motion data is calibrated in advance, where the first time delay parameter is used to describe the time delay from the motion data acquisition by the motion sensor 110 to the motion data output.

Correspondingly, the embodiment corrects the first acquisition time stamp according to the first delay parameter corresponding to the motion data, and marks the corrected first acquisition time stamp as a first correction time stamp, wherein the first correction time stamp describes the acquisition time of the motion data. At this time, according to the first correction time stamp, an optical anti-shake image corresponding to the motion data acquired by the image sensor 1220 may be obtained.

Optionally, in an embodiment, the first delay parameter is pre-generated according to the following steps:

(1) Acquiring a sample shake image sequence;

(2) According to a second acquisition time stamp of each sample jittering image in the sample jittering image sequence, acquiring sample motion data corresponding to each sample jittering image to obtain a sample motion data sequence;

(3) And searching to obtain a first time delay parameter according to the preset time delay parameter, the preset search step length, the sample jittering image sequence and the corresponding sample motion data sequence.

The present embodiment provides an optional calibration of the first delay parameter.

Firstly, an image sequence which is not subjected to electronic anti-shake processing is obtained as a sample and recorded as a sample shake image sequence. For example, the anti-shake functions of the optical anti-shake module 130 and the electronic anti-shake module 140 are turned off, and an image sequence with a certain duration is captured by the electronic device 100 as a sample shake image sequence.

In addition, sample motion data corresponding to each sample jittered image is acquired according to a second acquisition time stamp (for describing output time of the image sensor 1220 outputting the corresponding sample jittered image) of each sample jittered image in the sample jittered image sequence, so as to obtain a sample motion data sequence corresponding to the sample jittered image sequence.

After the sample jittering image sequence and the corresponding sample motion data sequence are obtained, searching to obtain the first delay parameter according to the preset search strategy and according to the preset delay parameter, the preset search step length, the sample jittering image sequence and the corresponding sample motion data sequence. The configuration of the preset search strategy, the preset delay parameter and the preset search step is not particularly limited, and can be configured by a person skilled in the art according to actual needs.

Optionally, in an embodiment, searching to obtain the first delay parameter according to the preset delay parameter, the preset search step length, the sample jittering image sequence and the corresponding sample motion data sequence includes:

(1) Acquiring a preset number of candidate delay parameters according to the preset delay parameters and the preset search step length;

(2) Respectively correcting the sample motion data sequences according to each candidate time delay parameter to obtain a preset number of corrected sample motion data sequences;

(3) According to each corrected sample motion data sequence, performing electronic anti-shake processing on the sample shake image sequence through an electronic anti-shake component 140 to obtain a preset number of anti-shake image sequences;

(4) Quantitatively scoring the anti-shake quality of each anti-shake image sequence to obtain an anti-shake quality score of each anti-shake image sequence;

(5) Updating the preset time delay parameter to a candidate time delay parameter corresponding to the anti-shake image sequence with the highest anti-shake quality score, and updating the preset searching step length according to the preset updating parameter;

(6) Repeating the steps until the preset search condition is met, and taking the preset time delay parameter when the preset search condition is met as a first time delay parameter.

The present embodiment provides a search strategy for optionally searching for the first delay parameter.

Firstly, obtaining a preset number of candidate delay parameters according to preset delay parameters and preset search step length. The preset number is not particularly limited, for example, when the value is 3, the time delay parameter D is preset ₀ For the center, generating [ D ] with a preset searching step delta D ₀ -Δd,D ₀ ,D ₀ +Δd]3 candidate delay parameters.

Then, respectively correcting the sample motion data sequence according to each candidate time delay parameter, namely updating the corresponding relation between the sample motion data in the sample motion data sequence and the sample jittering images in the sample jittering image sequence according to each candidate time delay parameter, and correspondingly obtaining a preset number of corrected sample motion data sequences;

Then, according to each corrected sample motion data sequence, an electronic anti-shake processing is performed on the sample shake image sequence through the electronic anti-shake component 140 to obtain a preset number of anti-shake image sequences, wherein for a corrected sample motion data sequence, each corrected sample motion data and sample shake image corresponding to the sample shake image sequence in the corrected sample motion data sequence are input into a sample shake image for anti-shake processing, and accordingly an anti-shake image sequence corresponding to the corrected sample motion data sequence is obtained.

As above, after the preset number of anti-shake image sequences are obtained, the anti-shake quality of each anti-shake image sequence is quantitatively scored according to a preset quantization scoring strategy, so as to obtain the anti-shake quality score of each anti-shake image sequence. The configuration of the quantization scoring strategy is not particularly limited, and can be configured by a person skilled in the art according to actual needs.

For example, in this embodiment, the quantization scoring of the anti-shake quality of the anti-shake image sequence by adopting the frame-by-frame feature matching and the average feature distance statistics may be expressed as:

where s represents an anti-shake quality score (the larger the value thereof is, the higher the anti-shake quality is), p represents a feature matching point set, i represents the frame number of the anti-shake image sequence, and N represents the total frame number of the anti-shake images in the anti-shake image sequence. It should be noted that the feature matching algorithm and its descriptors can be selected by those skilled in the art according to actual needs, including but not limited to SIFT, FAST, harris, ORB, etc.

After the quantization scoring of each anti-shake image sequence is completed and the anti-shake quality scoring of each anti-shake image sequence is correspondingly obtained, updating the preset time delay parameter to the candidate time delay parameter corresponding to the anti-shake image sequence with the highest anti-shake quality score, and updating the preset search step length according to the preset updating parameter and the preset updating strategy. The configuration of the preset updating policy is not particularly limited, and the preset updating policy may be configured by a person skilled in the art according to actual needs with the constraint of reducing the preset search step.

For example, the preset update policy configured in this embodiment may be expressed as:

Δd′＝Δd/M；M＞1；

wherein Δd' represents a preset search step after updating, Δd represents a preset search step before updating, and M represents a preset updating parameter.

After the updating of the preset time delay parameter and the preset searching step length is completed, repeating the steps to continue searching until the preset searching condition is met, and taking the preset time delay parameter when the preset searching condition is met as the first time delay parameter. The configuration of the preset search condition is not particularly limited here, and may be configured by those skilled in the art according to actual needs.

Optionally, in an embodiment, the preset search condition includes:

the anti-shake quality score of any anti-shake image sequence is greater than or equal to a score threshold; or alternatively

The updated preset searching step length is smaller than a step length threshold value; or alternatively

The number of searches reaches a number threshold.

It should be noted that, in this embodiment, the scoring threshold, the step threshold and the number threshold are not limited in particular, and may be empirically determined by those skilled in the art according to actual needs.

Optionally, in an embodiment, acquiring the first deflection pose data of the lens 1210, the first position pose data of the image sensor 1220, and the first rotation pose data from the optical anti-shake assembly 130 includes:

correcting the first correction time stamp according to the second time delay parameter corresponding to the optical anti-shake component 130 to obtain a second correction time stamp;

the first deflection posture data of the lens 1210, the first position posture data of the image sensor 1220, and the first rotation posture data are acquired from the optical anti-shake assembly 130 according to the second correction time stamp.

It can be understood that, in this embodiment, in order to eliminate the time delay from acquiring the deflection gesture data of the lens 1210, the position gesture data and the rotation gesture data of the image sensor 1220 to outputting the gesture data, the optical anti-shake assembly 130 acquires the first deflection gesture data, the first position gesture data and the first rotation gesture data corresponding to the optical anti-shake image accurately, and the second time delay parameter corresponding to the optical anti-shake assembly 130 is calibrated in advance, where the second time delay parameter is used to describe the time delay from the optical anti-shake assembly 130 acquiring the gesture data until outputting the gesture data.

Accordingly, in the present embodiment, when the first deflection gesture data of the lens 1210, the first position gesture data of the image sensor 1220, and the first rotation gesture data are obtained from the optical anti-shake assembly 130, the first correction timestamp is corrected according to the second delay parameter corresponding to the optical anti-shake assembly 130, so as to obtain the second correction timestamp; and then according to the second correction timestamp, acquiring the first deflection gesture data of the lens 1210, the first position gesture data of the image sensor 1220 and the first rotation gesture data from the optical anti-shake assembly 130, that is, acquiring the deflection gesture data of the lens 1210 with the output timestamp matched with the second correction timestamp from the optical anti-shake assembly 130 as the first deflection gesture data, acquiring the position gesture data of the image sensor 1220 with the output timestamp matched with the second correction timestamp from the optical anti-shake assembly 130 as the first position gesture data, and acquiring the rotation gesture data of the image sensor 1220 with the output timestamp matched with the second correction timestamp from the optical anti-shake assembly 130 as the first rotation gesture data.

It should be noted that, the calibration method of the second delay parameter in this embodiment is not particularly limited, and may be selected by those skilled in the art according to actual needs. For example, the present embodiment adopts the same calibration strategy as the first delay parameter (refer to the related description in the above embodiment, and will not be repeated here), and after the calibration of the first delay parameter is completed, the calibration of the second delay parameter is performed.

Optionally, in an embodiment, the motion data includes acceleration data and angular velocity data.

Referring to fig. 8, in order to better execute the image processing method provided in the present application, the present embodiment further provides an image processing apparatus 300, as shown in fig. 8, the image processing apparatus 300 includes an optical anti-shake module 310, an attitude acquisition module 320, an attitude correction module 330, and an electronic anti-shake module 340, and the image processing apparatus 300 is configured to execute steps in any of the image processing methods provided in the present application, such as:

the optical anti-shake module 310 is configured to obtain motion data of the electronic device, and control, according to the motion data, the lens to perform compensation deflection in an X-axis direction and a Y-axis direction of the lens through the optical anti-shake component, control the image sensor to perform compensation translation in the X-axis direction and the Y-axis direction of the image sensor, and control the image sensor to perform compensation rotation around a Z-axis of the image sensor;

the gesture obtaining module 320 is configured to obtain an optical anti-shake image of the corresponding motion data collected by the image sensor, and obtain first deflection gesture data of the lens, first position gesture data of the image sensor, and first rotation gesture data from the optical anti-shake component;

The gesture correction module 330 is configured to restore and correct the gesture of the optical anti-shake image according to the first deflection gesture data, the first position gesture data, and the first rotation gesture data, so as to obtain a gesture correction image;

the electronic anti-shake module 340 is configured to perform electronic anti-shake processing on the posture correction image through the electronic anti-shake component according to the motion data, so as to obtain an electronic anti-shake image.

It should be noted that, the image processing apparatus 300 provided in the embodiment of the present application and the image processing method in the above embodiment belong to the same concept, and the specific implementation process is referred to the above related embodiments, which are not repeated here.

Referring to fig. 1 and 9 in combination, the electronic device 100 provided in the present application may further include a memory 150 and a processor 160. It will be appreciated by those skilled in the art that the structure of the electronic device 100 shown in fig. 9 does not constitute a limitation of the electronic device 100, and may include more or fewer components than shown, or may combine certain components, or may have a different arrangement of components.

Wherein the memory 150 may be used to store computer programs and data. The memory 150 stores a computer program having executable code included therein. The computer program may be divided into various functional modules.

The processor 160 is a control center of the electronic device 100, connects various parts of the entire electronic device 100 using various interfaces and lines, and performs various functions of the electronic device 100 and processes data by running or executing computer programs stored in the memory 150 and calling data stored in the memory 150, thereby controlling the electronic device 100 as a whole.

In the embodiment of the present application, the processor 160 in the electronic device 100 loads executable codes corresponding to one or more computer programs into the memory 150, and the executable codes are executed by the processor 160 to perform steps in any of the image processing methods provided in the present application, for example:

acquiring motion data of the electronic device 100, controlling the lens 1210 to perform compensation deflection in the X-axis direction and the Y-axis direction of the lens 1210 through the optical anti-shake component 130 according to the motion data, controlling the image sensor 1220 to perform compensation translation in the X-axis direction and the Y-axis direction of the image sensor 1220, and controlling the image sensor 1220 to perform compensation rotation around the Z-axis of the image sensor 1220;

acquiring an optical anti-shake image of corresponding motion data acquired by the image sensor 1220, and acquiring first deflection attitude data of the lens 1210, first position attitude data and first rotation attitude data of the image sensor 1220 from the optical anti-shake assembly 130;

according to the motion data, the electronic anti-shake component 140 performs electronic anti-shake processing on the posture correction image to obtain an electronic anti-shake image.

It should be noted that, the electronic device 100 provided in the embodiment of the present application and the image processing method in the above embodiment belong to the same concept, and the specific implementation process refers to the above related embodiment, which is not repeated here.

The present embodiment also provides a computer-readable storage medium in which a computer program is stored, the computer program being capable of being loaded by a processor to perform steps in any of the image processing methods provided in the embodiments of the present application.

Wherein the storage medium may include: read Only Memory (ROM), random access Memory (RAM, random Access Memory), magnetic or optical disk, and the like.

Since the computer program stored in the computer readable storage medium may execute the steps in any one of the image processing methods provided in the embodiments of the present application, the beneficial effects that any one of the image processing methods provided in the embodiments of the present application can be achieved, which are detailed in the previous embodiments and are not described herein.

The image processing method, the device, the electronic equipment and the storage medium provided by the embodiment of the application are described in detail above. Specific examples are set forth herein to illustrate the principles and embodiments of the present application, with the description of the examples given above only to assist in understanding the present application. Meanwhile, those skilled in the art will have variations in the specific embodiments and application scope in light of the ideas of the present application, and the present description should not be construed as limiting the present application in view of the above.

Claims

1. An image processing method applied to an electronic device, wherein the electronic device comprises a lens, an image sensor, an optical anti-shake component and an electronic anti-shake component, the image processing method comprising:

acquiring motion data of the electronic equipment, controlling the lens to perform compensation deflection in the X-axis direction and the Y-axis direction of the lens through the optical anti-shake component according to the motion data, controlling the image sensor to perform compensation translation in the X-axis direction and the Y-axis direction of the image sensor, and controlling the image sensor to perform compensation rotation around the Z-axis of the image sensor;

Acquiring an optical anti-shake image corresponding to the motion data acquired by the image sensor, and acquiring first deflection posture data of the lens, first position posture data and first rotation posture data of the image sensor from the optical anti-shake assembly;

2. The method of claim 1, wherein the restoring and correcting the pose of the optical anti-shake image according to the first deflection pose data, the first position pose data, and the first rotation pose data to obtain a pose corrected image includes:

acquiring a first correction rotation matrix for performing rotation correction on the optical anti-shake image according to the first deflection posture data and the first rotation posture data;

performing rotation correction on the posture of the optical anti-shake image according to the first correction rotation matrix to obtain a rotation correction image;

Acquiring a first correction translation vector for carrying out translation correction on the rotation correction image according to the first position and posture data;

and carrying out translational correction on the posture of the rotation correction image according to the first correction translational vector to obtain the posture correction image.

3. The method of claim 2, wherein the obtaining a first correction rotation matrix for rotation correction of the optical anti-shake image based on the first yaw attitude data and the first rotation attitude data comprises:

acquiring a first deflection angle corresponding to the first deflection attitude data according to a first functional relation between the first deflection attitude data and the deflection angle;

acquiring a second deflection angle corresponding to the first deflection attitude data according to a second functional relation between the first deflection attitude data and the deflection angle;

acquiring a first rotation angle corresponding to the first rotation gesture data according to a third functional relation between the first rotation gesture data and the rotation angle;

and acquiring the first correction rotation matrix according to the first deflection angle, the second deflection angle and the first rotation angle.

4. A method according to claim 3, wherein said obtaining said first correction rotation matrix from said first deflection angle, said second deflection angle, and said first rotation angle comprises:

According to the first deflection angle, the second deflection angle and the first rotation angle, the first correction rotation matrix is obtained according to the following formula:

R＝R _X ·R _Y ·R _Z ；

wherein R represents the first correction rotation matrix, C represents cos (), S represents sin (), α represents the first deflection angle, β represents the second deflection angle, and γ represents the first rotation angle.

5. The method of claim 4, wherein rotationally correcting the pose of the optical anti-shake image according to the first correction rotation matrix to obtain a rotation corrected image, comprises:

acquiring a deflection center coordinate of a deflection center of the lens for compensation deflection under a pixel coordinate system;

converting the optical anti-shake image from a pixel coordinate system to an image coordinate system according to the deflection center coordinate;

and carrying out rotation correction on the posture of the optical anti-shake image under an image coordinate system according to the first correction rotation matrix to obtain a rotation correction image.

6. The method of claim 5, wherein said performing translational correction on the pose of the rotation corrected image based on the first correction translational vector to obtain the pose corrected image comprises:

Acquiring a rotation center coordinate of a rotation center of the image sensor for compensation rotation under a pixel coordinate system;

acquiring a correction vector for correcting the first correction translation vector according to the deflection center coordinate and the rotation center coordinate;

correcting the first correction translation vector according to the correction vector to obtain a corrected first correction translation vector;

and carrying out translational correction on the posture of the rotation correction image according to the corrected first correction translational vector to obtain the posture correction image.

7. The method of claim 6, wherein the electronic device is secured to a pre-set calibration tool such that the electronic device is relatively fixed in position with respect to a rectangular calibration plate in the pre-set calibration tool, the rectangular calibration plate including a center mark, the deflection center coordinates, the first functional relationship, and the second functional relationship being determined as follows:

starting a lens deflection anti-shake function of the optical anti-shake assembly, and closing a translation anti-shake function of the image sensor and a rotation anti-shake function of the image sensor;

transmitting a first driving control instruction to the preset calibration tool, so that the preset calibration tool drives the electronic equipment to translate in the X-axis direction and the Y-axis direction of the electronic equipment;

In the translation process of the electronic equipment, performing optical anti-shake processing through the optical anti-shake component, and acquiring a first image sequence of the rectangular calibration plate acquired by the image sensor in the translation process;

sending a stopping instruction to the preset calibration tool, so that the preset calibration tool stops driving the electronic equipment to translate, and acquiring a second image sequence of the rectangular calibration plate acquired by the image sensor in a static process;

acquiring a first position coordinate of the center mark in each first image in the first image sequence and a second position coordinate of the center mark in each second image in the second image sequence under a pixel coordinate system;

acquiring second deflection attitude data of the lens when the image sensor acquires each first image in the first image sequence from an optical anti-shake component, and determining average coordinates of all second position coordinates;

and fitting according to the average coordinates, the first position coordinates corresponding to each first image and the second deflection posture data to obtain the deflection center coordinates, the first functional relation and the second functional relation.

8. The method of claim 7, wherein said fitting the center of deflection coordinates, the first functional relationship, and the second functional relationship from the average coordinates and the first position coordinates and the second deflection pose data corresponding to each of the first images comprises:

according to the average coordinates and the first position coordinates and the second deflection posture data corresponding to each first image, the deflection center coordinates, the first functional relation and the second functional relation are obtained through fitting according to the following formula:

min||P ₁ -f(p ₀ ,h ₀ '；θ)|| ₂ ；

θ＝[C _x1 ,C _y1 ,θ ₀ ,θ ₁ ]；

wherein f () represents the first polynomial function or the second polynomial function, P ₁ Representing the first position coordinates, p ₀ Representing the average coordinates, h ₀ ' represents second deflection attitude data, C _x1 Representing the abscissa in the deflection center coordinates, C _y1 Representing the ordinate, θ, of the deflection center coordinates ₀ Polynomial coefficients, θ, representing the first polynomial function ₁ Polynomial coefficients representing the second polynomial function.

9. The method of claim 6, wherein the electronic device is fixed to a predetermined calibration tool such that the electronic device is fixed relative to a position of a rectangular calibration plate in the predetermined calibration tool, the rectangular calibration plate including a tetragonal marker, the center of rotation coordinate and the third functional relationship being determined as follows:

Opening the rotation anti-shake function of the image sensor of the optical anti-shake assembly, and closing the deflection anti-shake function of the lens and the translation anti-shake function of the image sensor;

sending a second driving control instruction to the preset calibration tool, so that the preset calibration tool drives the electronic equipment to rotate around the Z axis of the image sensor;

during the rotation process of the electronic equipment, performing optical anti-shake processing through the optical anti-shake component, and acquiring a third image sequence of the rectangular calibration plate acquired by the image sensor during the rotation process;

acquiring second rotation gesture data of each third image in the third image sequence acquired by the image sensor from the optical anti-shake component, and acquiring third position coordinates of the four-corner mark in each third image under a pixel coordinate system;

and fitting according to the second rotation gesture data and the third position coordinates corresponding to each third image to obtain the rotation center coordinates and the third functional relation.

10. The method of claim 9, wherein the third functional relationship comprises a third polynomial function, the fitting the rotation center coordinates and the third functional relationship from the second rotation pose data and the third position coordinates corresponding to each third image, comprising:

And according to the second rotation gesture data and the third position coordinates corresponding to each third image, fitting according to the following formula to obtain the rotation center coordinates and the third functional relation:

θ'＝[C _x2 ,C _y2 ,θ ₂ ]；

wherein P is _i A third position coordinate indicating the ith four-corner mark, h ₂ 'represents the second rotational gesture data, f' () represents the third polynomial function, r represents the preset search radius, θ ₂ Polynomial coefficients representing said third polynomial function, C _x2 Representing the abscissa in the rotation center coordinates, C _y2 Representing the ordinate in the rotation center coordinates, imW representing the width of the third image, imH representing the third imageIs high.

11. A method according to claim 3, wherein said obtaining a first correction translation vector for performing a translation correction on said rotation corrected image from said first position and orientation data comprises:

and taking the translation vector corresponding to the first position and posture data as the first correction translation vector according to a fourth functional relation of the first position and posture data and the translation vector.

12. The method of claim 11, wherein the electronic device is fixed to a pre-set calibration tool such that the electronic device is fixed relative to a position of a rectangular calibration plate in the pre-set calibration tool, the rectangular calibration plate including a center mark, the fourth functional relationship being determined according to the steps of:

Opening an image sensor translation anti-shake function of the optical anti-shake assembly, and closing a lens deflection anti-shake function and an image sensor rotation anti-shake function;

transmitting a third driving control instruction to the preset calibration tool, so that the preset calibration tool drives the electronic equipment to translate in the X-axis direction and the Y-axis direction of the electronic equipment;

in the translation process of the electronic equipment, performing optical anti-shake processing through the optical anti-shake component, and acquiring a fourth image sequence of the rectangular calibration plate acquired by the image sensor in the translation process;

acquiring second position and posture data of each fourth image in the fourth image sequence from the optical anti-shake component, and acquiring fourth position coordinates of the center mark in each fourth image under a pixel coordinate system;

and fitting according to the second position and posture data and the fourth position coordinates corresponding to each first image to obtain the fourth functional relation.

13. The method as recited in claim 12, further comprising:

starting a lens deflection anti-shake function, an image sensor translation anti-shake function and an image sensor rotation anti-shake function of the optical anti-shake assembly;

Transmitting a fourth driving control instruction to the preset calibration tool, so that the preset calibration tool drives the electronic equipment to rotate around the Z axis of the image sensor and drive the electronic equipment to translate in the X axis direction and the Y axis direction of the electronic equipment;

in the rotation and translation process of the electronic equipment, performing optical anti-shake processing through the optical anti-shake component, and acquiring a fifth image sequence of the rectangular calibration plate, which is acquired by the image sensor in the rotation and translation process;

acquiring third deflection attitude data of the lens when the image sensor acquires each fifth image in the fifth image sequence from the optical anti-shake component, and acquiring third position attitude data and third rotation attitude data when the image sensor acquires each fifth image;

acquiring a second correction rotation matrix for carrying out rotation correction on each fifth image according to the third deflection gesture data, the third rotation gesture data, the first functional relation, the second functional relation and the third functional relation corresponding to each fifth image;

acquiring a second correction translation vector for carrying out rotation correction on each fifth image according to the third position and posture data corresponding to each fifth image and the fourth functional relation;

Correcting the central pixel of each fifth image according to the second correction translation vector and the second correction rotation matrix, and carrying out joint correction on the first functional relation, the second functional relation, the third functional relation and the fourth functional relation according to the position difference of the central pixel of each fifth image before and after correction.

14. The method of any of claims 1-13, wherein the acquiring an optical anti-shake image of the motion data acquired by the image sensor comprises:

correcting the first acquisition time stamp of the motion data according to the first delay parameter corresponding to the motion data to obtain a first correction time stamp;

and acquiring an optical anti-shake image corresponding to the motion data acquired by the image sensor according to the first correction time stamp.

15. The method of claim 14, wherein the first delay parameter is pre-generated by:

acquiring a sample shake image sequence;

according to the second acquisition time stamp of each sample jittering image in the sample jittering image sequence, acquiring sample motion data corresponding to each sample jittering image to obtain a sample motion data sequence;

And searching to obtain the first delay parameter according to a preset delay parameter, a preset searching step length, the sample shaking image sequence and a corresponding sample motion data sequence.

16. The method of claim 15, wherein searching for the first delay parameter according to a preset delay parameter, a preset search step, the sample jittered image sequence and a corresponding sample motion data sequence comprises:

acquiring a preset number of candidate delay parameters according to the preset delay parameters and the preset search step length;

correcting the sample motion data sequences according to each candidate time delay parameter to obtain a preset number of corrected sample motion data sequences;

according to each corrected sample motion data sequence, carrying out electronic anti-shake processing on the sample shake image sequence through the electronic anti-shake component to obtain a preset number of anti-shake image sequences;

quantitatively scoring the anti-shake quality of each anti-shake image sequence to obtain an anti-shake quality score of each anti-shake image sequence;

updating the preset time delay parameter to a candidate time delay parameter corresponding to an anti-shake image sequence with the highest anti-shake quality score, and updating the preset search step length according to a preset updating parameter;

Repeating the steps until the preset search condition is met, and taking the preset time delay parameter when the preset search condition is met as the first time delay parameter.

17. The method of claim 14, wherein the acquiring first yaw pose data of the lens, first position pose data of the image sensor, and first rotation pose data from the optical anti-shake assembly comprises:

correcting the first correction time stamp according to a second time delay parameter corresponding to the optical anti-shake component to obtain a second correction time stamp;

and acquiring first deflection attitude data of the lens, first position attitude data and first rotation attitude data of the image sensor from the optical anti-shake assembly according to the second correction timestamp.

18. An image processing apparatus applied to an electronic device, the electronic device comprising a lens, an image sensor, an optical anti-shake component, and an electronic anti-shake component, the image processing apparatus comprising:

the optical anti-shake module is used for acquiring motion data of the electronic equipment, controlling the lens to perform compensation deflection in the X-axis direction and the Y-axis direction of the lens through the optical anti-shake component, controlling the image sensor to perform compensation translation in the X-axis direction and the Y-axis direction of the image sensor, and controlling the image sensor to perform compensation rotation around the Z-axis of the image sensor according to the motion data;

The gesture acquisition module is used for acquiring an optical anti-shake image corresponding to the motion data acquired by the image sensor, and acquiring first deflection gesture data of the lens, first position gesture data and first rotation gesture data of the image sensor from the optical anti-shake assembly;

19. An electronic device comprising a lens, an image sensor, an optical anti-shake component, an electronic anti-shake component, a processor, and a memory, the memory storing a computer program, characterized in that the processor performs the method according to any one of claims 1-17 by loading the computer program.

20. A computer-readable storage medium, on which a computer program is stored, characterized in that the method according to any one of claims 1-17 is performed when the computer program is loaded by a processor.