CN117135459A

CN117135459A - Image anti-shake method and electronic equipment

Info

Publication number: CN117135459A
Application number: CN202310413771.2A
Authority: CN
Inventors: 卢圣卿; 徐荣跃; 王宁; 王宇; 朱聪超
Original assignee: Honor Device Co Ltd
Current assignee: Honor Device Co Ltd
Priority date: 2023-04-07
Filing date: 2023-04-07
Publication date: 2023-11-28

Abstract

The application relates to the field of images, and provides an image anti-shake method and electronic equipment, wherein the image anti-shake method is applied to the electronic equipment, the electronic equipment comprises a first camera, the first camera is a movable camera, and the method comprises the following steps: starting a camera application; acquiring a first image, a first offset and first angle data; converting the first offset of the first image into second angle data of the first image; obtaining third angle data of the first image based on the second angle data of the first image and the first angle data of the first image, wherein the third angle data is used for representing the motion quantity of the electronic equipment and the first camera when the first image is acquired; and performing anti-shake processing on the first image based on the third angle data of the first image to obtain a second image. Based on the technical scheme, under the condition that the camera in the electronic equipment realizes flexible movement, the image anti-shake effect can be improved.

Description

Image anti-shake method and electronic equipment

Technical Field

The present application relates to the field of image processing, and in particular, to an image anti-shake method and an electronic device.

Background

With the development of shooting functions in electronic devices, camera applications are becoming more and more widely used in electronic devices. At present, due to the limitation of hardware of a camera, the angle of view of the camera has a certain limitation; when a user shoots a long-distance object, a long-focus camera is generally needed, however, the field angle of the long-focus camera is limited to a certain extent, and the shooting needs of the user in different scenes cannot be met. To overcome this limitation, flexible movement of the camera is required; meanwhile, after the camera moves, if the image anti-shake processing is performed by adopting the existing anti-shake method, the image anti-shake effect is poor.

Therefore, how to perform image anti-shake processing and improve the anti-shake effect of an image becomes a problem to be solved under the circumstance that the camera in the electronic device realizes flexible movement.

Disclosure of Invention

The application provides an image anti-shake method and electronic equipment, which can improve the anti-shake effect of an image under the condition that a camera in the electronic equipment flexibly moves.

In a first aspect, an image anti-shake method is provided and applied to an electronic device, the electronic device includes a first camera, the first camera is a movable camera, and the method includes:

starting a camera application program;

acquiring a first image, a first offset and first angle data; the first image is an image acquired by the first camera in real time, the first offset is used for representing offset between a lens center and an imaging center of the first camera when the first image is acquired, and the first angle data is data of a gyroscope sensor in the electronic equipment when the first image is acquired;

converting a first offset of the first image into second angle data of the first image;

obtaining third angle data of the first image based on the second angle data of the first image and the first angle data of the first image, wherein the third angle data is used for representing the motion quantity of the electronic equipment and the first camera when the first image is acquired;

And performing anti-shake processing on the first image based on third angle data of the first image to obtain a second image.

In the embodiment of the application, the electronic equipment comprises the movable camera, and the movement of the movable camera can overcome the limitation of the angle of view of the long-focus camera and meet the requirements of different shooting scenes; under the condition that the camera in the electronic equipment flexibly moves, in the scheme of the application, the translation quantity of the movable camera in the X direction and/or the Y direction can be converted into the rotation quantity identifiable by the gyroscope sensor, so as to obtain third angle data of the first image, wherein the third angle data can represent the movement quantity of the electronic equipment and the first camera when the first image is acquired; performing anti-shake processing on the first image through third angle data of the first image; as camera movements may include movements of the electronic device and movements of the movable camera; when the movable camera moves, a gyroscope sensor in the electronic equipment cannot detect the movement of the movable camera; compared with the prior art, the method and the device for processing the data of the gyroscope sensor directly adopt the data of the gyroscope sensor to perform anti-shake processing, the scheme of the application can convert the motion of the movable camera which cannot be identified by the gyroscope sensor into the rotation quantity which can be identified by the gyroscope sensor, and accurately identify the motion of the electronic equipment; and carrying out anti-shake processing on the first image based on the motion quantity of the electronic equipment and the first camera to obtain a second image, thereby improving the anti-shake effect of the image.

With reference to the first aspect, in certain implementation manners of the first aspect, the first image is any frame image in a first image stream, and the first image stream includes M frame images; the anti-shake processing is performed on the first image based on the third angle data of the first image to obtain a second image, including:

obtaining a first camera external parameter matrix based on the difference value between the third angle data of the first image and the virtual angle of the first image, wherein the virtual angle data of the first image is virtual angle data obtained by performing first path smoothing on the third angle data of the M frame image, and M is an integer greater than 1;

obtaining a first rotation correction matrix based on the first camera external reference matrix and the standard camera internal reference matrix, wherein the first rotation correction matrix is used for correcting the rotation movement of the electronic equipment and the first camera when the first image is acquired;

and multiplying the pixel points in the first image by the first rotation correction matrix to obtain the second image.

In one possible implementation, the first path smoothing process includes a filter process and a solver process; the objective function processed by the solver is to make the difference value between the third angle data of the M frame image before and after smoothing smaller.

In the embodiment of the application, a first camera external parameter matrix is obtained through the difference value between the third angle data of the first image and the virtual angle of the first image, wherein the first camera external parameter matrix is a camera external parameter obtained based on the movement of the movable camera; the second image can be obtained by multiplying the pixel points in the first image which participate outside the first camera; because the camera external parameters are obtained based on the movement of the movable camera when the anti-shake processing is carried out, the image shake introduced by the movement of the movable camera can be considered when the anti-shake processing is carried out, so that the image anti-shake effect is improved.

multiplying the characteristic points of the first image by the first rotation correction matrix to obtain mapped characteristic points;

obtaining a first translation compensation amount based on a difference value between the characteristic points of the first image and the mapped characteristic points, wherein the first translation compensation amount is used for correcting the translation movement of the electronic equipment and the first camera when the first image is acquired;

and carrying out anti-shake processing on the first image based on the first rotation correction matrix and the first translation compensation amount to obtain the second image.

In the embodiment of the application, a first camera external parameter matrix is obtained through the difference value between the third angle data of the first image and the virtual angle of the first image, wherein the first camera external parameter matrix is a camera external parameter obtained based on the movement of the movable camera; since the movement of the movable camera includes a rotational movement and a translational movement, the first rotational correction matrix is used for correcting the rotational movement of the movable camera; in the scheme of the application, the translational motion of the movable camera can be corrected through the first translational compensation amount; in the process of carrying out anti-shake processing on the first image, the rotation movement can be corrected through the first rotation correction matrix; the translational movement can be corrected by the first translational compensation amount; therefore, when the anti-shake processing is performed in the scheme of the application, both translational motion and rotational motion can be corrected, and the anti-shake effect of the image is improved.

With reference to the first aspect, in certain implementation manners of the first aspect, the performing the anti-shake processing on the first image based on the first rotation correction matrix and the first translation compensation amount to obtain the second image includes:

multiplying the pixel points of the processed first image with the first rotation correction matrix to obtain the processed first image;

and obtaining the second image based on the processed first image and the first translation compensation amount.

With reference to the first aspect, in some implementations of the first aspect, the first translation compensation amount is a virtual translation compensation amount of the first image, and the virtual translation compensation amount of the first image is a translation compensation amount obtained by performing second path smoothing on a translation compensation amount of the M frame image.

With reference to the first aspect, in certain implementation manners of the first aspect, the converting the first offset of the first image into the second angle data of the first image includes:

obtaining a tangent value based on a ratio of a first offset to a first distance of the first image, wherein the first camera is used for representing the distance between the electronic equipment and a shooting object when the first image is acquired;

And obtaining the second angle data based on the tangent value.

With reference to the first aspect, in certain implementation manners of the first aspect, the acquiring the first image, the first offset, and the first angle data includes:

and acquiring the first offset through camera driving of the electronic equipment.

With reference to the first aspect, in certain implementations of the first aspect, the first camera includes a movable tele camera.

In a second aspect, there is provided an electronic device comprising: the system comprises one or more processors, a memory and a first camera, wherein the first camera is a movable camera; the memory is coupled with the one or more processors, the memory is for storing computer program code, the computer program code comprising computer instructions that the one or more processors call to cause the electronic device to perform:

starting a camera application program;

With reference to the second aspect, in certain implementations of the second aspect, the first image is any one frame image in a first image stream, and the first image stream includes M frame images; the one or more processors invoke the computer instructions to cause the electronic device to perform:

With reference to the second aspect, in certain implementations of the second aspect, the one or more processors invoke the computer instructions to cause the electronic device to perform:

With reference to the second aspect, in some implementations of the second aspect, the first translational compensation amount is a virtual translational compensation amount of the first image, and the virtual translational compensation amount of the first image is a translational compensation amount obtained by performing second path smoothing on the translational compensation amount of the M-frame image.

And obtaining the second angle data based on the tangent value.

With reference to the second aspect, in certain implementations of the second aspect, the first camera includes a movable tele camera.

It should be appreciated that the extensions, definitions, explanations and illustrations of the relevant content in the first aspect described above also apply to the same content in the second aspect.

In a third aspect, an electronic device is provided, the electronic device comprising: one or more processors, memory, and a display screen; the memory is coupled with the one or more processors, the memory is for storing computer program code, the computer program code comprising computer instructions that the one or more processors call to cause the electronic device to perform any of the image anti-shake methods of the first aspect.

In a fourth aspect, a chip system is provided, the chip system being applied to an electronic device, the chip system comprising one or more processors for invoking computer instructions to cause the electronic device to perform any of the image anti-shake methods of the first aspect.

In a fifth aspect, there is provided a computer readable storage medium storing computer program code which, when executed by an electronic device, causes the electronic device to perform any one of the image anti-shake methods of the first aspect.

In a sixth aspect, there is provided a computer program product comprising: computer program code which, when run by an electronic device, causes the electronic device to perform any of the image anti-shake methods of the first aspect.

Drawings

FIG. 1 is a schematic diagram of a hardware system suitable for use in an electronic device of the present application;

FIG. 2 is a schematic diagram of an application scenario according to an embodiment of the present application;

FIG. 3 is a schematic diagram of a camera arrangement on an electronic device;

fig. 4 is a schematic side view of a movable tele camera according to the present application;

FIG. 5 is a schematic side view of a movable tele camera according to the present application;

FIG. 6 is a schematic side view of a movable tele camera according to an embodiment of the present application;

fig. 7 is a light path diagram of a movable tele camera according to an embodiment of the present application;

FIG. 8 is a schematic side view of a movable tele camera according to the present application;

FIG. 9 is a schematic side view of a movable tele camera according to an embodiment of the present application;

FIG. 10 is a schematic diagram of another embodiment of a movable tele camera;

FIG. 11 is a schematic view of a field angle of a movable tele camera according to an embodiment of the present application;

fig. 12 is a schematic diagram of a view angle and a gyro sensor when an electronic device or a tele camera moves according to an embodiment of the present application;

Fig. 13 is a schematic flowchart of a shooting method when an anti-shake of an image is turned on according to an embodiment of the present application;

FIG. 14 is a schematic view of a movable tele camera motion according to an embodiment of the present application;

FIG. 15 is a schematic diagram of converting motion of a movable tele camera into rotation provided by an embodiment of the present application;

FIG. 16 is a schematic view of a movable tele camera with a changeable angle of view before and after movement according to an embodiment of the present application;

FIG. 17 is a schematic view of a change in field angle after movement of another movable tele camera according to an embodiment of the present application;

FIG. 18 is a schematic flow chart of an image anti-shake method according to an embodiment of the present application;

FIG. 19 is a schematic view of a rotational amount detected by a gyro sensor according to an embodiment of the present application;

FIG. 20 is a schematic diagram of image feature point detection according to an embodiment of the present application;

FIG. 21 is a schematic diagram of a pixel map according to an embodiment of the present application;

FIG. 22 is a schematic diagram of a path smoothing process for a translational compensation amount according to an embodiment of the present application;

FIG. 23 is a schematic diagram of an electronic image stabilization process for a pixel according to an embodiment of the present application;

FIG. 24 is a schematic flow chart of an image anti-shake method provided by an embodiment of the application;

FIG. 25 is a schematic diagram of an electronic device according to an embodiment of the application;

fig. 26 is a schematic diagram of an electronic device suitable for use with the present application.

Detailed Description

In embodiments of the present application, the following terms "first", "second", etc. are used for descriptive purposes only and are not to be construed as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Thus, a feature defining "a first" or "a second" may explicitly or implicitly include one or more such feature. In the description of the present embodiment, unless otherwise specified, the meaning of "plurality" is two or more.

First, the technical terms related to the present application will be briefly described.

1. Calibrating

The process of solving the parameters may be referred to as calibration; for example, in an embodiment of the present application, the optical image stabilization calibration may refer to converting coordinates adopted by the motion correction of the optical image stabilization into coordinates that can be recognized by the motion correction of the electronic image stabilization.

For example, it may refer to converting the original code of the optical stabilization image collected by the hall sensor into data of pixel units.

2. Electronic anti-shake (Electric Image Stabilization, EIS)

The electronic anti-shake can also be called electronic image stabilization, and the EIS technology refers to anti-shake processing based on motion sensor data, and motion conditions among image frames in an image sequence are calculated through data acquired by a motion sensor during each frame of image exposure; and correct for motion between image frames to produce a relatively stable image sequence.

Fig. 1 shows a hardware system suitable for use in the electronic device of the application.

The electronic device 100 may be a mobile phone, a smart screen, a tablet computer, a wearable electronic device, an in-vehicle electronic device, an augmented reality (augmented reality, AR) device, a Virtual Reality (VR) device, a notebook computer, an ultra-mobile personal computer (UMPC), a netbook, a personal digital assistant (personal digital assistant, PDA), a projector, etc., and the specific type of the electronic device 100 is not limited in the embodiments of the present application.

The electronic device 100 may include a processor 110, an external memory interface 120, an internal memory 121, a universal serial bus (universal serial bus, USB) interface 130, a charge management module 140, a power management module 141, a battery 142, an antenna 1, an antenna 2, a mobile communication module 150, a wireless communication module 160, an audio module 170, a speaker 170A, a receiver 170B, a microphone 170C, an earphone interface 170D, a sensor module 180, keys 190, a motor 191, an indicator 192, a camera 193, a display 194, and a subscriber identity module (subscriber identification module, SIM) card interface 195, etc. The sensor module 180 may include a pressure sensor 180A, a gyro sensor 180B, an air pressure sensor 180C, a magnetic sensor 180D, an acceleration sensor 180E, a distance sensor 180F, a proximity sensor 180G, a fingerprint sensor 180H, a temperature sensor 180J, a touch sensor 180K, an ambient light sensor 180L, a bone conduction sensor 180M, and the like.

The configuration shown in fig. 1 does not constitute a specific limitation on the electronic apparatus 100. In other embodiments of the application, electronic device 100 may include more or fewer components than those shown in FIG. 1, or electronic device 100 may include a combination of some of the components shown in FIG. 1, or electronic device 100 may include sub-components of some of the components shown in FIG. 1. The components shown in fig. 1 may be implemented in hardware, software, or a combination of software and hardware.

Processor 110 may include one or more processing units. For example, the processor 110 may include at least one of the following processing units: application processors (application processor, AP), modem processors, graphics processors (graphics processing unit, GPU), image signal processors (image signal processor, ISP), controllers, video codecs, digital signal processors (digital signal processor, DSP), baseband processors, neural-Network Processors (NPU). The different processing units may be separate devices or integrated devices.

The controller can generate operation control signals according to the instruction operation codes and the time sequence signals to finish the control of instruction fetching and instruction execution.

A memory may also be provided in the processor 110 for storing instructions and data. In some embodiments, the memory in the processor 110 is a cache memory. The memory may hold instructions or data that the processor 110 has just used or recycled. If the processor 110 needs to reuse the instruction or data, it can be called directly from the memory. Repeated accesses are avoided and the latency of the processor 110 is reduced, thereby improving the efficiency of the system.

For example, in an embodiment of the present application, the processor 110 may perform: starting a camera application program; acquiring a first image, a first offset and first angle data; the first image is an image acquired by the first camera in real time, the first offset is used for representing the offset between the lens center and the imaging center of the first camera when the first image is acquired, and the first angle data is data of a gyroscope sensor in the electronic equipment when the first image is acquired; converting the first offset of the first image into second angle data of the first image; obtaining third angle data of the first image based on the second angle data of the first image and the first angle data of the first image, wherein the third angle data is used for representing the motion quantity of the electronic equipment and the first camera when the first image is acquired; and performing anti-shake processing on the first image based on the third angle data of the first image to obtain a second image.

The connection relationships between the modules shown in fig. 1 are merely illustrative, and do not constitute a limitation on the connection relationships between the modules of the electronic device 100. Alternatively, the modules of the electronic device 100 may also use a combination of the various connection manners in the foregoing embodiments.

The wireless communication function of the electronic device 100 may be implemented by the antenna 1, the antenna 2, the mobile communication module 150, the wireless communication module 160, the modem processor, the baseband processor, and the like.

The antennas 1 and 2 are used for transmitting and receiving electromagnetic wave signals. Each antenna in the electronic device 100 may be used to cover a single or multiple communication bands. Different antennas may also be multiplexed to improve the utilization of the antennas. For example: the antenna 1 may be multiplexed into a diversity antenna of a wireless local area network. In other embodiments, the antenna may be used in conjunction with a tuning switch.

The electronic device 100 may implement display functions through a GPU, a display screen 194, and an application processor. The GPU is a microprocessor for image processing, and is connected to the display 194 and the application processor. The GPU is used to perform mathematical and geometric calculations for graphics rendering. Processor 110 may include one or more GPUs that execute program instructions to generate or change display information.

For example, the motor 191 may generate vibration. The motor 191 may be used for incoming call alerting as well as for touch feedback. The motor 191 may generate different vibration feedback effects for touch operations acting on different applications. The motor 191 may also produce different vibration feedback effects for touch operations acting on different areas of the display screen 194. Different application scenarios (e.g., time alert, receipt message, alarm clock, and game) may correspond to different vibration feedback effects. The touch vibration feedback effect may also support customization.

The display screen 194 may be used to display images or video.

The electronic device 100 may implement a photographing function through an ISP, a camera 193, a video codec, a GPU, a display screen 194, an application processor, and the like.

The ISP is used to process data fed back by the camera 193. For example, when photographing, the shutter is opened, light is transmitted to the camera photosensitive element through the lens, the optical signal is converted into an electric signal, and the camera photosensitive element transmits the electric signal to the ISP for processing and is converted into an image visible to naked eyes. The ISP can carry out algorithm optimization on noise, brightness and color of the image, and can optimize parameters such as exposure, color temperature and the like of a shooting scene. In some embodiments, the ISP may be provided in the camera 193.

The camera 193 is used to capture still images or video. The object generates an optical image through the lens and projects the optical image onto the photosensitive element. The photosensitive element may be a charge coupled device (charge coupled device, CCD) or a Complementary Metal Oxide Semiconductor (CMOS) phototransistor. The photosensitive element converts the optical signal into an electrical signal, which is then transferred to the ISP to be converted into a digital image signal. The ISP outputs the digital image signal to the DSP for processing. The DSP converts the digital image signal into a standard Red Green Blue (RGB), YUV, etc. format image signal. In some embodiments, electronic device 100 may include 1 or N cameras 193, N being a positive integer greater than 1.

Illustratively, the gyroscopic sensor 180B may be used to determine a motion pose of the electronic device 100. In some embodiments, the angular velocity of electronic device 100 about three axes (i.e., x-axis, y-axis, and z-axis) may be determined by gyro sensor 180B. The gyro sensor 180B may be used for photographing anti-shake. For example, when the shutter is pressed, the gyro sensor 180B detects the shake angle of the electronic device 100, calculates the distance to be compensated by the lens module according to the angle, and makes the lens counteract the shake of the electronic device 100 through the reverse motion, so as to realize anti-shake. The gyro sensor 180B can also be used for scenes such as navigation and motion sensing games.

Illustratively, in embodiments of the present application, the gyro sensor 180B may be used to collect shake information, which may be used to represent pose changes of the electronic device during shooting.

For example, the acceleration sensor 180E may detect the magnitude of acceleration of the electronic device 100 in various directions (typically, x-axis, y-axis, and z-axis). The magnitude and direction of gravity may be detected when the electronic device 100 is stationary. The acceleration sensor 180E may also be used to recognize the gesture of the electronic device 100 as an input parameter for applications such as landscape switching and pedometer.

Illustratively, a distance sensor 180F is used to measure distance. The electronic device 100 may measure the distance by infrared or laser. In some embodiments, for example, in a shooting scene, the electronic device 100 may range using the distance sensor 180F to achieve fast focus.

Illustratively, ambient light sensor 180L is used to sense ambient light level. The electronic device 100 may adaptively adjust the brightness of the display 194 based on the perceived ambient light level. The ambient light sensor 180L may also be used to automatically adjust white balance when taking a photograph. Ambient light sensor 180L may also cooperate with proximity light sensor 180G to detect whether electronic device 100 is in a pocket to prevent false touches.

The hardware system of the electronic device 100 is described in detail above, and the software system of the electronic device 100 is described below. The software system may employ a layered architecture, an event driven architecture, a microkernel architecture, a micro-service architecture, or a cloud architecture, and the embodiment of the present application exemplarily describes the software system of the electronic device 100.

Fig. 2 is a schematic diagram of an image flow obtained by opening and closing an image anti-shake for the same shooting scene according to an embodiment of the present application.

As shown in fig. 2, the image flow shown in (a) in fig. 2 may represent an image flow obtained by the electronic device by turning off the image anti-shake; the image flow shown in fig. 2 (b) may represent an image flow obtained after the electronic device turns on the anti-shake of the image; as can be seen from the combination of (a) in fig. 2 and (b) in fig. 2, after the anti-shake of the image is turned on in the electronic device, the stability of the image stream can be improved, and larger shake between image frames in the image stream is avoided, so that a relatively stable image stream is generated.

Fig. 3 shows a schematic view of a camera 193 arrangement on an electronic device.

Illustratively, the electronic device provided by the present application may include one or more cameras 193, where the one or more cameras 193 may be located on the front of the electronic device 100 or on the back of the electronic device 100. The camera 193 located on the front side of the electronic device 100 may be referred to as a front camera, and the camera located on the back side of the electronic device 100 may be referred to as a rear camera. In the present application, the camera may also be referred to as a camera module.

For example, in an embodiment of the present application, the electronic device 100 may include 5 cameras 193, the 5 cameras 193 including 2 front cameras and 3 rear cameras. Referring to fig. 3 (a) and 3 (b), the 3 rear cameras are arranged in a row from top to bottom on the rear cover of the electronic device 100, and the 3 rear cameras are a main camera 1931, a telephoto camera 1932, and a wide-angle camera 1933 in order of arrangement.

The focal length of the wide-angle camera 1933 is shorter than that of the main camera 1931, and the focal length of the telephoto camera 1932 is longer than that of the main camera 1931, and the longer the focal length is, the smaller the angle range is, so that the angle range of view of the wide-angle camera 1933 is larger than that of the main camera 1931, and the angle range of view of the telephoto camera 1932 is smaller than that of the Yu Zhu camera 1931.

It should be appreciated that the foregoing is merely an example, and that the electronic device 100 may also include other types of cameras, such as ultra-wide angle cameras, black and white cameras, multispectral cameras, etc., as the present application is not limited in this regard.

Exemplary, fig. 4 shows a schematic side view structure of a movable tele camera 1932 according to the present application.

As shown in fig. 4, the movable tele camera comprises an optical lens, an OIS controller, a motor assembly comprising a prism and a photosensitive element, wherein the plane of the photosensitive element is perpendicular to the plane of a lens included in the optical lens; in addition, the prism in the motor component is arranged in an inclined state, so that light rays emitted by the optical lens can be refracted onto the photosensitive element; on the basis, the motor in the motor component can also enable the light path emitted by the optical lens to deviate through controlling the prism to rotate, so that the shooting angle range is enlarged. The motor assembly may further comprise an electric motor for driving the motor. Here, the motor in the motor assembly is not the same motor as the OIS motor in the OIS controller.

In some embodiments, referring to fig. 5 (a) and 5 (b), taking the coordinate system shown in fig. 5 (a) as an example, the motor in the motor assembly may control the cube structure in which the prism is located to rotate up and down around the x-axis, i.e. nodding, to expand the field of view in the z-axis direction. For example, as shown in (a) of fig. 6, the dotted line position indicates the initial position of the prism, and the motor may rotate the prism around the x-axis and in the z-axis forward direction to expand the field of view in the z-axis forward direction; alternatively, as shown in fig. 6 (b), the motor may also rotate the prism around the x-axis and in the negative z-axis direction to expand the field of view in the negative z-axis direction.

Fig. 7 is a schematic view of the optical path corresponding to the prism shown in fig. 6 (a) when the prism moves in the direction of the nodding. As shown in fig. 7, when the prism is not moving, the two incident light rays are reflected by the prism, and the imaging points on the photosensitive element are P and Q respectively. When the prism moves to the position of the prism 'in fig. 7 along the nodding direction shown in fig. 7, the same two incident light rays are reflected by the prism', and the imaging points on the photosensitive element are P 'and Q', respectively, wherein P 'corresponds to P and Q' corresponds to Q before and after the movement.

Here, since the nodding motion is in the optical path plane, the prism is displaced when nodding, and the displacements are almost equal. From this, it is clear that the length of the line segment PQ before the nodding movement is substantially equal to the length of the line segment P 'Q' after the nodding movement. Therefore, when the nod moves, the image does not shake in the y-axis direction, namely the problem of image rotation does not occur, and other distortions are avoided.

In other embodiments, referring to fig. 8 (a) and 8 (b), taking the coordinate system shown in fig. 8 (a) as an example, the motor in the motor assembly may control the cube structure in which the prism is located to rotate left and right around the y-axis, i.e. to swing, so as to expand the field of view in the x-axis direction. For example, as shown in (a) of fig. 9, the dotted line position indicates the initial position of the prism, and the motor may rotate the prism around the y-axis and in the x-axis forward direction to expand the field of view in the x-axis forward direction; alternatively, as shown in (b) of fig. 9, the motor may also rotate the prism around the y-axis and in the negative x-axis direction to expand the field of view in the negative y-axis direction.

Referring to fig. 10 for example, fig. 10 shows a schematic view of the optical path corresponding to the prism shown in fig. 9 (a) when moving in the panning direction. As shown in fig. 10, the incident light and the reflected light are symmetrical about the optical axis, and the optical axis is rotated along with the rotation of the prism. When the prism is moved by shaking the head, such as the angle of shaking the head theta, the horizontal coordinate of the incident point on the prism surface is slightly changed, the optical path is changed, and the reflection angle is changed. Whereas a change in reflection angle will result in an image rotated approximately a theta angle in the x-axis direction and a delta angle (cross talk) in the y-axis direction.

As shown in fig. 10, when the prism is not moving, an incident light beam is reflected by the prism, and the imaging point on the photosensitive element is P2. When the prism moves to the position of the prism ' shown in fig. 10 in the panning direction shown in fig. 10, the same incident light is reflected by the prism ', and the imaging point on the photosensitive element is P2'. Here, before and after the prism shaking head, P2 and P2 'are coplanar, but the line segment P2' is not parallel to the x-axis and not parallel to the y-axis, in other words, the imaging point moves obliquely in the xy plane before and after the movement.

Further, before the prism rotates, if one of the two incident light rays is O1 (assumed to be the center of the prism), the imaging point at the photosensitive element is P1; the other ray incidence point is O2 (assuming non-center of prism), and the corresponding imaging point at the photosensitive element is P2. After the prism is rotated by an angle theta, the imaging point of the incident point O1 on the photosensitive element is P1', and the imaging point of the incident point O2 on the photosensitive element is P2'. The rotation of the image at O1 is smaller and the rotation at O2 is larger due to the change of the optical path, so that the translation corresponding to the change of P1 to P1 'is different from the translation corresponding to the change of P2 to P2'.

Illustratively, the change in the angle of view at the time of photographing with the movable tele camera as shown in fig. 4 is shown in fig. 11; the initial field angle range corresponding to the tele camera is FOV0, and the initial field angle range is rectangular; when the motor component is started, the prism is controlled to perform spot head movement and the movement amplitude is maximum, the field angle range corresponding to the tele camera can be changed upwards to FOV1 or downwards to FOV2; when the motor component is started and then the prism is controlled to perform head shaking motion and the motion amplitude is maximum, the field angle range corresponding to the tele camera can be changed leftwards to FOV3 or rightwards to FOV4. When the motor assembly is started, the control prism performs both nodding and panning (the sequence is not limited), and the range of the field angle corresponding to the tele camera can be changed from the furthest upper right angle to the FOV5, from the lower right angle to the FOV6, from the lower left angle to the FOV7, from the upper left angle to the FOV8, and the like.

If the angle of rotation of the motor component for controlling the prism to perform nodding motion and panning motion is smaller, the FOV which can be realized by the tele camera is within the visual field range consisting of FOV3 to FOV 8. In addition, when all FOVs are overlapped, the maximum FOV which can be realized by the tele camera can be obtained. The maximum FOV is far larger than the FOV0 in the initial field angle range, namely the expansion of the field angle range can be realized under the condition that the motor controls the prism to rotate by the long-focus camera.

As can be seen from the above description, if the electronic device includes a movable camera, the camera movement may include the movement of the whole machine of the electronic device and the movement of the motor; in general, if the whole electronic equipment moves, a gyroscope sensor in the electronic equipment can detect the rotation quantity; the motor can generate translation in the X-axis and Y-axis directions; EIS correction is typically based on motion estimation between two frames of images in three axes, X, Y and Z, to stabilize the image stream and produce a relatively smooth sequence of images.

It should be noted that, the gyro sensor in the electronic device can only sense the motion of the electronic device, and if the electronic device does not generate motion and the camera device in the electronic device generates motion, the gyro sensor in the electronic device cannot sense the motion, as shown in fig. 12.

Case one: the electronic device moves from position 1 to position 2 and the movable camera in the electronic device is not moved.

Illustratively, the electronic device moves from position 1 to position 2, as shown in (a) of fig. 12; in the position 1, the field angle of the movable camera in the electronic equipment is FOV1; when the electronic device is moved to position 2, the field angle of the movable camera in the electronic device is FOV2, as shown in (b) in fig. 12; the electronic device moves from position 1 to position 2, and the change in the gyro sensor is shown in fig. 12 (c).

And a second case: the electronic device remains unchanged in position 1 and the movable camera moves.

Illustratively, the electronic device remains unchanged at position 1 and the movable camera in the electronic device moves, as shown in (d) of fig. 12; before the movable camera moves, the field angle of the movable camera in the electronic equipment is FOV1; after the movable camera moves, the field angle of the movable camera in the electronic device is FOV2, as shown in (e) in fig. 12; the gyro sensor in the electronic apparatus before and after the movement of the movable camera is as shown in (f) of fig. 12.

And a third case: the electronic device moves from position 1 to position 2 and the movable camera in the electronic device moves.

Illustratively, the electronic device moves from position 1 to position 2, as shown in (g) of fig. 12; in the position 1, the field angle of the movable camera in the electronic equipment is FOV1; when the electronic equipment moves to the position 2, the field angle of the movable camera in the electronic equipment is FOV2; after the movable camera moves at the position 2, the field angle of the movable camera in the electronic device is FOV3, as shown in (h) of fig. 12; in the case where the electronic device is moved from position 1 to position 2 and the movable camera in the electronic device at position 2 is moved, the gyro sensor in the electronic device is as shown in (i) of fig. 12.

As can be seen from fig. 12, when the whole electronic device is moved, or the whole electronic device is not moved and the movable camera in the electronic device is moved, the field angle of the camera in the electronic device is changed; it will be appreciated that if a moveable camera is included in the electronic device, the movement between image frames in the image stream is derived from the movement of the electronic device and the movement of the moveable camera; therefore, if the electronic device includes a movable camera, the electronic image stabilization processing is performed only based on the data (for example, the rotation amount) in the gyro sensor, and a relatively stable image stream cannot be generated.

In view of the above, the embodiments of the present application provide an image anti-shake method and an electronic device, where a gyro sensor in the electronic device cannot detect movement of a movable camera when the movable camera moves; in the scheme of the application, the translation amount of the movable camera in the X direction and/or the Y direction can be converted into the rotation amount identifiable by the gyroscope sensor; compared with the prior art, the method and the device for processing the data of the gyroscope sensor directly adopt the data of the gyroscope sensor to perform anti-shake processing, the scheme of the application can convert the motion of the movable camera which cannot be identified by the gyroscope sensor into the rotation quantity which can be identified by the gyroscope sensor, and accurately identify the motion of the electronic equipment; and carrying out anti-shake processing on the first image based on the motion quantity of the electronic equipment and the first camera to obtain a second image, thereby improving the anti-shake effect of the image.

The following describes in detail schematic flowcharts of an image anti-shake method according to an embodiment of the present application with reference to fig. 13 to 24.

Fig. 13 exemplarily shows a schematic flowchart of a photographing method in a case where the electronic apparatus 100 turns on the image anti-shake method. The method 200 includes S210 to S290; s210 to S290 are described in detail below, respectively.

S210, starting the movable camera.

Illustratively, the movable camera may be a tele camera, as shown in fig. 4; the working principle of the tele camera is shown in fig. 5 to 11; and will not be described in detail herein.

Alternatively, after the camera application is started, the wide-angle camera and the movable camera may be started at the same time.

Optionally, after the camera application is started, the wide-angle camera may be started first; and subsequently, after detecting the operation of the user, starting the movable camera.

For example, a user may instruct an electronic device to open a camera application by clicking an icon of a "camera" application; or when the electronic equipment is in the screen locking state, the user can instruct the electronic equipment to start the camera application through a gesture of sliding rightward on the display screen of the electronic equipment. Or the electronic equipment is in a screen locking state, the screen locking interface comprises an icon of the camera application program, and the user instructs the electronic equipment to start the camera application program by clicking the icon of the camera application program. Or when the electronic equipment runs other applications, the applications have the authority of calling the camera application program; the user may instruct the electronic device to open the camera application by clicking on the corresponding control. For example, while the electronic device is running an instant messaging type application, the user may instruct the electronic device to open the camera application, etc., by selecting a control for the camera function.

It should be appreciated that the above is illustrative of the operation of opening a camera application; the camera application program can be started by voice indication operation or other operation indication electronic equipment; the present application is not limited in any way.

S220, acquiring a Raw image.

For example, raw image may refer to Raw image data acquired by a movable camera.

S230, front-end processing.

Alternatively, the front-end processing may include a correlation algorithm that converts the acquired Raw image into a YUV image; the application does not limit the algorithm in any way.

By way of example, the front-end processing may refer to an image processing algorithm that converts a Raw image to a YUV image as performed in ISP.

S240, EIS processing.

Illustratively, the EIS process is used to perform image stabilization processing on motion between image frames in an image stream, generating a relatively stable image stream; for example, the image flow before EIS processing is as shown in (a) in fig. 2; the EIS-processed image stream is shown in fig. 2 (b).

In the embodiment of the application, since the movable camera is included in the electronic equipment, the movement of the camera comprises the movement of the movable camera and the movement of the electronic equipment, and the movement of the electronic equipment can be estimated through the data of the gyroscope sensor; the movement of the movable camera is the translation amount on the X axis and the Y axis; in the embodiment of the application, the offset of the movable camera on the X axis and the Y axis can be converted into the rotation amount of the gyroscope sensor when EIS is carried out; for example, fig. 14 is a schematic diagram schematically showing the positional relationship between a lens and a sensor before and after movement of a movable camera; for example, before the movable camera moves, the optical path diagram of the incident light is as shown in (a) in fig. 14; after the movable camera rotates, the optical path diagram of the incident light is shown in (b) of fig. 14; as can be seen from fig. 14 (a) and 14 (b), before the movable camera moves, the center point of the sensor receiving the incident light is point 1; after the movable camera moves, the center point of the sensor for receiving incident light is point 2; it can be understood that after the movable camera moves, the center point of the image acquired by the sensor moves from the point 1 to the point 2, and an offset is generated; in the embodiment of the present application, as shown in fig. 15, the offset amount of the point 1 to the point 2 can be converted into an equivalent angle, equivalent angle=arctan (distance between the point 1 and the point 2/distance between the center point and the point 1); wherein, the distance between the point 1 and the point 2 can represent the offset of the lens center and the imaging center, and the offset can be obtained through a camera driver; the distance between the center point and point 1 may represent the distance of the camera, which may be obtained by the camera's internal parameters.

It should be understood that in embodiments of the present application, the movement of the movable camera means that the motor pushes the prism to rotate, thereby changing the optical path to realize the change of the imaging FOV position; as shown in fig. 16.

Illustratively, as shown in (a) of fig. 16, the electronic device 100 includes a lens, an image sensor, and a gyro sensor therein; before the movable camera can move, the field angle of the movable camera is FOV1, as shown in (b) in fig. 16; as shown in (c) of fig. 16, the movable camera moves; after the movable camera moves, the field angle of the movable camera is FOV2, as shown in (d) in fig. 16; in the embodiment of the application, after the movable camera moves in the electronic equipment, the movement of the movable camera can be converted into the change of the pose of the electronic equipment; the amount of offset of the center point of the lens of the electronic device 100 from the imaging center point of the image sensor as shown in (c) in fig. 16 may be converted into rotation of the electronic device; when the electronic apparatus is in the pose as in (e) of fig. 16, the field angle of the movable camera is FOV2 as shown in (d) of fig. 16.

Further, since the movement of the camera includes a rotational movement and a translational movement; only the rotational movement amount of the camera can be detected through the data of the gyroscope sensor, and the translational movement amount of the camera cannot be accurately identified; if only the rotation motion is compensated, the shake of the image is larger when the camera is at a higher zoom magnification; in the embodiment of the application, in order to improve the accuracy of EIS processing, on one hand, the motion quantity of a movable camera can be converted into the rotation quantity of a gyroscope sensor; on the other hand, the offset of the two frames of images can be obtained by estimating the offset of the key points in the images; the translational motion of the camera is compensated through the offset of the two frames of images, so that the accuracy of EIS processing can be improved; the anti-shake effect of the image stream is improved.

Illustratively, as shown in (a) of fig. 17, the electronic device acquires an N-1 th frame image while in pose 1; the electronic equipment generates motion, and an N frame image is acquired when the electronic equipment is in the pose 2; the offset is generated by the same key point in the two frames of images of the motion of the electronic equipment, for example, the offset is L by the A1 point of the N-1 frame image and the A2 point of the N frame image; the movement of the electronic device from pose 1 to pose 2 may include rotational movement and translational movement; wherein the translational movement of the electronic device can be represented by (b) as in fig. 17; the electronic device rotational movement is represented by (c) as in fig. 17; as shown in (b) of fig. 17, the translational movement of the electronic device generates an offset amount L1; as shown in (c) of fig. 17, the rotational movement of the electronic device generates an offset amount L2; therefore, in performing EIS processing, it is necessary to consider rotation amount compensation and translation amount compensation.

It should be appreciated that the above-described changes in the pose of the electronic device may include: the movement of the electronic device and/or the movement of a movable camera in the electronic device is not limited in this regard.

Alternatively, the implementation of S240 may be described with reference to S2401 to S2410 in subsequent fig. 18.

S250, main body detection processing.

Illustratively, the subject detection process is for detecting a target object among photographic objects; in the embodiment of the present application, the subject detection process may use any existing algorithm, and the embodiment of the present application is not limited in this regard.

S260, main body tracking processing.

Illustratively, a subject tracking process may be used to achieve a focus tracking of a certain target object; alternatively, the subject tracking may implement a zooming process on the moving photographic subject; in an embodiment of the present application, the subject tracking process may employ any existing algorithm, and the embodiment of the present application is not limited in this regard.

It should be noted that, S250 and S260 may be executed when the electronic device starts the main angle mode or the intelligent snapshot mode; in other photographing modes of the camera application, S270 may be directly performed after S240 is performed in the method 200.

Further, since the coordinate information of the photographing object in the image needs to be considered in performing S250 and S260, the EIS offset output in S240 needs to be acquired.

S270, back-end processing.

Illustratively, backend processing includes, but is not limited to: the method comprises the steps of brightening treatment, denoising treatment, saturation adjustment treatment, clipping treatment or deformation treatment; in the embodiment of the present application, the back-end processing may use any existing algorithm, which is not limited in any way by the embodiment of the present application.

S280, driving and controlling the movable camera to move.

Illustratively, the movable camera may be driven to perform motion according to the position information of the shooting object in the image output in S270 to acquire the next frame of image; for example, the driving of the movable camera can control the movable camera to perform motion acquisition of the next frame of image according to the EIS offset and the offset of the region of interest output by the S270; for example, in the main angle mode, the driving of the movable camera may control the movable camera to move toward the main body moving direction according to the moving direction of the main body output at S270, and collect the next frame image.

For example, when performing EIS processing, it is necessary to acquire the amount of motion of the movable camera, convert the amount of motion of the movable camera into the rotation amount of the gyro sensor, and further implement EIS processing of the image; for example, the offset of the center of the movable camera lens from the imaging center, such as the offset between point 1 and point 2 in fig. 15, is acquired.

S290, displaying the processed image.

The processed image may be an EIS processed image, for example.

Alternatively, if in the preview shooting mode, the processed image may be displayed; among them, the preview photographing mode includes, but is not limited to: for example, a shooting mode such as a night view preview mode, a video preview mode, a shooting preview mode, and a portrait preview mode.

Alternatively, if a photographing operation, or a video recording operation, or the like is detected, the processed image may be saved.

FIG. 18 is a schematic flow chart of an image anti-shake method provided by an embodiment of the present application; the method includes S2401 to S2410; s2401 to S2410 are described in detail below.

S2401, acquiring first data and first angle data of a gyroscope sensor when a movable camera acquires a first image.

The first data is used for representing the offset between the center of the lens and the imaging center, such as the offset between the point 1 and the point 2 in fig. 15; for example, the optical center offset; alternatively, the first data may be acquired by a camera drive.

For example, the angle of the electronic device may be estimated from the first angle data of the gyro sensor.

S2402, converting the first data into angle data of the gyroscope sensor to obtain second angle data.

Illustratively, the second angle data may refer to equivalent angle data, as shown in fig. 15; the offset of point 1 to point 2 can be converted to an equivalent angle, equivalent angle = arctan (distance between point 1 and point 2/distance between the center point and point 1); wherein, the distance between the point 1 and the point 2 can represent the offset of the lens center and the imaging center, and the offset can be obtained through a camera driver; the distance between the center point and point 1 may represent the distance of the camera, which may be obtained by the camera's internal parameters.

S2403, obtaining third angle data based on the first angle data and the second angle data.

For example, the first angle data may be added to the third angle data to obtain the third angle data.

It should be appreciated that the third angle data may be real angle data of the electronic device; at this time, the angle data of the gyro sensor includes both the motion data of the electronic apparatus and the motion data of the motion of the movable camera in the electronic apparatus.

S2404, acquiring third angle data of each frame of image in the acquired image stream by the electronic equipment, and performing path smoothing processing to obtain smoothed virtual angle data.

Optionally, path smoothing processing can be performed on third angle data of each frame of image through a filtering algorithm, so as to obtain virtual angle data after smoothing processing.

Alternatively, the third angle data of each frame image may be subjected to path smoothing processing by a solver, so as to obtain smoothed angle data.

For example, the filtering algorithm or solver described above may refer to existing correlation algorithms, as the application is not limited in this regard.

For example, as shown in fig. 19, fig. 19 (a) shows angle data of a gyro sensor in the conventional scheme; namely, only the angle data of the movement of the electronic equipment can be identified; in fig. 19 (b), according to the scheme of the present application, after the movable camera moves in the electronic device, the translation amount of the movable camera in the X axis and the Y axis is converted into the angle data after the rotation amount; fig. 19 (c) shows virtual angle data obtained by performing a path smoothing process on angle data in the embodiment of the present application; it will be understood that the angle data of (a) in fig. 19 is used to represent the pose change of the electronic device, and cannot represent the movement of the movable camera in the electronic device; the angle data shown in (b) of fig. 19 is used to represent the pose change of the electronic device and the movement of the movable camera in the electronic device; the angle data shown in fig. 19 (c) is used to represent virtual angle data obtained by performing a path smoothing process on the angle data shown in fig. 19 (b).

It should be appreciated that the electronic device captures an image stream in a pose as shown in (b) in fig. 19; correcting the image stream acquired by the electronic equipment in the pose shown in (b) of fig. 19 into the image stream corresponding to the virtual pose shown in (c) of fig. 19, and obtaining the image stream after the rotational motion compensation of the camera.

In the embodiment of the application, the stability of the angle data can be ensured by carrying out the path smoothing processing on the third angle data; thereby avoiding larger image jumps generated when EIS processing is performed based on the third angle data.

S2405, obtaining a camera external parameter matrix according to the difference value between the third angle data corresponding to each image frame in the image stream and the virtual angle data after the smoothing processing.

For example, for a frame of image, a camera external parameter matrix of the frame of image may be obtained based on a difference value between the third angle data of the frame of image acquired by the electronic device and the virtual angle data after the smoothing process.

For example, the camera extrinsic matrix R may be:

where x represents the offset of the x-axis in a three-dimensional spatial coordinate system (e.g., world coordinate system); y represents the offset of the y-axis in the three-dimensional space coordinate system; z represents the offset of the z-axis in a three-dimensional space coordinate system (e.g., world coordinate system).

S2406, obtaining a rotation correction matrix based on the camera external reference matrix and the standard camera internal reference matrix.

Illustratively, the rotation correction matrix is H, which is a matrix obtained based on the following equation;

wherein K represents a standard camera reference matrix; f represents the focal length of the camera; c _x Representing the position of the optical center of the image sensor on the X-axis; c _y Representing the position of the optical center of the image sensor on the Y-axis; r represents a camera outlier matrix, as shown in S245.

S2407, mapping the key points in the first image to the virtual angle data after the smoothing processing based on the rotation correction matrix, and obtaining the mapped key points.

For example, feature point detection may be performed on the first image, and feature points (e.g., key points) in the first image are acquired; the mapped key point P' can be obtained by multiplying the key point P in the first image with the rotation transformation matrix H; p' = PH and,wherein K represents a standard camera reference matrix; f represents the focal length of the camera; c _x Representing the position of the optical center of the image sensor on the X-axis; c _y Representing the position of the optical center of the image sensor on the Y-axis; r represents a camera outlier matrix, as shown in S245.

Exemplary, the mapping process may be performed on the keypoints in the first image through the transformation matrix H, so as to obtain mapped keypoints; in the process, feature point detection and feature point pairing processing are needed, key points in the first image can be determined through feature point detection, and translational compensation quantity between a group of feature points (comprising the key points in the first image and the mapped key points) can be determined through feature point pairing; obtaining the translation compensation quantity between two frames of images by averaging the translation compensation quantity between a plurality of groups of characteristic points; as shown in fig. 20 (a) and 20 (b).

For example, as shown in fig. 21, the key point P point is a key point in an image acquired by the electronic device in the pose shown in (b) of fig. 19, and the key point P point in the image is mapped to the virtual pose shown in (c) of fig. 19, so as to obtain a P' point.

S2408, obtaining a translation compensation amount based on the offset between the mapped key points and the key points in the first image.

Illustratively, the mapped keypoints obtained by S246 may be A2 points in the nth frame image shown in (b) of fig. 17, and the keypoints in the first image may be A1 points in the N-1 th frame image shown in (b) of fig. 17; the offset L1, i.e., the shift compensation of the N-1 frame image, can be obtained from the coordinate offset between the two points.

S2409, performing path smoothing processing on the translation compensation quantity of the multi-frame images in the image stream to obtain the processed translation compensation quantity.

It should be understood that, in the embodiment of the present application, the path smoothing is performed on the translational compensation amount, so that it is ensured that the stability of the image in the image stream is determined when the pixel point in the image is compensated by the translational compensation amount; the problem of pixel point jump between images is avoided.

Illustratively, the motion between image frames in the image stream is caused by rotational motion of the camera and translational motion of the camera; as shown in fig. 22, the image stream 1 in fig. 22 is an image stream obtained by mapping the pose of the electronic device as shown in (b) in fig. 19 to an image stream corresponding to the virtual pose as shown in (c) in fig. 19, that is, the image stream 1 is an image stream after correcting the rotational movement of the camera; the image stream 2 is a target image stream, i.e., an image stream after correcting the rotational motion of the camera and the translational motion of the camera; the difference between image stream 1 and image stream 2 is due to the translational movement of the camera; correcting the image stream 1 by the translation compensation amount to obtain an image stream 2; for example, from the offset between the key point in the 1 st frame image in the image stream 1 and the key point in the 1 st frame image in the image stream 2, the translational compensation amount 1 (Δl1) can be obtained; according to the offset between the key point in the 2 nd frame image in the image stream 1 and the key point in the 2 nd frame image in the image stream 2, a translation compensation amount 2 (delta L2) can be obtained; according to the offset between the key point in the 3 rd frame image in the image stream 1 and the key point in the 3 rd frame image in the image stream 2, a translation compensation amount 3 (delta L3) can be obtained; similarly, if the image stream 1 includes i frames of images, i amounts of translational compensation can be obtained; the path smoothing processing may be performed on the i translation compensation amounts by executing S248, resulting in smoothed translation compensation amounts.

S2410, performing compensation processing on pixel points in the first image based on the rotation correction matrix and the smoothed translational compensation amount to obtain a processed first image.

It will be appreciated that the differences between image frames in the image stream are caused by the rotational movement of the camera and the translational movement of the camera; in the scheme of the application, the rotation movement of the camera can be corrected through the smoothed angle data, and the translation movement of the camera can be corrected through the smoothed translation compensation quantity; by processing each frame of image in the image stream, a relatively stable image stream is generated.

Illustratively, the post-deformation grid points are obtained by multiplying the pre-deformation grid points by the rotation transformation matrix H; then compensating the calculated smooth translation compensation quantity to the grid point after deformation; it can be understood that the input image is corrected according to the rotation correction matrix, and then the translation compensation amount is compensated to the corrected image to obtain an output image; i.e. an image compensated for rotational movement of the camera and translational movement of the camera.

For example, as shown in fig. 23, the grid points before deformation can be made to become grid points after deformation by rotating the correction matrix; the grid point displacement between the grid points before deformation and the grid points after deformation can be compensated by the smoothed translation compensation quantity; and the mapping from the input image to the output image is realized according to the deformed grid points, so that the processed output image is obtained.

Alternatively, in one implementation, after step S2405 when performing EIS processing, a position offset of each pixel point in the first image may be calculated according to the rotation transformation matrix H; and obtaining an image subjected to anti-shake processing through an interpolation algorithm, and realizing electronic image stabilizing processing.

In the embodiment of the application, in the process of performing anti-shake processing on the first image, image shake generated due to translational movement of the electronic device can be corrected by the translational compensation amount; image shake due to rotational movement of the electronic device can be corrected by the rotational correction matrix; the anti-shake effect of the image is improved.

Fig. 24 is a schematic flowchart of an image anti-shake method according to an embodiment of the present application. The method 300 may be performed by the electronic device shown in fig. 1; the method 300 includes S310 to S350, and S310 to S350 are described in detail below, respectively.

S310, starting a camera application program.

S320, acquiring a first image, a first offset and first angle data.

The first image is an image acquired by the first camera in real time, the first offset is used for representing the offset between the lens center of the first camera and the imaging center when the first image is acquired, and the first angle data is data of a gyroscope sensor in the electronic equipment when the first image is acquired.

The first image may be, for example, a Raw image acquired by a movable camera in the electronic device.

S330, converting the first offset of the first image into second angle data of the first image.

Illustratively, the first offset of the first image may be as the offset between points 1 and 2 in fig. 15; the second angle data of the first image may refer to an equivalent angle as shown in fig. 15.

S340, obtaining third angle data of the first image based on the first angle data of the first image and the second angle data of the first image.

Illustratively, the first angle data of the first image may be an angle 0 as shown in (a) in fig. 19; the third angle data of the first image may refer to an angle 0 as shown in (b) of fig. 19.

S350, performing anti-shake processing on the first image based on third angle data of the first image to obtain a second image.

Optionally, in one implementation, the first image is any one frame image in a first image stream, and the first image stream includes M frame images; performing anti-shake processing on the first image based on third angle data of the first image to obtain a second image, including:

obtaining a first camera external parameter matrix based on the difference value between the third angle data of the first image and the virtual angle of the first image, wherein the virtual angle data of the first image is virtual angle data obtained by performing first path smoothing on the third angle data of M frames of images, and M is an integer larger than 1; based on the first camera external reference matrix and the standard camera internal reference matrix, a first rotation correction matrix is obtained, and the first rotation correction matrix is used for correcting the rotation movement of the electronic equipment and the first camera when the first image is acquired; and multiplying the pixel points in the first image by the first rotation correction matrix to obtain a second image.

obtaining a first camera external parameter matrix based on the difference value between the third angle data of the first image and the virtual angle of the first image, wherein the virtual angle data of the first image is virtual angle data obtained by performing first path smoothing on the third angle data of M frames of images, and M is an integer larger than 1; based on the first camera external reference matrix and the standard camera internal reference matrix, a first rotation correction matrix is obtained, and the first rotation correction matrix is used for correcting the rotation movement of the electronic equipment and the first camera when the first image is acquired; multiplying the characteristic points of the first image by the first rotation correction matrix to obtain mapped characteristic points; obtaining a first translation compensation amount based on the difference value between the characteristic points of the first image and the mapped characteristic points, wherein the first translation compensation amount is used for correcting the translation movement of the electronic equipment and the first camera when the first image is acquired; and performing anti-shake processing on the first image based on the first rotation correction matrix and the first translation compensation amount to obtain a second image.

Alternatively, the implementation is described with reference to S2401 to S2410 shown in fig. 18, which will not be repeated here.

Optionally, in one implementation, performing anti-shake processing on the first image based on the first rotation correction matrix and the first translation compensation amount to obtain a second image, including:

multiplying the pixel points of the processed first image with a first rotation correction matrix to obtain the processed first image; and obtaining a second image based on the processed first image and the first translation compensation amount.

Optionally, in one implementation, the first translation compensation amount is a virtual translation compensation amount of the first image, and the virtual translation compensation amount of the first image is a translation compensation amount obtained by performing the second path smoothing processing on the translation compensation amount of the M-frame image.

Optionally, in one implementation, converting the first offset of the first image into the second angle data of the first image includes:

obtaining a tangent value based on the ratio of the first offset of the first image to the first distance, wherein the first camera is used for representing the distance between the electronic equipment and the shooting object when the first image is acquired; based on the tangent value, second angle data is obtained.

Optionally, in one implementation, acquiring the first image, the first offset, and the first angle data includes:

The first offset is obtained by a camera drive of the electronic device.

Optionally, in one implementation, the first camera comprises a movable tele camera.

It should be understood that the above description is intended to aid those skilled in the art in understanding the embodiments of the present application, and is not intended to limit the embodiments of the present application to the specific values or particular scenarios illustrated. It will be apparent to those skilled in the art from the foregoing description that various equivalent modifications or variations can be made, and such modifications or variations are intended to be within the scope of the embodiments of the present application.

The image anti-shake method provided by the embodiment of the application is described in detail above with reference to fig. 1 to 24; an embodiment of the device of the present application will be described in detail with reference to fig. 25 and 26. It should be understood that the apparatus in the embodiments of the present application may perform the methods of the foregoing embodiments of the present application, that is, specific working procedures of the following various products may refer to corresponding procedures in the foregoing method embodiments.

Fig. 25 is a schematic structural diagram of an electronic device according to an embodiment of the present application. The electronic device 400 includes a processing module 410 and an acquisition module 420; the electronic device 400 includes a first camera, which is a movable camera.

Wherein, the processing module 410 is configured to: starting a camera application program; the obtaining module 420 is configured to: acquiring a first image, a first offset and first angle data; the first image is an image acquired by the first camera in real time, the first offset is used for representing offset between a lens center and an imaging center of the first camera when the first image is acquired, and the first angle data is data of a gyroscope sensor in the electronic equipment when the first image is acquired; the processing module 410 is further configured to: converting a first offset of the first image into second angle data of the first image; obtaining third angle data of the first image based on the second angle data of the first image and the first angle data of the first image, wherein the third angle data is used for representing the motion quantity of the electronic equipment and the first camera when the first image is acquired; and performing anti-shake processing on the first image based on third angle data of the first image to obtain a second image.

Optionally, as an embodiment, the first image is any one frame image in a first image stream, and the first image stream includes M frame images; the processing module 410 is specifically configured to:

Optionally, as an embodiment, the processing module 410 is specifically configured to:

Optionally, as an embodiment, the first translation compensation amount is a virtual translation compensation amount of the first image, and the virtual translation compensation amount of the first image is a translation compensation amount obtained by performing a second path smoothing process on the translation compensation amount of the M frame image.

and obtaining the second angle data based on the tangent value.

Optionally, as an embodiment, the obtaining module 420 is specifically configured to:

Optionally, as an embodiment, the first camera includes a movable tele camera.

The electronic device 400 is embodied as a functional unit. The term "module" herein may be implemented in software and/or hardware, and is not specifically limited thereto.

For example, a "unit" may be a software program, a hardware circuit or a combination of both that implements the functions described above. The hardware circuitry may include application specific integrated circuits (application specific integrated circuit, ASICs), electronic circuits, processors (e.g., shared, proprietary, or group processors, etc.) and memory for executing one or more software or firmware programs, merged logic circuits, and/or other suitable components that support the described functions.

Thus, the elements of the examples described in the embodiments of the present application can be implemented in electronic hardware, or in a combination of computer software and electronic hardware. Whether such functionality is implemented as hardware or software depends upon the particular application and design constraints imposed on the solution. Skilled artisans may implement the described functionality in varying ways for each particular application, but such implementation decisions should not be interpreted as causing a departure from the scope of the present application.

Fig. 26 shows a schematic structural diagram of an electronic device provided by the present application. The dashed line in fig. 26 indicates that the unit or the module is optional. The electronic device 800 may be used to implement the image anti-shake method described in the above method embodiments.

The electronic device 800 includes one or more processors 801, which one or more processors 801 may support the electronic device 800 to implement the image anti-shake method in the method embodiments. The processor 801 may be a general purpose processor or a special purpose processor. For example, the processor 801 may be a central processing unit (central processing unit, CPU), digital signal processor (digital signal processor, DSP), application specific integrated circuit (application specific integrated circuit, ASIC), field programmable gate array (field programmable gate array, FPGA), or other programmable logic device such as discrete gates, transistor logic, or discrete hardware components.

The processor 801 may be used to control the electronic device 800, execute software programs, and process data for the software programs. The electronic device 800 may also include a communication unit 805 to enable input (reception) and output (transmission) of signals.

For example, the electronic device 800 may be a chip, the communication unit 805 may be an input and/or output circuit of the chip, or the communication unit 805 may be a communication interface of the chip, which may be an integral part of a terminal device or other electronic device.

For another example, the electronic device 800 may be a terminal device, the communication unit 805 may be a transceiver of the terminal device, or the communication unit 805 may be a transceiver circuit of the terminal device.

Electronic device 800 may include one or more memories 802 having programs 804 stored thereon, the programs 804 being executable by processor 801 to generate instructions 803, such that processor 801 performs the methods described in the method embodiments described above in accordance with instructions 803.

Optionally, the memory 802 may also have data stored therein. Optionally, processor 801 may also read data stored in memory 802, which may be stored at the same memory address as program 804, or which may be stored at a different memory address than program 804.

The processor 801 and the memory 802 may be provided separately or may be integrated together, for example, on a System On Chip (SOC) of the terminal device.

Illustratively, the memory 802 may be used to store a related program 804 of the image anti-shake method provided in the embodiment of the present application, and the processor 801 may be used to call the related program 804 of the image anti-shake method stored in the memory 802 during video processing, to execute the image anti-shake method of the embodiment of the present application; for example, a camera application is started; acquiring a first image, a first offset and first angle data; the first image is an image acquired by the first camera in real time, the first offset is used for representing the offset between the lens center and the imaging center of the first camera when the first image is acquired, and the first angle data is data of a gyroscope sensor in the electronic equipment when the first image is acquired; converting the first offset of the first image into second angle data of the first image; obtaining third angle data of the first image based on the second angle data of the first image and the first angle data of the first image, wherein the third angle data is used for representing the motion quantity of the electronic equipment and the first camera when the first image is acquired; and performing anti-shake processing on the first image based on the third angle data of the first image to obtain a second image.

The application also provides a computer program product which, when executed by the processor 801, implements the method of any one of the method embodiments of the application.

The computer program product may be stored in a memory 802, such as program 804, with the program 804 ultimately being converted into an executable object file that can be executed by the processor 801 via preprocessing, compiling, assembling, and linking processes.

The application also provides a computer readable storage medium having stored thereon a computer program which when executed by a computer implements the method according to any of the method embodiments of the application. The computer program may be a high-level language program or an executable object program.

Such as memory 802. The memory 802 may be volatile memory or nonvolatile memory, or the memory 802 may include both volatile and nonvolatile memory. The nonvolatile memory may be a read-only memory (ROM), a Programmable ROM (PROM), an Erasable PROM (EPROM), an electrically Erasable EPROM (EEPROM), or a flash memory. The volatile memory may be random access memory (random access memory, RAM) which acts as an external cache. By way of example, and not limitation, many forms of RAM are available, such as Static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDR SDRAM), enhanced SDRAM (ESDRAM), synchronous DRAM (SLDRAM), and direct memory bus RAM (DR RAM).

It will be clearly understood by those skilled in the art that, for convenience and brevity of description, specific working processes and technical effects of the apparatus and device described above may refer to corresponding processes and technical effects in the foregoing method embodiments, which are not described in detail herein.

In the several embodiments provided by the present application, the disclosed systems, devices, and methods may be implemented in other manners. For example, some features of the method embodiments described above may be omitted, or not performed. The above-described apparatus embodiments are merely illustrative, the division of units is merely a logical function division, and there may be additional divisions in actual implementation, and multiple units or components may be combined or integrated into another system. In addition, the coupling between the elements or the coupling between the elements may be direct or indirect, including electrical, mechanical, or other forms of connection.

It should be understood that, in various embodiments of the present application, the size of the sequence number of each process does not mean that the execution sequence of each process should be determined by its functions and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

In addition, the terms "system" and "network" are often used interchangeably herein. The term "and/or" herein is merely one association relationship describing the associated object, meaning that there may be three relationships, e.g., a and/or B, may represent: a exists alone, A and B exist together, and B exists alone. In addition, the character "/" herein generally indicates that the front and rear associated objects are an "or" relationship.

In summary, the foregoing description is only a preferred embodiment of the present application, and is not intended to limit the scope of the present application. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application should be included in the protection scope of the present application.

Claims

1. An image anti-shake method is characterized by being applied to electronic equipment, wherein the electronic equipment comprises a first camera, the first camera is a movable camera, and the method comprises the following steps:

starting a camera application program;

2. The image anti-shake method according to claim 1, wherein the first image is any one frame image in a first image stream, and the first image stream includes M frame images; the anti-shake processing is performed on the first image based on the third angle data of the first image to obtain a second image, including:

3. The image anti-shake method according to claim 1, wherein the first image is any one frame image in a first image stream, and the first image stream includes M frame images; the anti-shake processing is performed on the first image based on the third angle data of the first image to obtain a second image, including:

4. The image stabilization method of claim 3, wherein the performing the stabilization process on the first image based on the first rotation correction matrix and the first translation compensation amount to obtain the second image comprises:

5. The image anti-shake method according to claim 3 or 4, wherein the first translational compensation amount is a virtual translational compensation amount of the first image, and the virtual translational compensation amount of the first image is a translational compensation amount obtained by performing a second path smoothing process on the translational compensation amount of the M-frame image.

6. The image anti-shake method according to any one of claims 1 to 5, characterized in that the converting the first offset amount of the first image into the second angle data of the first image includes:

and obtaining the second angle data based on the tangent value.

7. The image anti-shake method according to any one of claims 1 to 6, wherein the acquiring the first image, the first offset, and the first angle data includes:

8. The image anti-shake method according to any one of claims 1 to 7, characterized in that the first camera includes a movable tele camera.

9. An electronic device, the electronic device comprising: one or more processors, memory, and a first camera; the memory is coupled with the one or more processors, the memory is for storing computer program code, the computer program code comprising computer instructions that the one or more processors call to cause the electronic device to perform the image anti-shake method of any of claims 1-8.

10. A chip system for application to an electronic device, the chip system comprising one or more processors for invoking computer instructions to cause the electronic device to perform the image anti-shake method of any of claims 1-8.

11. A computer readable storage medium, characterized in that the computer readable storage medium stores a computer program, which when executed by a processor, causes the processor to perform the image anti-shake method according to any one of claims 1 to 8.