WO2021227693A1 - Photographic method and apparatus, and mobile terminal and chip system - Google Patents

Abstract

A photographic method and apparatus, and a mobile terminal and a chip system, which are applied to the technical field of video photographing. The photographic method comprises: acquiring an original image captured by a camera of a mobile terminal; performing jitter correction processing and target tracking processing on the original image, and obtaining a processed image and a preview area, which comprises a tracking target, in the processed image; and taking an image in the preview area as a preview picture and displaying same. The jitter correction processing is performed on a low-magnification original image captured by a camera to avoid the jitter of a picture, and the target tracking processing is performed thereon to avoid the shaking of a tracking target in the picture. After the jitter correction processing and the target tracking processing, an obtained preview area is a certain area in a processed image corresponding to the low-magnification original image, and a stable high-magnification photographic effect can be presented by taking an image in the area as a preview picture and displaying same.

Description

Shooting method, device, mobile terminal and chip system

This application claims the priority of a Chinese patent application filed with the State Intellectual Property Office on May 15, 2020, the application number is 202010417818.9, and the application name is "Photographing method, device, mobile terminal and chip system", the entire content of which is incorporated by reference In this application.

Technical field

This application relates to the field of video shooting, in particular to a shooting method, device, mobile terminal and chip system.

Background technique

With the development of camera technology, more and more mobile terminals are integrated with cameras, and users can take pictures anytime and anywhere with the mobile terminal.

When the user follows the subject through the camera on the mobile terminal, the picture shake often occurs, especially when following the subject in a high magnification scene, the picture shake is more serious, and it is even difficult to capture the subject of interest. .

Summary of the invention

The embodiments of the present application provide a shooting method, device, mobile terminal, and chip system, which solves the problem that the picture shakes in a high-magnification scene and the shooting object of interest cannot be captured.

In order to achieve the above objectives, this application adopts the following technical solutions:

In the first aspect, an embodiment of the present application provides a shooting method, including:

Obtain the original image collected by the camera of the mobile terminal;

Performing jitter correction processing and target tracking processing on the original image to obtain a processed image and a preview area containing the tracking target in the processed image;

The image in the preview area is displayed as a preview screen.

In a possible implementation manner of the first aspect, the performing jitter correction processing and target tracking processing on the original image to obtain a processed image and a preview area of the processed image containing the tracking target includes:

Performing jitter correction processing on the original image to obtain a processed image and a correction area in the processed image;

Finding a tracking target from the original image, and determining a tracking area in the original image based on the position of the tracking target in the original image;

Jointly crop the processed image based on the correction area and the tracking area to obtain a preview area in the processed image.

In a possible implementation manner of the first aspect, the performing jitter correction processing and target tracking processing on the original image to obtain a processed image and a preview area of the processed image containing the tracking target includes:

Performing jitter correction processing on the original image to obtain a processed image and a correction area in the processed image;

Find a tracking target from the correction area, determine a tracking area in the correction area based on the position of the tracking target in the correction area, and use the tracking area as a preview area.

In a possible implementation manner of the first aspect, the performing jitter correction processing and target tracking processing on the original image to obtain a processed image and a preview area of the processed image containing a tracking target includes:

Finding a tracking target from the original image, and determining a tracking area in the original image based on the position of the tracking target in the original image;

Performing a shake correction process on the image in the tracking area to obtain a processed image and a correction area in the processed image, and use the correction area as a preview area.

In a possible implementation manner of the first aspect, the jitter correction processing includes:

Acquiring an image to be corrected, and acquiring jitter information of the mobile terminal at the time when the image to be corrected is collected;

Performing a first cropping process on the image to be corrected to obtain an edge image and an intermediate image;

Compensating the intermediate image based on the shaking information and the edge image to obtain a processed image, wherein the area corresponding to the intermediate image is a correction area.

In a possible implementation manner of the first aspect, the target tracking processing includes:

Find the tracking target from the image to be tracked and/or N frames before the image to be tracked based on the attention mechanism, where N≥1;

Based on the position of the tracking target in the image to be tracked, a second cropping process is performed to determine a tracking area in the image to be tracked.

In a possible implementation of the first aspect, before displaying the image in the preview area as a preview screen, the method further includes:

Smoothing the preview area in the processed image to obtain a smoothed preview area;

Correspondingly, the displaying the image in the preview area as a preview screen includes:

The image in the preview area after the smoothing process is displayed as a preview screen.

In a possible implementation manner of the first aspect, the performing smoothing processing on the preview area includes:

Smoothing the preview area in the processed image by using at least two filters, respectively, to obtain a smooth sub-area corresponding to each filter;

Determine the weight of each filter through the decision maker;

Based on the weight of each filter, the smooth subregion corresponding to each filter is fused to obtain a smoothed preview region.

In a possible implementation of the first aspect, after obtaining the processed image and the processed image including the preview area of the tracking target, the method further includes:

Perform zoom processing on the camera based on the size of the tracking target in the preview area in the original image, where the zoomed camera is used to collect the next frame of the original image.

In a possible implementation manner of the first aspect, the number of cameras provided on the mobile terminal is at least two;

After performing zoom processing on the camera, the method further includes:

One of the cameras after zoom processing is selected as the main camera, where the main camera is a camera that collects the next frame of original image.

In a possible implementation manner of the first aspect, the selecting one of the zoomed cameras as the main camera includes:

According to the position of the tracking target in the preview area in the original image, one of the cameras after the zoom processing is selected as the main camera.

In a second aspect, an embodiment of the present application provides a photographing device, including:

The image acquisition unit is used to acquire the original image collected by the camera of the mobile terminal;

An image processing unit, configured to perform jitter correction processing and target tracking processing on the original image to obtain a processed image and a preview area containing the tracking target in the processed image;

The image display unit is used to display the image in the preview area as a preview picture.

In a third aspect, a mobile terminal is provided, including: a memory, a processor, and a computer program stored in the memory and running on the processor, wherein the processor executes the computer program when the computer program is executed. The steps of any one of the methods described in the first aspect of the present application are implemented.

In a fourth aspect, a chip system is provided, including: a processor coupled with a memory, and the processor executes a computer program stored in the memory to implement the steps of any one of the methods described in the first aspect of the present application .

In a fifth aspect, a computer-readable storage medium is provided, and the computer-readable storage medium stores a computer program that, when executed by one or more processors, implements the method described in any one of the first aspects of the present application A step of.

In a sixth aspect, embodiments of the present application provide a computer program product, which when the computer program product runs on a mobile terminal, causes the mobile terminal to execute the method described in any one of the above-mentioned first aspects.

The shooting method provided by this application uses a camera to collect the original image, the original image is a low-magnification image, the shake correction process avoids the shaking of the picture, and the target tracking process avoids the shaking of the tracking target in the picture. After shaking correction After processing and target tracking processing, the processed image after the shake correction processing can be obtained, and at the same time, a certain area in the processed image can be obtained. The size of the area is smaller than the original image size, and finally the image in the area is displayed as a preview screen In this way, a stable high-magnification shooting effect can be presented; that is, the shake correction processing and target tracking processing in the embodiments of the present application avoid image shake through cropping compensation, and avoid the shaking of the tracking target in the image through cropping. The way also achieves high magnification scene shooting.

It is understandable that the beneficial effects of the second aspect to the sixth aspect can be referred to the related description in the first aspect, and will not be repeated here.

Description of the drawings

FIG. 1 is a schematic diagram of an application scenario of a shooting method provided by an embodiment of this application;

FIG. 2 is a schematic flowchart of a shooting method provided by an embodiment of the application;

3 is a schematic diagram of a correction area after shake correction processing and a tracking area after target tracking processing provided by an embodiment of the application;

FIG. 4 is a schematic flowchart of another shooting method provided by an embodiment of the application;

5 is a schematic diagram of a correction area, a tracking area, and a preview area corresponding to the shooting method provided by the embodiment shown in FIG. 4;

FIG. 6 is a schematic flowchart of another shooting method provided by an embodiment of the application;

FIG. 7 is a schematic diagram of a correction area, a tracking area, and a preview area corresponding to the shooting method provided by the embodiment shown in FIG. 6; FIG.

FIG. 8 is a schematic flowchart of another shooting method provided by an embodiment of the application;

FIG. 9 is a schematic diagram of a correction area, a tracking area, and a preview area corresponding to the shooting method provided by an embodiment of the application;

FIG. 10 is a schematic diagram of a smoothing process in a shooting method provided by an embodiment of the application;

FIG. 11 is a schematic flowchart of another shooting method provided by an embodiment of the application;

FIG. 12 is a schematic block diagram of a photographing device provided by an embodiment of the application;

FIG. 13 is a schematic block diagram of a mobile terminal provided by an embodiment of this application;

FIG. 14 is a schematic diagram of the software structure of a mobile terminal provided by an embodiment of the application.

Detailed ways

In the following description, for the purpose of illustration rather than limitation, specific details such as a specific system structure and technology are proposed for a thorough understanding of the embodiments of the present application. However, it should be clear to those skilled in the art that the present application can also be implemented in other embodiments without these specific details. In other cases, detailed descriptions of well-known systems, devices, circuits, and methods are omitted to avoid unnecessary details from obstructing the description of this application.

It should be understood that when used in the specification and appended claims of this application, the term "comprising" indicates the existence of the described features, wholes, steps, operations, elements and/or components, but does not exclude one or more other The existence or addition of features, wholes, steps, operations, elements, components, and/or collections thereof.

It should also be understood that in the embodiments of the present application, "one or more" refers to one, two or more than two; "and/or" describes the association relationship of the associated objects, indicating that there may be three relationships; for example, A and/or B can mean the situation where A exists alone, A and B exist at the same time, and B exists alone, where A and B can be singular or plural. The character "/" generally indicates that the associated objects before and after are in an "or" relationship.

As used in the description of this application and the appended claims, the term "if" can be construed as "when" or "once" or "in response to determination" or "in response to detecting ". Similarly, the phrase "if determined" or "if detected [described condition or event]" can be interpreted as meaning "once determined" or "in response to determination" or "once detected [described condition or event]" depending on the context ]" or "in response to detection of [condition or event described]".

In addition, in the description of the specification of this application and the appended claims, the terms "first", "second", "third", etc. are only used to distinguish the description, and cannot be understood as indicating or implying relative importance.

The reference to "one embodiment" or "some embodiments" described in the specification of this application means that one or more embodiments of this application include a specific feature, structure, or characteristic described in combination with the embodiment. Therefore, the sentences "in one embodiment", "in some embodiments", "in some other embodiments", "in some other embodiments", etc. appearing in different places in this specification are not necessarily All refer to the same embodiment, but mean "one or more but not all embodiments" unless it is specifically emphasized otherwise. The terms "including", "including", "having" and their variations all mean "including but not limited to", unless otherwise specifically emphasized.

In order to illustrate the technical solutions described in the present application, specific embodiments are used for description below.

Figure 1 shows a schematic diagram of an application scenario of the shooting method provided by an embodiment of the present application. As shown in the figure, a user (person 1) holds a mobile phone to shoot a subject (person 2), and person 2 may be walking or running. In order to track and shoot person 2, person 1 may also need to walk, run, etc. When the user is walking or running, the phone will jitter as the user walks or runs. When shooting at high magnification, it will zoom in. This kind of jitter causes the blur phenomenon of person 2 in the phone; in addition, when person 2 is running or beating, the user may need to run to be able to follow person 2, and run as person 2 is beating or person 1 is shooting. In high-magnification shooting scenes, the shaking of Person 2 in the zoomed screen will also be enlarged, causing Person 2 in the mobile phone to appear on the left side of the phone screen, such as (a) in Figure 1, and once on the right side of the phone screen. Even only part of the screen of Person 2 appears in the screen of the mobile phone, such as (b) in Figure 1. In order to solve the above problems in high magnification scenes, a shooting method is provided. For details, please refer to the description of the shooting method below.

FIG. 2 shows a schematic flowchart of a photographing method provided by an embodiment of the present application. As an example and not a limitation, the method may be applied to a mobile terminal.

Step S201: Obtain the original image collected by the camera of the mobile terminal.

In the embodiment of the present application, the mobile terminal may be a movable device with a camera function, for example, a mobile phone, a camera, a video camera, a tablet computer, a surveillance camera, a notebook, etc., which can be used on the mobile terminal. Camera or another connected camera for shooting. The original image represents an image before subsequent shake correction processing and target tracking processing. The original image may be the original large image collected by the camera, or it may be the original large image collected by the camera after preprocessing. Image. The original large image may be an image generated by information collected by the sensor when all pixels in the sensor of the camera are working.

Step S202: Perform jitter correction processing and target tracking processing on the original image to obtain a processed image and a preview area containing the tracking target in the processed image.

In the embodiments of the present application, as the subject moves during the tracking and shooting process, the user will also move the body, causing the mobile terminal held by the user to shake. In addition, even if the user holds the mobile terminal through a certain exercise or In some postures with better stability, it is difficult to avoid the phenomenon of "hand shaking".

When shooting in a low-magnification scene, because the mobile terminal has a larger field of view when collecting images, the jitter phenomenon of the mobile terminal has a relatively small impact on the collected images. However, when shooting in a high-magnification scene, the field of view of the mobile terminal when capturing images is small, and the jitter phenomenon of the mobile terminal will have a relatively large impact on the captured images, such as image jitter in the video. Therefore, in high-magnification scenes, shake correction processing is required.

Currently, the methods of shake correction processing include: optical image stabilization, electronic image stabilization, and body sensor image stabilization.

Optical image stabilization is to correct the "optical axis shift" through the floating lens of the lens. The principle is to detect the slight movement through the gyroscope in the lens, and then transmit the signal to the microprocessor. The microprocessor immediately calculates the amount of displacement that needs to be compensated, and then through the compensation lens group, according to the lens shake direction and displacement amount To be compensated; thus effectively overcome the image blur caused by camera vibration.

The principle of sensor anti-shake technology is similar to that of lens anti-shake. It mainly installs the sensor on a free-floating bracket, and also needs a gyroscope to sense the direction and amplitude of the camera's shaking, and then control the sensor to perform corresponding displacement compensation. The main reason why so many correction directions are involved is to reduce the irregular hand shaking when taking pictures.

Both the optical anti-shake and the sensor anti-shake require the participation of hardware, while the electronic anti-shake can be implemented without hardware assistance, and the image on the sensor is mainly analyzed and processed by software. When the image is blurred, the edge image is used to compensate for the blurry part in the middle, so as to achieve "anti-shake". The anti-shake principle is more like the "post-processing" of the photo.

After turning on the electronic image stabilization, the viewfinder screen will have a very obvious cropping, but the cropped part is not that the edge sensor has stopped working, but the electronic image stabilization system will use this part of the data for shake compensation. It can also be understood that the image is cut into two parts, and the outermost part (edge part) is used to compensate the inner part (middle blurred part). It can still bring a certain anti-shake effect without hardware anti-shake.

The embodiments of this application can use electronic anti-shake, and without adding external hardware, the "anti-shake" processing is also realized, and after the electronic anti-shake is used for the shake correction processing, the middle compensated area in the processed image obtained ( The intermediate image) can become clearer. This area can become a correction area. At the same time, a correction detection frame can be generated, and the obtained coordinates can also be the coordinates of the correction detection frame. A schematic diagram of the post-correction area and the tracking area after the target tracking process is shown in Figure 3 (a). Assuming A is the original image, then the original image undergoes the shake correction process to obtain the processed image A', and the processed image A'and the original The size of the image A is the same. The difference is that the intermediate image of the processed image A'is clearer than the original image A, and B is the correction area. After the shake correction processing, the obtained can be the coordinates of the four corners of the correction area, or the correction area The coordinates of the center point and the distance between the center point coordinates of the correction area and the side length.

Figure 3 does not show the change in image clarity after the shake correction process, but only shows the positional relationship between the correction area and the processed image (original image). Although the correction area is mainly described in the embodiment of the application, it is undeniable that the correction is The region generation process is carried out with the shake correction process, that is, the process of the shake correction process first compensates the middle blurred part of the original image, and the processed image is obtained after the compensation. The original image is shaken corrected in the embodiment of the application. The processed image obtained after processing has the same size as the original image. Therefore, the processed image contains the edge image and the compensated image in the middle. After compensation, the corrected area is naturally obtained. The image in the corrected area is the clearer after compensation. The middle part of the image.

In the process of tracking and shooting, as the tracking target (ie tracking target) moves, the user will also move the body. When the tracking target and the user's pace are inconsistent, or the shooting angle or direction changes, the tracking target will be in the lens. The position of the camera often shakes, and in severe cases, the tracking target even jumps out of the camera screen.

As an example, when a user holds a mobile phone to photograph a running puppy, the user needs to follow the puppy to run. However, the running route of the puppy is not fixed, it is random, and the user cannot even predict the direction the puppy is running. As a result, the position of the puppy in the camera image has been trembling during the user's follow-up shot. Within a few seconds, the puppy will appear in the middle of the camera image, then on the left side of the camera image, and then for a while. Appears in the lower right corner of the lens screen. In this way, when looking back, the viewing effect is very poor.

In view of the foregoing, the embodiment of the present application also adds target tracking processing. In the target tracking process, the tracking target needs to be determined. For example, in the process of following a photo, the user selects an area on the touch screen of the mobile terminal (for example, selects an area within a preset range around the selected position by clicking, or by drawing Select an area in a circle), and set the target in the area as the target to be tracked; you can also detect the foreground image in the picture, and use the foreground image as the tracking target; you can also use the Attention algorithm to find the saliency in the current image The object serves as the tracking target.

After the tracking target is determined in the embodiment of the application, the obtained area may also be a region. For example, the coordinates of the tracking detection frame obtained after the target tracking processing are shown in Figure 3 (b), where A is the original image, and C is Tracking area, after target tracking processing, the obtained can be the coordinates of the four corners of the tracking area, or the center point coordinates of the tracking area and the distance between the center point coordinates of the tracking area and the side length.

In the embodiment of the present application, the same logic can be used in the shake correction processing process, for example, the principle of minimum scene change in the obtained correction detection frame. The target tracking process can also adopt the same logic, for example, the principle of maximizing the matching degree of the position and size of the tracking target in the obtained tracking detection frame. In this way, it is avoided that the preview picture sequence in the preview area corresponding to the original image sequence has larger picture shake and larger tracking target position shake.

To facilitate a clearer understanding of the solution, the principle of minimal changes in the scene in the correction detection frame obtained is illustrated by examples. After the image of the i-th frame is subjected to the shake correction processing, the picture in the correction detection frame obtained is B _i , and the frame i+1 is The image shake correction process is to ensure that the difference between the obtained picture B _i+1 in the correction detection frame and the _{background image in B i} is the smallest, that is, the continuous image frame (the picture in the correction detection frame corresponding to the continuous original image) The background image changes little between the time, so as to achieve a smoother viewing effect.

Through an example to illustrate the principle of maximizing the matching degree between the position and size of the tracking target in the tracking detection frame, after the target tracking processing is performed on the i-th frame image, the image in the tracking detection frame obtained is C _i , and the image in the i+1th frame is the target the tracking process is to ensure that the tracking and detection of the screen frame obtained C _{i + 1} and C _i in the target track position and size of the smallest difference, i.e., successive image frames (consecutive track detection frame picture corresponding to the original image) The position and size of the tracking target have minimal changes between them, achieving a smoother viewing effect.

For the convenience of description, the corrected detection frame obtained after the shake correction processing can be marked as the correction area, the tracking detection frame obtained after the target tracking processing is marked as the tracking area, and the jitter correction processing and target tracking are performed on the collected original image. The processed area is recorded as the preview area.

Step S203: Display the image in the preview area as a preview picture.

In the embodiment of the present application, the image in the preview area may be output to the display and displayed as a preview screen. During video shooting, the preview picture is an image frame in the displayed video stream.

In the embodiment of the application, the original image is collected by the camera, the original image is a low-magnification image, and then the shake correction process is used to avoid the shaking of the picture, and the shaking of the tracking target in the picture is avoided through the target tracking process. After the shake correction process and the target tracking process, After that, what is obtained is a certain area in the processed image corresponding to the original image, and finally the processed image in this area is displayed as a preview screen, so that a stable high-magnification effect can be presented. That is, the shake correction processing and target tracking processing in the embodiments of the present application not only avoid the shaking of the picture, but also avoid the shaking of the tracking target in the picture by cropping, and also delimit a certain area in the original image by cropping. The corresponding processed image is displayed as a preview screen, and high-magnification scene shooting is realized at the same time.

FIG. 4 shows a schematic flowchart of a shooting method provided by an embodiment of the present application. As an example and not a limitation, the method may be applied to a mobile terminal.

Step S401: Obtain the original image collected by the camera of the mobile terminal.

In the embodiment of the present application, the content of step S401 and step S201 are the same. For details, please refer to the description of step S201, which will not be repeated here.

Step S402: Perform a shake correction process on the original image to obtain a processed image and a correction area in the processed image.

In the embodiment of the present application, the process of the image stabilization processing is to use an electronic image stabilization (EIS) algorithm to avoid blurring through image cropping compensation.

Referring to FIG. 5, FIG. 5 is a schematic flow diagram of the shake correction processing and target tracking processing provided by an embodiment of the application; as shown in FIG. The size of the image A′ is the same as that of the original image A, and the correction area B is the area corresponding to the intermediate blurred image after cropping during the correction process.

Step S403: Find a tracking target from the original image, and determine a tracking area in the original image based on the position of the tracking target in the original image.

In the embodiment of the present application, when the target tracking processing is performed, the target tracking processing can be performed based on the original image. For example, the tracking target can be found from the original image, and the position of the tracking target in the original image can be determined in the original image. Tracking area.

The principle of determining the tracking area in the original image may also be determined according to the position and size of the tracking target in the original image and the composition model corresponding to the tracking target.

As shown in Fig. 5(b), after the original image A is subject to target tracking processing, the tracking area C is obtained.

Step S404: Jointly crop the processed image based on the correction area and the tracking area to obtain a preview area in the processed image.

In the embodiment of this application, the purpose of the joint cropping process is not to crop the processed image, but to finally determine an area, which is recorded as the preview area, and the picture corresponding to the preview area in the processed image can be used as the preview picture . At the same time, the correction area, the tracking area, and the preview area can be representations of coordinates. The coordinates corresponding to the correction area can be mapped to the original image or the processed image; similarly, the coordinates corresponding to the tracking area can also be mapped to the original image or the processed image; the coordinates corresponding to the preview area can also be mapped Mapping to the original image can also be mapped to the processed image. The mapping of the preview area to the corresponding picture in the processed image is the picture within the range indicated by the coordinates corresponding to the preview area in the processed image.

When performing joint cropping, the center point of the correction area (or the side length of the correction area) corresponding to the processed image (the processed image corresponding to the original image of the current frame) and the center point of the tracking area (may also include the tracking area) The side length of) is merged to obtain the preview area in the processed image. It can also be understood that both the correction area and the tracking area are frames with coordinates. After the two frames are fused, the coordinates of the fused frame obtained are mapped to the processed image to be the preview area. It can also be known from the above description that, in addition to its own processing of image sharpness, the correction area actually obtained is the coordinates relative to the original image or the coordinates relative to the processed image (the same size as the original image). .

In order to make the preview area corresponding to the original image of the current frame and the preview area corresponding to the original image of the previous frame have a smoother visual effect, you can also refer to the preview image corresponding to the original image of the previous frame, for example, the preview image corresponding to the original image of the previous frame. The center point of the correction area, or the center point of the correction area corresponding to the previous frame of the original image, and the center point of the tracking area , Correction area or tracking area) side length as a parameter.

As an example, take the center point of the correction area corresponding to the current frame of the original image, the center point of the correction area corresponding to the previous frame of the original image, the center point of the tracking area corresponding to the current image, and the tracking area corresponding to the previous frame of the original image When the center point of is combined to obtain the preview area as a parameter, the following formula can be used:

Among them, F is the preview area, α and β are constants,

Is the center point of the correction area corresponding to the original image of the previous frame

The center point of the correction area corresponding to the original image of the current frame

Is a function of variables,

Is the center point of the tracking area corresponding to the original image of the previous frame

The center point of the tracking area corresponding to the original image of the current frame

As a function of variables.

It can also be seen from the above formula that the preview area is related to the position of the correction area and tracking area corresponding to the original image of the current frame during joint cropping, and is also related to the position of the correction area and tracking area corresponding to the original image of the previous frame ( Or the preview area corresponding to the previous frame of the original image) is related.

The position correlation between the correction area and the tracking area corresponding to the original image of the previous frame is to enable the obtained preview area of the current frame and the preview area corresponding to the original image of the previous frame to have a relatively stable preview effect.

As shown in Figure 5 (c), the position comparison between the correction area B and the tracking area C, as shown in Figure 5 (d), is the preview area obtained after the correction area B and the tracking area C are jointly cropped D.

Step S404: Display the image in the preview area as a preview screen.

In the embodiment of the present application, the content of step S404 and step S203 are the same, and the description of step S203 can be referred to, and details are not repeated here.

In the embodiment of the application, the processed image and the correction area are obtained by performing jitter correction processing on the original image, and the target tracking processing is performed on the original image to obtain the tracking area. Finally, the processed image is jointly cropped based on the correction area and the tracking area, which can be used in high magnification scenes. Obtain a more stable and smooth video picture.

FIG. 6 shows a schematic flowchart of a shooting method provided by an embodiment of the present application. As an example and not a limitation, the method can be applied to the above-mentioned mobile terminal.

Step S601: Obtain the original image collected by the camera of the mobile terminal.

In the embodiment of the present application, the contents of step S601 and step S201 are the same. For details, please refer to the description of step S201, which will not be repeated here.

Step S602: Perform a shake correction process on the original image to obtain a processed image and a correction area in the processed image.

In the embodiment of the present application, the content of step S602 is the same as that of step S402. For details, please refer to the description of step S402, which will not be repeated here.

Step S603: Find a tracking target from the correction area, determine a tracking area in the correction area based on the position of the tracking target in the correction area, and use the tracking area as a preview area.

In the embodiment of the present application, the target tracking processing performed in this step is different from step S403 in the embodiment shown in FIG. 4 in that: in the embodiment shown in FIG. 4, the tracking area is determined from the original image, while The tracking area in the illustrated embodiment is determined from the image in the correction area obtained in step S602, for example, from the corresponding picture when the correction area is mapped in the processed image. Or it is determined from the corresponding picture when the correction area is mapped in the original image. Of course, when determining the tracking area, you also need to consider the location of the tracking target in the image within the correction area, or the composition model.

Referring to FIG. 7, FIG. 7 is a schematic diagram of the correction area, the tracking area, and the preview area corresponding to the shooting method provided by the embodiment of the application; as shown in FIG. 7 (a), the original image A is first subjected to shake correction processing to obtain a processed image A'and the correction area B, and then perform target tracking processing on the image in the correction area B (the corresponding picture when the correction area is mapped in the processed image) to obtain the tracking area C, which is the preview area. Since the target tracking processing requires a cropping process, the tracking area C is located in the correction area B. Of course, when determining the tracking area, it is also possible to perform target tracking processing on the corresponding picture when the correction area B is mapped in the original image, without limitation.

Step S604: Display the image in the preview area as a preview picture.

In the embodiment of the present application, the content of step S604 and step S203 are the same, and the description of step S203 can be referred to, and the details are not repeated here.

In the embodiment of this application, the original large image is shake-corrected to obtain the processed image and the correction area, and then the target tracking process is performed on the image in the corrected area in the processed image to obtain the tracking area. The obtained tracking area is the preview area. You can get a more stable and smooth video picture under high magnification scenes.

Fig. 8 shows a schematic flowchart of a shooting method provided by an embodiment of the present application. As an example and not a limitation, the method can be applied to the above-mentioned mobile terminal.

Step S801: Obtain the original image collected by the camera of the mobile terminal.

In the embodiment of the present application, the contents of step S801 and step S201 are the same. For details, please refer to the description of step S201, which will not be repeated here.

Step S802: Find a tracking target from the original image, and determine a tracking area in the original image based on the position of the tracking target in the original image;

In the embodiment of the present application, the content of step S802 and step S403 are the same, and the description of step S403 may be referred to and will not be repeated here.

Step S803: Perform a shake correction process on the image of the tracking area, determine a correction area in the tracking area, and use the correction area as a preview area.

In the embodiment of this application, the difference from the embodiment shown in FIG. 4 and FIG. 6 is that the embodiment of this application first performs target tracking processing on the original image, and after obtaining the tracking area, the image in the tracking area is shaken corrected. Processing to obtain the processed image and the correction area, it needs to be explained that the processed image obtained at this time is not the size of the original image, but the size of the tracking area.

Referring to FIG. 9, FIG. 9 is a schematic diagram of the correction area, the tracking area, and the preview area corresponding to the shooting method provided by the embodiment of the application; as shown in FIG. 9(a), the original image A is first subjected to target tracking processing to obtain the tracking area C, then perform a shake correction process on the image in the tracking area C to obtain a processed image C′ and a correction area B, and the correction area B is the preview area. Since the shake correction processing requires a cropping process, the correction area is located in the tracking area.

Step S804: Display the image in the preview area as a preview screen.

In the embodiment of the present application, the content of step S804 and step S203 are the same, and the description of step S203 can be referred to, and details are not described herein again.

In the embodiment of this application, the target tracking process is performed on the original large image to obtain the tracking area, and then the image in the tracking area is subjected to shake correction processing to obtain the processed image and the correction area in the processed image. The obtained correction area is the preview Area, you can get a more stable and smooth video picture in a high magnification scene.

As another embodiment of the present application, the process of the jitter correction processing includes:

Acquiring an image to be corrected, and acquiring jitter information of the mobile terminal at the time when the image to be corrected is collected;

Performing a first cropping process on the image to be corrected to obtain an edge image and an intermediate image;

Compensating the intermediate image based on the shaking information and the edge image to obtain a processed image, wherein the area corresponding to the intermediate image is a correction area.

In the embodiment of the present application, the image to be corrected is an image to be subjected to shake correction processing. For example, in the embodiment shown in FIG. 4 and FIG. 6, the original image is subjected to shake correction processing, that is, the original image is to be corrected Processed image. In the embodiment shown in FIG. 8, the image in the tracking area is subjected to shake correction processing, that is, the image in the tracking area is the image to be corrected.

In the process of shaking correction processing, it is necessary to use the image acquisition time to be corrected (if the image to be corrected is an image in the tracking area, the acquisition time of the original image corresponding to the image in the tracking area) The jitter information of the mobile terminal can be used to collect the jitter information of the mobile terminal by using the gyroscope set inside the mobile terminal, generate the jitter information of the mobile terminal corresponding to each moment, and obtain the image collection to be corrected from the jitter information of the mobile terminal. The jitter information of the mobile terminal corresponding to the moment.

Based on the jitter information, the image to be corrected can be reversely compensated. When reverse compensation is performed, the first cropping process is required, that is, the edge image and the intermediate blurred image are cut out, and the jitter information is calculated The amount of compensation, and then reversely compensate the intermediate blurred image through the edge image. After the shake correction process, the intermediate blurred image will become clearer after reverse compensation. Because the intermediate blurred image is cropped from the original image Therefore, after the first cropping process, an area will be obtained. The image in this area is a compensated image, and this area can be recorded as a correction area. When performing the shake correction processing, the ratio (for example, area ratio, length ratio, width ratio, etc.) of the finally determined correction area to the image to be subjected to the shake correction processing can be set in advance.

As another embodiment of the present application, the target tracking processing includes:

Find the tracking target from the image to be tracked and/or N frames before the image to be tracked based on the attention mechanism, where N≥1;

Based on the position of the tracking target in the image to be tracked, a second cropping process is performed to determine the position in the image to be tracked.

In the embodiment of the present application, when performing target tracking processing on the image to be tracked, the tracking target needs to be determined in advance. The tracking target can be found from the image to be tracked based on the attention mechanism, or from the image to be tracked. In the N frames before the image, the tracking target can be found, and the tracking target can also be found based on the image to be tracked and the N frames before the image to be tracked.

When an algorithm based on the attention mechanism performs target tracking processing, a convolutional neural network model needs to be constructed, and the input data or different parts of the feature map in the constructed convolutional neural network model correspond to different degrees of concentration. For example, at each target scale of concern, a classification network and an attention proposal network (APN) are used. APN can be composed of two fully connected layers, and output 3 parameters to indicate the position of the box. The next scale classification network only extracts features from this newly generated box image for classification. During training, the classification result obtained by controlling the latter scale by the loss function is better than that of the previous scale, so that APN can extract the target parts that are more conducive to fine classification. As the training progresses, the APN will become more and more focused. The subtle, distinguishing part of the target.

In the process of specifically determining the tracking target, the current image to be tracked can be input into the convolutional neural network model, or the original image corresponding to the original image corresponding to the image to be tracked can be N frames of original images (or N frames of original images corresponding to the original image). The preview screen) is input into the convolutional neural network model, and the N frames before the original image corresponding to the image to be tracked (or the preview screen corresponding to the N frame of the original image) and the current image to be tracked can also be input into the convolution In the neural network model, the tracking target is thus output, and the input image of the input convolutional neural network model is not limited here.

If it is based on the tracking target determined in N frames before the image to be tracked, after finding the tracking target, it is necessary to determine the location of the tracking target in the image to be tracked. The image to be tracked is about to be processed for target tracking. In the embodiment shown in Figures 4 and 8, the image to be tracked is the original image during target tracking processing. In the embodiment shown in Figure 6, the image to be tracked is It is the image within the correction area (the correction area is mapped to the picture in the original image or the picture in the processed image). When outputting the tracking area according to the location of the tracking target, the location and size of the tracking area can be determined according to the location and size of the tracking target. For example, the center point of the smallest bounding rectangle that can be set as the tracking target is the center point of the tracking area. The length and width can be set in advance.

It should be noted that the process of obtaining the tracking area by the second cropping process described above is only used as an example, and does not limit the process of obtaining the tracking area. In practical applications, other methods for obtaining the location and size of the tracking area can also be selected. There is no restriction here.

It should be noted here that the above-mentioned process of shake correction processing and target tracking processing is only used as an example. In practical applications, the process of shake correction processing and target tracking processing in the embodiments of FIG. 4, FIG. 6 and FIG. 8 can also be different. The process of the above-mentioned shake correction processing and target tracking processing. The specific method of shake correction and target tracking is not limited here.

As another embodiment of the present application, before displaying the image in the preview area as a preview screen, the method further includes:

Smoothing the preview area in the processed image to obtain a smoothed preview area;

Correspondingly, the displaying the image in the preview area as a preview screen includes:

The image in the preview area after the smoothing process is displayed as a preview screen.

In the embodiment of the present application, in order to have a smoother preview effect between the multiple frames of preview pictures, the image in the preview area may also be smoothed to obtain a smoothed preview area. The image in the preview area after the smoothing process is displayed as a preview screen. Since the final image that needs to be obtained is the image after the shake correction process, the preview image obtained after the smoothing process is the image in which the preview area is mapped to the processed image.

The smoothing process may be that the filter moves the position and size of the preview area with reference to the position and size of the tracking target in the previous frame of preview picture.

As another embodiment of the present application, the smoothing process on the preview area includes:

Smoothing the preview area by using at least two filters to obtain a smooth sub-area corresponding to each filter;

Determine the weight of each filter through the decision maker;

Based on the weight of each filter, the smooth subregion corresponding to each filter is fused to obtain a smoothed preview region.

In the embodiment of the present application, refer to FIG. 10, which is a schematic diagram of the smoothing process in the shooting method provided by the embodiment of the present application. As shown in the figure, a filter bank can be designed, and there are multiple filters in the filter bank. A different filtering algorithm is used in each filter, and the preview area is smoothed separately through each filter, and then the smooth sub-area processed by each filter can be obtained. Because the filters in the filter bank use different Therefore, the location of the smooth sub-region obtained by each filter may be different.

In order to fuse the smooth sub-regions corresponding to multiple filters, you can also set a weight for each filter, for example, based on the position and size of the tracking target in the continuous M (M greater than or equal to 1) frame preview screen Preview the location and size of the tracking target in the preview area, and determine the weight of each filter through the decision maker. After obtaining the weight of each filter, based on the weight of each filter, the smooth sub-region corresponding to each filter can be fused to obtain a smoothed preview area. The position of the preview area after smoothing may have changed relative to the position of the preview area before smoothing.

Of course, after the preview area is smoothed, other processing may be performed on the image in the preview area, and after other processing is performed, the processed image in the preview area is sent to display. For example, the image in the preview area after smoothing can also be processed by a super-division algorithm to improve the quality of the image. The super-resolution algorithm is a low-level image processing task, which maps low-resolution images to high-resolution, and has achieved the effect of enhancing image details.

As an example, a deep learning method can be used for super-resolution algorithm processing. First, a large number of high-resolution images are used to accumulate and perform model learning. For example, first, high-resolution images are reduced in quality according to a reduced-quality model to generate a training model , And then divide the image into blocks according to the correspondence between the low-frequency part and the high-frequency part of the high-resolution image, obtain prior knowledge through learning, and establish a learning model; then input the low-resolution image into the model, and use the input low-resolution image Based on the block, search for the highest matching high-frequency block in the established learning set to restore the low-resolution image. Finally, the high-frequency details of the image can be obtained and the quality of the image can be improved.

Since the image stabilization process provided by the embodiments of this application is an electronic image stabilization method, it may reduce the quality of the image. Therefore, before the preview display is performed, the image quality is improved by the super-resolution algorithm to avoid the image in the high-magnification shooting scene. The problem of reduced quality.

FIG. 11 shows a schematic flowchart of a shooting method provided by an embodiment of the present application. As an example and not a limitation, the method includes:

Step S1101: Obtain the original image collected by the camera of the mobile terminal.

Step S1102: Perform shake correction processing and target tracking processing on the original image to obtain a processed image and a preview area containing the tracking target in the processed image.

Step S1103: Display the image in the preview area as a preview screen.

In the embodiment of the present application, the content of step S1101 to step S1103 is consistent with the content of step S201 to step S203. For details, please refer to the description of step S201 to step S203.

Step S1104: Perform zoom processing on the camera based on the size of the tracking target in the preview area in the original image, where the zoomed camera is used to collect the next frame of the original image.

In the embodiment of the present application, since the preview area is obtained, cropping is performed. For example, the first cropping process may be performed during the jitter correction process, and the target tracking process may also be based on the found tracking process. The target is subjected to the second cutting process. Therefore, the size of the preview area picture is smaller than the picture size of the original image. The camera can be zoomed according to the position and proportion of the screen size (or tracking target) in the original image in the preview area. The process of zooming is to make the next frame of the original image captured by the camera perform at least twice After cropping, you can get a better composition preview screen. Of course, the process of zoom processing may include optical zoom and/or digital zoom.

It needs to be explained here that the zooming process does not make the position of the tracking target in the next frame of original image captured by the zoomed camera and the position of the tracking target in the preview area corresponding to the current frame of the original image have a higher size. The degree of matching, instead, the preview image (position and size of the tracking target) obtained after the jitter correction processing and target tracking processing of the next frame of the original image captured by the zoomed camera is required and the preview obtained from the original image of the current frame The picture (the position and size of the tracking target) has a high degree of matching. That is, when zooming, it is necessary to consider that the next frame of the original image also needs to be cropped.

When the number of cameras set on the mobile terminal is at least two, one of the zoomed cameras can also be selected as the main camera, and the main camera is used to collect the next frame of original image. Refer to the description of step S1205.

Step S1105: Select one of the zoomed cameras as the main camera, where the main camera is a camera that collects the next frame of original image.

In the embodiment of the present application, according to the position of the tracking target in the preview area in the original image, one of the cameras after the zoom processing may be selected as the main camera.

Of course, you can also select one of the zoom-processed cameras as the main camera according to the composition model. As an example, due to the different adjustable focal lengths of the cameras, the camera that captures the original image of the current frame will be estimated to be captured even if the original image of the current frame is zoomed. The position and size of the tracking target in the preview area corresponding to the original image of the next frame cannot match the position and size of the tracking target in the preview area corresponding to the original image of the current frame, or the matching degree is low. Therefore, you need to switch other Camera, switch to the other camera to estimate that the position and size of the tracking target in the preview area corresponding to the next frame of the original image collected and the position and size of the tracking target in the preview area corresponding to the current frame of the original image are more closely matched . In another case, because multiple cameras may have some differences in their positions on the mobile terminal, there may be differences in the location of the tracking target in the original image collected by each camera. In order to make the next frame of the original image correspond to the preview area The picture of the camera matches the composition model better. Therefore, you can switch to another camera, and the picture in the preview area corresponding to the next frame of the original image collected by the camera you switched to matches the composition model more closely.

It should be noted here that during zoom processing and camera switching processing, the next original image, the preview area corresponding to the next original image, and the preview area corresponding to the next original image are all calculated and estimated. .

The process of obtaining the composition model may be: generating a composition model based on the previous preview frame and the position and size of the tracking target in the preview frame, or it may be based on the previous frame preview frame and the tracking area in the preview frame, Choose a composition model with the highest matching degree from the preset composition models.

In the embodiment of the present application, by performing zoom processing on the camera and the operation of switching the camera, the preview image obtained after the jitter correction processing and target tracking processing of the captured original image of the next frame can have a good composition.

It should be understood that the size of the sequence number of each step in the foregoing embodiment does not mean the order of execution. The execution sequence of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiment of the present application.

Corresponding to the shooting method described in the above embodiment, FIG. 12 shows a structural block diagram of a shooting device provided in an embodiment of the present application. For ease of description, only parts related to the embodiment of the present application are shown.

Referring to FIG. 12, the device 12 includes:

The image acquisition unit 121 is configured to acquire the original image collected by the camera of the mobile terminal;

The image processing unit 122 is configured to perform jitter correction processing and target tracking processing on the original image to obtain a processed image and a preview area containing a tracking target in the processed image, wherein the preview area includes the tracking target;

The image display unit 123 is configured to display the image in the preview area as a preview screen.

As another embodiment of the present application, the image processing unit 122 includes:

The shake correction processing module 1221 is configured to perform shake correction processing on the original image to obtain a processed image and a correction area in the processed image;

The target tracking processing module 1222 is configured to find a tracking target from the original image, and determine a tracking area in the original image based on the position of the tracking target in the original image;

The joint cropping module 1223 is configured to perform joint cropping on the processed image based on the correction area and the tracking area to obtain a preview area in the processed image.

As another embodiment of the present application, the image processing unit 122 includes:

A jitter correction processing module, configured to perform jitter correction processing on the original image to obtain a processed image and a correction area in the processed image;

The target tracking processing module is configured to find a tracking target from the correction area, determine a tracking area in the correction area based on the position of the tracking target in the correction area, and use the tracking area as a preview area .

As another embodiment of the present application, the image processing unit 122 includes:

A target tracking processing module, configured to find a tracking target from the original image, and determine a tracking area in the original image based on the position of the tracking target in the original image;

The shake correction processing module is configured to perform shake correction processing on the image of the tracking area to obtain a processed image and a correction area in the processed image, and use the correction area as a preview area.

As another embodiment of the present application, the device 12 includes:

The smoothing processing unit 124 is configured to perform smoothing processing on the preview area before displaying the image in the preview area as a preview picture to obtain a smoothed preview area;

Correspondingly, the image display unit 123 is also used for:

The image in the preview area after the smoothing process is displayed as a preview screen.

As another embodiment of the present application, the smoothing processing unit 124 includes:

A smoothing processing module, configured to respectively perform smoothing processing on the preview area through at least two filters to obtain a smooth sub-area corresponding to each filter;

The weight generation module is used to determine the weight of each filter through the decision maker;

The fusion module is used to perform fusion processing on the smooth sub-region corresponding to each filter based on the weight of each filter to obtain a smoothed preview area.

As another embodiment of the present application, the device 12 further includes:

The zoom unit 125 is configured to perform zoom processing on the camera based on the size of the tracking target in the preview area in the original image after obtaining the preview area containing the tracking target in the processed image, wherein the zoom processing The rear camera is used to capture the next frame of the original image.

As another embodiment of the present application, the device 12 further includes:

The camera switching unit 126 is configured to select one of the zoomed cameras as the main camera after performing the zoom processing on the camera, where the main camera is a camera that collects the next frame of original image.

As another embodiment of the present application, the camera switching unit 126 is further configured to:

According to the position of the tracking target in the preview area in the original image, one of the cameras after the zoom processing is selected as the main camera.

It should be noted that the information interaction and execution process between the above-mentioned devices/units are based on the same concept as the method embodiment of this application, and its specific functions and technical effects can be found in the method embodiment section. I won't repeat it here.

Those skilled in the art can clearly understand that, for the convenience and conciseness of description, only the division of the above functional units and modules is used as an example. In practical applications, the above functions can be allocated to different functional units and modules as needed. Module completion, that is, the internal structure of the device is divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiments can be integrated into one processing unit, or each unit can exist alone physically, or two or more units can be integrated into one unit. The above-mentioned integrated units can be hardware-based Formal realization can also be realized in the form of a software functional unit. In addition, the specific names of the functional units and modules are only for the convenience of distinguishing each other, and are not used to limit the protection scope of the present application. For the specific working process of the units and modules in the foregoing system, reference may be made to the corresponding process in the foregoing method embodiment, which will not be repeated here.

The shooting method provided in the embodiments of this application can be applied to mobile terminals such as mobile phones, cameras, tablet computers, augmented reality (AR)/virtual reality (VR) devices, and notebook computers. There are no restrictions on the specific types of terminals.

The mobile terminal provided by the embodiment of the present application includes a memory, a processor, and a computer program stored in the memory and capable of running on the processor. When the processor executes the computer program, the implementation is as in the embodiment of the present application. Steps of any shooting method provided.

Take the mobile terminal as an example of a mobile phone. FIG. 13 shows a block diagram of a part of the structure of a mobile phone provided by an embodiment of the present application. Referring to FIG. 13, the mobile phone includes: a radio frequency (RF) circuit 1310, a memory 1320, an input unit 1330, a display unit 1340, a sensor 1350, an audio circuit 1360, a wireless fidelity (WiFi) module 1370, and a processor 1380 , And power supply 1390 and other components. Those skilled in the art can understand that the structure of the mobile phone shown in FIG. 13 does not constitute a limitation on the mobile phone, and may include more or fewer components than those shown in the figure, or a combination of some components, or different component arrangements.

The following is a detailed introduction to each component of the mobile phone in conjunction with Figure 13:

The RF circuit 1310 can be used for receiving and sending signals during the process of sending and receiving information or talking. In particular, after receiving the downlink information of the base station, it is processed by the processor 1380; in addition, the designed uplink data is sent to the base station. Generally, the RF circuit includes, but is not limited to, an antenna, at least one amplifier, a transceiver, a coupler, a low noise amplifier (LNA), a duplexer, and the like. In addition, the RF circuit 1310 can also communicate with the network and other devices through wireless communication. The above-mentioned wireless communication can use any communication standard or protocol, including but not limited to Global System of Mobile Communication (GSM), General Packet Radio Service (GPRS), Code Division Multiple Access (Code Division) Multiple Access (CDMA), Wideband Code Division Multiple Access (WCDMA), Long Term Evolution (LTE)), Email, Short Messaging Service (SMS), etc.

The memory 1320 may be used to store software programs and modules. The processor 1380 executes various functional applications and data processing of the mobile phone by running the software programs and modules stored in the memory 1320, such as processing images. The memory 1320 may mainly include a storage program area and a storage data area. The storage program area may store an operating system, an application program required by at least one function (such as an image playback function, etc.), etc.; Created data (such as the location of the correction area, tracking area, or preview area, etc.), etc. In addition, the memory 1320 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one magnetic disk storage device, a flash memory device, or other volatile solid-state storage devices.

The input unit 1330 can be used to receive input digital or character information, and generate key signal input related to user settings and function control of the mobile phone. Specifically, the input unit 1330 may include a touch panel 1331 and other input devices 1332. The touch panel 1331, also called a touch screen, can collect the user's touch operations on or near it (for example, the user uses any suitable objects or accessories such as fingers, stylus, etc.) on the touch panel 1331 or near the touch panel 1331. Operation), and drive the corresponding connection device according to the preset program. Optionally, the touch panel 1331 may include two parts: a touch detection device and a touch controller. Among them, the touch detection device detects the user's touch position, and detects the signal brought by the touch operation, and transmits the signal to the touch controller; the touch controller receives the touch information from the touch detection device, converts it into contact coordinates, and then sends it To the processor 1380, and can receive and execute the commands sent by the processor 1380. In addition, the touch panel 1331 can be implemented in multiple types such as resistive, capacitive, infrared, and surface acoustic wave. In addition to the touch panel 1331, the input unit 1330 may also include other input devices 1332. Specifically, the other input device 1332 may include, but is not limited to, a physical keyboard and function keys (such as volume control keys, switch keys, etc.).

The display unit 1340 may be used to display information input by the user or information provided to the user and various menus of the mobile phone. For example, a preview screen is displayed. The display unit 1340 may include a display panel 1341. Optionally, the display panel 1341 may be configured in the form of a liquid crystal display (LCD), an organic light-emitting diode (OLED), etc. Further, the touch panel 1331 can cover the display panel 1341. When the touch panel 1331 detects a touch operation on or near it, it transmits it to the processor 1380 to determine the type of the touch event, and then the processor 1380 responds to the touch event. Type provides corresponding visual output on the display panel 1341. Although in FIG. 13, the touch panel 1331 and the display panel 1341 are used as two independent components to implement the input and input functions of the mobile phone, but in some embodiments, the touch panel 1331 and the display panel 1341 can be integrated. Realize the input and output functions of the mobile phone.

The mobile phone may also include at least one sensor 1350, such as a light sensor, a motion sensor, and other sensors. Specifically, the light sensor may include an ambient light sensor and a proximity sensor. The ambient light sensor can adjust the brightness of the display panel 1341 according to the brightness of the ambient light. The proximity sensor can close the display panel 1341 and/or when the mobile phone is moved to the ear. Or backlight. As a kind of motion sensor, the accelerometer sensor can detect the magnitude of acceleration in various directions (usually three-axis), and can detect the magnitude and direction of gravity when it is stationary. It can be used to identify mobile phone posture applications (such as horizontal and vertical screen switching, related Games, magnetometer posture calibration), vibration recognition related functions (such as pedometer, percussion), etc.; as for mobile phone configurable gyroscope (to obtain mobile phone jitter information), barometer, hygrometer, thermometer, infrared sensor, etc. Other sensors will not be repeated here.

The audio circuit 1360, the speaker 1361, and the microphone 1362 can provide an audio interface between the user and the mobile phone. The audio circuit 1360 can transmit the electrical signal converted from the received audio data to the speaker 1361, which is converted into a sound signal for output by the speaker 1361; on the other hand, the microphone 1362 converts the collected sound signal into an electrical signal, and the audio circuit 1360 After being received, it is converted into audio data, and then processed by the audio data output processor 1380, and then sent to, for example, another mobile phone via the RF circuit 1310, or the audio data is output to the memory 1320 for further processing.

WiFi is a short-distance wireless transmission technology. The mobile phone can help users send and receive emails, browse web pages, and access streaming media through the WiFi module 1370. It provides users with wireless broadband Internet access. Although FIG. 13 shows the WiFi module 1370, it is understandable that it is not a necessary component of the mobile phone, and can be omitted as needed without changing the essence of the invention.

The processor 1380 is the control center of the mobile phone. It uses various interfaces and lines to connect various parts of the entire mobile phone. Various functions and processing data of the mobile phone can be used to monitor the mobile phone as a whole. Optionally, the processor 1380 may include one or more processing units; preferably, the processor 1380 may integrate an application processor and a modem processor, where the application processor mainly processes the operating system, user interface, application programs, etc. , The modem processor mainly deals with wireless communication. It can be understood that the foregoing modem processor may not be integrated into the processor 1380.

The mobile phone also includes a power supply 1390 (such as a battery) for supplying power to various components. Preferably, the power supply can be logically connected to the processor 1380 through a power management system, so that functions such as charging, discharging, and power consumption management can be managed through the power management system.

Although not shown, the mobile phone may also include a camera. Optionally, the position of the camera on the mobile phone may be front-mounted or rear-mounted, which is not limited in the embodiment of the present application.

Optionally, the mobile phone may include a single camera, a dual camera, or a triple camera, etc., which is not limited in the embodiment of the present application.

For example, a mobile phone may include three cameras, of which one is a main camera, one is a wide-angle camera, and one is a telephoto camera.

Optionally, when the mobile phone includes multiple cameras, the positions of the multiple cameras may be set according to actual conditions, which is not limited in the embodiment of the present application.

In addition, although not shown, the mobile phone may also include a Bluetooth module, etc., which will not be repeated here.

FIG. 14 is a schematic diagram of the software structure of a mobile terminal (mobile phone) according to an embodiment of the present application. Taking the Android system as the mobile phone operating system as an example, in some embodiments, the Android system is divided into four layers, namely the application layer, the application framework layer (framework, FWK), the system layer, and the hardware abstraction layer. Through the software interface communication between.

As shown in FIG. 14, the application layer can be a series of application packages, and the application packages can include applications such as short message, calendar, camera, video, navigation, gallery, and call.

The application framework layer provides an application programming interface (application programming interface, API) and a programming framework for applications in the application layer. The application framework layer may include some predefined functions, such as functions for receiving events sent by the application framework layer.

As shown in Figure 14, the application framework layer can include a window manager, a resource manager, and a notification manager.

The window manager is used to manage window programs. The window manager can obtain the size of the display screen, determine whether there is a status bar, lock the screen, take a screenshot, etc. The content provider is used to store and retrieve data and make these data accessible to applications. The data may include videos, images, audios, phone calls made and received, browsing history and bookmarks, phone book, etc.

The resource manager provides various resources for the application, such as localized strings, icons, pictures, layout files, video files, and so on.

The notification manager enables the application to display notification information in the status bar, which can be used to convey notification-type messages, and it can automatically disappear after a short stay without user interaction. For example, the notification manager is used to notify download completion, message reminders, and so on. The notification manager can also be a notification that appears in the status bar at the top of the system in the form of a chart or a scroll bar text, such as a notification of an application running in the background, or a notification that appears on the screen in the form of a dialog window. For example, text messages are prompted in the status bar, prompt sounds, electronic devices vibrate, and indicator lights flash.

The application framework layer can also include:

A view system, which includes visual controls, such as controls that display text, controls that display pictures, and so on. The view system can be used to build applications. The display interface can be composed of one or more views. For example, a display interface that includes a short message notification icon may include a view that displays text and a view that displays pictures.

The phone manager is used to provide the communication function of the mobile phone. For example, the management of the call status (including connecting, hanging up, etc.).

The system layer can include multiple functional modules. For example: sensor service module, physical state recognition module, 3D graphics processing library (for example: OpenGL ES), etc.

The sensor service module is used to monitor the sensor data uploaded by various sensors at the hardware layer to determine the physical state of the mobile phone;

Physical state recognition module, used to analyze and recognize user gestures, faces, etc.;

The 3D graphics processing library is used to implement 3D graphics drawing, image rendering, synthesis, and layer processing.

The system layer can also include:

The surface manager is used to manage the display subsystem and provides a combination of 2D and 3D layers for multiple applications.

The media library supports playback and recording of a variety of commonly used audio and video formats, as well as still image files. The media library can support multiple audio and video encoding formats, such as: MPEG4, H.264, MP3, AAC, AMR, JPG, PNG, etc.

The hardware abstraction layer is the layer between hardware and software. The hardware abstraction layer can include display drivers, camera drivers, sensor drivers, etc., used to drive related hardware at the hardware layer, such as display screens, cameras, sensors, and so on.

The above embodiments of the shooting method can be implemented on a mobile phone having the above hardware structure/software structure.

The embodiments of the present application also provide a computer-readable storage medium, where the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps in each of the foregoing method embodiments can be realized.

The embodiments of the present application provide a computer program product. When the computer program product runs on a mobile terminal, the steps in the foregoing method embodiments can be realized when the mobile terminal is executed.

If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer readable storage medium. Based on this understanding, the implementation of all or part of the processes in the above-mentioned embodiment methods in the present application can be accomplished by instructing relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium. When executed by the processor, the steps of the foregoing method embodiments can be implemented. Wherein, the computer program includes computer program code, and the computer program code may be in the form of source code, object code, executable file, or some intermediate forms. The computer-readable medium may at least include: any entity or device capable of carrying the computer program code to the camera device/mobile terminal, recording medium, computer memory, read-only memory (ROM, Read-Only Memory), and random access memory (RAM, Random Access Memory), electric carrier signal, telecommunications signal and software distribution medium. For example, U disk, mobile hard disk, floppy disk or CD-ROM, etc. In some jurisdictions, according to legislation and patent practices, computer-readable media cannot be electrical carrier signals and telecommunication signals.

An embodiment of the present application also provides a chip system, wherein the chip system includes a processor, the processor is coupled to a memory, and the processor executes a computer program stored in the memory to implement any The steps of an alarm method in a vehicle left behind by a child. The chip system may be a single chip or a chip module composed of multiple chips.

In the above-mentioned embodiments, the description of each embodiment has its own emphasis. For parts that are not described in detail or recorded in an embodiment, reference may be made to related descriptions of other embodiments.

A person of ordinary skill in the art may be aware that the units and method steps of the examples described in combination with the embodiments disclosed herein can be implemented by electronic hardware or a combination of computer software and electronic hardware. Whether these functions are executed by hardware or software depends on the specific application and design constraint conditions of the technical solution. Professionals and technicians can use different methods for each specific application to implement the described functions, but such implementation should not be considered beyond the scope of this application.

The above-mentioned embodiments are only used to illustrate the technical solutions of the present application, not to limit them; although the present application has been described in detail with reference to the foregoing embodiments, a person of ordinary skill in the art should understand that it can still implement the foregoing The technical solutions recorded in the examples are modified, or some of the technical features are equivalently replaced; these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the application, and should be included in Within the scope of protection of this application.

Claims (14)

1. A photographing method, characterized in that it comprises:

   Obtain the original image collected by the camera of the mobile terminal;

   Performing jitter correction processing and target tracking processing on the original image to obtain a processed image and a preview area containing the tracking target in the processed image;

   The image in the preview area is displayed as a preview screen.
2. The shooting method according to claim 1, wherein the performing shake correction processing and target tracking processing on the original image to obtain a processed image and a preview area of the processed image containing the tracking target comprises:

   Performing jitter correction processing on the original image to obtain a processed image and a correction area in the processed image;

   Finding a tracking target from the original image, and determining a tracking area in the original image based on the position of the tracking target in the original image;

   Jointly crop the processed image based on the correction area and the tracking area to obtain a preview area in the processed image.
3. The shooting method according to claim 1, wherein the performing shake correction processing and target tracking processing on the original image to obtain a processed image and a preview area of the processed image containing the tracking target comprises:

   Performing jitter correction processing on the original image to obtain a processed image and a correction area in the processed image;

   Find a tracking target from the correction area, determine a tracking area in the correction area based on the position of the tracking target in the correction area, and use the tracking area as a preview area.
4. The shooting method according to claim 1, wherein the performing shake correction processing and target tracking processing on the original image to obtain a processed image and a preview area of the processed image containing the tracking target comprises:

   Finding a tracking target from the original image, and determining a tracking area in the original image based on the position of the tracking target in the original image;

   Performing a shake correction process on the image in the tracking area to obtain a processed image and a correction area in the processed image, and use the correction area as a preview area.
5. The photographing method according to any one of claims 1 to 4, wherein the shake correction processing comprises:

   Acquiring an image to be corrected, and acquiring jitter information of the mobile terminal at the time when the image to be corrected is collected;

   Performing a first cropping process on the image to be corrected to obtain an edge image and an intermediate image;

   Compensating the intermediate image based on the shaking information and the edge image to obtain a processed image, wherein the area corresponding to the intermediate image is a correction area.
6. The shooting method according to any one of claims 1 to 4, wherein the target tracking processing comprises:

   Find the tracking target from the image to be tracked and/or N frames before the image to be tracked based on the attention mechanism, where N≥1;

   Based on the position of the tracking target in the image to be tracked, a second cropping process is performed to determine a tracking area in the image to be tracked.
7. The shooting method according to any one of claims 1 to 4, wherein before displaying the image in the preview area as a preview screen, the method further comprises:

   Smoothing the preview area in the processed image to obtain a smoothed preview area;

   Correspondingly, the displaying the image in the preview area as a preview screen includes:

   The image in the preview area after the smoothing process is displayed as a preview screen.
8. 8. The shooting method according to claim 7, wherein said smoothing said preview area comprises:

   Smoothing the preview area in the processed image by using at least two filters, respectively, to obtain a smooth sub-area corresponding to each filter;

   Determine the weight of each filter through the decision maker;

   Based on the weight of each filter, the smooth subregion corresponding to each filter is fused to obtain a smoothed preview region.
9. The photographing method according to any one of claims 1 to 4, wherein after obtaining the processed image and the processed image includes a preview area of the tracking target, the method further comprises:

   Perform zoom processing on the camera based on the size of the tracking target in the preview area in the original image, where the zoomed camera is used to collect the next frame of the original image.
10. 9. The shooting method of claim 9, wherein the number of cameras provided on the mobile terminal is at least two;

    After performing zoom processing on the camera, the method further includes:

    One of the cameras after zoom processing is selected as the main camera, where the main camera is a camera that collects the next frame of original image.
11. 10. The shooting method of claim 10, wherein the selecting one of the zoomed cameras as the main camera comprises:

    According to the position of the tracking target in the preview area in the original image, one of the cameras after the zoom processing is selected as the main camera.
12. A photographing device, characterized in that it comprises:

    The image acquisition unit is used to acquire the original image collected by the camera of the mobile terminal;

    An image processing unit, configured to perform jitter correction processing and target tracking processing on the original image to obtain a processed image and a preview area containing the tracking target in the processed image;

    The image display unit is used to display the image in the preview area as a preview picture.
13. A mobile terminal, characterized in that it includes a memory, a processor, and a computer program stored in the memory and running on the processor, wherein the processor executes the computer program as follows: The steps of the method according to any one of claims 1 to 11.
14. A chip system, characterized in that the chip system includes a processor, the processor is coupled with a memory, and the processor executes a computer program stored in the memory to implement any one of claims 1 to 11 Method steps.

Images