CN114071010A

CN114071010A - Shooting method and equipment

Info

Publication number: CN114071010A
Application number: CN202011296539.8A
Authority: CN
Inventors: 吴亮; 敖欢欢; 郭勇; 王妙锋; 王军
Original assignee: Huawei Technologies Co Ltd
Current assignee: Huawei Technologies Co Ltd
Priority date: 2020-07-30
Filing date: 2020-11-18
Publication date: 2022-02-18

Abstract

The embodiment of the application provides a shooting method and equipment, relates to the technical field of electronics, and can refer to an image collected by a first camera with a larger field angle, shoot the image by using a second camera with a smaller field angle and splice to obtain a target image with a larger field angle. The scheme comprises the following steps: the electronic equipment starts a photographing function; displaying the image and the guide frame acquired by the first camera on a preview interface; after the photographing operation of a user is detected, displaying a first image and a guide frame on a photographing interface; the first image is obtained according to the image collected by the first camera, the guide frame comprises a plurality of grids, and the single grid corresponds to the field angle of the second camera; displaying splicing information on a shooting interface, wherein the splicing information is used for indicating the shooting progress; generating a spliced image according to the acquired multi-frame target shooting images; and after shooting is finished, generating a target image according to the spliced image. The embodiment of the application is used for shooting the image.

Description

Shooting method and equipment

The present application claims priority of chinese patent application entitled "a shooting method and apparatus" filed by the national intellectual property office at 30/7/2020, application number 202010754701.X, the entire contents of which are incorporated herein by reference.

Technical Field

The embodiment of the application relates to the technical field of electronics, in particular to a shooting method and equipment.

Background

At present, electronic devices such as mobile phones or watches can use wide-angle cameras with small equivalent focal lengths to capture and obtain target images with large field of view (FOV). However, the sharpness of local details on the target image is low. In particular, when the electronic device captures a large scene or a distant landscape using a wide-angle camera, the user may not be able to clearly see the detailed contents on the target image.

Disclosure of Invention

The embodiment of the application provides a shooting method and equipment, which can refer to an image collected by a first camera with a larger field angle, use a second camera with a smaller field angle to shoot the image and splice to obtain a target image with a larger field angle, and the target image has higher definition, clear details and better shooting effect.

In order to achieve the above purpose, the embodiment of the present application adopts the following technical solutions:

on one hand, the embodiment of the application provides a shooting method, which is applied to electronic equipment, wherein the electronic equipment comprises a first camera and a second camera, and the equivalent focal length of the second camera is greater than that of the first camera. The method comprises the following steps: the electronic equipment starts a photographing function. And then, the electronic equipment displays the image acquired by the first camera and the guide frame on a preview interface. After the electronic equipment detects the photographing operation of a user, displaying a first image and a guide frame superposed on the first image on a photographing interface; the first image is obtained according to the image collected by the first camera, the guide frame comprises a plurality of grids, and the single grid corresponds to the size of the field angle of the second camera. The electronic equipment displays splicing information on a shooting interface, the splicing information is used for indicating the shooting progress, the splicing information corresponds to multi-frame target shooting images matched with a plurality of grids in the guide frame, and the target shooting images are acquired through a second camera. And the electronic equipment generates a spliced image according to the multi-frame target shooting images. And after shooting is finished, the electronic equipment generates a target image according to the spliced image.

In the scheme, the electronic equipment can reference a first image collected by a first camera with a smaller equivalent focal length and a larger field angle, use a second camera with a larger equivalent focal length and a smaller field angle to shoot a target shooting image, and splice to obtain the target image with the larger field angle, wherein the target image has higher definition, clear details and better shooting effect. And on the shooting interface, the first image is displayed as a background image, and a guide frame can be superimposed and displayed on the first image so as to guide the second camera to move to shoot a target shooting image matched with the grid in the guide frame. The electronic equipment can also display the splicing information on a shooting interface so as to indicate the current shooting progress for the user in real time.

In a possible implementation manner, the first image is a first frame image acquired by a first camera after the photographing operation of the user is detected; or the first image is an image obtained by fusing Q frame images collected by the first camera after the photographing operation of the user is detected, wherein Q is an integer larger than 1; or the first image is an image acquired by the first camera in the photographing process after the photographing operation of the user is detected.

That is, during the photographing process, the first image displayed as the background image on the photographing interface may be a fixed frame image or an image refreshed in real time.

In another possible implementation manner, in the target shooting image and the matched grid, the overlapping proportion of the same content of the first image is greater than or equal to a first preset value; or in the target shooting image and the matched grid, the similarity of the histogram of the characteristic parameters of the first image is greater than or equal to a second preset value; or in the target shooting image and the matched grid, the similarity of the first image in the same transformation domain is greater than or equal to a third preset value; or in the target shooting image and the matched grid, the feature matching degree of the first image is larger than or equal to a fourth preset value.

That is, the electronic apparatus may determine whether the target captured image matches the grid in various ways.

In another possible implementation manner, the method further includes: and the electronic equipment determines the configuration parameters of the second camera for acquiring the target shooting image according to the image block corresponding to the grid on the reference image. And the electronic equipment acquires a target shooting image matched with the grid according to the configuration parameters.

Therefore, the electronic equipment can determine the configuration parameters when the second camera acquires the target shot images corresponding to different grids according to the reference images corresponding to the global range, so that the overall effect of the spliced images obtained according to the target shot images is consistent with that of the reference images as much as possible, and the overall change of the spliced images is smooth and natural in transition.

In another possible implementation, the configuration parameters include at least one or more of the following: automatic exposure AE configuration parameters, automatic white balance AWB configuration parameters, or dynamic range correction DRC configuration parameters.

Therefore, the electronic equipment can enable the whole exposure effect, the AWB effect or the dynamic range of the spliced image obtained according to the target shooting image to be consistent with the reference image as much as possible, so that the whole change of the spliced image is smooth and natural in transition.

In another possible implementation manner, the generating, by the electronic device, a stitched image according to the multiple frames of target captured images includes: the electronic equipment generates a spliced image after performing first image processing on the first target shooting image and the second target shooting image, wherein the first image processing comprises image registration.

That is, the electronic device generates a stitched image after registering the first captured target image and the second captured target image.

In another possible implementation manner, the electronic device performs image registration on the first target captured image and the second target captured image, and includes: the electronic equipment respectively extracts features of the first target shot image and the second target shot image according to a preset image registration algorithm; and the electronic equipment registers the first target shooting image and the second target shooting image according to the obtained characteristic matching pair.

That is, the electronic device may register the first target captured image and the second target captured image according to the feature matching pair after extracting the features.

In another possible implementation manner, the method further includes: and the electronic equipment corrects the registered image according to the reference image.

That is, the electronic device may correct the result after registration according to the feature matching pair based on the reference image.

In another possible implementation manner, the electronic device performs image registration on the first target captured image and the second target captured image, and includes: the electronic equipment respectively extracts features of the first target shot image and the reference image, and the first target shot image and the reference image are registered according to the obtained feature matching pairs. The electronic equipment respectively extracts the characteristics of the second target shooting image and the reference image, and the second target shooting image and the reference image are registered according to the obtained characteristic matching pair.

That is, the electronic device may register the first target captured image and the reference image and register the second target captured image and the reference image according to the feature matching pair, thereby achieving the purpose of registering the first target captured image and the second target captured image.

In another possible implementation manner, the method further includes: the electronic equipment extracts features of the first target shot image and the second target shot image respectively, and the first target shot image and the second target shot image are registered according to the obtained feature matching pairs; the electronic equipment calculates a homography matrix according to the feature matching pairs of the overlapping areas of the first target shot image, the second target shot image and the reference image; and the electronic equipment deforms the second target shooting image according to the homography matrix.

That is, the electronic device may register the first target captured image and the reference image, register the second target captured image and the reference image, and register the first target captured image and the second target captured image, respectively, and calculate the homography matrix according to the feature matching pairs of the overlapping regions of the first target captured image, the second target captured image, and the reference image to deform the second target captured image, so that the fusion effect of the second target captured image is better.

In another possible implementation manner, the electronic device performs image registration on the first target captured image and the second target captured image, and includes: attaching the first target shooting image to a coordinate position with the same image content on the reference image; attaching the second target shooting image to a coordinate position with the same image content on the reference image; and if a hole exists between the first target shooting image and the second target shooting image, filling the hole according to the image content of the position corresponding to the hole on the reference image.

In this approach, the electronic device may register the first and second captured images of the object according to a coordinate system of the reference image.

In another possible implementation manner, the first image processing further includes a ghost-removing processing, and the electronic device generates a stitched image after performing the first image processing on the first target captured image and the second target captured image, including: keeping a ghost-free image of a moving object on the first target shooting image; deleting the image of the moving object on the second target shot image; carrying out image registration on the first target shooting image and the second target shooting image to generate a spliced image; and filling the holes in the spliced image according to the reference image.

In the scheme, the electronic equipment can keep the ghost-free image of the moving object on the first target shooting image, and carry out hole filling on the spliced image generated by the first target shooting image and the second target shooting image so as to achieve the purpose of removing ghost.

In another possible implementation manner, the first image processing further includes a ghost-removing processing, and the electronic device generates a stitched image after performing the first image processing on the first target captured image and the second target captured image, including: carrying out image registration on the first target shooting image and the second target shooting image; keeping a no-ghost image of the moving object on the registered image, wherein the no-ghost image is from the first target shooting image, the second target shooting image or the first image; deleting images of the moving object in other areas on the registered images; and filling the hole caused by deletion according to the reference image to generate a spliced image.

In the scheme, the electronic device may register the first target captured image and the second target captured image, retain the ghost-free image of the moving object on the registered image, delete the images of other moving objects on the registered image, and perform hole filling according to the reference image.

In another possible implementation manner, the first image processing further includes a ghost-removing processing, and the electronic device generates a stitched image after performing the first image processing on the first target captured image and the second target captured image, including: deleting images of the moving object on the first target shot image and the second target shot image; judging the motion track of the moving object according to the multi-frame images collected by the first camera so as to delete the image of the moving object on the first image and obtain a first image without ghost; attaching the first target shot image and the second target shot image, from which the image of the moving object is deleted, to the first image without the ghost; and filling the hole caused by deletion according to the reference image to generate a spliced image.

In the scheme, the electronic equipment can obtain a first image without ghost, attach the first target shot image and the second target shot image, from which the image of the moving object is deleted, to the first image without ghost, and perform hole filling according to the reference image to generate a stitched image.

In another possible implementation, the first image processing further includes a dodging and color-homogenizing process and an image fusion process.

That is to say, the electronic device generates a stitched image after performing registration, light and color evening processing and image fusion processing on the first target shot image and the second target shot image.

In another possible implementation manner, the electronic device generates a target image according to the stitched image, including: after the electronic equipment performs second image processing on the spliced image, a target image is generated, wherein the second image processing at least comprises one or more of the following steps: ghost image removing, dynamic range enhancement, high-frequency component fusion, light and color homogenizing treatment or cavity filling.

That is, the electronic device performs processes such as de-ghosting, dynamic range enhancement, high-frequency component fusion, light and color homogenizing, or hole filling on the stitched image, and then generates a target image.

In another possible implementation manner, the electronic device performs high-frequency component fusion on the spliced image, including: the electronic equipment extracts the high-frequency component image of the reference image and fuses the high-frequency component image and the spliced image so as to enhance the high-frequency details of the spliced image and enhance the definition and the details of the spliced image and the target image.

In another possible implementation manner, the electronic device performs light and color evening processing on the stitched image, including: the electronic equipment synthesizes images acquired by a plurality of frames of first cameras in a high dynamic range; and the electronic equipment performs light and color homogenizing treatment on the spliced image according to the obtained high dynamic range information.

Therefore, the electronic equipment can carry out dodging and color-homogenizing treatment on the spliced image according to the synthesized high dynamic range information, so that the dynamic range of the spliced image is enhanced.

In another possible implementation manner, the electronic device performs hole filling on the stitched image, including: splicing areas corresponding to holes of images acquired by a plurality of frames of first cameras, and performing super-resolution image synthesis; and filling holes in the spliced image according to the obtained image information.

Therefore, the filling of the holes and the hollow edges can have higher resolution, and the image quality and the user experience are improved.

In another possible implementation manner, the electronic device performs hole filling on the stitched image, including: performing image super-resolution processing on an area corresponding to the cavity of the spliced image on the reference image; and filling holes in the spliced image according to the obtained image information.

In another possible implementation manner, the reference image is a first frame image acquired by the first camera after the photographing operation of the user is detected; or the first image is an image obtained by fusing Q frame images collected by the first camera after the photographing operation of the user is detected, wherein Q is an integer larger than 1.

That is, the reference image in the photographing process is a fixed image of the same frame.

In another possible implementation manner, the electronic device determining that the shooting is finished includes: and after the grids in the guide frame are matched, determining that the shooting is finished.

That is, the electronic device automatically ends shooting after determining that the meshes in the guide frame are matched.

In another possible implementation manner, the electronic device generates a target image according to the stitched image, including: the spliced image is a target image; or if the edges of the spliced images are not aligned, cutting the spliced images to obtain target images with aligned edges; or if the edges of the stitched images are not aligned, filling the edge areas of the stitched images according to the first image to obtain the target images with aligned edges.

That is, the target image is the same as the stitched image, or an image obtained by performing edge cropping or edge filling on the stitched image.

In another possible implementation manner, the electronic device determining that the shooting is finished includes: before the grids in the guide frame are not matched, if the photographing stopping operation of the user is detected, the photographing is determined to be finished; or if the moving direction of the electronic equipment moves out of the guide frame, determining that the shooting is finished; or if the deviation range of the moving direction of the electronic equipment and the indication direction of the guide frame is larger than or equal to a fifth preset value, determining that the shooting is finished.

That is, the electronic device may determine to end the photographing in various ways.

In another possible implementation manner, the electronic device generates a target image according to the stitched image, including: generating a target image according to the matched spliced image corresponding to the whole row/column grid; or generating a target image according to the spliced image corresponding to the matched grid and the image area corresponding to the unmatched grid on the first image.

That is, before the grids in the guide frame are not matched, the electronic device may generate the target image according to the stitched image corresponding to the entire row/column grid, or perform image filling in combination with the image area corresponding to the unmatched grid on the first image to generate the target image.

In another possible implementation manner, the method further includes: the electronic equipment acquires a target zooming magnification, the guide frame corresponds to the target zooming magnification, and the target zooming magnification is larger than the zooming magnification of the first camera and smaller than the zooming magnification of the second camera. The electronic equipment generates a target image according to the spliced image, and the method comprises the following steps: and the electronic equipment cuts the spliced image to generate a target image, wherein the target image corresponds to the target zooming magnification.

In the scheme, the electronic equipment can splice the target shooting images collected by the second camera with the smaller field angle to obtain spliced images with the larger field angle and the clearness, and then cut the spliced images to obtain clear target images corresponding to the target zoom magnification. Moreover, the electronic equipment does not need to carry out image amplification through digital zooming, so that the high resolution of the second camera and the high definition of the second image can be reserved, and the zooming effect of the optical zooming is realized.

On the other hand, the embodiment of the application provides another shooting method, which is applied to electronic equipment, wherein the electronic equipment comprises a first camera and a second camera, and the equivalent focal length of the second camera is greater than that of the first camera. The method comprises the following steps: the electronic equipment starts a photographing function. The electronic equipment displays a third image and an image frame on the preview interface, the third image is an image acquired by the first camera, the image range of the third image in the image frame corresponds to the image range of the second image, and the second image is an image acquired by the second camera. After the electronic equipment detects the photographing operation of the user, a first image and an image frame are displayed on a photographing interface, and the first image is obtained according to the image collected by the first camera. The electronic equipment displays splicing information on a shooting interface, the splicing information is used for indicating the shooting progress, the splicing information corresponds to multiple frames of target shooting images acquired in the shooting process, and the adjacent target shooting images are matched with each other. And the electronic equipment generates a spliced image according to the multi-frame target shooting images. And after shooting is finished, the electronic equipment generates a target image according to the spliced image.

In the scheme, the electronic equipment can reference a first image collected by a first camera with a smaller equivalent focal length and a larger field angle, use a second camera with a larger equivalent focal length and a smaller field angle to shoot a target shooting image, and splice to obtain the target image with the larger field angle, wherein the target image has higher definition, clear details and better shooting effect. And the electronic equipment displays the image frame on the preview interface and the shooting interface so as to facilitate the user to move the second camera according to the real-time shooting range of the second camera. Moreover, the electronic equipment can also display the splicing information on the shooting interface so as to indicate the current shooting progress for the user in real time.

In a possible implementation manner, for a plurality of frames of target captured images, a third target captured image and a fourth target captured image that are matched satisfy: the deviation between the abscissa range of the third target photographed image and the abscissa range of the fourth target photographed image is less than or equal to a first preset threshold; or the deviation between the vertical coordinate range of the third target captured image and the vertical coordinate range of the fourth target captured image is less than or equal to a second preset threshold.

That is, the third captured target image and the fourth captured target image, which are matched, are substantially distributed in the left-right direction or substantially distributed in the up-down direction.

In another possible implementation manner, the third target captured image and the fourth target captured image that are matched further satisfy: the overlapping area between the third target photographed image and the fourth target photographed image is greater than or equal to a sixth preset value; or, a gap between the third target captured image and the fourth target captured image is less than or equal to a seventh preset value.

That is, between the third target captured image and the fourth target captured image that match, the overlapping area is small, or the gap is small.

In another aspect, an embodiment of the present application provides a shooting device, which is included in an electronic device. The device has the function of realizing the behavior of the electronic equipment in any one of the above aspects and possible designs, so that the electronic equipment executes the shooting method executed by the electronic equipment in any one of the possible designs of the above aspects. The function can be realized by hardware, and can also be realized by executing corresponding software by hardware. The hardware or software includes at least one module or unit corresponding to the above functions. For example, the apparatus may include an activation unit, a detection unit, a display unit, a generation unit, and the like.

In another aspect, an embodiment of the present application provides an electronic device, including: the first camera and the second camera are used for collecting images; a screen for displaying an interface; one or more processors; a memory; and one or more computer programs, the one or more computer programs being stored in the memory, the one or more computer programs including instructions which, when executed by the electronic device, cause the electronic device to perform the photographing method performed by the electronic device in any of the possible designs of the above aspects.

In another aspect, an embodiment of the present application provides an electronic device, including: one or more processors; and a memory having code stored therein. When executed by an electronic device, cause the electronic device to perform the photographing method performed by the electronic device in any of the possible designs of the above aspects.

In yet another aspect, an embodiment of the present application provides a computer-readable storage medium, which includes computer instructions, when the computer instructions are executed on an electronic device, cause the electronic device to perform the shooting method in any one of the possible designs of the above aspect.

In still another aspect, the present application provides a computer program product, which when run on a computer, causes the computer to execute the shooting method performed by the electronic device in any one of the possible designs of the above aspect.

In another aspect, an embodiment of the present application provides a chip system, which is applied to an electronic device. The chip system includes one or more interface circuits and one or more processors; the interface circuit and the processor are interconnected through a line; the interface circuit is used for receiving signals from a memory of the electronic equipment and sending the signals to the processor, and the signals comprise computer instructions stored in the memory; the computer instructions, when executed by the processor, cause the electronic device to perform the method of capturing in any of the possible designs of the above aspects.

For the advantageous effects of the other aspects, reference may be made to the description of the advantageous effects of the method aspects, which is not repeated herein.

Drawings

Fig. 1 is a schematic structural diagram of an electronic device according to an embodiment of the present disclosure;

fig. 2 is a schematic flowchart of a shooting method according to an embodiment of the present disclosure;

FIG. 3A is a schematic diagram of a set of interfaces provided by an embodiment of the present application;

fig. 3B is a schematic interface diagram provided in an embodiment of the present application;

FIG. 4 is a schematic view of another interface provided by an embodiment of the present application;

FIG. 5 is a schematic view of another set of interfaces provided by embodiments of the present application;

FIG. 6 is a schematic diagram of another set of interfaces provided by an embodiment of the present application;

FIG. 7A is a schematic view of another set of interfaces provided by embodiments of the present application;

FIG. 7B is a schematic diagram of another set of interfaces provided by an embodiment of the present application;

FIG. 8 is a schematic view of another set of interfaces provided by embodiments of the present application;

FIG. 9A is a schematic view of another set of interfaces provided by embodiments of the present application;

fig. 9B is a schematic diagram of a group of shooting sequences provided in the embodiment of the present application;

fig. 10A is a schematic diagram of a set of interfaces and a schematic diagram of a camera position provided in an embodiment of the present application;

FIG. 10B is a schematic view of an interface provided by an embodiment of the present application;

FIG. 11 is a schematic view of another set of interfaces provided by embodiments of the present application;

FIG. 12A is a schematic view of another set of interfaces provided by embodiments of the present application;

FIG. 12B is a schematic diagram of a set of images provided by an embodiment of the present application;

FIG. 12C is a schematic view of another set of images provided in accordance with an embodiment of the present application;

FIG. 12D is a schematic view of another set of images provided by an embodiment of the present application;

fig. 13A is a schematic diagram of an image fusion process provided in an embodiment of the present application;

fig. 13B is a schematic diagram of another image fusion process provided in the embodiment of the present application;

FIG. 13C is a schematic illustration of a set of interfaces provided by embodiments of the present application;

FIG. 14 is a schematic view of another set of interfaces provided by embodiments of the present application;

FIG. 15A is a schematic view of another set of interfaces provided by embodiments of the present application;

FIG. 15B is a schematic view of another set of interfaces provided by embodiments of the present application;

FIG. 15C is a schematic view of an interface provided by an embodiment of the present application;

FIG. 15D is a schematic view of another set of interfaces provided by embodiments of the present application;

FIG. 16 is a schematic view of another interface provided by an embodiment of the present application;

FIG. 17 is a schematic diagram of a set of images provided by an embodiment of the present application;

FIG. 18 is a schematic diagram of an identification of a set of target images provided by an embodiment of the present application;

fig. 19 is a schematic flowchart of another shooting method provided in the embodiment of the present application;

FIG. 20 is a schematic diagram of a set of interfaces provided by embodiments of the present application;

FIG. 21 is a schematic view of another set of interfaces provided by embodiments of the present application;

FIG. 22 is a schematic view of another set of interfaces provided by embodiments of the present application;

FIG. 23A is a schematic view of another set of interfaces provided by embodiments of the present application;

FIG. 23B is a schematic view of another set of interfaces provided by embodiments of the present application;

fig. 24 is a schematic flowchart of a zooming scheme provided in an embodiment of the present application;

FIG. 25 is a schematic diagram of a set of interfaces provided by an embodiment of the present application;

FIG. 26A is a set of schematic guide box diagrams and interface diagrams provided by embodiments of the present application;

FIG. 26B is a schematic view of an interface provided by an embodiment of the present application;

FIG. 27 is a schematic diagram of a set of interfaces provided by an embodiment of the present application;

FIG. 28 is a schematic view of another set of an interface and a target image provided by an embodiment of the present application;

FIG. 29 is a schematic view of another interface provided by an embodiment of the present application;

fig. 30 is a schematic structural diagram of another electronic device according to an embodiment of the present application.

Detailed Description

For ease of understanding, examples are given in part to illustrate concepts related to embodiments of the present application. As follows:

a first image: and shooting a background image displayed on the interface, wherein when the first camera is a wide-angle camera, the first image is a wide-angle image. The first image may be fixed and not refreshed, for example, the first frame wide-angle image acquired by the wide-angle camera after the photographing operation of the user is detected in the following embodiments or the following initial wide-angle image. The first image may also be refreshed in real time, for example, a wide-angle image acquired by a wide-angle camera in real time during the photographing process.

A second image: the second camera collects images in real time. For example, when the second camera is a tele camera, the second image is a tele image.

Shooting an image of a target: and a second image matched with the grid in the guide frame. For example, when the second camera is a telephoto camera, the target captured image may be a target telephoto image that matches the mesh in the guide frame.

A third image: and previewing the real-time changed image displayed on the interface, and acquiring the real-time changed image through the first camera in real time. For example, when the first camera is a wide-angle camera, the third image may be a wide-angle image displayed on the preview interface in the following embodiment.

Splicing information: the display device is used for indicating the shooting progress of the shooting process and indicating the matching progress of the grids in the guide frame in the shooting process. And the splicing information corresponds to the multi-frame target tele image matched with the grids in the guide frame in the photographing process. For example, the stitching information may be a thumbnail of a stitched image, a stitching frame, a matched grid or a border of a matched grid displayed on the shooting interface in the following embodiments.

Image frame: the real-time shooting range of the second camera is indicated. For example, when the second camera is a telephoto camera, the image frame is a telephoto frame in the following embodiments.

First image processing: and processing the two target long-focus images to be spliced, such as image registration, dodging and color evening, ghost image removal or image fusion and the like.

And second image processing: the processing performed on the stitched image before generating the target image, such as de-ghosting, dynamic range enhancement, high-frequency component fusion, dodging and color-homogenizing processing, or hole filling.

The first target long-focus image and the second target long-focus image are two frames of target long-focus images acquired in the photographing process respectively. The third target tele image and the fourth target tele image are two frames of target tele images acquired in the photographing process respectively.

The technical solutions in the embodiments of the present application will be described below with reference to the drawings in the embodiments of the present application. In the description of the embodiments herein, "/" means "or" unless otherwise specified, for example, a/B may mean a or B; "and/or" herein is merely an association describing an associated object, and means that there may be three relationships, e.g., a and/or B, which may mean: a exists alone, A and B exist simultaneously, and B exists alone. In addition, in the description of the embodiments of the present application, "a plurality" means two or more than two.

In the following, the terms "first", "second" are used for descriptive purposes only and are not to be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature. In the description of the present embodiment, "a plurality" means two or more unless otherwise specified.

At present, on an image with a large field angle, which is obtained by shooting with a wide-angle camera by electronic equipment such as a mobile phone, the local detail definition is low, and a user may not be able to clearly see the detail content on the image. Particularly, when the shot object is far away from the electronic equipment, the definition of an image obtained by long-distance shooting through the wide-angle camera is lower, and the detail content of the shot object is difficult to clearly present.

The embodiment of the application provides a shooting method, which can be applied to electronic equipment, and can refer to an image collected by a first camera with a larger field angle, and use a second camera (telophoto camera) with a smaller field angle to shoot the image and splice to obtain a target image with a larger field angle, so that the target image has higher definition and resolution, clear local details, a prominent main body and a better shooting effect.

The second camera has a large equivalent focal length and a small field angle, and may be, for example, a telephoto camera or an ultra-telephoto camera. Different from the second camera, the first camera related to the embodiment of the present application may be a camera with a small equivalent focal length and a large field angle, and for example, may be a wide-angle camera, a super wide-angle camera, a panoramic camera, or the like. For example, the equivalent focal length of the second camera may be 240mm, and the angle of view may be 10 °. As another example, the equivalent focal length of the second camera may be 125mm, and the angle of view may be 20 °. As another example, the equivalent focal length of the second camera may be 80mm, and the angle of view may be 30 °. For example, the equivalent focal length of the first camera may be 26mm, and the angle of view may be 80 °. As another example, the equivalent focal length of the first camera may be 16mm, and the angle of view may be 120 °.

Illustratively, the first camera is a wide-angle camera, and the second camera is a telephoto camera; or the first camera is an ultra-wide-angle camera, and the second camera is a long-focus camera; or the first camera is a wide-angle camera, and the second camera is an ultra-long-focus camera.

In the shooting method provided by the embodiment of the application, the field angle of the image shot by the second camera and the target image obtained by splicing is smaller than or equal to the field angle of the first camera. In some embodiments, the equivalent focal length of the second camera may be greater than or equal to a preset value, so that the zoom magnification of the second camera is larger, the field angle is smaller, and the image resolution is higher; in other embodiments, the ratio of the equivalent focal length of the second camera to the equivalent focal length of the first camera may be greater than or equal to a preset value 1, so that the ratio of the zoom magnification of the second camera to the zoom magnification of the first camera is larger, and the ratio of the field angle of the first camera to the field angle of the second camera is larger. Illustratively, the preset value 1 may be 2 or 3, etc. Therefore, the definition and the resolution of the target image obtained by the electronic equipment are higher, the local details are clearer, the main body is more prominent, and the shooting effect is better.

The equivalent focal length is the length of the diagonal of the image area of the photoelectric sensor chip of the camera, and is equivalent to the focal length of the 35mm camera lens corresponding to the actual focal length of the camera when the length of the diagonal of the 35mm camera frame is (42.27 mm). Zoom magnification describes the meaning of relative equivalent focal length, for example, if the zoom magnification of the wide-angle camera is defined to be equal to 1, the zoom magnification of other cameras (such as a tele camera) is equal to the ratio of the equivalent focal length of the tele camera to the equivalent focal length of the wide-angle camera serving as a reference. The size of the field of view determines the field of view of the camera, and the larger the field of view, the larger the field of view. The larger the equivalent focal length, the smaller the angle of view.

In the embodiment of the application, the target image obtained by splicing the images shot by the second camera may be in the form of a wide frame (including a landscape frame or a vertical frame), a square frame, an ultra-wide frame, a panoramic image, or the like. For example, the aspect ratio of the target image may be 2:1, 9:16, 1:1, or 2.39:1, etc. The target images of different frames can give different visual feelings to the user, so that the user can shoot by adopting the proper frame when aiming at different subjects or themes. For example, a scene subject may be photographed using an landscape frame to represent the characteristics of a wide scene and the atmosphere. For example, subjects such as tall buildings, tall towers, mountains, etc. can be photographed by using vertical pictures to show the effect of tall and tall pictures.

The shooting method provided by the embodiment of the application can be used for shooting the rear images and can also be used for shooting the front images.

For example, the electronic device may be a mobile phone, a tablet computer, a wearable device (e.g., a smart watch), an in-vehicle device, an Augmented Reality (AR)/Virtual Reality (VR) device, a notebook computer, an ultra-mobile personal computer (UMPC), a netbook, a Personal Digital Assistant (PDA), or other mobile terminals, or may be a professional camera or other devices.

Fig. 1 shows a schematic structural diagram of an electronic device 100. The electronic device 100 may include a processor 110, an external memory interface 120, an internal memory 121, a Universal Serial Bus (USB) interface 130, a charging management module 140, a power management module 141, a battery 142, an antenna 1, an antenna 2, a mobile communication module 150, a wireless communication module 160, an audio module 170, a speaker 170A, a receiver 170B, a microphone 170C, an earphone interface 170D, a sensor module 180, a key 190, a motor 191, an indicator 192, a camera 193, a display screen 194, a Subscriber Identification Module (SIM) card interface 195, and the like. The sensor module 180 may include a pressure sensor 180A, a gyroscope sensor 180B, an air pressure sensor 180C, a magnetic sensor 180D, an acceleration sensor 180E, a distance sensor 180F, a proximity light sensor 180G, a fingerprint sensor 180H, a temperature sensor 180J, a touch sensor 180K, an ambient light sensor 180L, a bone conduction sensor 180M, and the like.

Processor 110 may include one or more processing units, such as: the processor 110 may include an Application Processor (AP), a modem processor, a Graphics Processing Unit (GPU), an Image Signal Processor (ISP), a controller, a memory, a video codec, a Digital Signal Processor (DSP), a baseband processor, and/or a neural-Network Processing Unit (NPU), etc. The different processing units may be separate devices or may be integrated into one or more processors.

The controller may be, among other things, a neural center and a command center of the electronic device 100. The controller can generate an operation control signal according to the instruction operation code and the timing signal to complete the control of instruction fetching and instruction execution.

A memory may also be provided in processor 110 for storing instructions and data. In some embodiments, the memory in the processor 110 is a cache memory. The memory may hold instructions or data that have just been used or recycled by the processor 110. If the processor 110 needs to reuse the instruction or data, it can be called directly from memory. Avoiding repeated accesses reduces the latency of the processor 110, thereby increasing the efficiency of the system.

The electronic device 100 implements display functions via the GPU, the display screen 194, and the application processor. The GPU is a microprocessor for image processing, and is connected to the display screen 194 and an application processor. The GPU is used to perform mathematical and geometric calculations for graphics rendering. The processor 110 may include one or more GPUs that execute program instructions to generate or alter display information.

The display screen 194 is used to display images, video, and the like. The display screen 194 includes a display panel. The display panel may adopt a Liquid Crystal Display (LCD), an organic light-emitting diode (OLED), an active-matrix organic light-emitting diode (active-matrix organic light-emitting diode, AMOLED), a flexible light-emitting diode (FLED), a miniature, a Micro-oeld, a quantum dot light-emitting diode (QLED), and the like. In some embodiments, the electronic device 100 may include 1 or N display screens 194, with N being a positive integer greater than 1. In an embodiment of the present application, the display screen 194 may be used to display a preview interface, a shooting interface, and the like in the shooting mode.

The electronic device 100 may implement a shooting function through the ISP, the camera 193, the video codec, the GPU, the display 194, the application processor, and the like.

The ISP is used to process the data fed back by the camera 193. For example, when a photo is taken, the shutter is opened, light is transmitted to the camera photosensitive element through the lens, the optical signal is converted into an electrical signal, and the camera photosensitive element transmits the electrical signal to the ISP for processing and converting into an image visible to naked eyes. The ISP can also carry out algorithm optimization on the noise, brightness and skin color of the image. The ISP can also optimize parameters such as exposure, color temperature and the like of a shooting scene. In some embodiments, the ISP may be provided in camera 193.

The camera 193 is used to capture still images or video. The object generates an optical image through the lens and projects the optical image to the photosensitive element. The photosensitive element may be a Charge Coupled Device (CCD) or a complementary metal-oxide-semiconductor (CMOS) phototransistor. The light sensing element converts the optical signal into an electrical signal, which is then passed to the ISP where it is converted into a digital image signal. And the ISP outputs the digital image signal to the DSP for processing. The DSP converts the digital image signal into image signal in standard RGB, YUV and other formats. In some embodiments, the electronic device 100 may include 1 or N cameras 193, N being a positive integer greater than 1.

Cameras 193 may include cameras of different focal lengths, such as a first camera of a short focal length and a second camera of a long focal length, etc. The first camera has a small equivalent focal length (for example, 13mm, 16mm, 26mm, 30mm, or 40mm, etc.), and a large field angle (for example, the field angle may be 80 °, 120 °, or 150 °, etc.), and may be used to capture a large screen such as a landscape. For example, the current cameras with large field angles, such as wide-angle cameras, super-wide-angle cameras, and panoramic cameras, may all be referred to as first cameras. The second camera has a larger equivalent focal length (for example, 80mm, 125mm, 150mm, 240mm, or the like), a smaller field angle (for example, the field angle may be 10 °, 20 °, 30 °, or the like), and can be used to photograph a distant object, and the area that can be photographed is smaller. For example, both the current tele camera and the super-tele camera may be referred to as the second camera.

In some embodiments, the second camera is stationary and the user may move the second camera by moving the electronic device 100. In other embodiments, the second camera may be independently movable, and the user may directly move the second camera through a certain button, control, or operation without moving the mobile phone; or, the mobile phone can automatically control the second camera to move. When the second camera moves, the content of the picture shot by the second camera changes correspondingly.

In addition, the camera 193 may further include a depth camera for measuring an object distance of an object to be photographed, and other cameras. For example, the depth camera may include a three-dimensional (3D) depth-sensing camera, a time of flight (TOF) depth camera, a binocular depth camera, or the like.

The digital signal processor is used for processing digital signals, and can process digital image signals and other digital signals. For example, when the electronic device 100 selects a frequency bin, the digital signal processor is used to perform fourier transform or the like on the frequency bin energy.

Video codecs are used to compress or decompress digital video. The electronic device 100 may support one or more video codecs. In this way, the electronic device 100 may play or record video in a variety of encoding formats, such as: moving Picture Experts Group (MPEG) 1, MPEG2, MPEG3, MPEG4, and the like.

The internal memory 121 may be used to store computer-executable program code, which includes instructions. The processor 110 executes various functional applications of the electronic device 100 and data processing by executing instructions stored in the internal memory 121. The internal memory 121 may include a program storage area and a data storage area. The storage program area may store an operating system, an application program (such as a sound playing function, an image playing function, etc.) required by at least one function, and the like. The storage data area may store data (such as audio data, phone book, etc.) created during use of the electronic device 100, and the like. In addition, the internal memory 121 may include a high-speed random access memory, and may further include a nonvolatile memory, such as at least one magnetic disk storage device, a flash memory device, a universal flash memory (UFS), and the like.

In the embodiment of the present application, the processor 110 may implement, by executing the instruction stored in the internal memory 121, referring to the image acquired by the first camera, shooting the image by using the second camera and splicing to obtain the target image with a large field angle, so that the target image has high definition, clear details and good shooting effect.

The touch sensor 180K is also referred to as a "touch panel". The touch sensor 180K may be disposed on the display screen 194, and the touch sensor 180K and the display screen 194 form a touch screen, which is also called a touch screen. The touch sensor 180K is used to detect a touch operation applied thereto or nearby. The touch sensor can communicate the detected touch operation to the application processor to determine the touch event type. Visual output associated with the touch operation may be provided through the display screen 194. In other embodiments, the touch sensor 180K may be disposed on a surface of the electronic device 100, different from the position of the display screen 194.

It is to be understood that the illustrated structure of the embodiment of the present application does not specifically limit the electronic device 100. In other embodiments of the present application, electronic device 100 may include more or fewer components than shown, or some components may be combined, some components may be split, or a different arrangement of components. The illustrated components may be implemented in hardware, software, or a combination of software and hardware.

In an embodiment of the present application, a first camera and a second camera of the cameras 193 may be used to acquire images; the display screen 194 may be used to display a preview interface, a shooting interface, and the like when shooting; the processor 110 can refer to the image collected by the first camera with the larger field angle by operating the instruction stored in the internal memory 121, and use the second camera with the smaller field angle to shoot the image and splice to obtain the target image with the larger field angle, so that the target image has higher definition, clear details and better shooting effect.

The following describes a shooting method provided in an embodiment of the present application, taking an electronic device as a mobile phone having a structure shown in fig. 1 as an example. As shown in fig. 2, the method may include:

200. the mobile phone starts a photographing function.

In the embodiment of the application, when a user wants to use a mobile phone to shoot an image, the shooting function of the mobile phone can be started. For example, the mobile phone may start a camera application, or start another application with a shooting function (such as an AR application like a tremble or a river view cyberverse), so as to start a shooting function of the application.

For example, after detecting an operation of clicking a camera icon 301 shown in (a) of fig. 3A by a user, the mobile phone starts a photographing function of the camera application, and displays a preview interface shown in (b) of fig. 3A. As another example, the mobile phone displays an interface of a desktop or a non-camera application, starts a photographing function after detecting a voice instruction of the camera application opened by the user, and displays a preview interface as shown in (b) of fig. 3A.

It should be noted that the mobile phone may also start the photographing function in response to other operations of the user, such as a touch operation, a voice instruction, or a shortcut gesture, and the operation of triggering the mobile phone to start the photographing function is not limited in the embodiment of the present application.

In some embodiments, after the mobile phone starts the photographing function, a second camera with a smaller field angle is used to collect multiple frames of images, and the multiple frames of images are spliced into a target image with a larger field angle by the photographing method provided by the embodiments of the present application. The target image has high definition and resolution, clear local details, prominent main body and good shooting effect.

In other embodiments, after the mobile phone starts the photographing function and enters the target photographing mode, a second camera with a smaller field angle is used to collect multiple frames of images, and the multiple frames of images are spliced into a target image with a larger field angle by the photographing method provided by the embodiment of the application.

In some technical solutions, the mobile phone can reference the image collected by the first camera with the larger field angle through the shooting method provided by the embodiment of the application, and use the second camera with the smaller field angle to shoot the image and splice to obtain the target image with the larger field angle, so that the target image has higher definition and resolution, clear local details, prominent main body and better shooting effect.

For example, the target photographing mode may be specifically referred to as a wide frame mode, a wide view mode, a high definition mode, or the like, and the specific name of the target photographing mode is not limited in the embodiments of the present application.

In the following embodiments of the present application, the target photographing mode is described as an example of the wide view mode.

For example, after the mobile phone starts the photographing function, if an operation that the user clicks the control 302 shown in (b) of fig. 3A is detected, the wide view mode is entered as shown in (c) of fig. 3A. As another example, after detecting that the user clicks the control 303 shown in (b) in fig. 3A, the mobile phone displays an interface shown in (d) in fig. 3A; after detecting the operation of clicking the control 304 by the user, the mobile phone enters the panoramic mode as shown in (c) in fig. 3A.

For another example, when the mobile phone displays a desktop or a non-camera application interface, the mobile phone starts a photographing function after detecting a voice instruction of the user to enter the wide view mode, and enters the wide view mode as shown in (c) of fig. 3A.

It should be noted that the mobile phone may also start the photographing function and enter the wide view mode in response to other operations of the user, such as a touch operation, a voice instruction, or a shortcut gesture, and the specific operation of triggering the mobile phone to enter the wide view mode is not limited in the embodiment of the present application.

In some embodiments, the mobile phone may prompt the function of the shooting mode to the user in a manner of displaying information or voice broadcasting, etc. in the wide view mode. For example, referring to fig. 3B, the mobile phone displays a text prompt message on the preview interface: in the wide-view mode, images collected by the first camera with the larger field angle can be referred to, the second camera with the smaller field angle is used for shooting the images, and the target images with the larger field angle are obtained through splicing.

In some embodiments, in the wide view mode, the mobile phone may prompt the user with the specific cameras used by the first camera and the second camera. For example, referring to fig. 4, the mobile phone prompts the user by displaying text information on the preview interface, where the first camera is a wide-angle camera and the second camera is a telephoto camera.

In some technical solutions, the first camera and the second camera in the wide view mode are cameras set by a user, for example, the first camera is an ultra-wide-angle camera set by the user, and the second camera is a telephoto camera set by the user. The embodiment of the application does not limit the specific mode of setting the first camera and the second camera by the user.

In other technical solutions, the first camera and the second camera in the wide view mode are default cameras, for example, the first camera is default to be a wide-angle camera, and the second camera is default to be a telephoto camera. The user may also modify the camera types of the first camera and the second camera.

In the following embodiments of the present application, the first camera is a wide-angle camera, and the second camera is a telephoto camera.

201. The mobile phone displays the wide-angle image on the preview interface.

The mobile phone enters a preview state after starting a photographing function. In a preview state, the mobile phone may acquire a wide-angle image in real time through the wide-angle camera according to a preset acquisition frame rate 1, and display the acquired wide-angle image on a preview interface to present a global picture (or called a panoramic image) in a larger field angle for a user. The wide-angle camera is a first camera.

In some embodiments, the mobile phone may further display a guide frame for guiding the tele camera to move for shooting on the preview interface, so that the tele camera shoots multiple frames of images according to the guide frame in the shooting process. Wherein, the guide frame is displayed on the wide-angle image in a mode of a transparent suspension frame in an overlapping mode. The field angle of the guide frame is smaller than or equal to that of the wide-angle camera. That is, the angle of view of the target image generated by the mobile phone in accordance with the instruction of the guide frame is smaller than or equal to the angle of view of the wide-angle camera.

The guide frame can comprise at most R-R grids, and R is related to K1 and/or K2. Wherein, K1 is the ratio of the equivalent focal length of the telephoto camera to the equivalent focal length of the wide-angle camera, and is a value rounded up or rounded down. K2 is a value obtained by rounding up or rounding down the ratio of the angle of view of the wide camera to the angle of view of the telephoto camera. In some embodiments, R is K1; in other embodiments, R is K2; in other embodiments, R is the greater of K1 and K2.

In the embodiment of the application, the guide frame displayed on the guide interface by the mobile phone comprises M (rows) N (columns) grids. Wherein M is less than or equal to R, N is less than or equal to R, and at least one of M and N is more than 1. This M × N may be referred to as the specification of the lead frame. That is, the specification of the guide frame includes the number of meshes in the guide frame, the arrangement of the meshes, and the like. The wide-angle image corresponds to the guide frame having the largest number of meshes. For example, the size and the angle of view of the wide-angle image may be slightly larger than or equal to those of the guide frame having the largest number of meshes, and the angle of view of the telephoto camera corresponds to that of a single mesh. For example, the field angle of the tele camera may be slightly larger than or equal to the field angle corresponding to the single mesh.

For example, the equivalent focal length of the wide-angle camera is 125mm, the equivalent focal length of the tele camera is 25mm, the ratio K1 of the two equivalent focal lengths is 5, and R is K1 is 5. In this way, the number of grids in each row of the guide frame is less than or equal to 5, and the number of grids in each column is also less than or equal to 5. That is, the guide box may include 5 × 5 (i.e., 5 rows and 5 columns) meshes at most; the guide frame may also comprise less than 5x 5 meshes, for example may comprise 3x 3, 3x 4 or 4x 5 meshes. The field angle of the wide-angle camera corresponds to the 5 × 5 grids, and the single grid corresponds to the field angle of the telephoto camera.

In some embodiments, the ratio between the equivalent focal length of the tele camera and the equivalent focal length of the wide camera is less than or equal to the preset value 2. Therefore, the difference between the size of the guide frame and the size of the wide-angle image is small, the sizes of the guide frame and the grid are large, and a user can conveniently move the tele camera to enable the tele image to be matched with the grid. Illustratively, the preset value 2 may be 8 or 10, etc.

In the following embodiments, the guide frame is described as including 5 × 5 grids at most.

In some embodiments, the guide box is displayed by default in the middle of the image displayed by the preview interface. The specification of the guide frame is a default specification or a specification adopted by the mobile phone in the wide view mode at the last time.

In other embodiments, the position or specification of the guide frame displayed on the preview interface by the mobile phone may be set by the user.

For example, in some embodiments, after the user selects an area (e.g., a box or a circle) on the wide-angle image on the preview interface, the mobile phone determines the matching guide frame according to the area selected by the user. For example, the area corresponding to the guide frame determined by the mobile phone can cover the selection area of the user, or the proportion of the area that can cover the selection area of the user is greater than or equal to a preset proportion (for example, 90%), or is slightly greater than the selection area of the user, and the like. Illustratively, referring to fig. 5 (a), the cell phone prompts the user on the preview interface: please select the range of the target image to display the corresponding guide frame. As shown in fig. 5 (b), after detecting an operation of selecting an area by the user, the mobile phone determines that the target image that the user wants to capture is the size corresponding to the area. Then, the mobile phone determines the position or specification of the guide frame according to the area. For example, the guide box includes 3 × 3 grids, and as shown in (c) in fig. 5, the guide box 501 is displayed on the preview interface.

In other technical solutions, after detecting that the user clicks the setting control 601 on the preview interface as shown in (a) in fig. 6, the mobile phone displays the setting interface. As shown in (b) in fig. 6, the setting interface includes a setting control of the target image guide box, and after the mobile phone detects an operation of clicking the control by the user, as shown in (c) in fig. 6, the maximum guide box is displayed, and the maximum guide box includes the maximum number of grids (e.g., the above-mentioned 5 × 5 grids). After the mobile phone detects that the user performs frame selection on the maximum guide frame and clicks the operation of the determination control, the range of the position corresponding to the frame selection operation of the user is determined to be the position and the size of the guide frame. As shown in fig. 6 (d), the cell phone displays the determined guide box 602 including 2 × 3 grids on the preview interface.

In other embodiments, referring to fig. 7A (a), the mobile phone displays a maximum guide box including the maximum number of grids on the preview interface, and prompts the user to: please set a guide frame of the target image. As shown in fig. 7A (b), the user drags on the maximum guide frame to frame and select a plurality of grids, and the mobile phone determines that the position and range corresponding to the framing operation of the user are the position and size of the guide frame. As shown in fig. 7A (c), the mobile phone displays the determined guide box 701 including 3 × 3 grids on the preview interface, which further includes the wide-angle image.

In other technical solutions, the user may indicate a body on the preview interface, and the mobile phone determines the position and the specification of the guide frame according to the body indicated by the user, so that the guide frame can cover the body indicated by the user. For example, referring to (a) in fig. 7B, the mobile phone may prompt the user on a preview interface to select a subject to be photographed. As shown in fig. 7B (B), after detecting the operation of clicking the building by the user, the mobile phone determines that the connected whole building is the subject to be photographed, and thus determines that the guide frame is the guide frame 703 capable of covering the connected whole building on the preview interface shown in fig. 7B (c), and includes 3 × 3 grids in total.

In other embodiments, referring to fig. 8 (a), the mobile phone displays a plurality of specification controls, for example, 5 × 5, 4 × 4, 3 × 4, or 3 × 3, on the preview interface, and the mobile phone determines the number of grids included in the guide frame according to the specification control selected by the user. For example, when the mobile phone detects an operation of clicking the specification control of 3 × 3, as shown in (b) of fig. 8, a guide frame 801 including 3 × 3 grids is displayed on the preview interface, and the preview interface further includes a wide-angle image.

In some embodiments of the application, the mobile phone may further determine a shooting order of the grids in the guide frame according to an instruction operation of the user. In the photographing process, the user can move the tele camera according to the photographing sequence, so that the tele camera is sequentially matched with the grids specified by the sequence.

In one possible implementation, the user may also set the shooting order of the grids in the guide frame when setting the position or specification of the guide frame in the manner described in the above embodiments (e.g., the manner shown in fig. 5-8).

In another possible implementation manner, after the mobile phone determines the specification of the guide frame, multiple sequential modes corresponding to the guide frame of the specification may be displayed, and the mobile phone may determine one sequential mode according to the selection operation of the user. Illustratively, the specification of the guide box is 3 × 3, see (a) in fig. 9A, and the preview interface includes a plurality of sequential mode controls, such as controls 901 to 903. For example, after detecting the operation of the user clicking the sequence mode control 901, the mobile phone determines that the shooting sequence is to shoot images corresponding to a middle row of grids from left to right, then shoot images of an upper row of grids from left to right, and then shoot images of a lower row of grids from left to right. For another example, after detecting that the user clicks the sequence mode control 902, the mobile phone determines that the shooting sequence is to shoot the image of the middle row of grids from left to right, then shoot the image of the upper row of grids from right to left, and then shoot the image of the lower row of grids from left to right. For another example, after detecting that the user clicks the sequence mode control 903, the mobile phone determines that the shooting sequence is shooting from top to bottom by pressing a zigzag track.

In other embodiments, the guide frame of each specification corresponds to a shooting sequence by default, and the mobile phone determines the specification of the guide frame and then shoots by adopting the default shooting sequence. The mobile phone can also modify the shooting sequence according to the instruction operation of the user.

In some embodiments of the application, after determining the shooting order corresponding to the guide frame, the mobile phone may prompt the user to shoot according to the shooting order. For example, the mobile phone may display a shooting sequence prompt message on a preview interface. For example, when the user selects the shooting order corresponding to the order mode control 901, referring to (b) in fig. 9A, the mobile phone may display a reference number and a guide line with an arrow on the guide frame, where the reference number is used to indicate the shooting order of different rows of grids, and the arrow of the guide line indicates a direction used to indicate the shooting order of each row of grids. For another example, when the user selects the shooting order corresponding to the order mode control 902, referring to (c) in fig. 9A, the mobile phone may display a guide line with an arrow on the guide frame to indicate the shooting order corresponding to the different grids.

It is understood that the shooting sequence may also include other sequence modes, for example, sequence modes shown in (a) - (d) in fig. 9B, and the embodiment of the present application does not limit the specific form of the sequence modes.

In other embodiments, the mobile phone may display the guide frame on the shooting interface for a short time after acquiring the guide frame, and display the guide frame on the shooting interface after detecting the shooting operation of the user subsequently.

In other embodiments, the mobile phone does not display the guide frame on the preview interface, and displays the guide frame on the shooting interface after detecting the shooting operation of the user.

In addition, in other embodiments, in the preview state, the mobile phone may further acquire a tele image through the tele camera according to the preset acquisition frame rate 2. The tele camera is a second camera.

In still other embodiments, the mobile phone may further prompt the user of the real-time shooting range of the tele-camera through the tele frame, so as to present the local picture acquired by the tele-camera in real time for the user. Wherein, the long focus frame is displayed on the wide-angle image in a mode of a transparent suspension frame in an overlapping mode. The position and size of the tele frame on the wide image correspond to the shooting range of the tele camera. The field angle of the wide-angle image within the tele frame may be equal to or slightly smaller than the shooting range and the field angle of the tele camera. The picture range of the wide-angle image in the tele frame corresponds to the picture range of the tele camera. For example, the frame range of the wide image within the tele frame may be equal to or slightly smaller than the frame range of the tele image acquired by the tele camera. The acquisition frame rate 2 and the acquisition frame rate 1 may be the same or different, and are not limited. Illustratively, referring to (a) in fig. 10A, the preview interface includes a wide image 1001, a tele frame 1002, and a guide frame 1003.

When a user moves the mobile phone or shakes the mobile phone due to reasons such as hand shake, the relative position and size of the telephoto frame on the wide-angle image on the preview interface are basically kept unchanged, specifically, the relative positions and sizes are related to the lens centers of the wide-angle camera and the telephoto camera, the shot object distance and the size of the field angle. Illustratively, referring to (b) in fig. 10A, the distance between the lens centers of the wide camera and the telephoto camera is fixed, and the size of the field angle is also fixed when the object distance is fixed, in which case the telephoto camera and the wide camera move simultaneously when the mobile phone is moved, but the relative relationship between the field angle of the telephoto camera and the field angle of the wide camera remains unchanged, and thus the relative position and size of the telephoto frame on the wide image also remains substantially unchanged.

In other embodiments, when the equivalent focal length of the tele camera and the equivalent focal length of the wide camera are different greatly, the field angle of the tele camera and the field angle of the wide camera are also different greatly, the field angle of the tele camera is smaller, and the size of the tele frame is smaller, so that it is inconvenient for the user to view details in the shooting range of the tele camera. Therefore, the mobile phone can display the telephoto frame and the wide-angle image in the telephoto frame on the preview interface after being amplified, so that a user can conveniently know the shooting range and details of the telephoto camera. Illustratively, referring to (c) in fig. 10A, the preview interface includes a wide-angle image 1004, a telephoto frame 1005 displayed in an enlarged manner, and a guide frame 1006.

In other embodiments, the cell phone displays the target area image corresponding to the guide box on the wide-angle image on the preview interface without displaying the full wide-angle image. The ratio of the size of the target area image to the size of the guide frame is r, and r is more than or equal to 1. The target area image may be obtained by cropping and enlarging the complete wide-angle image.

In particular, when the equivalent focal length of the tele camera and the equivalent focal length of the wide camera are different greatly, the field angle of the tele camera and the field angle of the wide camera are also different greatly, the field angle of the tele camera is smaller, the size of the tele frame is smaller, and it is inconvenient for the user to view details in the shooting range of the tele camera. Under the condition, the mobile phone can amplify the target area image and the guide frame in equal proportion and then display the target area image and the guide frame on the preview interface, so that a user can conveniently know the shooting range and details of the telephoto camera.

For example, in a preview state, if the specification of the guide frame is a default specification, after the wide-angle image is acquired by the mobile phone, the wide-angle image may be cropped according to the guide frame of the default specification to obtain a target area image, and the target area image and the guide frame may be enlarged in equal proportion and displayed on a preview interface. If the specification of the guide frame corresponds to the area/body selected by the user, after the wide-angle image is collected by the mobile phone, the wide-angle image can be cut according to the guide frame to obtain a target area image, and the target area image and the guide frame are displayed on the preview interface after being amplified in equal proportion.

For example, the complete wide-angle image on the preview interface shown in (B) in fig. 8 may be replaced with the target area image 1007 in the wide-angle image on the preview interface shown in fig. 10B. In the preview interface shown in fig. 10B, the target area image and the guide frame are enlarged in equal proportion to the preview interface shown in fig. 8 (B), and the ratio r of the size of the target area image to the size of the guide frame is larger than 1.

202. After the mobile phone detects the photographing operation of the user, the wide-angle image and the guide frame superposed on the wide-angle image are displayed on the photographing interface.

When the user wants to start shooting the target image, the shooting operation can be triggered to enable the mobile phone to enter the shooting process. For example, after detecting an operation of clicking a shooting control on a preview interface by a user, the mobile phone determines that a shooting operation of the user is detected, and then enters a shooting process. For another example, after detecting that the user indicates to start the photographing operation, the mobile phone determines that the photographing operation of the user is detected, and then enters a photographing process. It can be understood that the manner for triggering the mobile phone to enter the photographing process may also include various other manners such as a gesture, and the embodiment of the present application is not limited.

In the shooting process, a wide-angle image is displayed on the shooting interface, and the wide-angle image is acquired by using a wide-angle camera (namely, a first camera). In some embodiments, during the photographing process, the wide-angle image on the photographing interface is used as a background image for providing a panoramic image for a user, and the user determines a moving path of the telephoto camera according to the range of the wide-angle image, so as to generate a spliced image of a plurality of telephoto images. In a possible implementation manner, the wide-angle image is a first frame image acquired by the wide-angle camera after the photographing operation of the user is detected, and in the photographing process, the mobile phone always displays the first frame image without refreshing and displaying the image acquired by the wide-angle camera. In another possible implementation manner, the mobile phone may acquire a Q (Q is an integer greater than 1) frame wide-angle image after detecting a photographing operation of the user, so as to fuse the Q frame wide-angle image into an initial wide-angle image with better quality as a background image. In the shooting process, the mobile phone always displays the initial wide-angle image, and does not refresh and send the image collected by the wide-angle camera to display. That is, after the mobile phone detects the photographing operation of the user, the wide-angle image displayed on the photographing interface for being a preview remains unchanged.

In other embodiments, in the photographing process, the first wide-angle image serving as the background image on the photographing interface is changed, and is an image acquired by the mobile phone through the wide-angle camera in real time according to the preset acquisition frame rate 3. Wherein, the acquisition frame rate 3 and the acquisition frame rate 1 may be the same or different.

In the shooting process, the mobile phone can also display the guide frame on the shooting interface, and the guide frame is superposed on the background image in a transparent suspension frame mode. In some embodiments, the mobile phone may continuously display the entire guide frame on the shooting interface; in other embodiments, the cell phone may display only the unmatched grids in the guide frame on the camera interface, instead of continuously displaying the entire guide frame.

In the shooting process, the mobile phone can collect the tele image through the tele camera according to the preset collection frame rate 4. The acquisition frame rate 4 may be the same as or different from the acquisition frame rate 2.

In the shooting process, the user can move the long-focus camera through the mobile phone, or the user can directly move the long-focus camera, or the mobile phone can automatically control the long-focus camera to move at a preset angle interval. In some embodiments, the cell phone may also display a tele frame on the capture interface. Along with the movement of the long-focus camera, the shooting range of the long-focus camera is changed, the content of a long-focus image acquired by the long-focus camera is correspondingly changed, and the position of a long-focus frame is correspondingly changed. The tele frame can prompt the dynamic change process of the shooting range of the tele camera to a user in real time in the moving process of the tele camera. During movement of the tele camera, the wide image on the shooting interface may serve as a background image to provide a panoramic image to the user, thereby guiding the user to move the tele camera so that the tele frame is matched with the grid in the guide frame one by one.

In other embodiments, if the mobile phone can automatically control the tele camera to move, the mobile phone can control the tele camera to automatically move according to the grid sequence according to the arrangement sequence of the grids in the guide frame, so that the tele camera is matched with the grids in the guide frame one by one without displaying the tele frame on the shooting interface. Moreover, even if the wide-angle image as the background image on the shooting interface is changed, the user does not specially move the mobile phone, so that the picture of the wide-angle image acquired by the mobile phone in real time is basically unchanged or slightly changed, and the positions of the guide frame and the grids in the picture are also basically unchanged, so that the mobile phone controls the tele-camera to automatically move according to the grid sequence, and the tele-camera can be matched with the grids in the guide frame one by one without displaying the tele-frame.

203. And the mobile phone generates a spliced image according to the acquired target tele image and displays a spliced image thumbnail on a shooting interface.

In the photographing process, along with the movement of the long-focus camera, the mobile phone can acquire a multi-frame target long-focus image matched with the grid in the guide frame through the long-focus camera. The mobile phone can splice the target tele image, so as to generate a target image.

Wherein, when the content of the tele image is matched with the content of the wide image in a certain grid of the guiding frame on the shooting interface, the tele image is matched with the grid, and the tele image can be called as a target tele image. Matching the content of the tele image with the content of the Wide image in the grid means that the content of the tele image is exactly or substantially identical to the content of the Wide image in the grid. For example, the overlapping ratio of the same content of the tele image and the wide image in the grid is greater than or equal to a preset value 3 (e.g., 80% or 90%). For another example, the similarity between the histogram of the tele image and the histogram of the wide image in the grid is greater than or equal to the preset value 4. The histogram here may be a histogram of a characteristic parameter value such as luminance. For another example, the similarity between the tele image and the wide image in the grid in the same transform domain (e.g., Fast Fourier Transform (FFT), Wavelet Transform (WT), Discrete Cosine Transform (DCT), etc.) is greater than or equal to the preset value 5. The similarity measure can be the sum of the differences of the scaling coefficients corresponding to different values. For another example, the feature matching degree between the tele image and the wide image in the grid is greater than or equal to a preset value 6, for example, the feature includes a corner, a convolutional neural network feature or a SIFI feature.

The method for obtaining the target tele image by the mobile phone may include: the method comprises the steps that in the mode 1, after a long-focus image in a certain frame is determined to be matched with a grid, a target long-focus image matched with the grid is shot and obtained; and 2, after the long-focus image in a certain frame is matched with the grid, determining that the long-focus image in the frame is the target long-focus image.

In some embodiments, after the mobile phone detects the photographing operation of the user, the mobile phone may prompt the user of the photographing order of the grids in the guide frame, so that the user moves the telephoto camera according to the photographing order during the photographing process.

If the mobile phone determines the shooting sequence of the grids in the guide frame in the preview state, the mobile phone can prompt the shooting sequence to a user in the shooting process, so that the user can move the mobile phone or directly move the tele camera according to the shooting sequence, and the tele image acquired by the tele camera is matched with the grids in the guide frame according to the shooting sequence. In one possible implementation, the mobile phone may prompt the user with the complete shooting sequence. In another possible implementation manner, the mobile phone may prompt only a part of the currently required shooting sequence according to the shooting situation, without prompting the user of the complete shooting sequence.

If the shooting sequence of the grids in the guide frame is not determined in the preview state of the mobile phone, the user can move the mobile phone or directly move the tele-camera according to the requirement, habit or will of the user, so that the tele-image acquired by the tele-camera is matched with the grids in the guide frame. For the same row or column of grids, the tele image should match the adjacent grids in the row/column as sequentially as possible. In the photographing process, if a certain grid is matched, the grid is not repeatedly matched.

The field angle of the tele image acquired by the tele camera in real time is small, so that the visual angle of the tele camera is likely to be greatly deviated by slightly moving the mobile phone or the tele camera, and the content of the tele image is greatly deviated, so that the tele image is not easily matched with the content of the next grid to be matched. In the shooting process, the wide-angle image on the shooting interface can provide a panoramic image in a large field angle for a user, so that the wide-angle image can provide reference for the user to move the mobile phone or the tele camera, and the user can accurately control the moving path (such as the moving direction and the moving amplitude) of the mobile phone or the tele camera according to the content of the wide-angle image and the corresponding position of the content of the tele image on the wide-angle image, so that the tele image can be quickly and accurately matched with the next grid.

And the long-focus frame can be displayed on the shooting interface in real time so as to guide the user to move the shooting range of the long-focus camera and the long-focus frame to the position of the first grid to be matched. Illustratively, the first grid to be matched is the first grid on the left side of the middle row as shown in fig. 11 (a), and the preview interface includes the wide-angle image and the guide frame. After detecting that the user clicks the shooting control 1100 on the preview interface shown in (a) in fig. 11, the mobile phone enters the shooting process of the target image and displays the shooting interface shown in (b), (c) or (d) in fig. 11, where the shooting interface includes the wide-angle image, the guide frame, and the tele frame 1101. Note that the imaging interface shown in fig. 11 (b) does not display the presentation information of the imaging order. The shooting interface shown in fig. 11 (c) displays the prompt information of the complete shooting order, and the shooting interface shown in fig. 11 (d) displays the prompt information of the partial shooting order. In some embodiments, as shown in fig. 11 (c) - (d), the cell phone may also prompt the user on the camera interface: please move the handset in the direction of the arrow and match the grid.

The photographing process will be described below by taking the photographing sequence shown in (c) of fig. 11 and displaying a prompt message of a part of the photographing sequence on the photographing interface as an example.

In some embodiments, the mobile phone may determine the configuration parameters when the long-focus camera acquires the target long-focus images corresponding to different grids according to the reference wide-angle image. The mobile phone performs Automatic Exposure (AE) configuration, Automatic White Balance (AWB) adjustment, Dynamic Range Correction (DRC) configuration, and the like on the target tele image according to the configuration parameters, thereby obtaining the target tele image. Because the spliced image is obtained by splicing a plurality of target tele images, the shooting environment of each target tele image is different, only a picture in a local range can be shot, and the whole effect of a global picture with a large visual field cannot be considered, the whole spliced image is possibly unnatural, obtrusive or unsmooth in the aspects of brightness, dynamic range or color and the like. The mobile phone determines configuration parameters when the tele-camera collects target tele-images corresponding to different grids according to the reference wide-angle image corresponding to the global range, so that the overall exposure effect, the AWB effect and the dynamic range of the spliced image obtained according to the target tele-image are consistent with those of the reference wide-angle image as much as possible, and the overall change of the spliced image is smooth and natural in transition.

The reference wide-angle image is used for determining configuration parameters of the acquired target tele image in the photographing process, and the reference wide-angle image is the same frame image in the photographing process. For example, the reference wide-angle image may be the initial wide-angle image, or a first frame image captured by the wide-angle camera after the photographing operation of the user is detected.

In some embodiments, to ensure that the quality of the reference wide-angle image is good (e.g., the dynamic range is high, the color configuration is good, etc.), so that the quality of the stitched image of the tele image obtained from the reference wide-angle image is also good, the reference wide-angle image may be the initial wide-angle image obtained by the multi-frame image fusion.

In one possible implementation, the reference wide-angle image is divided into a plurality of image blocks (patch) according to the grid of the guide frame, and each image block corresponds to one grid of the guide frame. For example, grid 1 corresponds to image patch 1, grid 2 corresponds to image patch 2, and grid 3 corresponds to image patch 3. The mobile phone sets configuration parameters such as AE, AWB or DRC of a target tele image of a grid corresponding to the image block according to parameters such as brightness, color and dynamic range of each image block of the reference wide-angle image. And the mobile phone acquires the target tele image according to the configuration parameters, and realizes the configuration of AE, AWB, DRC and the like according to the target tele image in the wide-angle image guide corresponding grid.

The first grid to be matched is taken as grid 1, and grid 1 corresponds to target tele-image 1. As for the above mode 1, the mobile phone may collect the tele image first, and after determining that the tele image is matched with the grid 1, set the configuration parameters of the tele camera according to the image block 1 corresponding to the grid 1 on the reference wide-angle image, so that the tele camera collects and obtains the target tele image 1 corresponding to the grid 1 according to the configuration parameters. As for the above mode 2, the mobile phone may set the configuration parameters of the tele camera according to the image block 1 corresponding to the grid 1 on the reference wide-angle image, so that the tele camera acquires the tele image according to the configuration parameters, until the tele image in a certain frame is matched with the grid 1, the mobile phone determines that the tele image in the certain frame is the target tele image 1.

The following description is made for different configuration parameters of the telephoto camera respectively:

AE configuration parameters:

the mobile phone can perform photometry on the image block 1 on the reference wide-angle image, so as to obtain brightness parameters such as a brightness value, a brightness average value, a maximum value and the like of the image block 1, and the brightness parameters can reflect the environment brightness condition. The exposure table in the mobile phone is preset with the corresponding relation between the environment brightness and the exposure parameters. The mobile phone can determine AE configuration parameters of the tele image to be acquired corresponding to the grid 1, such as exposure time of the tele image and exposure parameters such as ISO, according to the ambient brightness condition reflected by the brightness parameter of the image block 1 on the wide-angle image, in combination with the exposure table of the tele image. The mobile phone adopts the exposure parameters to automatically expose the target tele image 1 corresponding to the grid 1 so as to improve the exposure effect of the target tele image 1.

Because the overall exposure effect of the reference wide-angle image is good, the exposure parameters of the target tele image are determined according to the exposure information of the image blocks in the reference wide-angle image, and the exposure effect of the spliced image obtained according to different target tele images can be consistent with that of the reference wide-angle image from the whole situation to the greatest extent, so that the overall exposure effect of the spliced image is good.

In addition, because the brightness transition between different image blocks on the reference wide-angle image is natural, in some embodiments, the mobile phone can also adjust the exposure parameters of the target tele image 1 corresponding to the image block 1 according to the brightness condition of the adjacent image block of the image block 1, so that the brightness transition between the brightness of the target tele image 1 and the brightness of the adjacent target tele image is natural, and the problems that the mobile phone independently performs automatic exposure on the target tele image corresponding to each grid, so that the exposure effect difference of different target tele images is large, the brightness process of the spliced image is not smooth and natural enough, the splicing trace is obvious, and the like are avoided, thereby improving the overall quality of the spliced image. For example, if the image block 1 is darker and the image block 2 adjacent to the image block 1 is brighter, the mobile phone may increase the exposure parameter of the target tele image 1 determined according to the image block 1 by one brightness step, so that the brightness transition between the target tele image 1 and the target tele image 2 corresponding to the image block 2 is more natural.

AWB configuration parameters:

the mobile phone may determine the AWB configuration parameters of the target tele image 1 corresponding to the grid 1 according to the color distribution of the image block 1 on the reference wide-angle image, for example, determine WB values, that is, proportions of three primary RGB colors, of the target tele image 1 corresponding to the grid 1. The mobile phone correspondingly adjusts the RGB ratio of the target tele image 1 corresponding to the grid 1 according to the ratio so as to improve the color configuration effect of the target tele image.

The white balance of the whole image of the reference wide-angle image is good, and the color transition between different image blocks is natural, so that the white balance information of the image blocks of the reference wide-angle image can be started from the whole situation, the white balance effect of the target tele image is consistent with that of the reference wide-angle image as much as possible, the color transition between different target tele images included in the spliced image is natural, automatic white balance adjustment of the mobile phone aiming at the target tele image corresponding to each grid independently is avoided, the white balance effect difference of the different target tele images is large, the color transition of the spliced image is not smooth and natural, the splicing trace is obvious, and the like, and the overall quality of the spliced image can be improved.

DRC configuration parameters:

the mobile phone can determine the DRC configuration parameter of the target tele image 1 corresponding to the grid 1 according to the dynamic range of the image block 1 on the reference wide-angle image, and thus correspondingly adjust the dynamic range of the target tele image 1 corresponding to the grid 1, so that the luminance distribution of the target tele image 1 is relatively consistent with the luminance distribution of the corresponding image block. The dynamic range may include the brightness distribution of different pixels on the image, the brightness difference between different pixels, and so on. For example, the mobile phone may count the dynamic range of the image block 1 through a luminance histogram. For example, image block 1 includes a plurality of pixels with luminance lower than 100 and pixels with luminance higher than 200. The mobile phone can control the brightness of the pixel points of the target tele image, so that the target tele image comprises a plurality of pixel points with the brightness lower than 100 and pixel points with the brightness higher than 200, and the dynamic range of the target tele image is enlarged.

Because the dynamic range of the reference wide-angle image is large, the dynamic range of the target tele image 1 corresponding to the grid 1 is adjusted according to the dynamic range of the image block 1 on the reference wide-angle image, so that the dynamic range of the target tele image 1 can be increased, the brightness range of the target tele image 1 is large, the bright and dark layers are rich, and more image details of the bright part and the dark part can be provided. The mobile phone configures the DRC parameters of the target tele images according to the dynamic range information of the image blocks of the reference wide-angle image, so that the brightness distribution transition between different target tele images on the spliced image is natural from the global aspect, and the problems of large difference of the dynamic ranges of different target tele images, poor dynamic range of the spliced image, obvious splicing trace and the like caused by that the mobile phone performs DRC on the target tele images corresponding to each grid respectively are avoided, so that the overall quality of the spliced image can be improved.

After the tele image is matched with the grid 1 in the guide frame, the tele camera can move and continue to acquire the tele image to acquire a target tele image 2 matched with the next grid 2 to be matched. Wherein, in order to facilitate the splicing of the target tele image matched with the grid, the next grid 2 to be matched is adjacent to the matched at least one grid (e.g. grid 1).

Subsequently, with the movement of the tele camera, the mobile phone may also configure, according to the image block of the reference wide-angle image, configuration parameters of the tele camera to acquire a target tele image corresponding to another grid (e.g., the target tele image 2 corresponding to the grid 2, the target tele image 3 corresponding to the grid 3, etc.), which is not described again.

In some embodiments, after the target tele image is matched with the grid, the mobile phone may further prompt the grid to the user, so that the user can know the position of the currently matched grid and know the current shooting progress. The mobile phone can display the matched grids and other grids in a differentiated mode to prompt a user of the position of the currently matched grid, and the user can conveniently know the current shooting progress, the subsequent shooting direction and the moving direction of the telephoto camera. For example, the currently matched grid may be highlighted, bolded, displayed in a color different from that of other grids, displayed in a specific color, or transformed into a line type different from that of other grids.

For example, during the photographing process, the user moves a cell phone or a tele camera so that the target tele image 1 matches the content of the wide image in the leftmost grid 1 in the middle row of the guide box. As shown in (a) of fig. 12A, the boundary of the grid 1 is changed from a dotted line to a thick solid line, so that it is displayed differently from other grids, so that it is convenient for the user to know that the currently matched grid 1 corresponds to the current shooting progress corresponding to the grid 1. When the content of the target tele image 2 is matched with the content of the wide image in the grid 2 in the guide frame, the mobile phone can also distinguish the grid 2 from other grids for displaying, so that a user can conveniently know that the grid 2 is currently matched, and the current shooting progress corresponds to the grid 2.

In some embodiments of the application, in the photographing process, the mobile phone may be spliced with the previously obtained spliced image after obtaining a new target tele image each time, so as to generate a new spliced image. That is to say, when all grids in the guide frame are not matched, a stitched image is generated according to the multi-frame target tele images corresponding to the matched grids. In other embodiments, after obtaining the target tele image matched with all the grids in the guide frame, or after finishing shooting, the mobile phone generates the stitched image according to the target tele image. The embodiment of the application does not limit the splicing time of the spliced images.

The process of generating the spliced image by splicing the different target long-focus images by the mobile phone can comprise the processes of image registration, image dodging and color evening, image fusion and the like. The image registration refers to a process of matching and overlaying different images. For example, the mobile phone may perform feature extraction on two frames of images to be registered to obtain feature points, find matched feature point pairs by performing similarity measurement, then obtain image space coordinate transformation parameters by the matched feature point pairs, and finally perform image registration by the coordinate transformation parameters. The mobile phone can calculate a homography matrix of the target tele image 2 relative to the target tele image 1, and thus register the target tele image 2 with the target tele image 1 according to the homography matrix. For example, the image registration algorithm may include a SURF feature matching algorithm, an SKB feature matching algorithm, an ORB feature matching algorithm, a mesh registration algorithm, an optical flow registration algorithm, or a convolutional neural network (AI) registration algorithm, and the specific type of the image registration algorithm is not limited in the embodiments of the present application.

The image dodging refers to brightness equalization of registered images after image registration, so that brightness transition between adjacent target tele images is natural. The brightness equalization of image stitching is a mature technology and is not limited herein.

The image dodging and color evening refers to that after image registration, statistics of a brightness histogram and a color histogram are carried out on a part where registered images are overlapped, and a cumulative distribution function of image brightness and image color is obtained through curve fitting (such as a spline curve). The brightness and color distribution of one image can be taken as the standard, and the brightness and color of other images are corrected according to the brightness and color cumulative distribution function, so that the purposes of light and color evening are achieved; the brightness and color parameters of a plurality of images can be used as a common optimization target to carry out iterative optimization, so that the purpose of light and color homogenization of all the images is achieved.

The image fusion refers to a process of extracting relevant information in each image to the maximum extent and integrating the relevant information into a high-quality image by subjecting a plurality of frames of images to image processing, computer technology and the like. For example, the image fusion algorithm may include an alpha fusion algorithm, a poisson fusion algorithm, or a convolutional neural network fusion (AI fusion) algorithm, etc. The embodiment of the present application does not limit the specific type of the image fusion algorithm.

For example, after the mobile phone obtains the target tele image 2, the target tele image 2 and the target tele image 1 may be subjected to registration, dodging, homogenizing, and fusion, so as to splice a spliced image with a large field angle.

In some embodiments, the target tele image 2 and the target tele image 1 may be registered and fused directly according to an image registration algorithm, thereby generating a stitched image. In this case, a certain overlap ratio (for example, 20%) needs to be present between the target tele image 2 to be registered and the target tele image 1, so as to ensure that the registration result is accurate. For example, the mobile phone may extract features from the target tele image 1 and the target tele image 2 according to a preset image registration algorithm, and perform image registration according to feature matching between the two.

In other embodiments, the target tele image 2 is registered and fused directly with the target tele image 1, thereby generating a stitched image. Then, the mobile phone can correct the registered image or the spliced image according to the reference wide-angle image, so that registration errors caused by fewer characteristic point pairs between the long-focus images of the target to be registered are avoided. For example, on the stitched image generated by directly registering and fusing the target tele image 2 and the target tele image 1, part of the image content may have distortion, and the mobile phone may adjust the coordinates of the distorted image content on the stitched image according to the coordinate position of the same content on the reference wide-angle image, so as to correct the distortion and improve the quality of the stitched image.

In other embodiments, the target tele image 2 and the target tele image 1 are registered and fused from the reference wide image. In this case, the overlap ratio between the target tele image 2 to be registered and the target tele image 1 may be small (for example, may be 10% or 5%, etc.); or, even if there is no overlap between the target tele image 2 and the target tele image 1, the mobile phone can accurately register and fuse the target tele image 2 and the target tele image 1 according to the reference wide image. Therefore, the requirement of the target long-focus image to be matched is relaxed, the mobile phone can quickly and easily obtain the target long-focus image in the process that a user moves the mobile phone or the long-focus camera, the shooting time of the user is shortened, and the registration error caused by fewer characteristic point pairs between the target long-focus images to be registered can be avoided. Moreover, if the overlapping proportion between the target long-focus images is small or no overlapping exists, the shooting process can be finished by shooting a few target long-focus images by the user, so that the number of shooting frames and the shooting time can be reduced, and the shooting efficiency and the shooting experience of the user are improved.

In one possible implementation, the target tele image 2, the target tele image 1 and the reference wide image may be registered and fused together. Referring to fig. 13A, the target tele image 2, the target tele image 1, and the reference wide image may respectively extract features, and two of the features are matched, so that more feature matching pairs may be obtained in an overlapping area of the three. Therefore, the mobile phone can obtain more and more accurate feature matching pairs, so that a more accurate homography matrix is calculated, and the target tele image 2 is deformed according to the homography matrix, so that better registration, splicing and fusion effects are achieved.

In another possible implementation, the target tele image 2 and the target tele image 1 may be registered and fused with the reference wide image, respectively, without registration and fusion between the target tele image 2 and the target tele image 1. For example, as shown in fig. 13B, the target tele image 2 and the reference wide image may be respectively extracted and matched to calculate a homography matrix, and the target tele image 2 is deformed according to the homography matrix to be better fused with the target tele image 1. In this case, the overlap ratio between the target tele image 2 to be registered and the target tele image 1 may be small (for example, may be 10% or 5%, etc.). Or even if the target tele image 2 and the target tele image 1 are not overlapped, a good fusion splicing effect can be realized due to the common wide-angle image. Therefore, the requirement of the target long-focus image to be matched is relaxed, and the mobile phone can quickly and easily obtain the target long-focus image in the process of moving the mobile phone or the long-focus camera by a user, so that the shooting time of the user is shortened. Moreover, if the overlapping proportion between the target long-focus images is small or no overlapping exists, the shooting process can be finished by shooting a few target long-focus images by the user, so that the number of shooting frames and the shooting time can be reduced, and the shooting efficiency and the shooting experience of the user are improved.

In yet another possible implementation, the target tele image 1 may be registered to the coordinate system of the reference Wide image, i.e. the target tele image 1 may be fitted to the coordinate position of the same content on the reference Wide image. Similarly, the target tele image 2 may also be registered according to the coordinate system of the reference Wide image, i.e. the target tele image 2 may be attached to the coordinate position of the reference Wide image where the same content is located. If there is no cavity between the target tele image 1 and the target tele image 2 attached to the reference wide image, the whole formed by attaching the target tele image 1 and the target tele image 2 is the stitched image.

In addition, the acquisition angle of the telephoto camera may be deflected due to the hand shake of the user or the rotation of the mobile phone, so that the image offset between the target telephoto image 2 and the target telephoto image 1 is relatively large, and a void is easily generated between the target telephoto image 2 and the target telephoto image 1, for example, a void exists between two frames of images or a void edge is generated due to the dislocation of the upper/lower edge positions of the two frames of images.

The mobile phone can detect the hole through various methods. For example, the target tele image 1 and the target tele image 2 may be both attached to the reference wide image at the same position, so as to determine whether a gap occurs between the attached target tele image 1 and the target tele image 2, thereby detecting whether a hole occurs. For another example, when the target tele image 1 and the target tele image 2 are distributed left and right, and the target tele image 1 and the target tele image 2 are both matched to the coordinate system of the wide image, if the abscissa of the leftmost pixel of the right image is greater than the abscissa of the rightmost pixel of the left image, the mobile phone may determine that the hole is detected.

In addition, the mobile phone may also measure and calculate a spatial position relationship when the target tele image 1 and the target tele image 2 are shot through an Inertial Measurement Unit (IMU) of the mobile phone (e.g., a gyroscope), and sequentially detect whether a hole occurs.

When the mobile phone determines that a hole exists between the target tele image 2 and the target tele image 1, the content at the same position on the reference wide image may be filled in the hole between the target tele image 1 and the target tele image 2. In this way, the whole of the target tele image 1, the target tele image 2, and the filled portion is a stitched image.

Therefore, the mobile phone can fill the cavity between the object tele images to be spliced according to the reference wide-angle image, the object tele images cannot be registered and spliced due to the cavity, the object tele images do not need to be cut according to the cavity, the image field angle and the image resolution are not lost during splicing, and the spliced images have larger field angle and image resolution. In other existing technical solutions, when the tele images to be stitched deviate up/down due to the angle problem of manual rotation, the cropping is performed at the minimum image height, so that the image resolution and the image field angle are lost.

For example, the reference wide image may be referred to as an image 1210 shown in (a) of fig. 12B, the target tele image 1 may be referred to as an image 1211 shown in (a) of fig. 12B, and the target tele image 2 may be referred to as an image 1212 shown in (B) of fig. 12B. A hole 1213 exists between the target tele image 1 and the target tele image 2, and the cell phone can fill the hole 1213 according to the content of the corresponding position of the reference wide image. In this way, as shown in (B) of fig. 12B, the target tele image 1, the target tele image 2, and the entire image 1214 formed by the filled portion are a stitched image.

Further, in the event that the cell phone determines that there is a hole between the target tele image 2 and the target tele image 1, in other embodiments, the cell phone may direct the user to take a new frame of the target tele image 2 over to fill the hole. For example, in a possible implementation, in the case of a hole, the grid 2 corresponding to the target tele image 2 is not displayed differently from other grids, so as to guide the user to shoot one frame of the target tele image 2 matching the grid 2 again. In another possible implementation manner, the mobile phone prompts the user in a manner of displaying information or playing voice, and the like, and please acquire a frame of target tele image again to correspond to the current grid.

The hole filling adopts information on the wide-angle image, and other surrounding areas are the result of fusion splicing of the target tele image 2 and the target tele image 1, so that the hole filling area has obvious difference from the surrounding areas in terms of resolution and resolving power. Therefore, the mobile phone can perform image super-resolution processing on the cavity area, reduce the resolution difference between the cavity filling area and the surrounding area, and achieve better user experience. In addition, the mobile phone can also adopt an AI local search method or an AI image restoration method to predict and fill the hollow area. The embodiment of the application is not limited to the specific method of filling and super-resolution of the image.

In some embodiments of the application, the mobile phone may further perform ghost-removing processing on the shot moving object in the stitching process.

In the photographing process, a certain time is needed for photographing of the multi-frame target tele images, and when a moving object appears in the photographing range, images of the moving object may appear on adjacent N (N is an integer greater than 2) frames of target tele images, so that ghosts (or called ghosts, virtual images and the like) of the moving object appear on the spliced image. In this case, the mobile phone can also perform ghost image removing processing in the photographing process.

In the following, the method for removing ghosts of moving objects is explained by taking the splicing scene of the target tele-image 2 and the target tele-image 1 as an example through two situations.

(1) Object moving at slow speed

The moving speed of the moving object is slow, and no ghost image appears on the wide-angle image of the moving object. Moving objects appear on the adjacent target tele images 2 and the target tele images 1, and images of the moving objects on different target tele images are mutually distinguished and are not communicated, so that a plurality of images, namely ghosts, of the moving objects appear on the spliced image easily. The detection of the ghost area (also referred to as a motion area) is a well-established technology at present, and may be estimated and detected by, for example, an optical flow method or by combining with gyroscope data of a mobile phone, which is not described in detail herein.

In this case, the mobile phone may detect the position of the moving object first (an optical flow method, an object detection and tracking method, a semantic segmentation method, a luminance subtraction and dilation corrosion method, or the like may be used). Then, the mobile phone can keep the complete image (also called as a ghost-free image) of the moving object on the target tele image 1, the target tele image 2 or the wide-angle image on the spliced image of the target tele image 2 and the target tele image 1, delete the images of the moving objects in other areas, and adopt the content of the deleted area corresponding to the deleted area on the reference wide-angle image to perform cavity filling on the deleted area, thereby avoiding the occurrence of ghost of the moving object on the spliced image. The wide-angle image used for filling the cavity may be a frame of recently acquired wide-angle image, a frame of wide-angle image acquired by the mobile phone in the photographing process, or a reference wide-angle image, and the like.

For example, as a simple implementation, the mobile phone may keep a complete image of the moving object over the wide-angle image. For another example, in order to make the position of the moving object on the final target image closer to the position that the user finally sees, the mobile phone may keep the complete image of the moving object on the last frame of target tele-image.

The positions of the moving object on the target tele image 2, the target tele image 1 and the wide image may be different, and the mobile phone may determine to keep a complete image of the moving object on a certain image according to corresponding measurements.

When the mobile phone keeps the complete image of the moving object on the target tele image 1 on the spliced image of the target tele image 2 and the target tele image 1, the mobile phone can delete the image of the moving object on the target tele image 2, and adopt the content of the corresponding position on the wide image to perform cavity filling on the deleted area.

For example, the mobile phone may keep the complete image of the moving object on the target tele image 1 and delete the image of the moving object on the target tele image 2 before the target tele image 1 and the target tele image 2 are registered. Then, the mobile phone can perform registration and fusion on the target tele image 1 and the deleted target tele image 2, and perform cavity filling on the deleted blank area by using the content of the corresponding position on the reference wide-angle image, so as to generate a spliced image.

For another example, the mobile phone may perform registration and fusion on the target tele image 1 and the target tele image 2 to generate a stitched image. Then, the mobile phone can delete the images of the moving object on the stitched image (including the images of the moving object on the target tele image 1 and the target tele image 2). And the mobile phone attaches the complete image of the moving object on the target tele image 1 to the corresponding position on the spliced image. And the mobile phone performs cavity filling on the spliced image by referring to the content of the corresponding position on the wide-angle image, wherein the cavity position is the area where the deleted image of the moving object on the target tele image 2 is located.

When the mobile phone retains the complete image of the moving object on the target tele image 2 on the spliced image of the target tele image 2 and the target tele image 1, the mobile phone can delete the image of the moving object on the target tele image 1, and perform cavity filling on the deleted area by adopting the content of the corresponding position on the wide-angle image.

When the mobile phone keeps the complete image of the moving object on the wide-angle image on the spliced image of the target tele image 2 and the target tele image 1, the mobile phone can delete the image of the moving object on the target tele image 1 and the target tele image 2, and adopt the content of the corresponding position on the wide-angle image to perform cavity filling on the deleted area.

For example, the reference wide-angle image may be referred to as (a) in fig. 12C. The target tele image 1 can be seen in (b) of fig. 12C, including an image 1221 of a moving object. The target tele image 2 can be seen in (C) of fig. 12C, including an image 1222 of a moving object. The mobile phone deletes the images of the moving object on the target tele image 1 and the target tele image 2, and fills the holes in the deleted areas by referring to the content of the corresponding position on the wide-angle image, thereby obtaining the stitched image shown in (d) of fig. 12C.

(2) Fast moving object

When the moving speed of the moving object is fast, there may be the following cases:

in case a, no ghost of the moving object appears on the reference wide-angle image. The mobile phone can perform the ghost removing processing by adopting the same method as the slow moving object, and perform the cavity filling on the ghost area deleted from the spliced image by adopting the content of the corresponding position on the reference wide-angle image.

And b, the ghost of the moving object appears on the reference wide-angle image, and the ghost of the moving object appears on the multi-frame wide-angle image acquired by the mobile phone in real time. Under the condition, the mobile phone can perform processing such as motion vector estimation or optical flow estimation by combining a plurality of frames of wide-angle images acquired in the photographing process, so that the motion track of the moving object is judged, and the connected region of the moving object on the wide-angle images is removed to obtain the complete image of the moving object without ghost. That is, the mobile phone can directly obtain or obtain after processing a wide-angle image without ghosts of moving objects.

When the moving speed of the moving object is high, the moving object appears on the adjacent target tele image 2 and the target tele image 1, and the images of the moving object on different target tele images are mutually communicated, so that ghosting of the moving object appears on the spliced image easily.

In this case, in some embodiments, the mobile phone may attach the target tele image 1 and the target tele image 2 to the wide-angle image without the ghost, so as to mark a connected region of the moving object image, i.e., a ghost region, on the entire image after the target tele image 1 and the target tele image 2 are attached. The wide-angle image without the ghost may be a wide-angle image of a frame without the ghost, or a wide-angle image obtained after the ghost removal processing is performed. When the wide-angle image without the ghost is the reference wide-angle image, the mobile phone may attach the target tele-image 1 and the target tele-image 2 to the wide-angle image without the ghost in the registration process, and mark a connected region.

In other embodiments, the mobile phone may perform processing such as motion vector estimation in combination with the multiple frames of wide-angle images acquired during the photographing process, so as to determine the connected regions of the target tele image 2 and the moving object image on the target tele image 1. The mobile phone can delete the image areas of the moving object on the spliced image of the target tele image 2 and the target tele image 1, and perform cavity filling on the deleted areas according to the content of the corresponding position on the wide-angle image without the ghost so as to avoid the ghost of the moving object on the spliced image.

For example, the reference wide-angle image may be referred to as (a) in fig. 12D. The target tele image 1 can be seen in (b) of fig. 12D, including a ghost 1231 of the moving object. The target tele image 2 can be seen in (c) of fig. 12D, including a ghost 1232 of the moving object. The mobile phone deletes the ghosts of the moving object on the target tele image 1 and the target tele image 2, and fills the holes in the deleted areas by referring to the content of the corresponding position on the wide-angle image, thereby obtaining the stitched image shown in (D) in fig. 12D.

Therefore, the mobile phone can eliminate the ghost of a moving object on the spliced image of the target long-focus image in the shooting process, and presents a clear and ghost-free spliced image for a user on the shooting interface, so that the image display effect in the shooting process is improved.

The above description has been given by taking an example in which ghosting appears on the target tele image 1 and the target tele image 2. It will be appreciated that ghosting may also occur on successive multi-frame target tele images, such as a lengthy cat dragged by a body having a ghosted area on successive multi-frame target tele images.

The above description has been given by taking an example in which the target tele image 2 is stitched with the target tele image 1 to obtain a stitched image. Similarly, the mobile phone may adopt the same stitching method to stitch the subsequent target tele image k (k is an integer greater than 2) with the previously generated stitched image, so as to generate a new stitched image. For example, after obtaining a stitched image from the target tele image 2 and the target tele image 1, the mobile phone may stitch the target tele image 3 with the previously generated stitched image, thereby generating a new stitched image. The target tele image 3 is stitched with the previous stitched image, or the target tele image 2 adjacent to the target tele image 3 in the target tele image 3 is stitched with the previous stitched image. The stitching process of the subsequent target tele images is not described in detail here.

Wherein, the non-first frame target long-focus image can be spliced with one or more adjacent target long-focus images. For example, for the target tele images corresponding to the grid of the first row and the first column in the guide frame, the target tele images corresponding to the grid of the first row and the second column may be registered and stitched only, and may also be registered and stitched with all target tele images adjacent thereto.

In some embodiments of the application, after obtaining the target tele image 1 matched with the grid 1 (i.e., the first grid to be matched), the mobile phone may match and fuse the thumbnail of the target tele image 1 with the wide image serving as the background image, and then display the thumbnail of the target tele image 1 on the shooting interface, so that the user sees the real picture of the target tele image 1 matched with the grid 1. The thumbnail of the target tele image 1 is superimposed over an area of the wide image having the same image content as the target tele image 1. In this case, the position and size of the thumbnail of the target tele image 1 may have some deviation from the matched grid 1, or may just coincide with the position and size of the matched grid 1. Illustratively, referring to the shooting interface shown in (b) in fig. 12A, the thumbnail of the target tele image 1 is overlaid over the area of the same image content as the wide image.

The thumbnail of the target tele image 1 is an image obtained by down-sampling the target tele image 1 acquired by the tele camera. The wide-angle images displayed on the preview interface and the shooting interface of the mobile phone are also images obtained by down-sampling images acquired by the wide-angle camera. Since the thumbnail of the target tele image and the wide image displayed on the interface are from different cameras and data sources, the display effect of the interface may be different before and after the thumbnail of the target tele image is attached.

In some embodiments, in the photographing process, the mobile phone may display the stitched image thumbnail on the photographing interface, so that the user can know the current real-time photographing progress conveniently. For example, the mobile phone may match and fuse the stitched image thumbnail with the wide-angle image serving as the background image, and then attach the stitched image thumbnail to an area where the wide-angle image and the stitched image thumbnail have the same image content.

Wherein the stitched image thumbnail may be obtained in different ways. In one implementation, the mobile phone performs registration and stitching on the downsampled adjacent target tele images to obtain a stitched image thumbnail, and attaches the stitched image thumbnail to an area where the wide-angle image and the stitched image thumbnail have the same image content. In the implementation mode, based on the target long-focus image with lower resolution after down-sampling, the processing process of obtaining the spliced image thumbnail by the mobile phone is simpler, the processing time is shorter, the spliced image thumbnail can be obtained in real time and displayed on a shooting interface in time, and the occurrence of blocking is avoided. The process of obtaining the spliced image thumbnail by the mobile phone and the process of obtaining the spliced image by the mobile phone are processed in parallel.

It should be noted that, as can be seen from the above description, the stitched images can be obtained by real-time stitching in the photographing process; or uniformly splicing the target long-focus images matched with all grids after the target long-focus images are obtained or shooting is finished. In the scheme of displaying the spliced image thumbnail on the shooting interface, in order to facilitate the user to know the shooting progress, the mobile phone can obtain and display the spliced image thumbnail in real time according to the target long-focus image obtained each time in the shooting process.

In another implementation, the mobile phone down-samples the stitched image to obtain a stitched image thumbnail, and fits the stitched image thumbnail over an area of the wide-angle image having the same image content as the stitched image thumbnail. On the shooting interface, the position and the size of the spliced image thumbnail may have some deviation from the positions and the sizes of the matched grids, and may be just consistent with the positions and the sizes of the matched grids. The mobile phone is attached to the spliced image thumbnail on the wide-angle image of the shooting interface, so that a user can feel a real and real-time splicing process and splicing progress.

The following describes how the thumbnail of the stitched image is displayed on the shooting interface in both the fixed and unfixed cases of the background image.

Case 1, background image fixed as reference wide-angle image

In the embodiment of the application, the user can move the tele camera through moving the mobile phone, or the user can directly move the tele camera, or the mobile phone can automatically control the tele camera to move. In case 1, the background image on the shooting interface is fixed as the reference wide-angle image, the relative position of the guide frame and the background image is kept unchanged, the content of the background image corresponding to each grid is also kept unchanged, and the position of the tele frame relative to the background image can be changed in real time.

For example, according to the shooting sequence shown in fig. 11 (c), the mobile phone matches the grid in the middle row of the guide frame, then matches the grid in the upper row of the guide frame, and then matches the grid in the lower row of the guide frame during the shooting process. After the mobile phone obtains the target tele image corresponding to the middle grid in the middle row, the shooting interface with the thumbnail of the stitched image is displayed as (c) in fig. 12A. Then, the telephoto camera continues to move, and as shown in fig. 12A (c), the mobile phone displays a real-time telephoto frame corresponding to the photographing range of the telephoto camera on the photographing interface. After the mobile phone obtains the target tele image matched with the rightmost grid above the guide frame, the shooting interface with the thumbnail of the stitched image displayed thereon can be seen in (d) of fig. 12A. After the mobile phone obtains the target tele image matched with the leftmost grid below the guide frame, the shooting interface with the thumbnail of the stitched image displayed thereon can be seen in (e) of fig. 12A. After the mobile phone obtains the target tele image matched with the last grid in the guide frame, the shooting interface with the stitched image thumbnail displayed may refer to (f) in fig. 12A, and the mobile phone generates the stitched image thumbnail corresponding to the 3 × 3 grid. As another example, in the case where the shooting interface displays the shooting order prompt information, the schematic diagrams of the shooting interface may be referred to in fig. 14 (a) - (e).

Case 2, wide-angle image that background image is not fixed and is collected in real time by mobile phone

In this case, when the user moves the mobile phone to synchronously move the wide-angle camera and the telephoto camera, the content of the background image corresponding to each mesh is not changed, but the background image is not fixed and changes in real time, so that the relative position of the guide frame and the background image changes in real time along with the change of the content of the background image. It is understood that, since the position of the telephoto frame is determined by the lens centers of the wide-angle camera and the telephoto camera, the object distance of the photographing, and the size of the angle of view, in the case where the object distance is substantially constant, the relative position of the telephoto frame to the background image is also substantially constant in view of the relative positions of the lens centers of the wide-angle camera and the telephoto camera. For example, during the photographing process, the tele frame is always located near the middle of the background image.

In the photographing process, when the mobile phone moves, the wide-angle camera also moves, the content of the wide-angle image serving as the background image also changes, but the content of the background image corresponding to each grid is kept unchanged. It is also understood that during the photographing process, the correspondence between the grid and the content of the wide-angle image within the grid remains unchanged, and the grid is bound to the content of the wide-angle image as the background image.

The cell phone can realize the binding of the grid and the wide-angle image content in various ways. For example, the mobile phone may record the content of the wide-angle image corresponding to each grid after the photographing operation is detected; in the subsequent shooting process, the mobile phone can match the grid with the content of the real-time acquired wide-angle image in an image matching mode, so that the binding of the grid and the content of the wide-angle image is realized. For another example, the mobile phone may record the coordinate position of each grid and the corresponding wide-angle image content after detecting the photographing operation; in the subsequent photographing process, the mobile phone can determine the translation amount and/or rotation amount of the mobile phone according to the data of the inertial measurement units such as the gyroscope and the like, so that the new coordinate position of each grid and the corresponding wide-angle image content is calculated according to the translation amount and/or rotation amount, and the binding of the grids and the wide-angle image content is realized.

For example, when the mobile phone moves to the right, the wide camera and the tele camera move to the right simultaneously, the field angle of the wide image and the image content move to the right, the position of the guide frame on the screen shifts to the left as the wide image content moves to the right, and the position of the tele frame relative to the wide image remains substantially unchanged. For example, in the case that the background image is a wide-angle image acquired by a mobile phone in real time, after the leftmost grid in the middle row is matched, the schematic diagram of the shooting interface may be as shown in (a) of fig. 13C, and a thumbnail of the target tele image 1 is displayed on the shooting interface; after the grids in the middle of the middle row are matched, the schematic diagram of the shooting interface can be seen in (b) in fig. 13C, and the spliced image thumbnail is displayed on the shooting interface; after the rightmost grid in the middle row is matched, the schematic diagram of the shooting interface can be seen in (C) of fig. 13C, and the thumbnail of the stitched image is displayed on the shooting interface. As shown in (a) - (C) of fig. 13C, the wide image is shifted to the right, the guide frame is shifted to the left, and the tele frame is located substantially at the middle of the wide image.

Fig. 12A illustrates an example in which the thumbnail of the target tele image is substantially aligned with the edge of the matched grid. During the photographing process, as shown in (a) - (C) of fig. 13C, when the thumbnail of the target tele image deviates from the position of the grid, the edge of the stitched image may be jagged and not substantially aligned with the edge of the matched grid.

In addition, under the condition that the user directly moves the tele-camera or the mobile phone automatically controls the tele-camera to move, the tele-camera and the wide-angle camera can move asynchronously. Under the condition that the background image is not fixed and is a wide-angle image acquired by the mobile phone in real time, for example, when the background image changes in real time due to shaking of the mobile phone and the like, the relative position of the guide frame and the background image changes in real time along with the change of the content of the background image, the content of the background image corresponding to each grid is basically kept unchanged, and the relative position of the tele frame and the background image can change in real time along with the movement of the tele camera.

In an embodiment of the present application, the stitched image thumbnail is substantially the same size, location, and content of the overlaid image as the wide-angle image. The size of the stitched image thumbnail is small, so that the user cannot conveniently view the image details. In some embodiments, the mobile phone may enlarge and display the stitched image thumbnail on the shooting interface in response to a preset operation such as a click or a long press of the user, so that the user can clearly see specific details of an object on the target tele stitched image acquired by the tele camera at any time.

For example, as shown in fig. 15A (a), after the mobile phone detects the user's operation of clicking the stitched image thumbnail on the shooting interface, the stitched image thumbnail is displayed in an enlarged manner on the shooting interface as shown in fig. 15A (b). After detecting that the user clicks the stitched image again or detecting the return operation of the user, the mobile phone resumes to display the stitched image thumbnail as shown in (a) in fig. 15A.

In other embodiments, in order to avoid that the size of the stitched image thumbnail is small (especially when the difference between the equivalent focal lengths of the telephoto camera and the wide-angle camera is large, for example, the ratio of the equivalent focal lengths is larger than a certain preset value), which causes inconvenience for a user to view details of the image, the mobile phone may automatically display the stitched image thumbnail in an enlarged manner on the shooting interface. Similarly, the mobile phone can also enlarge and display the thumbnail of the target tele image 1 on the shooting interface. The magnification display times can be default values, can also be related to the proportion of the equivalent focal lengths of the wide-angle camera and the long-focus camera, and can also be numerical values set by a user, without limitation.

In order to guide a user to move a mobile phone or a tele camera conveniently, the thumbnail of the amplified and displayed target tele image 1 or the thumbnail of the spliced image cannot shield the grid to be matched. Illustratively, referring to (a) - (B) in fig. 15B, when shooting is performed in a grid sequence from left to right, the right sides of the thumbnail of the enlarged target tele image 1 and the thumbnail of the stitched image on the shooting interface are aligned with the right side of the recently matched grid, that is, aligned with the left side of the grid to be matched, so that the thumbnail of the target tele image 1 and the thumbnail of the stitched image are prevented from blocking the grid to be matched on the right side. When shooting is carried out according to the grid sequence from top to bottom, the lower side of the amplified thumbnail of the target tele image 1 or the spliced image thumbnail is aligned with the lower side of the grid which is matched recently, namely aligned with the upper side of the grid to be matched, so that the condition that the thumbnail of the target tele image 1 and the spliced image thumbnail shield the grid to be matched on the lower side is avoided. For another example, the grid in the middle row of the guide frame usually corresponds to the image content that the user wants to capture most, so the thumbnail of the target tele image 1 and the stitched image thumbnail do not obscure the unmatched grid in the middle row as much as possible.

In other embodiments, the mobile phone displays the target area image corresponding to the guide frame on the wide-angle image on the shooting interface, but does not display the complete wide-angle image. The ratio of the size of the target area image to the size of the guide frame is r, and r is more than or equal to 1. The target area image may be obtained by cropping and enlarging the complete wide-angle image. Therefore, the sizes of the guide frame and the grid displayed on the shooting interface are larger, and the mobile phone can conveniently carry out mobile shooting and matching according to the grid with the larger size.

In particular, when the equivalent focal length of the tele camera is different from that of the wide camera by a large amount, if a complete wide image and a corresponding display guide frame are displayed on the shooting interface, the sizes of the guide frame, the grid, and the thumbnail/stitched image thumbnail of the target tele image 1 are small, which is inconvenient for the user to view. The mobile phone can display the target area image, the guide frame and the thumbnail/stitched image thumbnail of the target tele image 1 on the shooting interface after the thumbnail/stitched image thumbnail of the target tele image 1 is enlarged in equal proportion, so that a user can conveniently view the specific content of the thumbnail/stitched image thumbnail of the target tele image 1 with a larger size.

For example, the complete wide-angle image on the shooting interface shown in (d) in fig. 12A described above may be replaced with the target area image 1500 in the wide-angle image on the shooting interface shown in fig. 15C. In contrast to the shooting interface shown in fig. 12A (d), the target area image, the guide frame, and the stitched image thumbnail are enlarged in equal proportion on the shooting interface shown in fig. 15C, and the ratio r of the size of the target area image to the size of the guide frame is greater than 1.

In the scheme described in the above embodiment, during the photographing process, the mobile phone displays the thumbnail of the target tele image 1 and the thumbnail of the stitched image on the photographing interface. In other embodiments, during the photographing process, the mobile phone may not display the thumbnail of the target tele image 1 and the thumbnail of the stitched image, but only display a stitched frame on the wide image, where the stitched frame is a frame of the thumbnail of the target tele image 1 or a frame of the thumbnail of the stitched image. Therefore, the mobile phone can prompt the user of the current shooting progress through the splicing frame, and the thumbnail of the target tele image 1 and the thumbnail of the spliced image do not need to be acquired and displayed, so that the processing load of the mobile phone can be reduced. In other embodiments, during the photographing process, the cell phone may not display the thumbnail of the target tele image 1 and the thumbnail of the stitched image, but merely highlight (e.g., highlight or bold) the matched grid or the border of the matched grid. Therefore, the mobile phone can prompt the user of the current shooting progress by highlighting the spliced grid, and does not need to acquire and display the thumbnail of the target tele image 1 and the spliced image thumbnail, so that the processing load of the mobile phone can be reduced.

In addition, in some embodiments, in the photographing process, the mobile phone may also give a corresponding prompt to the user according to the current real-time photographing situation. For example, when the cell phone is moving too fast and the tele-image is not in time to match the grid, the cell phone may prompt the user to please move the cell phone slowly. For another example, when the moving direction of the mobile phone is opposite to the direction indicated by the shooting sequence, or the mobile phone moves towards the direction of the matched grid, the mobile phone may prompt the user to please move the mobile phone along the indicated direction. Or the mobile phone directly terminates the shooting process and generates a final target image according to the acquired target tele image. As another example, when the tele frame is offset farther above the grid to be matched. The mobile phone can prompt the user to ask for moving the mobile phone downwards; when the tele frame deviates far from the lower part of the grid to be matched, the mobile phone can prompt the user to please move the mobile phone upwards.

It is worth noting that, in the process of photographing, for the wide-angle image collected by the mobile phone in real time, on one hand, the wide-angle image can be used for displaying as a background image under the condition that the background image is not fixed; on the other hand, the method can also be used for carrying out motion vector estimation on the moving object, thereby determining a connected region of the moving object and assisting in removing ghost. During the photographing process, the reference wide-angle image with better quality can be used as a reference for carrying out AE, AWB and DRC configuration on the target tele image, as a reference for registering and fusing multi-frame target tele images, and as a reference for carrying out processing such as hole filling and the like.

204. And after the mobile phone determines that the shooting is finished, generating a target image according to the spliced image.

After the mobile phone determines that the shooting is finished, the mobile phone can generate a target image according to a spliced image obtained by a plurality of frames of target tele images.

In some embodiments, after the shooting is finished, before the target image is generated according to the stitched image, the mobile phone may further perform processing such as ghost removal, dynamic range enhancement, or hole filling on the stitched image to improve the quality of the stitched image, so that the target image is generated according to the processed stitched image to improve the quality of the target image.

Removing ghost images:

if the mobile phone does not perform the ghost removing processing on the moving object in the photographing process, the mobile phone can perform the ghost removing processing on the spliced image after the photographing is finished and before the target image is generated according to the spliced image, so that the target image is generated according to the spliced image after the ghost is removed. For example, when moving objects appear on adjacent multi-frame target images, the mobile phone may process the ghosts on the stitched image obtained after the shooting is finished. In one possible implementation, the handset may retain a complete image of the moving object on the last frame of the target tele-image in order to bring the position of the moving object on the final target image closer to what the user last viewed. For another example, when the mobile phone generates the stitched image after the shooting is finished, the mobile phone may process the ghosts on the stitched image together.

In this case, similar to the above-mentioned method for removing ghosts during the photographing process, for a slow-moving object, the mobile phone may delete the image of the moving object from the stitched image, retain the complete image of the moving object on a certain frame of the target tele-image or a certain frame of the wide-angle image, and perform hole filling by using the content of the corresponding position on the wide-angle image. For a fast moving object, the mobile phone can perform motion vector estimation according to the multi-frame wide-angle image, so that a connected region of the moving object on the wide-angle image is removed to obtain a complete moving object image without ghost. And deleting the image area of the moving object on the spliced image by the mobile phone, and filling the hole in the deleted area by adopting the content of the corresponding position on the wide-angle image without the ghost.

The embodiment of the present application does not limit whether the ghost-removing processing is performed during the shooting process or after the shooting is finished.

And (3) enhancing the dynamic range:

in the embodiment of the application, the dynamic range of the spliced image of the multi-frame target tele image may be small, the bright layers are few, the brightness range is small, the bright-dark contrast is insufficient, and the details of the dark part and the bright part are insufficient. In some embodiments, after the shooting is finished, before the target image is generated according to the stitched image, the mobile phone may enhance the dynamic range of the whole stitched image, for example, a method (HDRnet) for enhancing the dynamic range of the shot picture is adopted, so that the stitched image is more abundant in brightness and gradation and larger in brightness and darkness range, and then the target image is generated according to the stitched image with the enhanced dynamic range.

For example, in the process of enhancing the dynamic range, the mobile phone may process the spliced image through a convolutional neural network (AI network), so as to directly obtain the effect of the high dynamic range, thereby obtaining the target image with the high dynamic range.

In the process of enhancing the dynamic range, the mobile phone can also adjust the brightness distribution of the spliced image according to the brightness distribution condition of the reference wide-angle image, and increase the brightness range of the spliced image, so that the dynamic range of the spliced image is enhanced. For example, as can be seen from the luminance histograms of the reference wide-angle image and the stitched image, the luminance value range in the corresponding region of the reference wide-angle image and the stitched image is 30-250, and the luminance value range of the stitched image is 180-230. The mobile phone can adjust the brightness value of partial pixel points (such as randomly selected pixel points or edge pixel points) with the brightness value close to 180 on the spliced image to be between 30 and 180; adjusting the brightness value of the partial pixel point with the brightness value close to 230 on the spliced image to be between 230 and 250; therefore, the brightness range of the spliced image is enlarged, and the dynamic range of the spliced image is enhanced.

In addition, the mobile phone can also extract a high-frequency component image of the reference wide-angle image, and fuse the high-frequency component image and the stitched image to increase high-frequency details of the stitched image (which may also be referred to as high-frequency component fusion). For example, the high-frequency component image includes pixel points of an upper edge portion of a reference wide-angle image or pixel points of a transition portion of an edge of a different object. The mobile phone can extract the high-frequency component image of the reference wide-angle image by adopting methods such as wavelet transformation or deep learning, so that high-frequency details with large frequency and texture changes are extracted and superposed on the spliced image to enhance the high-frequency details of the spliced image and enhance the definition and details of the spliced image and the target image.

In the process of enhancing the dynamic range, the mobile phone can also synthesize a high dynamic range of a plurality of frames of wide-angle images, and apply the synthesized high dynamic range effect to the spliced image through a dodging and color-homogenizing algorithm, namely, the spliced image is subjected to dodging and color-homogenizing treatment according to the synthesized high dynamic range information, so that the dynamic range of the spliced image is enhanced. In addition, the mobile phone can fold the details of the spliced image back to the wide-angle image with the effect of high dynamic range, and finally obtain the target image with high dynamic range and abundant texture details.

For example, in some embodiments, the mobile phone may synthesize a wide-angle image with a high dynamic range by using different exposures of multiple frames of the wide-angle image, and then perform a process of dodging and evening colors or a process of style migration on the stitched image based on the high dynamic range of the wide-angle image, so that the stitched image also has an effect of the high dynamic range.

For example, in some embodiments, the handset may synthesize a wide-angle image with a high dynamic range using different exposures of multiple frames of the wide-angle image. Then, the mobile phone can extract details and textures in the spliced image, and the images are folded back on the wide-angle image with the high dynamic range to perform fusion of details, colors and brightness, so that the target image with the high dynamic range and high definition details is finally obtained.

The algorithm for enhancing the dynamic range is not limited to a conventional algorithm or an algorithm of a convolutional neural network. The algorithm for synthesizing the wide-angle image with the high dynamic range by the multi-frame wide-angle image is not limited to the traditional algorithm or the algorithm of the convolution neural network.

Filling the holes:

in embodiments of the present application, there may be some holes between different target tele images on the stitched image. In some embodiments, the mobile phone may further perform hole filling on the stitched image, and then generate the target image according to the stitched image after the hole filling. Because the target tele images may be staggered, the edges of the stitched images may not be smooth enough, and empty edges may exist between image portions from different target tele images, the mobile phone may further perform void filling on the empty edges according to the reference wide-angle image to obtain a neat and attractive rectangular or square stitched image. In this way, the mobile phone fills the blank edge according to the corresponding content of the reference wide-angle image, and does not need to cut the target tele image according to the blank edge, so that the image field angle and the image resolution are lost during splicing, and the spliced image can have a larger field angle and image resolution, so that the target image generated according to the spliced image can have a larger field angle and image resolution. In addition, for filling the cavity and the cavity edge, the mobile phone can adopt the corresponding area of the multi-frame wide-angle image to perform high-resolution synthesis, and can also use an image super-resolution algorithm to process the corresponding area of the single-frame reference wide-angle image, so that the cavity and the cavity edge can have higher resolution, and the user experience is improved.

For example, the stitched image with the blank edge obtained by the mobile phone according to the multi-frame target tele image may refer to an image framed by a solid line shown in (a) in fig. 15D, and the stitched image obtained by the mobile phone after performing the hole filling processing according to the reference wide-angle image may refer to an image framed by a solid line shown in (b) in fig. 15D.

In the photographing process, the reference wide-angle image with good quality is used as a reference to perform AE, AWB adjustment and DRC configuration on the target tele image, is used as a reference to register and fuse the multi-frame target tele image, is used as a reference to perform hole filling, and is used as a reference to enhance the dynamic range of the whole stitched image after the photographing is finished. In addition, in the whole photographing process, each frame of target tele image is processed according to the same reference wide-angle image, so that the effect of each target tele image is consistent with that of the reference wide-angle image as much as possible, the overall effect of the final spliced image and the target image is consistent with that of the reference wide-angle image, and the final spliced image and the target image are natural, smooth in transition and good in quality.

In addition, in the process of generating the target image from the stitched image, the mobile phone may need to spend a certain time to perform processing such as dynamic range enhancement. Therefore, in some embodiments, the mobile phone can prompt the user that the generated target image is being processed in the processing process, so as to avoid the user from mistakenly thinking that the mobile phone is jammed or has other abnormal conditions. For example, the mobile phone may prompt the user through text prompt, voice prompt, or a rotating circle mark. For example, referring to fig. 16, the mobile phone can transmit the text message: is processing, please later; and a rotating circle mark to prompt the user that the target image is currently being processed.

The method for determining the shooting end by the mobile phone can be various, and the method for generating the target image by the mobile phone according to the spliced image can also be various.

In some cases, before the mesh in the guide frame does not complete shooting (i.e., the mesh in the guide frame does not complete matching), the mobile phone determines that shooting is finished after detecting the stop photographing operation of the user. For example, the mobile phone determines that the shooting is finished after detecting the operation of the user clicking the stop shooting control 1200 shown in (e) in fig. 12A. It can be understood that the operation of stopping photographing may also be other gesture operations or user voice indication operations, and the operation of triggering the mobile phone to end the photographing process is not limited in the embodiment of the present application. In other cases, before the grid in the guide frame is not finished shooting, if the mobile phone moving direction is seriously deviated from the indication direction of the guide frame (for example, the deviation range of the moving direction and the indication direction of the guide frame is larger than or equal to a certain preset value) or the mobile phone moves out of the guide frame, the shooting is automatically ended.

In some embodiments, if the mobile phone determines that the shooting is finished before the grid in the guide frame is not finished shooting, a part corresponding to the guide frame is cut from the reference wide-angle image to generate the target image.

In other embodiments, if the mobile phone determines that the shooting is finished before the grid in the guide frame is not shot, and the shooting is performed according to the rows of the grid in the shooting process, the mobile phone omits the part, which is not shot in the whole row of the grid, of the stitched image, and generates the target image according to the stitched image corresponding to the shot whole row of the grid, so that the target image is the image corresponding to the whole row of the grid. For example, in the case shown in (e) in fig. 12A, if the mobile phone detects a photographing stop operation of the user, the target image shown in (a) in fig. 17 is generated from the stitched image corresponding to the meshes of the middle row and the upper row of the guide frame.

Similarly, if the mobile phone determines that the shooting is finished before the grid in the guide frame is not shot, and the shooting is carried out according to the columns of the grid in the shooting process, the mobile phone removes the part which is not shot of the whole column of the grid on the spliced image, and generates the target image according to the shot spliced image corresponding to the whole column of the grid, so that the target image is the image corresponding to the whole column of the grid.

In other embodiments, if the mobile phone determines that the shooting is finished before the grid in the guide frame is not finished, and the shooting is performed according to the rows of the grid in the shooting process, the mobile phone supplements the image corresponding to the latest row of the grid on the spliced image according to the reference wide-angle image, so as to generate the target image. Similarly, if the mobile phone determines that the shooting is finished before the grid in the guide frame is not finished and the shooting is carried out according to the grid columns in the shooting process, the mobile phone supplements the image corresponding to the latest grid column on the spliced image according to the reference wide-angle image, so as to generate the target image.

In other embodiments, if the mobile phone determines that the shooting is finished before the grid in the guide frame is not finished, the shooting process is abnormal, and the mobile phone does not generate the target image. Moreover, the mobile phone can also prompt the user to stop shooting or abnormal shooting and the like.

In other cases, after the grids in the guide frame are completely shot, the mobile phone automatically finishes shooting, and generates a target image according to the spliced image. In some embodiments, the size of the target image may be the same as the size of the guide frame. If the size of the stitched image is inconsistent with the size of the guide frame due to reasons such as misalignment between the target tele images during stitching, the mobile phone can cut or fill a void (fill according to the reference wide-angle image) in the stitched image according to the size of the guide frame, so that the size of the stitched image is consistent with the size of the guide frame. For example, the target image generated by the mobile phone can be seen in (b) of fig. 17. In other embodiments, if the edges of the stitched images are not aligned due to misalignment between the target tele images during stitching or the like, the mobile phone may fill the stitched images according to the reference wide-angle image, so as to obtain target images with regular shapes such as a rectangle or a square, where the positions and the sizes of the target images may be different from those of the guide frame.

The resolution ratio of the wide-angle image collected by the wide-angle camera and the resolution ratio of the long-focus image collected by the long-focus camera are both large. Because the field angle of the wide-angle camera is far larger than that of the telephoto camera, the number of the pixel points corresponding to the telephoto image in the unit field angle is far larger than that of the pixel points corresponding to the wide-angle image in the unit field angle. For example, the resolution of a wide image acquired by a wide camera is 4000 × 3000, i.e., 1200 ten thousand pixels, and a tele image acquired by a tele camera is 3264 × 2448, i.e., 800 ten thousand pixels. If the field angle of the wide-angle camera is 5 times of the field angle of the telephoto camera, the number of pixels corresponding to the telephoto image in the unit field angle is 5 times of the number of pixels corresponding to the wide-angle image in the unit field angle. That is, the resolution of the telephoto image in the unit angle of view is 800/(1200/5) × 3.33 times the resolution of the wide-angle image. That is, the resolution of the tele image is greater than that of the wide image within the unit field angle, and the tele image is clearer and the details are clearer than the wide image.

Because the resolution of the tele image in the unit field angle is greater than that of the wide image, the definition of the target image generated by the mobile phone according to the spliced image of the multi-frame target tele image is higher than that of the wide image, the details are clearer, and the shooting effect is better. And the mobile phone can obtain a high-definition target image with a large field angle by splicing the multiple frames of target long-focus images.

It should be further noted that the tele image in the foregoing embodiment is a single frame image acquired by the tele camera. In other embodiments, the tele image involved in the foregoing embodiments may also be a frame of image with better quality generated after registration and fusion of multiple frames (e.g., 2 frames or 3 frames, etc.) of tele images acquired by a tele camera.

In some embodiments of the present application, the target image obtained by stitching according to the target tele image, which is stored in the gallery by the mobile phone, may be distinguished from other images and is specifically identified, so as to facilitate a user to intuitively know the type of image. For example, referring to (a) in fig. 18, a text label 1801 of "cp" is displayed on the target image obtained by the mobile phone. For another example, referring to fig. 18 (b), a specific symbol mark 1802 is displayed on the target image obtained by the mobile phone.

Scheme without guide box shown:

the photographing method described in the above embodiment may be referred to as a scheme of displaying the guide frame. Other embodiments of the present application provide another shooting method, and unlike the above embodiments, the mobile phone does not display a guide frame on the preview interface and the shooting interface.

The following mainly describes the differences from the above-described scheme of displaying the guide frame, and details of the same will not be described. As shown in fig. 19, the photographing method may include:

1900. the mobile phone starts a photographing function.

In some embodiments, after the mobile phone starts the photographing function, the photographing can be performed through the scheme that the guide frame is not displayed, which is provided by the embodiments of the present application.

In other embodiments, after the mobile phone starts the photographing function and enters the target photographing mode, photographing may be performed by the scheme of not displaying the guide frame provided by the embodiment of the present application. For example, the target photographing mode is the wide view mode described above.

For other relevant descriptions about step 1900, reference may be made to the description in step 200 above, and further description is omitted here.

1901. The mobile phone displays the wide-angle image and the tele frame on the preview interface.

In the scheme of not displaying the guide frame, the mobile phone displays the wide-angle image on the preview interface. Different from the scheme for displaying the guide frame, in the scheme for not displaying the guide frame, the mobile phone displays the tele frame on the preview interface so as to facilitate a user to know the real-time shooting range of the tele camera; and the mobile phone does not display the guide frame on the preview interface. And, in the scheme of not displaying the guide frame, the mobile phone is on the preview interface. Illustratively, referring to fig. 20 (a), the preview interface includes a wide image and a tele frame.

For other relevant descriptions in step 1901, reference may be made to the description in step 201 above, and details are not repeated here.

1902. After the mobile phone detects the photographing operation of the user, the wide-angle image and the telephoto frame superposed on the wide-angle image are displayed on the photographing interface.

Unlike the above-described scheme for displaying the guide frame, in step 1902, after the mobile phone detects the photographing operation of the user, the mobile phone does not display the guide frame on the photographing interface, and displays the wide-angle image and the tele frame on the photographing interface.

For other relevant descriptions in step 1902, reference may be made to the description in step 202 above, and further description is omitted here.

1903. And the mobile phone generates a spliced image according to the acquired target tele image and displays a spliced image thumbnail on a shooting interface.

In the scheme without displaying the guide frame, when the user wants to shoot the target image, the composition can be performed by referring to the wide-angle image mobile phone or the tele camera on the preview interface, so that the shooting range of the tele camera and the tele frame are located at the starting position of the area that the user wants to shoot. The user may then trigger a photographing operation. After the mobile phone detects the photographing operation of the user, the configuration parameters of the tele camera may be determined according to the image block of the reference wide-angle image corresponding to the photographing range of the tele camera, for example, the configuration parameters may include configuration parameters of AE, AWB, DRC, or the like. The mobile phone acquires a tele image through the tele camera according to the configuration parameters such as AE, AWB or DRC, and obtains a first frame of target tele image, namely a target tele image 1. For the process of determining the configuration parameters such as AE, AWB, DRC, etc. by the mobile phone, reference may be made to the relevant description in the above scheme of displaying the guide frame, and details are not repeated here.

It should be noted that, unlike the case where the guide frame is displayed, the target tele image is matched with the grid, and in the case where the guide frame is not displayed, the target tele image is not matched with the grid of the guide frame because the guide frame is not displayed on the photographing interface. For example, when a frame long-focus image acquired by the mobile phone matches the target long-focus image 1, the frame long-focus image may be the target long-focus image 2. Wherein the matching of the target tele image 2 with the target tele image 1 comprises: a deviation between the abscissa range of the target tele image 2 and the abscissa range of the target tele image 1 is less than or equal to a preset threshold 1, or a deviation between the ordinate range of the target tele image 2 and the ordinate range of the target tele image 1 is less than or equal to a preset threshold 2. That is, the target tele image 2 and the target tele image 1 are arranged substantially side-by-side or substantially side-by-side.

Matching the target tele image 2 with the target tele image 1 further comprises: the overlapping area between the target tele image 2 and the target tele image 1 is greater than or equal to a preset value 8, or the gap between the target tele image 2 and the target tele image 1 is less than or equal to a preset value 9, and so on. For example, when the mobile phone determines that the similarity between the acquired matching content on the tele image and the reference wide image and the matching content of the target tele image 1 on the reference wide image is greater than or equal to the preset value 8, it determines to obtain the target tele image 2 matched with the target tele image 1.

In the scheme of not displaying the guide frame, since the guide frame is not displayed on the shooting interface, the user cannot move the mobile phone or the telephoto camera according to the instruction of the guide frame. In this case, the user can move the mobile phone or directly move the telephoto camera according to the position of the obtained image content of the target telephoto image 1 with respect to the wide-angle image of the global scope, and the need, habit, or intention of the user. In the moving process of the tele camera, the mobile phone can adjust the configuration parameters of the tele camera in real time according to the image block of the reference wide-angle image corresponding to the shooting range of the tele camera, so as to acquire the target tele image according to the configuration parameters, and guide the target tele image to carry out the configuration of AE, AWB, DRC and the like according to the wide-angle image.

In some embodiments, the mobile phone may perform stitching with the previously obtained stitched image after obtaining a new target tele image each time in the photographing process, so as to generate a new stitched image. For example, the mobile phone may stitch the target tele image 1 and the target tele image 2 to obtain a stitched image. Subsequently, after the long-focus image in a certain frame is matched with the obtained target long-focus image, the mobile phone obtains a new target long-focus image, and the new target long-focus image and the spliced image obtained before can be spliced to obtain a new spliced image. In other embodiments, after the shooting is finished, the mobile phone generates the stitched image according to the target tele image. The embodiment of the application does not limit the splicing time of the spliced images. The image stitching process may refer to the description in the above scheme for displaying the guide frame, and is not described herein again.

In some embodiments of the present application, similar to the scheme of displaying the guide frame, after obtaining the target tele image 1, the mobile phone may display a thumbnail of the target tele image 1 on the shooting interface. For example, a shooting interface on which a thumbnail of the target tele image 1 is displayed may be referred to as (b) in fig. 20. In the shooting process, the mobile phone can display the spliced image thumbnail on the shooting interface so as to prompt the current real-time shooting progress to the user. For example, the shooting interface on which the stitched image thumbnail is displayed may be referred to as (c) - (d) in fig. 20. In other embodiments, the mobile phone displays the splicing frame on the shooting interface during the shooting process. In other embodiments, to avoid the situation that the size of the stitched image thumbnail is small and the user cannot conveniently view the image details, the mobile phone may automatically enlarge and display the thumbnail of the target tele image 1 and the stitched image thumbnail on the shooting interface. For example, a shooting interface for enlarging a thumbnail of the display target tele image 1 may be referred to as (a) in fig. 21, and a shooting interface for enlarging a thumbnail of the display stitched image may be referred to as (b) - (c) in fig. 21. In other embodiments, the mobile phone displays the target area image corresponding to the guide frame on the wide-angle image on the shooting interface, but does not display the complete wide-angle image. The ratio of the size of the target area image to the size of the guide frame is r, and r is more than or equal to 1. The target area image may be obtained by cropping and enlarging the complete wide-angle image.

1904. And after the mobile phone determines that the shooting is finished, generating a target image according to the spliced image.

Similar to step 204, in step 1904, after the shooting is finished, before the target image is generated according to the stitched image, the mobile phone may further perform processing such as removing ghost, enhancing a dynamic range, or filling a hole on the stitched image to improve the quality of the stitched image, so as to generate the target image according to the processed stitched image and improve the quality of the target image, which is not described herein again.

In the scheme of not displaying the guide frame, the mobile phone detects that the photographing operation of the user is stopped and then finishes photographing. Or, the mobile phone may preset a maximum shooting frame number, and the mobile phone automatically ends shooting after acquiring the target tele image with the preset frame number. And after finishing shooting, the mobile phone generates a target image according to the spliced image of the target tele image.

In some embodiments, if the edges of the stitched image are not aligned, the mobile phone may minimally crop the edges of the stitched image to obtain a stitched image in a regular shape such as a rectangle or a square, so as to generate the target image according to the stitched image. For example, a schematic diagram of a stitched image thumbnail obtained by the mobile phone according to the target telephoto camera may be shown in fig. 22 (a), and a schematic diagram of a new stitched image thumbnail obtained after the mobile phone performs cropping may be shown in fig. 22 (b).

In other embodiments, if the edges of the stitched image are not aligned, for the blank portion, the mobile phone may perform hole filling according to the content of the corresponding position on the reference wide-angle image to obtain a stitched image in a regular shape such as a rectangle or a square, so as to generate the target image according to the stitched image. For example, a schematic diagram of a stitched image thumbnail obtained by the mobile phone according to the target telephoto camera may be shown in (a) in fig. 23A, and a schematic diagram of a new stitched image thumbnail obtained after the mobile phone performs hole filling according to the reference wide-angle image may be shown in (b) in fig. 23A.

In other embodiments, the mobile phone generates the target image according to the stitched image, regardless of whether the edges of the stitched image are aligned, and the size of the target image is consistent with the size of the stitched image.

In other embodiments, in the scenario where the guide frame is not displayed, the size of the target image obtained by the mobile phone may not be the size desired by the user. Thus, in one possible solution, the user may also edit the generated target image in the gallery to obtain a target image of a desired size.

In another possible technical solution, the user may also edit the stitched image to obtain a stitched image with an ideal size, so as to generate a target image with an ideal size. For example, referring to (a) in fig. 23B, after the shooting is finished, the mobile phone may prompt the user on the shooting interface to set the range of the target image on the stitched image. After detecting the operation of setting the dashed box 2300 by the user and clicking the determination control, the mobile phone cuts the stitched image according to the dashed box 2300, thereby generating the target image within the user indication range as shown in (B) in fig. 23B.

In the scheme of not displaying the guide frame, because the resolution of the tele image is greater than that of the wide image within the unit field angle, the target image generated by the mobile phone according to the spliced image of the multi-frame target tele image has higher definition, clearer details and better shooting effect than the wide image. And the mobile phone can obtain a high-definition target image with a large field angle by splicing the multiple frames of target long-focus images.

In addition, in the scheme of not displaying the guide frame, the user can move the mobile phone or the tele-camera to acquire the tele image according to the requirement of the user, so that the target tele image at the position desired by the user is obtained, and the spliced image with any size or shape desired by the user is obtained according to the target tele image at the position desired by the user. The mobile phone can obtain target images with corresponding sizes or shapes according to the spliced images with any specifications, and users can conveniently shoot and obtain target images of wide frames, square frames or panoramic frames with various sizes.

Hybrid zoom scheme:

other embodiments of the present application further provide a shooting method, which can splice the long-focus images with a small field angle to obtain a spliced image with a large field angle and a clear field angle, and then cut the spliced image to obtain clear target images corresponding to different target zoom magnifications. In the scheme, the mobile phone does not need to carry out image amplification through digital zooming, so that the high resolution of the telephoto camera and the high definition of the telephoto image can be reserved, and the zooming effect of optical zooming is realized. This scheme combines the stitching and cropping process of the tele image, which may also be referred to as a hybrid zoom scheme.

The following mainly describes the differences from the above-described scheme of displaying the guide frame, and details of the same will not be described. As shown in fig. 24, the compound zoom scheme may include:

2400. the mobile phone starts a photographing function.

In some embodiments, after the mobile phone starts the photographing function, the mobile phone can perform processing by using the hybrid zoom method provided in the embodiments of the present application.

In other embodiments, after the mobile phone starts the photographing function and enters the target photographing mode, the processing may be performed by the hybrid zoom method provided in the embodiments of the present application.

In the following embodiments, the target photographing mode is described as an example of the compound zoom mode.

For other relevant descriptions about step 2400, refer to the description in step 200 above, and are not described here again.

2401. The mobile phone displays the wide-angle image on the preview interface.

Unlike the above scheme for displaying the guide frame, in the step 2401 of the hybrid zoom mode, the mobile phone may not display the guide frame on the preview interface first, and after the target zoom magnification is obtained subsequently, display the guide frame corresponding to the target zoom magnification on the preview interface.

In some embodiments, the handset does not display the tele frame on the preview interface. In other embodiments, the handset continues to display the tele frame on the preview interface. In other embodiments, the mobile phone does not display the tele frame on the preview interface first, and displays the tele frame on the preview interface after acquiring the target zoom magnification subsequently.

2402. And the mobile phone acquires the target zoom magnification.

The target zooming magnification is larger than that of the wide-angle camera (namely, the first camera) and smaller than that of the long-focus camera (namely, the second camera). The target zoom magnification is a zoom magnification of a target image that can be obtained based on a stitched image of the target tele image, and is a zoom magnification of a final image that the user wants to capture and obtain, and is not a zoom magnification of a wide image that is a background image. The zoom magnification of the wide-angle image as the background image does not change before and after the user sets the target zoom magnification.

In some embodiments, the target zoom magnification is a zoom magnification set by a user. In a possible implementation manner, the mobile phone may prompt the user to set the target zoom magnification in a manner of displaying information or playing voice. Illustratively, referring to fig. 25 (a) - (c), the handset prompts the user on the preview interface: in this mode, you can set a zoom magnification to capture a high-definition image corresponding to the zoom magnification.

In the embodiment of the present application, the manner in which the user sets the target zoom magnification may be various. For example, referring to fig. 25 (a), a preview interface of the mobile phone includes a plurality of selectable zoom magnification controls, such as a 1.5X control, a 2X control, a 2.5X control, a 3X control, a 3.5X control, a 4X control, and a 4.5X control, and the mobile phone determines a corresponding target zoom magnification according to the zoom magnification control selected by the user.

For another example, after detecting that the user clicks the zoom magnification setting control on the preview interface, the mobile phone may display a setting interface, and the user may set the target zoom magnification based on the setting page.

As another example, as shown in (b) in fig. 25, after detecting a zoom/zoom operation of a user on a preview interface, the mobile phone acquires a corresponding target zoom magnification after the zoom/zoom operation.

Further exemplarily, as shown in (c) of fig. 25, after the mobile phone detects a drag operation of the user on the zoom magnification adjustment lever, the corresponding target zoom magnification after the drag operation is acquired.

In another example, after the mobile phone detects a voice instruction for setting the zoom magnification by the user, the target zoom magnification set by the voice of the user is obtained.

It should be noted that, in the case that the target zoom magnification is greater than the zoom magnification of the wide-angle camera and smaller than the zoom magnification of the telephoto camera, the mobile phone may perform processing by using the hybrid zoom method provided in the embodiment of the present application. If the target zoom magnification is less than or equal to the zoom magnification of the wide-angle camera, the mobile phone can directly generate a target image corresponding to the target zoom magnification according to the image collected by the wide-angle camera or the ultra-wide-angle camera without adopting the mixed zoom scheme. If the target zoom magnification is greater than or equal to the zoom magnification of the long-angle camera, the mobile phone may also generate a target image corresponding to the target zoom magnification directly according to the image acquired by the long-focus camera or the ultra-long-focus camera without adopting the hybrid zoom scheme.

In other embodiments, the target zoom magnification may also be a default zoom magnification (e.g., default zoom magnification of a wide-angle camera) or a last-used zoom magnification. The mobile phone can also modify the target zoom magnification according to the operation of the user.

In some embodiments of the present application, the mobile phone displays a guide frame corresponding to the target zoom magnification in an overlapping manner on the wide-angle image of the preview interface. Unlike the guide frame displayed on the mobile phone in the above-described scheme of displaying the guide frame, the guide frame in the hybrid zoom scheme corresponds to the target zoom magnification and is a guide frame of the minimum specification of the image area size corresponding to the field angle including the target zoom magnification.

In the embodiment of the present application, the image area corresponding to the field angle of the target zoom magnification is located at the middle position of the wide-angle image by default. For example, the zoom magnification of the wide-angle camera is 1X, the zoom magnification of the tele camera is 5X, and the guide frame may include 5 × 5 meshes at most. If the target zoom magnification is 2.5X, the field angle of the target zoom magnification corresponds to the image region 2601; a guide frame corresponding to the target zoom magnification of 2.5X, which can be referred to as a broken line guide frame 2602 shown in (a) in fig. 26A, includes 3 × 3 meshes, and the size of the mesh corresponds to the field angle of the telephoto camera; the size of the image area corresponding to the field angle of the target zoom magnification is 2.5 times the size of the grid. For example, when the target zoom magnification is 2.5X, the schematic diagram of the preview interface may be as shown in (c) of fig. 26A, including a guide frame 2604 corresponding to the target zoom magnification. If the target zoom magnification is 3X, the image area corresponding to the target zoom magnification 3X matches the size of the guide frame, and the guide frame corresponding to the target zoom magnification 3X can be referred to the broken-line guide frame 2603 shown in (b) of fig. 26A, where the guide frame 2603 includes 3 × 3 meshes, and the image size corresponding to the field angle of the target zoom magnification is 3 times the mesh size.

In some embodiments, in the preview state, the mobile phone may prompt the user in a manner of displaying information, voice broadcasting, and the like, and please shoot according to the guide frame in the shooting process, so as to cut the generated spliced image and obtain the target image according with the target zoom magnification. For example, referring to (d) in fig. 26A, the mobile phone may prompt the user on the preview interface by displaying information: please shoot by pressing the dashed guide frame in the shooting process to obtain a stitched image, which is used to crop the image corresponding to the zoom magnification pointed by you!

In some embodiments, the mobile phone may further display, in an overlapping manner, an object frame corresponding to the target zoom magnification on the wide-angle image of the preview interface. The position and size of the target frame are matched with the position and size of the image area corresponding to the field angle of the target zoom magnification. After the mobile phone obtains the target zoom magnification, a target frame corresponding to the target zoom magnification can be displayed on the preview interface, so that the position and size of the image area size corresponding to the field angle of the current target zoom magnification are prompted to a user, and the user can conveniently know the size of the target image which can be obtained according to the current target zoom magnification. The guide frame is a guide frame including the minimum specification of the target frame. The size of the guide frame is larger than or equal to the size of the target frame, that is, larger than or equal to the image size corresponding to the field angle of the target zoom magnification.

In some embodiments of the present application, the target frame is located at an intermediate position of the wide-angle image by default. For example, the zoom magnification of the wide-angle camera is 1X, the zoom magnification of the tele camera is 5X, and the guide frame may include 5 × 5 meshes at most. If the target zoom magnification is 2.5X, a target frame corresponding to the target zoom magnification of 2.5X may be referred to as a solid-line rectangular frame 2601 shown in (b) in fig. 26A. If the target zoom magnification is 3X, the size of the target frame corresponding to the target zoom magnification of 3X is the same as the size of the border of the guide frame. For example, when the target zoom magnification is 2.5X, the schematic diagram of the preview interface may be as shown in fig. 26B, including a guide frame 2604 corresponding to the target zoom magnification and a target frame 2605 corresponding to the target zoom magnification.

In an embodiment of the present application, the target frame on the preview interface is located at the middle of the wide-angle image by default. In some technical solutions, the user may also move the position of the target frame, and the position of the guide frame also changes correspondingly with the change of the position of the target frame. For example, on the preview interface shown in (a) in fig. 27, if the mobile phone detects an operation of dragging the target frame to the right by the user, both the target frame and the guide frame move to the right as shown in (b) in fig. 27. Alternatively, the user may move the position of the guide frame and the position of the target frame changes accordingly.

In some other embodiments, after the target zoom magnification is obtained, the mobile phone displays a target frame on the preview interface for a short time, so as to prompt the user of the size of the image size corresponding to the field angle of the current target zoom magnification, which is convenient for the user to determine whether the current target zoom magnification is appropriate, and then stops displaying the target frame. In some embodiments, the cell phone may record the position and size of the target frame. After the mobile phone determines that the shooting is finished, the spliced image can be cut according to the recorded position and size of the target frame, so that the target image is obtained.

2403. After the mobile phone detects the photographing operation of the user, the wide-angle image and a guide frame superposed on the wide-angle image are displayed on a photographing interface, and the guide frame corresponds to the target zooming magnification.

On the shooting interface, the mobile phone can display a wide-angle image and a guide frame superposed on the wide-angle image.

In some embodiments, the cell phone may also display a tele frame on the capture interface.

Optionally, the mobile phone can also display the target frame on the shooting interface.

In some embodiments, in the photographing process, the mobile phone may prompt the user temporarily or continuously through modes such as information display, voice broadcast, and the like, and please shoot according to the guide frame in the photographing process, so as to cut the spliced image obtained by shooting, and obtain a target image meeting the target zoom magnification.

In some embodiments, as described above, in a compound zoom scheme, the handset displays the captured full wide-angle image on the preview interface and the capture interface. In other embodiments, similar to the above scheme for displaying the guide frame, the mobile phone may replace the complete wide-angle image displayed on the preview interface and the shooting interface with the target area image corresponding to the guide frame on the wide-angle image.

2404. And the mobile phone generates a spliced image according to the acquired target tele image and displays a spliced image thumbnail on a shooting interface.

For the related description of step 2404, reference may be made to the description in step 203, which is not described herein again. Illustratively, referring to (a) in fig. 28, a thumbnail of the target tele image 1, as well as the wide image, the guide frame, and the tele frame, is displayed on the photographing interface. Referring to (b) - (c) in fig. 28, a stitched image thumbnail, as well as a wide image, a guide frame, and a tele frame, is displayed on the shooting interface.

2405. And after the mobile phone determines that the shooting is finished, cutting the spliced image to generate a target image.

Similar to step 204, in step 2405, after the shooting is finished, before the target image is generated according to the stitched image, the mobile phone may further perform processing such as removing ghost, enhancing a dynamic range, or filling a hole on the stitched image to improve the quality of the stitched image, so that the target image is generated according to the processed stitched image to improve the quality of the target image, which is not repeated herein.

There are various conditions for the mobile phone to determine the end of shooting. For example, in the first case, after the shooting of all the meshes in the guide frame is completed, the mobile phone automatically ends the shooting, and cuts the stitched image to the size of the target frame to generate the target image. Under the condition that the size of the guide frame is the same as that of the target frame, the mobile phone does not need to cut the spliced image, and can directly generate the target image according to the spliced image. For example, in the case shown in (c) in fig. 28, the target image generated by the mobile phone may be referred to as (d) in fig. 28.

In the second case, before the grid in the guide frame does not finish shooting, the mobile phone determines that shooting is finished after detecting the shooting stopping operation of the user. In this case, the mobile phone cuts and enlarges the reference wide-angle image or the wide-angle image, thereby generating an image of the target zoom magnification by digital zooming. If the shooting interface does not display the target frame, after the mobile phone determines that shooting is finished, the position and the size of the target frame can be determined according to the target zoom magnification, and therefore the reference wide-angle image or the wide-angle image is cut according to the target frame to obtain the target image.

When the shooting interface does not display the target frame, the mobile phone can determine the position and the size of an image area corresponding to the field angle of the target zoom magnification after the shooting is determined to be finished, and accordingly the spliced image is cut according to the position and the size of the image area to obtain the target image. Or, under the condition that the target frame is not displayed on the shooting interface, after the shooting is determined to be finished, the mobile phone can determine the position and the size of the target frame according to the target zoom magnification, so that the spliced image is cut according to the position and the size of the target frame to obtain the target image. Or, under the condition that the target frame is not displayed on the shooting interface, after the mobile phone acquires the target zoom magnification, the position and the size of the target frame can be determined according to the target zoom magnification, and the position and the size of the target frame are recorded, so that after shooting is finished, the spliced image is cut according to the position and the size of the target frame to obtain the target image. The position and size of the image area corresponding to the field angle of the target zoom magnification are the position and size of the target frame.

Compared with a target image corresponding to a target zooming magnification realized through a wide-angle image and digital zooming, the scheme corresponding to the first case obtains the target image of a large field angle corresponding to the target zooming magnification through splicing and cutting the long-focus image of a small field angle with high resolution and clearness, the resolution and the clarity of the target image are also higher, the image quality is better, and the zooming effect of optical zooming can be realized.

Another zoom scheme exists at present, in which images acquired by two cameras with different FOV sizes are fused and then cropped to realize zooming. Wherein, the image with small FOV is located in the middle area, which can improve the definition in the middle of the target image frame, but the definition at the edge of the frame is poor. According to the scheme corresponding to the first condition in the embodiment of the application, the definition of the whole target image is high, and the zooming effect of optical zooming is achieved.

In addition, in the shooting method described in each of the above embodiments, the tele image acquired in real time is not displayed in the tele frame, and is only used to indicate the real-time shooting range corresponding to the tele camera. In other embodiments, the image of the tele image acquired by the tele camera in real time after the downsampling processing is displayed in the tele frame, so as to present the tele image acquired by the tele camera in real time for the user. The position of the tele image is opposite to the position of the same content on the wide image. In other embodiments, the tele image acquired by the tele camera in real time is displayed within the tele frame. In other embodiments, the tele-frame is located at a predetermined position on the interface, such as at the lower left or lower right corner of the interface. Illustratively, referring to FIG. 29, the tele frame 2900 is located in the lower left corner of the interface.

The above description is given by taking the first camera as the wide-angle camera and the second camera as the telephoto camera as examples, and when the first camera and/or the second camera are/is other cameras, the target image can still be obtained by shooting with the shooting method provided by the above embodiments, which is not described in detail in the embodiments of the present application.

The above description is given by taking the electronic device as a mobile phone as an example, and when the electronic device is a tablet computer or other devices such as a smart watch, the target image can still be obtained by shooting with the shooting method provided by the above embodiment, which is not described in detail in the embodiments of the present application.

It will be appreciated that in order to implement the above-described functions, the electronic device comprises corresponding hardware and/or software modules for performing the respective functions. The present application is capable of being implemented in hardware or a combination of hardware and computer software in conjunction with the exemplary algorithm steps described in connection with the embodiments disclosed herein. Whether a function is performed as hardware or computer software drives hardware depends upon the particular application and design constraints imposed on the solution. Skilled artisans may implement the described functionality in varying ways for each particular application, with the embodiment described in connection with the particular application, but such implementation decisions should not be interpreted as causing a departure from the scope of the present application.

In this embodiment, the electronic device may be divided into functional modules according to the above method example, for example, each functional module may be divided corresponding to each function, or two or more functions may be integrated into one processing module. The integrated module may be implemented in the form of hardware. It should be noted that the division of the modules in this embodiment is schematic, and is only a logic function division, and there may be another division manner in actual implementation.

Embodiments of the present application also provide an electronic device including one or more processors and one or more memories. The one or more memories are coupled to the one or more processors for storing computer program code comprising computer instructions which, when executed by the one or more processors, cause the electronic device to perform the associated method steps described above to implement the photographing method in the above embodiments.

An embodiment of the present application further provides an electronic device, as shown in fig. 30, including: a display screen 3001, one or more processors 3002, a plurality of cameras 3003, memory 3004, and one or more computer programs 3005, which may be connected via one or more communication buses 3006. Wherein the one or more computer programs 3005 are stored in the memory 3004 and configured to be executed by the one or more processors 3002, the one or more computer programs 3005 comprising instructions that may be used to perform the steps in the above embodiments. All relevant contents of the steps related to the above method embodiment may be referred to the functional description of the corresponding entity device, and are not described herein again.

For example, the processor 3002 may be specifically the processor 110 shown in fig. 1, the memory 3004 may be specifically the internal memory 121 shown in fig. 1, the camera 3003 may be specifically the camera 193 shown in fig. 1, and the display screen 3001 may be specifically the display screen 194 shown in fig. 1.

Embodiments of the present application further provide a computer-readable storage medium, in which computer instructions are stored, and when the computer instructions are run on an electronic device, the electronic device is caused to execute the above related method steps to implement the shooting method in the above embodiments.

Embodiments of the present application further provide a computer program product, which when running on a computer, causes the computer to execute the above related steps to implement the shooting method performed by the electronic device in the above embodiments.

In addition, embodiments of the present application also provide an apparatus, which may be specifically a chip, a component or a module, and may include a processor and a memory connected to each other; the memory is used for storing computer execution instructions, and when the device runs, the processor can execute the computer execution instructions stored in the memory, so that the chip can execute the shooting method executed by the electronic equipment in the above-mentioned method embodiments.

The electronic device, the computer-readable storage medium, the computer program product, or the chip provided in this embodiment are all configured to execute the corresponding method provided above, so that the beneficial effects achieved by the electronic device, the computer-readable storage medium, the computer program product, or the chip may refer to the beneficial effects in the corresponding method provided above, and are not described herein again.

Through the description of the above embodiments, those skilled in the art will understand that, for convenience and simplicity of description, only the division of the above functional modules is used as an example, and in practical applications, the above function distribution may be completed by different functional modules as needed, that is, the internal structure of the device may be divided into different functional modules to complete all or part of the above described functions.

In the several embodiments provided in the present application, it should be understood that the disclosed apparatus and method may be implemented in other ways. For example, the above-described device embodiments are merely illustrative, and for example, the division of the modules or units is only one logical functional division, and there may be other divisions when actually implemented, for example, a plurality of units or components may be combined or may be integrated into another device, or some features may be omitted, or not executed. In addition, the shown or discussed mutual coupling or direct coupling or communication connection may be an indirect coupling or communication connection through some interfaces, devices or units, and may be in an electrical, mechanical or other form.

The units described as separate parts may or may not be physically separate, and parts displayed as units may be one physical unit or a plurality of physical units, that is, may be located in one place, or may be distributed in a plurality of different places. Some or all of the units can be selected according to actual needs to achieve the purpose of the solution of the embodiment.

In addition, functional units in the embodiments of the present application may be integrated into one processing unit, or each unit may exist alone physically, or two or more units are integrated into one unit. The integrated unit can be realized in a form of hardware, and can also be realized in a form of a software functional unit.

The integrated unit, if implemented in the form of a software functional unit and sold or used as a stand-alone product, may be stored in a readable storage medium. Based on such understanding, the technical solutions of the embodiments of the present application may be essentially or partially contributed to by the prior art, or all or part of the technical solutions may be embodied in the form of a software product, where the software product is stored in a storage medium and includes several instructions to enable a device (which may be a single chip, a chip, or the like) or a processor (processor) to execute all or part of the steps of the methods described in the embodiments of the present application. And the aforementioned storage medium includes: various media capable of storing program codes, such as a usb disk, a removable hard disk, a Read Only Memory (ROM), a Random Access Memory (RAM), a magnetic disk, or an optical disk.

The above description is only for the specific embodiments of the present application, but the scope of the present application is not limited thereto, and any person skilled in the art can easily conceive of the changes or substitutions within the technical scope of the present application, and shall be covered by the scope of the present application. Therefore, the protection scope of the present application shall be subject to the protection scope of the claims.

Claims

1. A shooting method is applied to electronic equipment, the electronic equipment comprises a first camera and a second camera, and the equivalent focal length of the second camera is larger than that of the first camera, and the shooting method is characterized by comprising the following steps:

starting a photographing function;

displaying the image and the guide frame acquired by the first camera on a preview interface;

after the photographing operation of a user is detected, displaying a first image and the guide frame superposed on the first image on a photographing interface; the first image is obtained according to the image collected by the first camera, the guide frame comprises a plurality of grids, and the single grid corresponds to the size of the second camera view angle;

displaying splicing information on the shooting interface, wherein the splicing information is used for indicating the shooting progress, the splicing information corresponds to multi-frame target shooting images matched with a plurality of grids in the guide frame, and the target shooting images are acquired through the second camera;

generating a spliced image according to the multi-frame target shooting images;

and after shooting is finished, generating a target image according to the spliced image.

2. The method according to claim 1, wherein the first image is a first frame image collected by the first camera after a photographing operation of a user is detected;

or the first image is an image obtained by fusing Q frames of images collected by the first camera after the photographing operation of the user is detected, wherein Q is an integer larger than 1;

or the first image is an image acquired by the first camera in the photographing process after the photographing operation of the user is detected.

3. The method according to claim 1 or 2, wherein the overlapping proportion of the same content of the first image in the target captured image and the matched grid is greater than or equal to a first preset value;

or in the target shooting image and the matched grid, the similarity of the histogram of the characteristic parameters of the first image is greater than or equal to a second preset value;

or in the target shooting image and the matched grid, the similarity of the first image in the same transformation domain is greater than or equal to a third preset value;

or in the target shooting image and the matched grid, the feature matching degree of the first image is greater than or equal to a fourth preset value.

4. The method according to any one of claims 1-3, further comprising:

determining configuration parameters of the second camera for acquiring the target shot image according to the image blocks corresponding to the grids on the reference image;

and acquiring the target shooting image matched with the grid according to the configuration parameters.

5. The method of claim 4, wherein the configuration parameters comprise at least one or more of: automatic exposure AE configuration parameters, automatic white balance AWB configuration parameters, or dynamic range correction DRC configuration parameters.

6. The method according to any one of claims 1 to 5, wherein the plurality of target captured images include a first target captured image and a second target captured image, and wherein generating the stitched image from the plurality of target captured images comprises:

and generating a spliced image after performing first image processing on the first target shooting image and the second target shooting image, wherein the first image processing comprises image registration.

7. The method of claim 6, wherein image registering the first target captured image and the second target captured image comprises:

respectively extracting features of the first target shooting image and the second target shooting image according to a preset image registration algorithm;

and registering the first target shooting image and the second target shooting image according to the obtained feature matching pair.

8. The method of claim 7, further comprising:

and correcting the registered image according to the reference image.

9. The method of claim 6, wherein image registering the first target captured image and the second target captured image comprises:

respectively extracting features of the first target shooting image and the reference image, and registering the first target shooting image and the reference image according to the obtained feature matching pair;

and respectively extracting features of the second target shooting image and the reference image, and registering the second target shooting image and the reference image according to the obtained feature matching pair.

10. The method of claim 9, further comprising:

respectively extracting features of the first target shot image and the second target shot image, and registering the first target shot image and the second target shot image according to the obtained feature matching pair;

calculating a homography matrix according to the feature matching pairs of the overlapping regions of the first target photographed image, the second target photographed image and the reference image;

and deforming the second target shooting image according to the homography matrix.

11. The method of claim 6, wherein image registering the first target captured image and the second target captured image comprises:

attaching the first target shooting image to a coordinate position with the same image content on a reference image;

attaching the second target shot image to a coordinate position with the same image content on the reference image;

and if a hole exists between the first target shooting image and the second target shooting image, filling the hole according to the image content of the position corresponding to the hole on the reference image.

12. The method according to any one of claims 6 to 11, wherein the first image processing further comprises a de-ghosting process, and wherein the generating of the stitched image after the first image processing of the first target captured image and the second target captured image comprises:

reserving a ghost-free image of a moving object on the first target shooting image;

deleting the image of the moving object on the second target shot image;

performing image registration on the first target shooting image and the second target shooting image to generate a spliced image;

and filling the holes in the spliced image according to the reference image.

13. The method according to any one of claims 6 to 11, wherein the first image processing further comprises a de-ghosting process, and wherein the generating of the stitched image after the first image processing of the first target captured image and the second target captured image comprises:

performing image registration on the first target captured image and the second target captured image;

keeping a ghost-free image of a moving object on the registered image, wherein the ghost-free image is from the first target shooting image, the second target shooting image or the first image;

deleting images of the moving object in other areas on the registered images;

and filling the hole caused by deletion according to the reference image to generate a spliced image.

14. The method according to any one of claims 6 to 11, wherein the first image processing further comprises a de-ghosting process, and wherein the generating of the stitched image after the first image processing of the first target captured image and the second target captured image comprises:

deleting images of a moving object on the first target shot image and the second target shot image;

judging the motion track of a moving object according to the multi-frame images collected by the first camera so as to delete the images of the moving object on the first image and obtain a first image without ghost;

attaching the first target photographed image and the second target photographed image, from which the image of the moving object is deleted, to the first image without the ghost;

15. The method according to any one of claims 6 to 14, wherein the first image processing further comprises a dodging process and an image fusion process.

16. The method according to any one of claims 1-15, wherein the generating a target image from the stitched image comprises:

and after second image processing is carried out on the spliced image, generating the target image, wherein the second image processing at least comprises one or more of the following steps: ghost image removing, dynamic range enhancement, high-frequency component fusion, light and color homogenizing treatment or cavity filling.

17. The method of claim 16, wherein high frequency component fusing the stitched image comprises:

and extracting a high-frequency component image of the reference image, and fusing the high-frequency component image and the spliced image.

18. The method of claim 16, wherein homogenizing the stitched image comprises:

synthesizing images acquired by a plurality of frames of the first camera in a high dynamic range;

and carrying out light and color evening processing on the spliced image according to the obtained high dynamic range information.

19. The method of claim 16, wherein hole filling the stitched image comprises:

performing super-resolution image synthesis on the images acquired by the first cameras and the areas corresponding to the holes of the spliced images;

and filling holes in the spliced image according to the obtained image information.

20. The method of claim 16, wherein hole filling the stitched image comprises:

performing image super-resolution processing on an area corresponding to the cavity of the spliced image on the reference image;

21. The method according to any one of claims 4 to 20, wherein the reference image is a first frame image acquired by the first camera after a photographing operation of the user is detected;

or the first image is an image obtained by fusing Q frame images collected by the first camera after the photographing operation of the user is detected, wherein Q is an integer larger than 1.

22. The method of any one of claims 1-21, wherein the determining that the shot is complete comprises:

and after the grids in the guide frame are matched, determining that the shooting is finished.

23. The method of claim 22, wherein generating the target image from the stitched image comprises:

the spliced image is the target image;

or if the edges of the spliced images are not aligned, cutting the spliced images to obtain the target images with aligned edges;

or if the edges of the spliced image are not aligned, filling the edge area of the spliced image according to the first image to obtain the target image with aligned edges.

24. The method of any one of claims 1-21, wherein the determining that the shot is complete comprises:

before the grids in the guide frame are not matched, if the photographing stopping operation of the user is detected, the photographing is determined to be finished;

or if the moving direction of the electronic equipment moves out of the guide frame, determining that the shooting is finished;

or if the deviation range of the moving direction of the electronic equipment and the indication direction of the guide frame is larger than or equal to a fifth preset value, determining that the shooting is finished.

25. The method of claim 24, wherein generating the target image from the stitched image comprises:

generating the target image according to the spliced image corresponding to the matched whole row/column grid;

or generating the target image according to the spliced image corresponding to the matched grid and the image area corresponding to the unmatched grid on the first image.

26. The method of any one of claims 1-25, further comprising:

acquiring a target zooming magnification, wherein the guide frame corresponds to the target zooming magnification, and the target zooming magnification is larger than the zooming magnification of the first camera and smaller than the zooming magnification of the second camera;

the generating of the target image according to the stitched image comprises:

and cutting the spliced image to generate the target image, wherein the target image corresponds to the target zooming magnification.

27. A shooting method is applied to electronic equipment, the electronic equipment comprises a first camera and a second camera, and the equivalent focal length of the second camera is larger than that of the first camera, and the shooting method is characterized by comprising the following steps:

starting a photographing function;

displaying a third image and an image frame on a preview interface, wherein the third image is an image acquired by the first camera, the picture range of the third image in the image frame corresponds to the picture of a second image, and the second image is an image acquired by the second camera;

after the photographing operation of a user is detected, displaying a first image and the image frame on a photographing interface, wherein the first image is obtained according to an image collected by the first camera;

displaying splicing information on the shooting interface, wherein the splicing information is used for indicating the shooting progress, corresponds to multiple frames of target shooting images acquired in the shooting process, and adjacent target shooting images are matched with each other;

generating a spliced image according to the multi-frame target shooting images;

28. The method according to claim 27, wherein for the third target captured image and the fourth target captured image which are matched in the multiple frames of target captured images, the following conditions are satisfied:

the deviation between the abscissa range of the third target photographed image and the abscissa range of the fourth target photographed image is less than or equal to a first preset threshold;

or the deviation between the vertical coordinate range of the third target shooting image and the vertical coordinate range of the fourth target shooting image is smaller than or equal to a second preset threshold value.

29. The method of claim 28, wherein the matching of the third captured image of the object and the fourth captured image of the object further satisfies:

an overlapping area between the third target photographed image and the fourth target photographed image is greater than or equal to a sixth preset value;

or a gap between the third target photographed image and the fourth target photographed image is less than or equal to a seventh preset value.

30. An electronic device, comprising:

the first camera and the second camera are used for collecting images;

a screen for displaying an interface;

one or more processors;

a memory;

and one or more computer programs, wherein the one or more computer programs are stored in the memory, the one or more computer programs comprising instructions which, when executed by the electronic device, cause the electronic device to perform the photographing method of any of claims 1-29.

31. A computer-readable storage medium comprising computer instructions which, when run on a computer, cause the computer to perform the photographing method according to any one of claims 1 to 29.

32. A computer program product, characterized in that it causes a computer to carry out the shooting method according to any one of claims 1-29, when said computer program product is run on the computer.