WO2022022715A1

WO2022022715A1 - Photographing method and device

Info

Publication number: WO2022022715A1
Application number: PCT/CN2021/109836
Authority: WO
Inventors: 吴亮; 敖欢欢; 郭勇; 王妙锋; 王军
Original assignee: 华为技术有限公司
Priority date: 2020-07-30
Filing date: 2021-07-30
Publication date: 2022-02-03

Abstract

Provided are a photographing method and a device, which relate to the technical field of electronics. With reference to an image collected by a first camera with a relatively large angle of view, a second camera with a relatively small angle of view can be used to photograph images, and splice the images to obtain a target image with a relatively large angle of view, wherein the target image has a higher definition, clear details and a better photographing effect. The solution comprises: an electronic device starting a photographing function; displaying, on a preview interface, an image collected by a first camera and a guide box; after a photographing operation of a user is detected, displaying a first image and a guide box on a photographing interface, wherein the first image is obtained according to the image collected by the first camera, the guide box comprises a plurality of grids, and a single grid corresponds to the size of an angle of view of a second camera; displaying splicing information on the photographing interface, wherein the splicing information is used for indicating photographing progress; generating a spliced image according to acquired multiple frames of target photographed images; and after photographing is finished, generating a target image according to the spliced image. The embodiments of the present application are used for photographing images.

Description

A shooting method and equipment

This application requires a Chinese patent application with an application number of 202010754701.X and an application title of "A Shooting Method and Equipment" to be submitted to the State Intellectual Property Office on July 30, 2020, as well as a national knowledge application on November 18, 2020 The Chinese Patent Office, the application number is 202011296539.8, and the application title is "a shooting method and apparatus" for the priority of the Chinese patent application, the entire contents of which are incorporated in this application by reference.

technical field

The embodiments of the present application relate to the field of electronic technologies, and in particular, to a photographing method and device.

Background technique

At present, electronic devices such as mobile phones or watches can use a wide-angle camera with a smaller equivalent focal length to capture a target image with a larger field of view (FOV). However, the clarity of local details on this target image is low. In particular, when an electronic device uses a wide-angle camera to shoot a larger scene or a distant scenery, the user may not be able to clearly see the details on the target image.

SUMMARY OF THE INVENTION

Embodiments of the present application provide a shooting method and device, which can refer to an image collected by a first camera with a larger field of view, use a second camera with a smaller field of view to shoot images, and stitch them to obtain a target image with a larger field of view , and the clarity of the target image is high, the details are clear, and the shooting effect is better.

In order to achieve the above purpose, the embodiment of the present application adopts the following technical solutions:

On the one hand, an embodiment of the present application provides a shooting method, which is applied to an electronic device. The electronic device includes a first camera and a second camera, and the equivalent focal length of the second camera is greater than the equivalent focal length of the first camera. The method includes: the electronic device starts a photographing function. Then, the electronic device displays the image collected by the first camera and the guide frame on the preview interface. After detecting the user's photographing operation, the electronic device displays the first image on the photographing interface, and a guide frame superimposed on the first image; wherein the first image is obtained according to the image collected by the first camera, and the guide frame includes a plurality of Grid, a single grid corresponds to the size of the field of view of the second camera. The electronic device displays splicing information on the shooting interface, the splicing information is used to indicate the shooting progress, and the splicing information corresponds to the multi-frame target shooting images matched with the multiple grids in the guide frame, and the target shooting images are acquired by the second camera. The electronic device generates a stitched image according to the multi-frame target captured images. After the shooting, the electronic device generates the target image according to the stitched image.

In this solution, the electronic device can refer to the first image captured by the first camera with a smaller equivalent focal length and a larger field of view, and use the second camera with a larger equivalent focal length and a smaller field of view to capture an image of the target , and splicing to obtain a target image with a larger field of view, and the target image has high definition, clear details and better shooting effect. In addition, on the shooting interface, the first image is displayed as a background image, and a guide frame can be superimposed on the first image to guide the second camera to move to shoot a target image that matches the grid in the guide frame. The electronic device may also display stitching information on the shooting interface to indicate the current shooting progress in real time for the user.

In a possible implementation manner, the first image is the first frame image collected by the first camera after detecting the user's photographing operation; or, the first image is the first frame image collected by the first camera after detecting the user's photographing operation The obtained image of Q frames after image fusion, where Q is an integer greater than 1; or, the first image is an image acquired by the first camera during the photographing process after detecting the user's photographing operation.

That is to say, during the photographing process, the first image displayed as the background image on the photographing interface may be a fixed frame of image or an image refreshed in real time.

In another possible implementation manner, in the target captured image and the matched grid, the overlap ratio of the same content of the first image is greater than or equal to the first preset value; or, the target captured image and the matched grid , the similarity of the histogram of the feature parameters of the first image is greater than or equal to the second preset value; or, in the target captured image and the matched grid, the similarity of the first image in the same transformation domain is greater than or equal to the third preset value; or, in the target captured image and the matched grid, the feature matching degree of the first image is greater than or equal to the fourth preset value.

That is, the electronic device can determine whether the target captured image matches the grid in various ways.

In another possible implementation manner, the method further includes: the electronic device determines, according to the image blocks corresponding to the grid on the reference image, configuration parameters of the image captured by the second camera to capture the target. The electronic device acquires the captured image of the target matched with the grid according to the configuration parameters.

In this way, the electronic device can determine the configuration parameters when the second camera collects the target captured images corresponding to different grids according to the reference image corresponding to the global scope, so that the overall effect of the stitched image obtained according to the target captured image can be as consistent as possible with the reference image, The overall change of the stitched image is smoother and the transition is natural.

In another possible implementation manner, the configuration parameters include at least one or more of the following: automatic exposure AE configuration parameters, automatic white balance AWB configuration parameters, or dynamic range correction DRC configuration parameters.

In this way, the electronic device can make the overall exposure effect, AWB effect or dynamic range of the stitched image obtained from the target captured image as consistent as possible with the reference image, so that the overall change of the stitched image is smoother and the transition is natural.

In another possible implementation manner, the multi-frame captured images of the target include a first captured image of the target and a second captured image of the target, and the electronic device generates a stitched image according to the multi-frame captured images of the target, including: the electronic device captures an image of the first target After performing first image processing with the second target captured image, a stitched image is generated, and the first image processing includes image registration.

That is, after the electronic device registers the captured image of the first target and the captured image of the second target, a stitched image is generated.

In another possible implementation manner, the electronic device performs image registration on the captured image of the first target and the captured image of the second target, including: the electronic device, according to a preset image registration algorithm, separately registers the captured image of the first target and the captured image of the second target. The second target captured image extracts features; the electronic device registers the first target captured image and the second target captured image according to the obtained feature matching pairs.

That is to say, after the electronic device extracts the features, the captured image of the first target and the captured image of the second target can be registered according to the feature matching pair.

In another possible implementation manner, the method further includes: the electronic device corrects the registered image according to the reference image.

That is, the electronic device can correct the result of the registration according to the feature matching pair according to the reference image.

In another possible implementation manner, the electronic device performs image registration on the captured image of the first target and the captured image of the second target, including: the electronic device extracts features from the captured image of the first target and the reference image respectively, and according to the obtained features The matched pair registers the first target captured image and the reference image. The electronic device extracts features from the captured image of the second target and the reference image, respectively, and registers the captured image of the second target and the reference image according to the obtained feature matching pair.

That is to say, the electronic device can register the first target captured image and the reference image according to the feature matching pair, and register the second target captured image and the reference image, so as to achieve the registration of the first target captured image and the second target captured image the goal of.

In another possible implementation manner, the method further includes: the electronic device extracts features from the captured image of the first target and the captured image of the second target, respectively, and matches the paired registration of the captured image of the first target and the second target according to the obtained features Shooting an image; the electronic device calculates a homography matrix according to the feature matching pair of the overlapping area of the first target shooting image, the second target shooting image and the reference image; the electronic device deforms the second target shooting image according to the homography matrix .

That is to say, the electronic device can register the first target captured image and the reference image, the second target captured image and the reference image, and the first target captured image and the second target captured image, respectively. The feature matching pair of the overlapping area of the captured image, the second target captured image and the reference image calculates a homography matrix to deform the second target captured image so that the fusion effect of the second target captured image is better.

In another possible implementation manner, the electronic device performs image registration on the captured image of the first target and the captured image of the second target, including: fitting the captured image of the first target to a coordinate position having the same image content on the reference image ; The second target photographed image is attached to the coordinate position with the same image content on the reference image; If there is a hole between the first target photographed image and the second target photographed image, then according to the image content of the corresponding position of the hole on the reference image Fill in the holes.

In this solution, the electronic device may register the captured image of the first target and the captured image of the second target according to the coordinate system of the reference image.

In another possible implementation manner, the first image processing further includes ghost removal processing, and the electronic device performs first image processing on the first target captured image and the second target captured image to generate a stitched image, including: retaining the first image A ghost-free image of a moving object on the target captured image; delete the image of the moving object on the second target captured image; perform image registration on the first target captured image and the second target captured image to generate a stitched image; Fill the holes in the image.

In this solution, the electronic device can retain the ghost-free image of the moving object on the captured image of the first target, and fill holes in the spliced image generated by the captured image of the first target and the captured image of the second target, so as to achieve a ghost-free image. Purpose.

In another possible implementation manner, the first image processing further includes ghost removal processing, and the electronic device performs first image processing on the first target captured image and the second target captured image to generate a stitched image, including: The target captured image and the second target captured image are registered; the ghost-free image of the moving object is retained on the registered image, and the ghost-free image comes from the first target captured image, the second target captured image or the first image ; Delete the images of moving objects in other areas on the registered image; fill the holes caused by the deletion according to the reference image to generate a stitched image.

In this solution, the electronic device can first register the first target captured image and the second target captured image, retain the ghost-free image of the moving object on the registered image, and delete the registered image of other moving objects. image, with hole filling based on the reference image.

In another possible implementation manner, the first image processing further includes ghost removal processing, and the electronic device performs first image processing on the first target captured image and the second target captured image to generate a stitched image, including: deleting the first target image. The image of the moving object on the target photographed image and the second target photographed image; according to the multi-frame images collected by the first camera, the motion trajectory of the moving object is judged, so as to delete the image of the moving object on the first image, and obtain a first ghost-free image. image; fit the first target captured image and the second target captured image after deleting the image of the moving object onto the ghost-free first image; fill the holes caused by the deletion according to the reference image to generate a stitched image.

In this solution, the electronic device can obtain the ghost-free first image, and attach the first target captured image and the second target captured image after deleting the image of the moving object to the ghost-free first image, and Hole filling is performed based on the reference image to generate a stitched image.

In another possible implementation manner, the first image processing further includes uniform light and color processing and image fusion processing.

That is to say, after the electronic device performs registration, uniform light and color processing, and image fusion processing on the captured image of the first target and the captured image of the second target, a stitched image is generated.

In another possible implementation manner, the electronic device generates the target image according to the spliced image, including: after the electronic device performs second image processing on the spliced image, generating the target image, and the second image processing includes at least one or more of the following: De-ghosting, dynamic range enhancement, high frequency component fusion, dodging, or hole filling.

That is to say, the electronic device generates the target image after the stitched image is processed such as ghost removal, dynamic range enhancement, high-frequency component fusion, uniform light and color processing, or hole filling.

In another possible implementation manner, the electronic device performs high-frequency component fusion on the spliced image, including: the electronic device extracts the high-frequency component image of the reference image, and fuses the high-frequency component image with the spliced image to enhance the quality of the spliced image. High frequency detail to enhance the clarity and detail of the stitched and target images.

In another possible implementation manner, the electronic device performs uniform light and color processing on the spliced image, including: the electronic device performs high dynamic range synthesis on the images collected by the multiple frames of the first camera; The range information performs uniform light and color processing on the stitched image.

In this way, the electronic device can perform uniform light and color uniform processing on the stitched image according to the synthesized high dynamic range information, thereby enhancing the dynamic range of the stitched image.

In another possible implementation manner, the electronic device fills holes in the spliced images, including: splicing the regions corresponding to the holes of the images on the images collected by the multiple frames of the first camera, and performing super-resolution image synthesis; The image information of the stitched image is filled with holes.

In this way, the filling of holes and empty edges can have a higher resolution, thereby improving image quality and user experience.

In another possible implementation manner, the electronic device performs hole filling on the stitched image, including: performing image super-resolution processing on the area on the reference image corresponding to the hole in the stitched image; Fill in the holes.

In another possible implementation manner, the reference image is the first frame image collected by the first camera after detecting the user's photographing operation; or, the first image is the first frame image collected by the first camera after detecting the user's photographing operation The fused image of the resulting Q frames of images, where Q is an integer greater than 1.

That is to say, the reference image in the photographing process is a fixed image of the same frame.

In another possible implementation manner, the electronic device determines that the shooting ends, including: after the grids in the guide frame are matched, determining that the shooting ends.

That is, after determining that the grids in the guide frame are matched, the electronic device automatically ends shooting.

In another possible implementation manner, the electronic device generates the target image according to the spliced image, including: the spliced image is the target image; or, if the edges of the spliced image are not aligned, the spliced image is cropped to obtain the target whose edges are aligned or, if the edges of the spliced image are not aligned, fill the edge area of the spliced image according to the first image to obtain a target image with aligned edges.

That is, the target image is the same as the stitched image, or an image that has been edge cropped or edge filled for the stitched image.

In another possible implementation manner, the electronic device determines that the shooting is ended, including: before the grids in the guide frame are not matched, if a user's operation to stop taking pictures is detected, determining that the shooting is ended; If the moving direction moves out of the guide frame, the shooting is determined to end; or, if the deviation range between the moving direction of the electronic device and the indicated direction of the guide frame is greater than or equal to the fifth preset value, the shooting is determined to be completed.

That is to say, there may be various ways for the electronic device to determine to end the shooting.

In another possible implementation manner, the electronic device generates the target image according to the stitched image, including: generating the target image according to the stitched image corresponding to the matched whole row/column grid; or, according to the matched grid corresponding to the grid. The target image is generated by stitching the image and the image area corresponding to the unmatched grid on the first image.

That is to say, before the matching of the grids in the guide frame is completed, the electronic device can generate the target image according to the stitched images corresponding to the entire row/column grid, or combine with the image area corresponding to the unmatched grid on the first image. Image padding to generate the target image.

In another possible implementation manner, the method further includes: the electronic device obtains a target zoom magnification, the guide frame corresponds to the target zoom magnification, and the target zoom magnification is greater than the zoom magnification of the first camera and smaller than the zoom magnification of the second camera . The electronic device generates a target image according to the stitched image, including: the electronic device cuts the stitched image to generate a target image, and the target image corresponds to the target zoom magnification.

In this solution, the electronic device can stitch the target image captured by the second camera with a smaller field of view to obtain a stitched image with a larger and clearer field of view, and then crop the stitched image to obtain the target zoom A clear target image corresponding to the magnification. In addition, the electronic device does not need to perform image magnification through digital zoom, so the high resolution of the second camera and the high resolution of the second image can be retained, and the zoom effect of optical zoom can be achieved.

On the other hand, an embodiment of the present application provides another shooting method, which is applied to an electronic device. The electronic device includes a first camera and a second camera, and the equivalent focal length of the second camera is greater than the equivalent focal length of the first camera. The method includes: the electronic device starts a photographing function. The electronic device displays a third image and an image frame on the preview interface, the third image is an image captured by the first camera, the screen range of the third image in the image frame corresponds to the screen range of the second image, and the second image is the second image. image captured by the camera. After detecting the user's photographing operation, the electronic device displays the first image and the image frame on the photographing interface, and the first image is obtained according to the image collected by the first camera. The electronic device displays stitching information on the shooting interface. The stitching information is used to indicate the shooting progress. The stitching information corresponds to the multi-frame target shooting images obtained during the shooting process, and the adjacent target shooting images match each other. The electronic device generates a stitched image according to the multi-frame target captured images. After the shooting, the electronic device generates the target image according to the stitched image.

In this solution, the electronic device can refer to the first image captured by the first camera with a smaller equivalent focal length and a larger field of view, and use the second camera with a larger equivalent focal length and a smaller field of view to capture an image of the target , and splicing to obtain a target image with a larger field of view, and the target image has high definition, clear details and better shooting effect. In addition, the electronic device displays an image frame on the preview interface and the shooting interface, so as to facilitate the user to move the second camera according to the real-time shooting range of the second camera. Moreover, the electronic device may also display stitching information on the shooting interface to indicate the current shooting progress for the user in real time.

In a possible implementation manner, in the multi-frame target captured images, the matching third target captured image and fourth target captured image satisfy: the abscissa range of the third target captured image and the fourth target captured image The deviation between the abscissa ranges is less than or equal to the first preset threshold; or, the deviation between the ordinate range of the third target captured image and the ordinate range of the fourth target captured image is less than or equal to the second preset threshold.

That is to say, the matched images of the third target and the images of the fourth target are basically distributed left and right or basically distributed up and down.

In another possible implementation manner, the matching third target captured image and fourth target captured image also satisfy: the overlapping area between the third target captured image and the fourth target captured image is greater than or equal to the sixth predetermined target image. or, the gap between the captured image of the third target and the captured image of the fourth target is less than or equal to the seventh preset value.

That is, between the matched third target captured image and the fourth target captured image, the overlapping area is smaller, or the gap is smaller.

On the other hand, an embodiment of the present application provides a photographing apparatus, and the apparatus is included in an electronic device. The device has the function of implementing the behavior of the electronic device in any of the above aspects and possible designs, so that the electronic device executes the photographing method performed by the electronic device in any of the possible designs in the above-mentioned aspects. This function can be implemented by hardware or by executing corresponding software by hardware. The hardware or software includes at least one module or unit corresponding to the above-mentioned functions. For example, the apparatus may include an activation unit, a detection unit, a display unit, a generation unit, and the like.

In another aspect, an embodiment of the present application provides an electronic device, including: a first camera and a second camera for capturing images; a screen for displaying an interface; one or more processors; a memory; and one or more One or more computer programs, one or more computer programs are stored in memory, the one or more computer programs include instructions that, when executed by an electronic device, cause the electronic device to perform any of the above aspects. Possible designs for the electronic device to perform Shooting method.

In another aspect, an embodiment of the present application provides an electronic device, including: one or more processors; and a memory, where codes are stored in the memory. When the code is executed by the electronic device, the electronic device is caused to execute the photographing method executed by the electronic device in any possible design of the above aspect.

In another aspect, an embodiment of the present application provides a computer-readable storage medium, including computer instructions, which, when the computer instructions are executed on an electronic device, enable the electronic device to execute the photographing method in any of the possible designs in the foregoing aspect.

In another aspect, an embodiment of the present application provides a computer program product, which, when the computer program product runs on a computer, enables the computer to execute the photographing method performed by the electronic device in any of the possible designs in the above aspect.

In another aspect, an embodiment of the present application provides a chip system, and the chip system is applied to an electronic device. The chip system includes one or more interface circuits and one or more processors; the interface circuit and the processor are interconnected by lines; the interface circuit is used for receiving signals from the memory of the electronic device and sending signals to the processor, and the signals are included in the memory Stored computer instructions; when the processor executes the computer instructions, the electronic device is made to execute the photographing method in any possible design of the above aspect.

For the beneficial effects corresponding to the above-mentioned other aspects, reference may be made to the description of the beneficial effects of the method, which will not be repeated here.

Description of drawings

FIG. 1 is a schematic structural diagram of an electronic device according to an embodiment of the present application;

2 is a schematic flowchart of a photographing method provided by an embodiment of the present application;

3A is a schematic diagram of a group of interfaces provided by an embodiment of the present application;

3B is a schematic diagram of an interface provided by an embodiment of the present application;

Fig. 4 is another interface schematic diagram provided by the embodiment of the present application;

FIG. 5 is another set of interface schematic diagrams provided by an embodiment of the present application;

6 is another set of interface schematic diagrams provided by the embodiment of the present application;

7A is another set of interface schematic diagrams provided by an embodiment of the present application;

FIG. 7B is another set of interface schematic diagrams provided by an embodiment of the present application;

FIG. 8 is another set of interface schematic diagrams provided by an embodiment of the present application;

9A is another set of interface schematic diagrams provided by an embodiment of the present application;

9B is a schematic diagram of a group of shooting sequences provided by an embodiment of the present application;

FIG. 10A is a set of schematic interface diagrams and a schematic diagram of camera positions provided by an embodiment of the present application;

FIG. 10B is a schematic diagram of an interface provided by an embodiment of the present application;

FIG. 11 is another set of interface schematic diagrams provided by this embodiment of the present application;

12A is another set of interface schematic diagrams provided by an embodiment of the present application;

FIG. 12B is a schematic diagram of a group of images provided by an embodiment of the present application;

FIG. 12C is a schematic diagram of another group of images provided by this embodiment of the present application;

FIG. 12D is a schematic diagram of another set of images provided by an embodiment of the present application;

13A is a schematic diagram of an image fusion process provided by an embodiment of the present application;

13B is a schematic diagram of another image fusion process provided by an embodiment of the present application;

FIG. 13C is a schematic diagram of a group of interfaces provided by an embodiment of the present application;

FIG. 14 is another set of interface schematic diagrams provided by an embodiment of the present application;

15A is another set of interface schematic diagrams provided by an embodiment of the present application;

15B is another set of interface schematic diagrams provided by an embodiment of the present application;

15C is a schematic diagram of an interface provided by an embodiment of the present application;

15D is another set of interface schematic diagrams provided by an embodiment of the present application;

FIG. 16 is another interface schematic diagram provided by an embodiment of the present application;

FIG. 17 is a schematic diagram of a group of images provided by an embodiment of the present application;

18 is a schematic diagram of the identification of a group of target images provided by an embodiment of the present application;

19 is a schematic flowchart of another photographing method provided by an embodiment of the present application;

FIG. 20 is a schematic diagram of a group of interfaces provided by an embodiment of the present application;

FIG. 21 is another set of interface schematic diagrams provided by this embodiment of the present application;

FIG. 22 is another set of interface schematic diagrams provided by this embodiment of the present application;

23A is another set of interface schematic diagrams provided by an embodiment of the present application;

FIG. 23B is another set of interface schematic diagrams provided by an embodiment of the present application;

FIG. 24 is a schematic flowchart of a zoom solution provided by an embodiment of the present application;

FIG. 25 is a schematic diagram of a group of interfaces provided by an embodiment of the present application;

26A is a set of schematic diagrams and interface diagrams of a guide frame provided by an embodiment of the present application;

26B is a schematic diagram of an interface provided by an embodiment of the present application;

FIG. 27 is a schematic diagram of a group of interfaces provided by an embodiment of the present application;

FIG. 28 is another set of interface schematic diagrams and target image schematic diagrams provided by an embodiment of the present application;

FIG. 29 is another interface schematic diagram provided by this embodiment of the application;

FIG. 30 is a schematic structural diagram of another electronic device provided by an embodiment of the present application.

detailed description

For ease of understanding, the examples provide some descriptions of concepts related to the embodiments of the present application for reference. As follows:

The first image: the background image displayed on the shooting interface. When the first camera is a wide-angle camera, the first image is a wide-angle image. The first image may be fixed and not refreshed, for example, the first frame of wide-angle image or the following initial wide-angle image captured by the wide-angle camera after the user's photographing operation is detected in the following embodiments. The first image may also be refreshed in real time, for example, a wide-angle image collected in real time by the wide-angle camera during the photographing process.

Second image: the image captured by the second camera in real time. For example, when the second camera is a telephoto camera, the second image is a telephoto image.

Target Capture Image: A second image that matches the grid in the guide frame. For example, when the second camera is a telephoto camera, the image captured by the target may be a telephoto image of the target that matches the grid in the guide frame.

The third image: the real-time changing image displayed on the preview interface, acquired through real-time acquisition by the first camera. For example, when the first camera is a wide-angle camera, the third image may be a wide-angle image displayed on the preview interface in the following embodiments.

Stitching information: used to indicate the shooting progress of the photographing process, indicating the matching progress of the grids in the guide frame during the photographing process. The stitching information corresponds to the multi-frame target telephoto images matched with the grids in the guide frame during the photographing process. For example, the splicing information may be the spliced image thumbnails, splicing frames, matched grids or borders of matched grids displayed on the shooting interface in the following embodiments.

Image frame: used to indicate the real-time shooting range of the second camera. For example, when the second camera is a telephoto camera, the image frame is the telephoto frame in the following embodiments.

The first image processing: the processing of the two target telephoto images to be spliced, for example, including image registration, uniform light and color, ghost removal or image fusion, etc.

Second image processing: The processing performed on the stitched image before generating the target image, such as ghost removal processing, dynamic range enhancement, high-frequency component fusion, uniform light and color processing, or hole filling, etc.

Wherein, the first target telephoto image and the second target telephoto image are respectively two frames of target telephoto images obtained during the photographing process. The third target telephoto image and the fourth target telephoto image are respectively two frames of target telephoto images obtained during the photographing process.

The technical solutions in the embodiments of the present application will be described below with reference to the accompanying drawings in the embodiments of the present application. Wherein, in the description of the embodiments of the present application, unless otherwise stated, “/” means or means, for example, A/B can mean A or B; “and/or” in this document is only a description of the associated object The association relationship of , indicates that there can be three kinds of relationships, for example, A and/or B, can indicate that A exists alone, A and B exist at the same time, and B exists alone. In addition, in the description of the embodiments of the present application, "plurality" refers to two or more than two.

Hereinafter, the terms "first" and "second" are only used for descriptive purposes, and should not be construed as indicating or implying relative importance or implicitly indicating the number of indicated technical features. Thus, a feature defined as "first" or "second" may expressly or implicitly include one or more of that feature. In the description of this embodiment, unless otherwise specified, "plurality" means two or more.

At present, on images with a wide field of view captured by electronic devices such as mobile phones with a wide-angle camera, the clarity of local details is low, and users may not be able to clearly see the details on the images. In particular, when the object to be photographed is far away from the electronic device, the sharpness of the image obtained by the wide-angle camera at a long distance is lower, and it is difficult to clearly present the details of the object to be photographed.

The embodiment of the present application provides a shooting method, which can be applied to electronic equipment, and can refer to an image collected by a first camera with a larger field of view, and use a second camera (telephoto camera) with a smaller field of view to shoot images and stitch them together A target image with a larger field of view is obtained, so that the target image has higher clarity and resolution, clear local details, prominent subject, and better shooting effect.

Wherein, the equivalent focal length of the second camera is relatively large, and the field of view angle is relatively small, for example, it may be a telephoto camera or an ultra-telephoto camera. Different from the second camera, the first camera involved in the embodiments of the present application may be a camera with a smaller equivalent focal length and a larger field of view, such as a wide-angle camera, an ultra-wide-angle camera, or a panoramic camera. For example, the equivalent focal length of the second camera may be 240mm, and the field angle may be 10°. For another example, the equivalent focal length of the second camera may be 125mm, and the field of view angle may be 20°. For another example, the equivalent focal length of the second camera may be 80mm, and the field of view angle may be 30°. For example, the equivalent focal length of the first camera may be 26mm, and the field angle may be 80°. For another example, the equivalent focal length of the first camera may be 16mm, and the field of view angle may be 120°.

Exemplarily, the first camera is a wide-angle camera, and the second camera is a telephoto camera; or, the first camera is an ultra-wide-angle camera, and the second camera is a telephoto camera; or, the first camera is a wide-angle camera, and the second camera is a Super telephoto camera.

In the shooting method provided by the embodiment of the present application, the field of view of the image captured by the second camera and the target image obtained by splicing is smaller than or equal to the field of view of the first camera. In some embodiments, the equivalent focal length of the second camera may be greater than or equal to a preset value, so that the zoom magnification of the second camera is larger, the field of view is smaller, and the image resolution is higher; in other embodiments, The ratio of the equivalent focal length of the second camera to the first camera may be greater than or equal to the preset value of 1, so that the ratio of the zoom magnification of the second camera to the first camera is larger, and the difference between the field of view of the first camera and the second camera The proportion is larger. Exemplarily, the preset value 1 may be 2 or 3 or the like. In this way, the target image obtained by the electronic device has higher definition and resolution, clearer local details, more prominent subject, and better shooting effect.

Among them, the equivalent focal length refers to the length of the diagonal of the image area of the photoelectric sensor chip of the camera. When it is equivalent to the diagonal length of the 35mm camera frame (42.27mm), the actual focal length of the camera corresponds to the focal length of the 35mm camera lens. The zoom ratio describes the meaning of the relative equivalent focal length. For example, if the zoom ratio of a wide-angle camera is defined as 1, then the zoom ratio of other cameras (such as telephoto cameras) is equal to the equivalent focal length of the telephoto camera and the wide-angle camera as the benchmark. ratio of equivalent focal lengths. The size of the field of view determines the field of view of the camera. The larger the field of view, the larger the field of view. The larger the equivalent focal length, the smaller the field of view.

In the embodiment of the present application, the target image obtained by splicing the images captured by the second camera may be in the form of a wide format (including a horizontal format or a vertical format), a square format, an ultra-wide format, or a panorama. For example, the aspect ratio of the target image can be 2:1, 9:16, 1:1, or 2.39:1, etc. Among them, target images of different frames can give users different visual feelings, so for different themes or themes, appropriate frames can be used to shoot. For example, landscape subjects can be shot in horizontal format to show the wide and atmospheric characteristics of the scene. For another example, high-rise buildings, towers, mountains and other themes can be shot in vertical format to express a towering and upright picture effect.

The shooting methods provided in the embodiments of the present application can be used for rear image shooting, and can also be used for front image shooting.

For example, the electronic device may be a mobile phone, a tablet, a wearable device (eg, a smart watch), an in-vehicle device, an augmented reality (AR)/virtual reality (VR) device, a laptop, a super mobile personal computer Mobile terminals such as (ultra-mobile personal computer, UMPC), netbook or personal digital assistant (personal digital assistant, PDA), can also be professional cameras and other equipment, the embodiment of the present application does not make any restrictions on the specific type of electronic equipment.

Exemplarily, FIG. 1 shows a schematic structural diagram of an electronic device 100 . The electronic device 100 may include a processor 110, an external memory interface 120, an internal memory 121, a universal serial bus (USB) interface 130, a charge management module 140, a power management module 141, a battery 142, an antenna 1, an antenna 2 , mobile communication module 150, wireless communication module 160, audio module 170, speaker 170A, receiver 170B, microphone 170C, headphone jack 170D, sensor module 180, buttons 190, motor 191, indicator 192, camera 193, display screen 194, and Subscriber identification module (subscriber identification module, SIM) card interface 195 and so on. The sensor module 180 may include a pressure sensor 180A, a gyroscope sensor 180B, an air pressure sensor 180C, a magnetic sensor 180D, an acceleration sensor 180E, a distance sensor 180F, a proximity light sensor 180G, a fingerprint sensor 180H, a temperature sensor 180J, a touch sensor 180K, and ambient light. Sensor 180L, bone conduction sensor 180M, etc.

The processor 110 may include one or more processing units, for example, the processor 110 may include an application processor (application processor, AP), a modem processor, a graphics processor (graphics processing unit, GPU), an image signal processor (image signal processor, ISP), controller, memory, video codec, digital signal processor (digital signal processor, DSP), baseband processor, and/or neural-network processing unit (NPU) Wait. Wherein, different processing units may be independent devices, or may be integrated in one or more processors.

The controller may be the nerve center and command center of the electronic device 100 . The controller can generate an operation control signal according to the instruction operation code and timing signal, and complete the control of fetching and executing instructions.

A memory may also be provided in the processor 110 for storing instructions and data. In some embodiments, the memory in processor 110 is cache memory. This memory may hold instructions or data that have just been used or recycled by the processor 110 . If the processor 110 needs to use the instruction or data again, it can be called directly from memory. Repeated accesses are avoided and the latency of the processor 110 is reduced, thereby increasing the efficiency of the system.

The electronic device 100 implements a display function through a GPU, a display screen 194, an application processor, and the like. The GPU is a microprocessor for image processing, and is connected to the display screen 194 and the application processor. The GPU is used to perform mathematical and geometric calculations for graphics rendering. Processor 110 may include one or more GPUs that execute program instructions to generate or alter display information.

Display screen 194 is used to display images, videos, and the like. Display screen 194 includes a display panel. The display panel can be a liquid crystal display (LCD), an organic light-emitting diode (OLED), an active-matrix organic light-emitting diode or an active-matrix organic light-emitting diode (active-matrix organic light). emitting diode, AMOLED), flexible light-emitting diode (flex light-emitting diode, FLED), Miniled, MicroLed, Micro-oLed, quantum dot light-emitting diode (quantum dot light emitting diodes, QLED) and so on. In some embodiments, the electronic device 100 may include one or N display screens 194 , where N is a positive integer greater than one. In the embodiment of the present application, the display screen 194 may be used to display a preview interface and a photographing interface and the like in the photographing mode.

The electronic device 100 may implement a shooting function through an ISP, a camera 193, a video codec, a GPU, a display screen 194, an application processor, and the like.

The ISP is used to process the data fed back by the camera 193 . For example, when taking a photo, the shutter is opened, the light is transmitted to the camera photosensitive element through the lens, the light signal is converted into an electrical signal, and the camera photosensitive element transmits the electrical signal to the ISP for processing, converting it into an image visible to the naked eye. ISP can also perform algorithm optimization on image noise, brightness, and skin tone. ISP can also optimize the exposure, color temperature and other parameters of the shooting scene. In some embodiments, the ISP may be provided in the camera 193 .

Camera 193 is used to capture still images or video. The object is projected through the lens to generate an optical image onto the photosensitive element. The photosensitive element may be a charge coupled device (CCD) or a complementary metal-oxide-semiconductor (CMOS) phototransistor. The photosensitive element converts the optical signal into an electrical signal, and then transmits the electrical signal to the ISP to convert it into a digital image signal. The ISP outputs the digital image signal to the DSP for processing. DSP converts digital image signals into standard RGB, YUV and other formats of image signals. In some embodiments, the electronic device 100 may include 1 or N cameras 193 , where N is a positive integer greater than 1.

The camera 193 may include cameras with different focal lengths, for example, a first camera with a short focal length and a second camera with a long focal length. Among them, the equivalent focal length of the first camera is small (for example, 13mm, 16mm, 26mm, 30mm or 40mm, etc.), and the field of view angle is large (for example, the field of view angle can be 80°, 120°, or 150°, etc.), which can be used for Take larger pictures such as landscapes. For example, a current wide-angle camera, an ultra-wide-angle camera, a panoramic camera, and other cameras with a larger field of view may be referred to as the first camera. The equivalent focal length of the second camera is larger (for example, 80mm, 125mm, 150mm or 240mm, etc.), and the field of view angle is small (for example, the field of view angle can be 10°, 20° or 30°, etc.), which can be used to shoot distant objects, the area that can be photographed is small. For example, both the current telephoto camera and the ultra-telephoto camera can be called the second camera.

In some embodiments, the second camera is fixed, and the user can move the second camera by moving the electronic device 100 . In other embodiments, the second camera can be moved independently, and the user can directly move the second camera through a certain key, control or operation without moving the mobile phone; or, the mobile phone can automatically control the movement of the second camera. When the second camera moves, the content of the picture captured by the second camera also changes accordingly.

In addition, the camera 193 may further include a depth camera for measuring the object distance of the object to be photographed, and other cameras. For example, the depth camera may include a 3-dimensions (3D) depth camera, a time of flight (TOF) depth camera, or a binocular depth camera, and the like.

A digital signal processor is used to process digital signals, in addition to digital image signals, it can also process other digital signals. For example, when the electronic device 100 selects a frequency point, the digital signal processor is used to perform Fourier transform on the frequency point energy and so on.

Video codecs are used to compress or decompress digital video. The electronic device 100 may support one or more video codecs. In this way, the electronic device 100 can play or record videos of various encoding formats, such as: Moving Picture Experts Group (moving picture experts group, MPEG) 1, MPEG2, MPEG3, MPEG4 and so on.

Internal memory 121 may be used to store computer executable program code, which includes instructions. The processor 110 executes various functional applications and data processing of the electronic device 100 by executing the instructions stored in the internal memory 121 . The internal memory 121 may include a storage program area and a storage data area. The storage program area can store an operating system, an application program required for at least one function (such as a sound playback function, an image playback function, etc.), and the like. The storage data area may store data (such as audio data, phone book, etc.) created during the use of the electronic device 100 and the like. In addition, the internal memory 121 may include high-speed random access memory, and may also include non-volatile memory, such as at least one magnetic disk storage device, flash memory device, universal flash storage (UFS), and the like.

In the embodiment of the present application, the processor 110 can realize the reference to the image collected by the first camera by running the instructions stored in the internal memory 121, use the second camera to capture the image and stitch to obtain the target image with a larger field of view, so as to The target image has high definition, clear details and better shooting effect.

Touch sensor 180K, also called "touch panel". The touch sensor 180K may be disposed on the display screen 194, and the touch sensor 180K and the display screen 194 form a touch screen, also called a touch screen. The touch sensor 180K is used to detect a touch operation on or near it. The touch sensor can pass the detected touch operation to the application processor to determine the type of touch event. Visual output related to touch operations may be provided through display screen 194 . In other embodiments, the touch sensor 180K may also be disposed on the surface of the electronic device 100 , which is different from the location where the display screen 194 is located.

It can be understood that the structures illustrated in the embodiments of the present application do not constitute a specific limitation on the electronic device 100 . In other embodiments of the present application, the electronic device 100 may include more or less components than shown, or combine some components, or separate some components, or arrange different components. The illustrated components may be implemented in hardware, software, or a combination of software and hardware.

In the embodiment of the present application, the first camera and the second camera in the camera 193 can be used to capture images; the display screen 194 can be used to display a preview interface and a shooting interface when taking pictures; The instructions of the memory 121 can realize the reference to the image collected by the first camera with a larger field of view, use the second camera with a smaller field of view to capture the image and stitch to obtain the target image with a larger field of view, so that the target image has a larger field of view. The sharpness is high, the details are clear, and the shooting effect is better.

The photographing method provided by the embodiment of the present application will be described below by taking the electronic device as a mobile phone having the structure shown in FIG. 1 as an example. As shown in Figure 2, the method may include:

200. The mobile phone starts the camera function.

In the embodiment of the present application, when the user wants to use the mobile phone to take an image, the camera function of the mobile phone can be activated. For example, the phone can launch the camera app, or launch other apps with a shooting function (such as AR apps such as Douyin or Hetu cyberverse), thereby launching the app's camera function.

Exemplarily, after the mobile phone detects that the user clicks the camera icon 301 shown in (a) in FIG. 3A , the mobile phone starts the photographing function of the camera application, and displays the preview interface shown in (b) in FIG. 3A . In another example, the mobile phone displays a desktop or a non-camera application interface, starts the photographing function after detecting the user's voice command to open the camera application, and displays a preview interface as shown in (b) of FIG. 3A .

It should be noted that the mobile phone can also activate the camera function in response to other user touch operations, voice commands or quick gestures and other operations, and the embodiment of the present application does not limit the operation of triggering the mobile phone to activate the camera function.

In some embodiments, after starting the camera function, the mobile phone can use the shooting method provided by the embodiment of the present application to use a second camera with a smaller field of view to collect multiple frames of images, and stitch the multiple frames of the images into a field of view target image with larger corners. The clarity and resolution of the target image are high, the local details are clear, the main body is prominent, and the shooting effect is better.

In other embodiments, after the mobile phone has activated the camera function and entered the target camera mode, the shooting method provided by the embodiment of the present application can use the second camera with a smaller field of view to collect multiple frames of images, and then record the multiple frames of the The images are stitched into a target image with a larger field of view.

In some technical solutions, the mobile phone can use the shooting method provided by the embodiment of the present application to refer to the image collected by the first camera with a larger field of view, use the second camera with a smaller field of view to shoot images, and stitch them to obtain the field of view. The larger the target image, the higher the clarity and resolution of the target image, the clear local details, the prominent subject, and the better shooting effect.

For example, the target photographing mode may be specifically referred to as a wide-frame mode, a wide-view mode, or a high-definition image mode, and the embodiment of the present application does not limit the specific name of the target photographing mode.

In the following embodiments of the present application, the target photographing mode is taken as the wide-view mode as an example for description.

Exemplarily, after the mobile phone starts the photographing function, if it is detected that the user clicks the control 302 shown in (b) in FIG. 3A , the wide view mode is entered as shown in (c) in FIG. 3A . In another example, after the mobile phone detects that the user clicks the control 303 shown in (b) in FIG. 3A , the interface shown in (d) in FIG. 3A is displayed; after the mobile phone detects the operation that the user clicks the control 304 , as shown in (c) in Figure 3A to enter the wide-view mode.

For another example, when the mobile phone displays a desktop or a non-camera application interface, it detects the user's voice command to enter the wide-view mode and starts the camera function, and enters the wide-view mode as shown in (c) of FIG. 3A .

It should be noted that the mobile phone can also activate the camera function and enter the wide-view mode in response to other user touch operations, voice commands, or shortcut gestures.

In some embodiments, the mobile phone can prompt the user of the function of the shooting mode in the wide-view mode by displaying information or voice broadcasting. Exemplarily, referring to FIG. 3B, the mobile phone displays text prompt information on the preview interface: in the wide-view mode, you can refer to the image captured by the first camera with a larger field of view, use the second camera with a smaller field of view to capture the image and Stitching obtains a target image with a larger field of view.

In some embodiments, in the wide view mode, the mobile phone can prompt the user about the specific cameras used by the first camera and the second camera. Exemplarily, referring to FIG. 4 , the mobile phone displays text information on the preview interface to prompt the user, the first camera is a wide-angle camera, and the second camera is a telephoto camera.

In some technical solutions, the first camera and the second camera in the wide-view mode are cameras set by the user, for example, the first camera is an ultra-wide-angle camera set by the user, and the second camera is a telephoto camera set by the user. This embodiment of the present application does not limit the specific manner in which the user sets the first camera and the second camera.

In other technical solutions, the first camera and the second camera in the wide-view mode are default cameras, for example, the first camera is a wide-angle camera by default, and the second camera is a telephoto camera by default. The user can also modify the camera types of the first camera and the second camera.

In the following embodiments of the present application, the first camera is a wide-angle camera and the second camera is a telephoto camera as an example for description.

201. The mobile phone displays a wide-angle image on the preview interface.

After the phone starts the camera function, it enters the preview state. In the preview state, the mobile phone can collect wide-angle images in real time through the wide-angle camera according to the preset frame rate 1, and display the obtained wide-angle images on the preview interface, so as to present the global image within a larger field of view to the user ( or panoramic image). The wide-angle camera is the first camera.

In some embodiments, the mobile phone can also display a guide frame for guiding the telephoto camera to move and shoot on the preview interface, so that the telephoto camera can shoot multiple frames of images according to the guide frame during the photographing process. Among them, the guide frame is superimposed and displayed on the wide-angle image in the form of a transparent floating frame. The field of view of the guide frame is smaller than or equal to the field of view of the wide-angle camera. That is to say, the field of view of the target image generated by the mobile phone according to the instructions of the guide frame is smaller than or equal to the field of view of the wide-angle camera.

The maximum number of grids that the guide frame can include is R*R, and R is related to K1 and/or K2. Among them, K1 is the ratio of the equivalent focal length of the telephoto camera to the equivalent focal length of the wide-angle camera, rounded up or down. K2 is the ratio of the field of view of the wide-angle camera to the field of view of the telephoto camera, rounded up or down. In some embodiments, R is K1; in other embodiments, R is K2; in other embodiments, R is the greater of K1 and K2.

In this embodiment of the present application, the guide frame displayed by the mobile phone on the guide interface includes M (rows)*N (columns) grids. Wherein, M≤R, N≤R, and at least one of M and N is greater than 1. This M*N can be called the specification of the lead frame. That is to say, the specification of the guide frame includes the number of grids in the guide frame and the arrangement of the grids. The wide-angle image corresponds to the lead frame with the highest number of grids. For example, the size and field of view of the wide-angle image can be slightly larger or equal to the size and field of view of the guide frame with the largest number of grids, and the field of view of the telephoto camera corresponds to the corresponding field of view of a single grid. For example, the field of view of the telephoto camera may be slightly larger than or equal to the field of view corresponding to a single grid.

For example, the equivalent focal length of the wide-angle camera is 125mm, the equivalent focal length of the telephoto camera is 25mm, the ratio K1 of the equivalent focal lengths of the two is 5, and R=K1=5. In this way, the number of grids in each row of the guide frame is less than or equal to 5, and the number of grids in each column is also less than or equal to 5. That is to say, the guide frame may include at most 5*5 (ie, 5 rows and 5 columns) grids; the guide frame may also include less than 5*5 grids, such as 3*3, 3*4 or 4* 5 grids. Among them, the field of view of the wide-angle camera corresponds to the 5*5 grids, and a single grid corresponds to the field of view of the telephoto camera.

In some embodiments, the ratio between the equivalent focal length of the telephoto camera and the equivalent focal length of the wide-angle camera is less than or equal to a preset value of 2. In this way, the difference between the size of the guide frame and the size of the wide-angle image is small, and the sizes of the guide frame and the grid are larger, which is convenient for the user to move the telephoto camera so that the telephoto image matches the grid. Exemplarily, the preset value 2 may be 8 or 10 or the like.

In the following embodiments, the guide frame includes at most 5*5 grids as an example for description.

In some embodiments, the guide frame is displayed by default in the middle of the image displayed in the preview interface. The specification of the guide frame is the default specification, or the specification that the phone last adopted in the wide view mode.

In other embodiments, the position or specification of the guide frame displayed by the mobile phone on the preview interface may be set by the user.

For example, in some technical solutions, on the preview interface, after the user selects an area on the wide-angle image (for example, draws a box or a circle), the mobile phone determines a matching guide frame according to the area selected by the user. For example, the area corresponding to the guide frame determined by the mobile phone can cover the user's selection area, or the ratio of covering the user's selection area is greater than or equal to a preset ratio (such as 90%), or slightly larger than the user's selection area. Exemplarily, see (a) in FIG. 5 , the mobile phone prompts the user on the preview interface: please select the range of the target image to display the corresponding guide frame. As shown in (b) of FIG. 5 , after detecting the user's operation of selecting an area, the mobile phone determines that the target image the user wants to shoot is the size corresponding to the area. Then, the mobile phone determines the position or size of the guide frame according to the area. For example, the guide frame includes 3*3 grids, and as shown in (c) of FIG. 5 , the guide frame 501 is displayed on the preview interface.

In other technical solutions, the mobile phone displays the setting interface after detecting that the user clicks the setting control 601 on the preview interface as shown in (a) of FIG. 6 . As shown in (b) in Figure 6, the setting interface includes the setting control of the target image guide frame. After the mobile phone detects the operation of the user clicking on the control, as shown in (c) in Figure 6, the largest guide frame is displayed. The largest lead frame includes the largest number of grids (eg 5*5 grids above). After the mobile phone detects that the user performs a frame selection on the largest guide frame and clicks to confirm the control, the range of the position corresponding to the user's frame selection operation is determined as the position and size of the guide frame. As shown in (d) of FIG. 6 , the mobile phone displays the determined guide frame 602 including 2*3 grids on the preview interface.

In other technical solutions, see (a) in FIG. 7A , the mobile phone displays the largest guide frame including the maximum number of grids on the preview interface, and prompts the user: please set the guide frame of the target image. As shown in (b) of FIG. 7A , the user drags on the largest guide frame to select multiple grids, and the mobile phone determines that the position and range corresponding to the user's frame selection operation are the position and size of the guide frame. As shown in (c) of FIG. 7A , the mobile phone displays a determined guide frame 701 including 3*3 grids on the preview interface, and the preview interface also includes a wide-angle image.

In other technical solutions, the user can indicate the main body on the preview interface, and the mobile phone determines the position and size of the guide frame according to the main body indicated by the user, so that the guide frame can cover the main body indicated by the user. Exemplarily, referring to (a) in FIG. 7B , the mobile phone may prompt the user on the preview interface to select the subject to be photographed. As shown in (b) in FIG. 7B , after the mobile phone detects the operation of the user clicking on the building, it determines that the entire connected building is the subject to be photographed, and thus determines that the guide frame is on the preview interface as shown in (c) in FIG. 7B . The guide frame 703 that can cover the entire connected building includes a total of 3*3 grids.

In other technical solutions, see (a) in FIG. 8 , the mobile phone displays multiple specification controls on the preview interface, such as 5*5, 4*5, 4*4, 3*4 or 3*3, etc. Determines the number of grids included in the guide box based on the user-selected specification control. For example, after the mobile phone detects the operation of clicking the 3*3 specification control, as shown in (b) of FIG. 8, a guide frame 801 including 3*3 grids is displayed on the preview interface, and the preview interface also includes Wide angle image.

In some embodiments of the present application, the mobile phone may also operate according to the user's instruction to determine the shooting sequence of the grids in the guide frame. During the photographing process, the user can move the telephoto camera according to the photographing sequence, so that the telephoto camera is sequentially matched with the grid specified in the sequence.

In a possible implementation manner, when the user sets the position or specification of the guide frame through the methods described in the above embodiments (eg, the methods shown in FIGS. 5-8 ), the user may also set the shooting order of the grids in the guide frame.

In another possible implementation manner, after determining the specification of the guide frame, the mobile phone can display multiple sequence modes corresponding to the guide frame of the specification, and the mobile phone can determine one sequence mode according to the user's selection operation. Exemplarily, the size of the guide frame is 3*3, see (a) in FIG. 9A , and the preview interface includes multiple sequential mode controls, such as controls 901 to 903 . For example, after the mobile phone detects the user's operation of clicking the sequence mode control 901, the mobile phone determines that the shooting sequence is to first shoot the image corresponding to the middle row of grids from left to right, then shoot the image of the upper row of grids from left to right, and then from left to right. Go to the right to take an image of the lower row of grids. For another example, after the mobile phone detects the user's operation of clicking the sequence mode control 902, it determines that the photographing sequence is to first photograph the image of the grid in the middle row from left to right, then photograph the image of the grid in the upper row from right to left, and then from the left. Go to the right to take an image of the lower row of grids. For another example, after the mobile phone detects that the user clicks on the sequence mode control 903, the mobile phone determines that the shooting sequence is to shoot according to the "zigzag" track from top to bottom.

In other embodiments, the guide frame of each specification corresponds to a shooting order by default, and the mobile phone uses the default shooting order to shoot after determining the specification of the guide frame. The mobile phone can also modify the shooting sequence according to the user's instruction operation.

In some embodiments of the present application, after determining the shooting sequence corresponding to the guide frame, the mobile phone may prompt the user to shoot according to the shooting sequence. For example, the mobile phone can display shooting sequence prompt information on the preview interface. Exemplarily, after the user selects the shooting sequence corresponding to the sequence mode control 901, see (b) in FIG. 9A , the mobile phone can display a label and a guide line with an arrow on the guide frame, and the label is used to indicate different rows of grids. , the arrows of the guide line indicate the direction to indicate the shooting order of each row of grids. For another example, after the user selects the shooting sequence corresponding to the sequence mode control 902, referring to FIG. 9A (c), the mobile phone can display a guide line with an arrow on the guide frame to indicate the shooting sequence corresponding to different grids.

It can be understood that the shooting sequence may also include other sequential modes, for example, may also include sequential modes as shown in (a)-(d) in FIG. limited.

In some other embodiments, after acquiring the guide frame, the mobile phone may briefly display the guide frame on the photographing interface, and then display the guide frame on the photographing interface after the user's photographing operation is subsequently detected.

In some other embodiments, the mobile phone does not display the guide frame on the preview interface, but displays the guide frame on the shooting interface after detecting the user's photographing operation.

In addition, in other embodiments, in the preview state, the mobile phone can also collect telephoto images through the telephoto camera according to the preset acquisition frame rate 2 . The telephoto camera is the second camera.

In still other embodiments, the mobile phone may also prompt the user with the real-time shooting range of the telephoto camera through the telephoto frame, so as to present the partial picture captured by the telephoto camera in real time to the user. Among them, the telephoto frame is superimposed and displayed on the wide-angle image in the form of a transparent floating frame. The position and size of the telephoto frame on the wide-angle image corresponds to the shooting range of the telephoto camera. The field of view of the wide-angle image in the telephoto frame may be equal to or slightly smaller than the shooting range and field of view of the telephoto camera. The picture range of the wide-angle image in the telephoto frame corresponds to that of the telephoto camera. For example, the frame range of the wide-angle image in the telephoto frame may be equal to or slightly smaller than the frame range of the telephoto image captured by the telephoto camera. The collection frame rate 2 and the collection frame rate 1 may be the same or different, and are not limited. Exemplarily, referring to (a) in FIG. 10A , the preview interface includes a wide-angle image 1001 , a telephoto frame 1002 and a guide frame 1003 .

Among them, when the user moves the mobile phone or the user's hand shakes and other reasons cause the mobile phone to shake, the relative position and size of the telephoto frame on the wide-angle image on the preview interface remain basically unchanged. The object distance and the size of the field of view are related. Exemplarily, see (b) in FIG. 10A , the distance between the lens centers of the wide-angle camera and the telephoto camera is fixed, and when the object distance is constant, the size of the field of view is also fixed. For mobile phones, the telephoto camera and the wide-angle camera move at the same time, but the relative relationship between the telephoto camera’s field of view and the wide-angle camera’s field of view remains unchanged, so the relative position and size of the telephoto frame on the wide-angle image are also Basically unchanged.

In some other embodiments, when the equivalent focal length of the telephoto camera is quite different from the equivalent focal length of the wide-angle camera, the field angle of the telephoto camera is also quite different from the field angle of the wide-angle camera. The field of view is small, and the size of the telephoto frame is small, making it inconvenient for users to view details within the shooting range of the telephoto camera. Therefore, the mobile phone can enlarge the telephoto frame and the wide-angle image in the telephoto frame and display them on the preview interface, so as to facilitate the user to know the shooting range and details of the telephoto camera. Exemplarily, see (c) in FIG. 10A , the preview interface includes a wide-angle image 1004 , a zoomed-in telephoto frame 1005 , and a guide frame 1006 .

In other embodiments, the mobile phone displays the target area image corresponding to the guide frame on the wide-angle image on the preview interface, but does not display the complete wide-angle image. The ratio of the size of the target area image to the size of the guide frame is r, and r≥1. The target area image can be obtained by cropping and zooming in on the full wide-angle image.

In particular, when the equivalent focal length of the telephoto camera is quite different from the equivalent focal length of the wide-angle camera, the field of view of the telephoto camera is also quite different from that of the wide-angle camera. Small, the size of the telephoto frame is small, which is not convenient for users to view the details within the shooting range of the telephoto camera. In this case, the mobile phone can enlarge the target area image and the guide frame proportionally and display them on the preview interface, so that the user can easily know the shooting range and details of the telephoto camera.

For example, in the preview state, if the specification of the guide frame is the default specification, after the mobile phone collects the wide-angle image, the wide-angle image can be cropped according to the guide frame of the default specification to obtain the target area image, and the target area image and The guide frame is proportionally enlarged and displayed on the preview interface. If the specification of the guide frame corresponds to the area/subject selected by the user, after the mobile phone collects the wide-angle image, the wide-angle image can be cropped according to the guide frame to obtain the target area image, and the target area image and the guide frame are proportional to each other. After zooming in, it will be displayed on the preview interface.

Exemplarily, the complete wide-angle image on the preview interface shown in (b) of FIG. 8 can be replaced with the target area image 1007 in the wide-angle image on the preview interface shown in FIG. 10B . Among them, compared with the preview interface shown in (b) in FIG. 8 , on the preview interface shown in FIG. 10B , the target area image and the guide frame are enlarged in equal proportions, and the size of the target area image is the same as the guide frame. The size of the ratio r is greater than 1.

202. After detecting the user's photographing operation, the mobile phone displays a wide-angle image and a guide frame superimposed on the wide-angle image on the photographing interface.

When the user wants to start shooting the target image, a shooting operation can be triggered to make the mobile phone enter the shooting process. For example, after the mobile phone detects the user's operation of clicking the photographing control on the preview interface, it determines that the user's photographing operation is detected, thereby entering the photographing process. For another example, after the mobile phone detects the operation of the user's voice instructing to start taking a photo, it determines that the user's photo-taking operation is detected, thereby entering the photo-taking process. It can be understood that the manner for triggering the mobile phone to enter the photographing process may also include gestures and other manners, which are not limited in the embodiments of the present application.

During the photographing process, a wide-angle image is displayed on the photographing interface, and the wide-angle image is acquired by using a wide-angle camera (ie, the first camera). In some embodiments, during the photographing process, the wide-angle image on the photographing interface is used as a background image to provide the user with a panoramic image, and the user determines the moving path of the telephoto camera according to the range of the wide-angle image, thereby generating multiple telephoto images stitched image. In a possible implementation manner, the wide-angle image is the first frame of image collected by the wide-angle camera after detecting the user's photographing operation. During the photographing process, the mobile phone always displays the first frame of image, not the image collected by the wide-angle camera. Perform refresh display. In another possible implementation, the mobile phone may collect Q (Q is an integer greater than 1) frames of wide-angle images after detecting the user's photo-taking operation, so as to fuse the Q frames of wide-angle images into an initial wide-angle image with better quality image as background image. During the photographing process, the mobile phone always displays the initial wide-angle image, and does not refresh and display the image captured by the wide-angle camera. That is to say, after the mobile phone detects the user's photographing operation, the wide-angle image displayed on the photographing interface and used as a preview remains unchanged.

In other embodiments, during the photographing process, the first wide-angle image serving as the background image on the photographing interface changes, and is an image acquired by the mobile phone through real-time acquisition of the wide-angle camera according to the preset acquisition frame rate 3 . The acquisition frame rate 3 and the acquisition frame rate 1 may be the same or different.

During the photographing process, the mobile phone can also display the above-mentioned guide frame on the photographing interface, and the guide frame is superimposed on the background image in the form of a transparent floating frame. In some embodiments, the mobile phone can continuously display the entire guide frame on the shooting interface; in other embodiments, the mobile phone can only display the unmatched grids in the guide frame on the shooting interface, instead of continuously displaying the entire guide frame .

During the photographing process, the mobile phone can collect telephoto images through the telephoto camera according to the preset acquisition frame rate 4 . The acquisition frame rate 4 and the acquisition frame rate 2 may be the same or different.

During the photographing process, the user can move the telephoto camera by moving the mobile phone, or the user can directly move the telephoto camera, or the mobile phone can automatically control the telephoto camera to move at preset angular intervals. In some embodiments, the mobile phone can also display a telephoto frame on the shooting interface. As the telephoto camera moves, the shooting range of the telephoto camera changes, the content of the telephoto image collected by the telephoto camera also changes accordingly, and the position of the telephoto frame also changes accordingly. The telephoto frame can prompt the user in real time the dynamic change process of the shooting range of the telephoto camera during the movement of the telephoto camera. During the movement of the telephoto camera, the wide-angle image on the shooting interface can be used as a background image to provide the user with a panoramic image, so as to guide the user to move the telephoto camera to match the telephoto frame with the grids in the guide frame one by one.

In other embodiments, if the mobile phone can automatically control the telephoto camera to move, the mobile phone can control the telephoto camera to automatically move according to the grid order according to the grid arrangement in the guide frame, so that the telephoto camera is connected to the guide frame. The grids are matched one by one without the need to display a telephoto frame on the shooting interface. Moreover, even if the wide-angle image used as the background image on the shooting interface changes, because the user does not specifically move the mobile phone, the wide-angle image captured by the mobile phone in real time is basically unchanged or changes little, and the guide frame and grid are on the screen. The position of the telephoto camera is basically unchanged, so the mobile phone controls the telephoto camera to automatically move in the grid order, and the telephoto camera can still be matched with the grid in the guide frame one by one without displaying the telephoto frame.

203. The mobile phone generates a stitched image according to the acquired target telephoto image, and displays a thumbnail of the stitched image on the shooting interface.

During the photographing process, with the movement of the telephoto camera, the mobile phone can collect multiple frames of target telephoto images that match the grid in the guide frame through the telephoto camera. The mobile phone can stitch the target telephoto image to generate the target image.

Wherein, when the content of the telephoto image matches the content of the wide-angle image in a certain grid of the guide frame on the shooting interface, the telephoto image matches the grid, and the telephoto image may be called the target telephoto image. The content of the telephoto image matches the content of the wide-angle image in the grid means that the content of the telephoto image is exactly or substantially the same as the content of the wide-angle image in the grid. For example, the overlap ratio of the telephoto image and the same content of the wide-angle image in the grid is greater than or equal to a preset value of 3 (eg, 80% or 90%, etc.). For another example, the similarity between the histogram of the telephoto image and the wide-angle image in the grid is greater than or equal to the preset value of 4. The histogram here may be a histogram of characteristic parameter values such as brightness. For another example, the telephoto image and the wide-angle image in the grid are in the same transform domain (such as fast Fourier transform (FFT), wavelet transform (WT) or discrete cosine transform (discrete cosine transform), The similarity of DCT), etc.) is greater than or equal to the preset value of 5. The measure of similarity can be the sum of the differences of the scale coefficients corresponding to different values. For another example, the feature matching degree between the telephoto image and the wide-angle image in the grid is greater than or equal to a preset value of 6, for example, the feature includes a corner point, a convolutional neural network feature, or a SIFI feature.

The ways for the mobile phone to obtain the target telephoto image may include: mode 1. After the mobile phone determines that a certain frame of telephoto image matches the grid, the mobile phone captures and obtains the target telephoto image that matches the grid; After matching with the grid, it is determined that this frame of telephoto image is the target telephoto image.

In some embodiments, after detecting the user's photographing operation, the mobile phone can prompt the user of the photographing sequence of the grid in the guide frame, so that the user can move the telephoto camera according to the photographing sequence during the photographing process.

If the mobile phone determines the shooting order of the grid in the guide frame in the preview state, the mobile phone can prompt the user for the shooting order during the shooting process, so that the user can move the mobile phone or directly move the telephoto camera according to the shooting order, so that the The telephoto image captured by the telephoto camera is matched to the grid in the guide frame in the order in which it was taken. In a possible implementation manner, the mobile phone may prompt the user for the complete shooting sequence. In another possible implementation manner, the mobile phone may only prompt a part of the currently required shooting sequence according to the shooting situation, instead of prompting the user for the complete shooting sequence.

If the mobile phone does not determine the shooting order of the grids in the guide frame in the preview state, the user can move the mobile phone or directly move the telephoto camera according to their needs, habits or wishes, so that the telephoto image collected by the telephoto camera and the guide frame grid to match. Among them, for the grid of the same row or column, the telephoto image should be matched with the adjacent grids in the row/column as far as possible. During the photographing process, if a grid has already been matched, the grid will not be matched again.

Among them, due to the small field of view of the telephoto image collected in real time by the telephoto camera, a slight movement of the mobile phone or the telephoto camera may cause the telephoto camera to have a larger angle of view shift, thereby causing a larger content shift in the telephoto image. , so it is not easy to match the content of the next grid to be matched. During the shooting process, the wide-angle image on the shooting interface can provide the user with a panoramic image within a large field of view, so the wide-angle image can provide a reference for the user to move the mobile phone or telephoto camera. The corresponding position of the content of the focus image on the wide-angle image is used to accurately control the movement path (such as the moving direction and amplitude) of the mobile phone or telephoto camera, so that the telephoto image can be quickly and accurately matched with the next grid.

Moreover, the telephoto frame can be displayed in real time on the shooting interface, so as to guide the user to move the shooting range of the telephoto camera and the telephoto frame to the position where the first grid to be matched is located. Exemplarily, the first grid to be matched is the first grid on the left side of the middle row as shown in (a) of FIG. 11 , and the preview interface includes a wide-angle image and a guide frame. After the mobile phone detects that the user clicks the shooting control 1100 on the preview interface shown in (a) in FIG. 11 , it enters the process of taking pictures of the target image and displays (b), (c) or (d) in FIG. 11 . As shown in the shooting interface, the shooting interface includes a wide-angle image, a guide frame and a telephoto frame 1101 . Wherein, the photographing interface shown in (b) of FIG. 11 does not display the prompt information of the photographing sequence. The photographing interface shown in (c) of FIG. 11 displays prompt information of the complete photographing sequence, and the photographing interface shown in (d) of FIG. 11 displays prompt information of part of the photographing sequence. In some embodiments, as shown in (c)-(d) of FIG. 11 , the mobile phone may also prompt the user on the shooting interface: please move the mobile phone in the direction of the arrow and match the grid.

The photographing process will be described below by taking the photographing sequence shown in (c) of FIG. 11 and the prompt information of a part of the photographing sequence displayed on the photographing interface as an example.

In some embodiments, the mobile phone may determine, according to the reference wide-angle image, configuration parameters when the telephoto camera collects the target telephoto images corresponding to different grids. The mobile phone performs automatic exposure (AE) configuration, automatic white balance (AWB) adjustment and dynamic range correction (DRC) on the target telephoto image according to the configuration parameters, so as to obtain the target length. focus image. Since the stitched image is obtained by stitching multiple target telephoto images, the shooting environment of each target telephoto image is different, and only a local range of pictures can be taken, and the overall effect of the global picture with a large field of view cannot be taken into account. Stitched images may be unnatural, sharp, or not smooth in terms of brightness, dynamic range, or color. The mobile phone determines the configuration parameters of the telephoto camera when collecting the target telephoto images corresponding to different grids according to the reference wide-angle image corresponding to the global scope, which can make the overall exposure effect, AWB effect and dynamic range of the stitched image obtained according to the target telephoto image. Try to be consistent with the reference wide-angle image, so that the overall change of the stitched image is smooth and the transition is natural.

Wherein, the reference wide-angle image is used to determine the configuration parameters of the telephoto image of the acquisition target during the photographing process, and the reference wide-angle image is the same frame image during the photographing process. For example, the reference wide-angle image may be the above-mentioned initial wide-angle image, or the first frame image collected by the wide-angle camera after detecting the user's photographing operation.

In some embodiments, in order to ensure better quality of the reference wide-angle image (eg, higher dynamic range, better color configuration, etc.), so that the quality of the stitched image of the telephoto image obtained from the reference wide-angle image is also better, The image may be an initial wide-angle image obtained by fusing the above-mentioned multiple frames of images.

In a possible implementation manner, the reference wide-angle image is correspondingly divided into a plurality of image patches (patches) according to the grid of the guide frame, and each image patch corresponds to a grid of the guide frame. For example, grid 1 corresponds to image block 1, grid 2 corresponds to image block 2, and grid 3 corresponds to image block 3. The mobile phone sets configuration parameters such as AE, AWB or DRC of the target telephoto image of the grid corresponding to the image block according to the parameters such as brightness, color and dynamic range of each image block of the reference wide-angle image. The mobile phone collects the target telephoto image according to the configuration parameters, and realizes the configuration of AE, AWB and DRC according to the wide-angle image to guide the target telephoto image in the corresponding grid.

The first grid to be matched is taken as grid 1, and grid 1 corresponds to target telephoto image 1 as an example for description. For the above method 1, the mobile phone can first collect the telephoto image, and after determining that the telephoto image matches grid 1, set the configuration parameters of the telephoto camera according to the image block 1 corresponding to grid 1 on the reference wide-angle image, so that the telephoto camera can The configuration parameters are collected to obtain the target telephoto image 1 corresponding to the grid 1 . For the above method 2, the mobile phone can set the configuration parameters of the telephoto camera according to the image block 1 corresponding to grid 1 on the reference wide-angle image, so that the telephoto camera can collect telephoto images according to the configuration parameters, until a certain frame of telephoto images and grid 1 After matching, the mobile phone determines that the telephoto image of the frame is the target telephoto image 1 .

The following describes the different configuration parameters of the telephoto camera:

AE configuration parameters:

The mobile phone can perform light metering on the image block 1 on the reference wide-angle image, so as to obtain brightness parameters such as the brightness value, brightness average value, and maximum value of the image block 1, which can reflect the ambient brightness. The exposure meter in the mobile phone is preset with a corresponding relationship between ambient brightness and exposure parameters. The mobile phone can determine the AE configuration parameters of the telephoto image to be collected corresponding to grid 1 according to the ambient brightness situation reflected by the brightness parameter of the image block 1 on the wide-angle image, combined with the exposure table of the telephoto image, for example, determine the exposure time and the exposure time of the telephoto image. Exposure parameters such as ISO. The mobile phone uses this exposure parameter to automatically expose the target telephoto image 1 corresponding to grid 1, so as to improve the exposure effect of the target telephoto image 1.

Since the overall exposure effect of the reference wide-angle image is good, the exposure parameters of the target telephoto image can be determined according to the exposure information of the image blocks in the reference wide-angle image, and the exposure effect of the stitched images obtained from different target telephoto images can be made global as much as possible. Consistent with the reference wide-angle image, so that the overall exposure of the stitched image is also better.

In addition, since the brightness transition between different image blocks on the reference wide-angle image is relatively natural, in some embodiments, the mobile phone can also adjust the target telephoto corresponding to the image block 1 according to the brightness of the adjacent image blocks of the image block 1 The exposure parameters of image 1 make the brightness transition between the brightness of the target telephoto image 1 and the adjacent target telephoto images more natural, and avoid the mobile phone independently automatically exposing the target telephoto image corresponding to each grid, resulting in different The exposure effect of the target telephoto image varies greatly, the brightness process of the spliced image is not smooth and natural, and the splicing traces are obvious, which can improve the overall quality of the spliced image. For example, if image block 1 is dark and image block 2 adjacent to image block 1 is brighter, the mobile phone can increase the exposure parameter of target telephoto image 1 determined according to image block 1 by one brightness level, so that the target telephoto The brightness transition between image 1 and target telephoto image 2 corresponding to image block 2 is relatively natural.

AWB configuration parameters:

The mobile phone can determine the AWB configuration parameters of the target telephoto image 1 corresponding to the grid 1 according to the color distribution of the image block 1 on the reference wide-angle image, for example, determine the WB value of the target telephoto image 1 corresponding to the grid 1, that is, RGB The ratio of the three primary colors. The mobile phone correspondingly adjusts the RGB ratio of the target telephoto image 1 corresponding to the grid 1 according to the ratio, so as to improve the color configuration effect of the target telephoto image.

Among them, since the white balance of the entire image of the reference wide-angle image is good, and the color transition between different image blocks is relatively natural, according to the white balance information of the image blocks of the reference wide-angle image, the white balance information of the image block of the reference wide-angle image can be taken from the global perspective, so that the white balance of the target telephoto image can be achieved from a global perspective. The balance effect is as consistent as possible with the reference wide-angle image, so that the color transition between different target telephoto images included in the spliced image is also more natural, avoiding the mobile phone to perform automatic white balance adjustment for the target telephoto image corresponding to each grid. The white balance effect of different target telephoto images is quite different, the color transition of the stitched image is not smooth and natural enough, and the stitching traces are obvious, which can improve the overall quality of the stitched image.

DRC configuration parameters:

The mobile phone can determine the DRC configuration parameters of the target telephoto image 1 corresponding to the grid 1 according to the dynamic range of the image block 1 on the reference wide-angle image, so as to adjust the dynamic range of the target telephoto image 1 corresponding to the grid 1 accordingly, so that the target The brightness distribution of the telephoto image 1 is relatively consistent with the brightness distribution of the corresponding image block. The dynamic range may include the brightness distribution of different pixels on the image, the brightness difference between different pixels, and the like. For example, the mobile phone can count the dynamic range of image block 1 through the luminance histogram. For example, the image block 1 includes a plurality of pixels whose brightness is lower than 100 and pixels whose brightness is higher than 200. The mobile phone can control the brightness of the pixels of the target telephoto image, so that the target telephoto image includes multiple pixels with a brightness lower than 100 and pixels with a brightness higher than 200, thereby increasing the dynamic range of the target telephoto image.

Among them, since the dynamic range of the reference wide-angle image is relatively large, adjusting the dynamic range of the target telephoto image 1 corresponding to the grid 1 according to the dynamic range of the image block 1 on the reference wide-angle image can increase the dynamic range of the target telephoto image 1. The dynamic range makes the target telephoto image 1 have a larger brightness range, richer levels of light and dark, and can provide more image details in bright and dark parts. The mobile phone configures the DRC parameters of the target telephoto image according to the dynamic range information of the image block of the reference wide-angle image, which can make the brightness distribution transition between different target telephoto images on the spliced image more natural from a global perspective, and avoid the mobile phone to separate each network. The DRC of the corresponding target telephoto image results in a large difference in the dynamic range of different target telephoto images, poor dynamic range of the stitched image, and obvious stitching traces, which can improve the overall quality of the stitched image.

After the telephoto image is matched with the grid 1 in the guide frame, the telephoto camera can move and continue to collect the telephoto image to acquire the target telephoto image 2 that matches the next grid 2 to be matched. Wherein, in order to facilitate the splicing of the target telephoto image matched with the grid, the next grid 2 to be matched is adjacent to at least one grid (eg grid 1 ) that has already been matched.

Subsequently, with the movement of the telephoto camera, the mobile phone can also configure the telephoto camera to collect the target telephoto image corresponding to other grids according to the image block of the reference wide-angle image (for example, the target telephoto image 2 corresponding to grid 2, the grid The configuration parameters of the corresponding target telephoto image 3, etc.) are not repeated here.

In some embodiments, after the target telephoto image is matched with the grid, the mobile phone can also prompt the grid to the user, so that the user can know the location of the currently matched grid and the current shooting progress. Among them, the mobile phone can display the matched grid and other grids in a differentiated manner to prompt the user where the currently matched grid is located, so that the user can know the current shooting progress, the subsequent shooting direction and the telephoto camera. direction of movement. For example, the currently matched grid can be highlighted, bolded, displayed in a color different from other grids, displayed in a specific color, or transformed into a line type different from other grids for display, etc.

For example, during the photographing process, the user moves the mobile phone or the telephoto camera so that the target telephoto image 1 matches the content of the wide-angle image in the leftmost grid 1 in the middle row of the guide frame. As shown in (a) of FIG. 12A , the boundary of grid 1 is changed from a dotted line to a thick solid line, so that it is displayed differently from other grids, so that the user can know that grid 1 is currently matched, and the current shooting progress is the same as that of other grids. Grid 1 corresponds. When the content of the target telephoto image 2 matches the content of the wide-angle image in grid 2 in the guide frame, the mobile phone can also display grid 2 differently from other grids, so that the user can know that grid 2 is currently matched. The current shooting progress corresponds to grid 2.

In some embodiments of the present application, during the photographing process, the mobile phone may spliced with the previously obtained spliced image every time a new target telephoto image is obtained, thereby generating a new spliced image. That is, when all grids in the guide frame are not matched, a stitched image is generated according to the multi-frame target telephoto images corresponding to the matched grids. In other embodiments, the mobile phone generates a stitched image according to the target telephoto image after obtaining the target telephoto image that matches all the grids in the guide frame, or after shooting ends. This embodiment of the present application does not limit the stitching timing of the stitched images.

The process of splicing different target telephoto images to generate a spliced image by a mobile phone may include image registration, image uniformity and color uniformity, and image fusion. Among them, image registration refers to the process of matching and stacking different images. For example, the mobile phone can perform feature extraction on the two frames of images to be registered to obtain feature points, find the matching feature point pairs by performing similarity measurement, and then obtain the image space coordinate transformation parameters through the matching feature point pairs, and finally use the coordinate transformation parameters. Image registration. The mobile phone can calculate the homography matrix of the target telephoto image 2 relative to the target telephoto image 1, so as to register the target telephoto image 2 with the target telephoto image 1 according to the homography matrix. For example, the image registration algorithm may include SURF feature matching algorithm, SKB feature matching algorithm, ORB feature matching algorithm, grid registration algorithm, optical flow registration algorithm or convolutional neural network (artificial intelligence (AI) network) A registration algorithm, etc., the embodiment of the present application does not limit the specific type of the image registration algorithm.

Among them, the image homogenization refers to the brightness equalization of the registered images after image registration, so that the brightness transition between adjacent target telephoto images is natural. Brightness equalization of image stitching is a mature technology, which is not limited here.

Image homogenization means that after image registration, the statistics of brightness histogram and color histogram are performed for the overlapping part of the registered images, and the cumulative distribution of image brightness and image color is obtained by curve fitting (such as spline curve). function. The brightness and color distribution of one image can be used as the criterion, and the brightness and color of other images can be corrected according to its cumulative distribution function of brightness and color to achieve the purpose of uniform light and color; it can also be based on the brightness of multiple images. And color parameters as the common optimization goal, iterative optimization, to achieve the purpose of uniform light and color of all images.

Image fusion refers to the process of extracting the relevant information from each image to the maximum extent and synthesizing it into a high-quality image through image processing and computer technology. For example, the image fusion algorithm may include an alpha fusion algorithm, a Poisson fusion algorithm, or a convolutional neural network fusion (AI fusion) algorithm, and the like. The embodiment of the present application does not limit the specific type of the image fusion algorithm.

For example, after the mobile phone obtains the target telephoto image 2, the target telephoto image 2 and the target telephoto image 1 can be registered, lighted and evened, and fused to form a stitched image with a larger field of view. .

In some embodiments, the target telephoto image 2 and the target telephoto image 1 can be directly registered and fused according to an image registration algorithm, thereby generating a stitched image. In this case, a certain overlap ratio (for example, 20%) is required between the target telephoto image 2 to be registered and the target telephoto image 1, so as to ensure a more accurate registration result. For example, the mobile phone can extract features from the target telephoto image 1 and the target telephoto image 2 respectively according to a preset image registration algorithm, and perform image registration according to the feature matching pair between the two.

In other embodiments, the target telephoto image 2 and the target telephoto image 1 are directly registered and fused to generate a stitched image. Then, the mobile phone can also correct the registered image or the stitched image according to the reference wide-angle image, so as to avoid registration errors caused by less feature point pairs between the telephoto images of the target to be registered. For example, on the spliced image generated after the target telephoto image 2 and the target telephoto image 1 are directly registered and fused, some image contents may be distorted. The coordinates of the distorted image content, so as to correct the distortion and improve the quality of the stitched image.

In other embodiments, the target telephoto image 2 and the target telephoto image 1 are registered and fused according to the reference wide-angle image. In this case, the overlap ratio between the target telephoto image 2 to be registered and the target telephoto image 1 may be small (for example, it may be 10% or 5%, etc.); There is no overlap between the telephoto images 1, and the mobile phone can also accurately register and fuse the target telephoto image 2 and the target telephoto image 1 according to the reference wide-angle image. In this way, the requirements for the target telephoto image to be matched are relatively loose. When the user moves the mobile phone or the telephoto camera, the mobile phone can quickly and easily obtain the target telephoto image, reducing the user's shooting time, and avoiding the need to be matched. Registration errors caused by fewer feature point pairs between quasi-target telephoto images. Moreover, if the overlap ratio between the target telephoto images is small or non-overlapping, the user can complete the photographing process by taking a few target telephoto images, thereby reducing the number of shooting frames and the shooting time, and improving the shooting efficiency and user shooting experience. .

In a possible implementation manner, the target telephoto image 2, the target telephoto image 1 and the reference wide-angle image can be registered and fused together. Referring to FIG. 13A , the target telephoto image 2, the target telephoto image 1 and the reference wide-angle image can extract features respectively and match them in pairs, and more feature matching pairs can be obtained in the overlapping area of the three. In this way, the mobile phone can obtain more and more accurate feature matching pairs, thereby calculating a more accurate homography matrix, and deforming the target telephoto image 2 according to the homography matrix, so as to achieve better registration, stitching and fusion effects. .

In another possible implementation manner, the target telephoto image 2 and the target telephoto image 1 can be registered and fused with the reference wide-angle image respectively, and no registration is required between the target telephoto image 2 and the target telephoto image 1 and fusion. For example, as shown in Fig. 13B, the target telephoto image 2 and the reference wide-angle image can be extracted and matched respectively, so as to calculate the homography matrix, and the target telephoto image 2 can be deformed according to the homography matrix, so that the target telephoto image 2 is longer than the target image. Focus image 1 blends better. In this case, the overlap ratio between the target telephoto image 2 to be registered and the target telephoto image 1 may be small (for example, it may be 10% or 5%, etc.). Alternatively, even if there is no overlap between the target telephoto image 2 and the target telephoto image 1, since there is a common wide-angle image, a good fusion and splicing effect can be achieved. In this way, the requirements for the target telephoto image to be matched are relatively loose. When the user moves the mobile phone or the telephoto camera, the mobile phone can quickly and easily obtain the target telephoto image, reducing the user's shooting time. Moreover, if the overlap ratio between the target telephoto images is small or non-overlapping, the user can complete the photographing process by taking a few target telephoto images, thereby reducing the number of shooting frames and the shooting time, and improving the shooting efficiency and user shooting experience. .

In another possible implementation manner, the target telephoto image 1 may be registered according to the coordinate system of the reference wide-angle image, that is, the target telephoto image 1 may be fitted to the coordinate position of the same content on the reference wide-angle image. Similarly, the target telephoto image 2 can also be registered according to the coordinate system of the reference wide-angle image, that is, the target telephoto image 2 can be fitted to the coordinate position of the same content on the reference wide-angle image. If there is no void between the target telephoto image 1 and the target telephoto image 2 that are attached to the reference wide-angle image, the whole formed by the attachment of the target telephoto image 1 and the target telephoto image 2 is the stitched image.

In addition, the acquisition angle of the telephoto camera may be deflected due to the user's hand shaking or the rotation of the mobile phone, so that the image shift between the target telephoto image 2 and the target telephoto image 1 is large, and it is easy to make the target telephoto image. There is a gap between 2 and the target telephoto image 1, for example, there is a gap between the two frames of images, or the upper/lower edge positions of the two frames of images are misplaced, resulting in a gap, etc.

Among them, the mobile phone can detect the hole through various methods. For example, the target telephoto image 1 and the target telephoto image 2 can both be attached to the position of the same content on the reference wide-angle image to determine whether there is a gap between the attached target telephoto image 1 and the target telephoto image 2 , thereby detecting the presence of voids. For another example, when the target telephoto image 1 and the target telephoto image 2 are distributed left and right, after both the target telephoto image 1 and the target telephoto image 2 are matched to the coordinate system of the wide-angle image, if the horizontal direction of the leftmost pixel in the right image is If the coordinates are greater than the abscissa of the rightmost pixel in the left image, the phone can determine that a hole has been detected.

In addition, the mobile phone can also measure and calculate the spatial positional relationship between the target telephoto image 1 and the target telephoto image 2 when shooting through an inertial measurement unit (IMU) (such as a gyroscope) of the mobile phone, and sequentially to detect the presence of voids.

When the mobile phone determines that there is a hole between the target telephoto image 2 and the target telephoto image 1, the mobile phone can fill the content of the same position on the reference wide-angle image into the hole between the target telephoto image 1 and the target telephoto image 2. . In this way, the whole formed by the target telephoto image 1 , the target telephoto image 2 and the filled portion is a stitched image.

In this way, the mobile phone can fill in the holes between the telephoto images of the target to be spliced according to the reference wide-angle image, and the target telephoto images cannot be registered and spliced due to the holes, and the target telephoto image does not need to be cropped according to the holes. As a result, the image field angle and image resolution are lost during stitching, and the stitched image can have a larger field angle and image resolution. In other existing technical solutions, when the telephoto image to be spliced is shifted up/down due to the angle of manual rotation, cropping is performed at the minimum image height, which will lose image resolution and image view. field angle.

Exemplarily, the reference wide-angle image can refer to the image 1210 shown in FIG. 12B (a), the target telephoto image 1 can refer to the image 1211 shown in FIG. 12B (a), and the target telephoto image 2 can refer to FIG. Image 1212 shown in (b) in 12B. There is a hole 1213 between the target telephoto image 1 and the target telephoto image 2, and the mobile phone can fill the hole 1213 according to the content of the corresponding position of the reference wide-angle image. In this way, as shown in (b) of FIG. 12B , the overall image 1214 formed by the target telephoto image 1 , the target telephoto image 2 and the filling portion is a stitched image.

In addition, when the mobile phone determines that there is a void between the target telephoto image 2 and the target telephoto image 1, in other embodiments, the mobile phone may guide the user to take a new frame of the target telephoto image 2 to fill the gap. Empty. For example, in a possible implementation manner, in the presence of holes, the grid 2 corresponding to the target telephoto image 2 will not be displayed differently from other grids, so as to guide the user to shoot a frame matching grid 2 again. Target telephoto image 2. In another possible implementation manner, the mobile phone prompts the user by displaying information or voice playback, etc., to collect a frame of the target telephoto image again to correspond to the current grid.

Among them, the information on the wide-angle image is used for the hole filling, and the other surrounding areas are the result of fusion and splicing of the target telephoto image 2 and the target telephoto image 1, resulting in obvious differences in the resolution and resolution of the hole filling area from the surrounding area. . Therefore, the mobile phone can perform image super-resolution processing on the cavity area to reduce the resolution difference between the cavity filled area and the surrounding area, so as to achieve a better user experience. In addition, the mobile phone can also use the AI local search method or the AI image restoration method to predict and fill in the hollow area. The embodiments of the present application do not limit the specific methods for filling and image super-resolution.

In some embodiments of the present application, the mobile phone may also perform ghost removal processing on the photographed moving objects during the splicing process.

During the photographing process, since it takes a certain amount of time to photograph multiple frames of the target telephoto image, when there is a moving object within the shooting range, the target telephoto image may appear on the adjacent N frames (N is an integer greater than 2) of the target telephoto image. The image of the object, resulting in ghosts (or ghosts, ghosts, etc.) of moving objects appear on the stitched image. In this case, the mobile phone can also perform ghost removal processing during the photographing process.

The following takes the splicing scene of the target telephoto image 2 and the target telephoto image 1 as an example, and describes the method for removing the ghost image of the moving object through two cases.

(1), slow moving objects

Moving objects move at a slower speed, and moving objects do not appear ghosted on wide-angle images. Moving objects appear on both the adjacent target telephoto image 2 and target telephoto image 1. The images of moving objects on different target telephoto images are distinguished from each other and are not connected, which easily leads to multiple moving objects appearing on the spliced image. Images are ghosts. Among them, the detection of ghost image area (also called motion area) is a relatively mature technology at present. For example, it can be estimated and detected by optical flow method or combined with gyroscope data of mobile phone, which will not be described in detail here.

In this case, the mobile phone can first detect the position of the moving object (the optical flow method, the object detection and tracking method, the method of semantic segmentation, and the method of brightness difference and expansion corrosion can be used). Then, on the spliced image of the target telephoto image 2 and the target telephoto image 1, the mobile phone can retain the complete image of the moving object on the target telephoto image 1, the target telephoto image 2 or the wide-angle image (also called ghost-free image). ), delete the images of moving objects in other areas, and fill the deleted area with the content corresponding to the deleted area on the reference wide-angle image, so as to avoid ghosting of moving objects on the spliced image. Wherein, the wide-angle image used for hole filling may be a recently collected frame of wide-angle image, a certain frame of wide-angle image previously collected by the mobile phone during the photographing process, or a reference wide-angle image, etc., which are not limited in the embodiments of the present application.

For example, as a simple implementation solution, the mobile phone can retain the complete image of the moving object on the wide-angle image. For another example, in order to make the position of the moving object on the final target image closer to what the user sees last, the mobile phone can retain the complete image of the moving object on the last frame of the target telephoto image.

The positions of the moving objects mentioned here may be different on the three images of the target telephoto image 2, the target telephoto image 1 and the wide-angle image. The mobile phone can determine to retain the complete image of the moving object on a certain image according to the corresponding measurement.

Wherein, when the mobile phone retains the complete image of the moving object on the target telephoto image 1 on the spliced image of the target telephoto image 2 and the target telephoto image 1, the mobile phone can delete the image of the moving object on the target telephoto image 2 , and fill the deleted area with the content of the corresponding position on the wide-angle image.

For example, the mobile phone can retain the complete image of the moving object on the target telephoto image 1 and delete the image of the moving object on the target telephoto image 2 before the target telephoto image 1 and the target telephoto image 2 are registered. Then, the mobile phone can register and fuse the target telephoto image 1 and the deleted target telephoto image 2, and use the content of the corresponding position on the reference wide-angle image to fill the deleted blank area to generate a stitched image. .

For another example, the mobile phone can first register and fuse the target telephoto image 1 and the target telephoto image 2 to generate a stitched image. Then, the mobile phone can delete the images of the moving objects on the stitched image (including the images of the moving objects on the target telephoto image 1 and the target telephoto image 2). The mobile phone fits the complete image of the moving object on the target telephoto image 1 to the corresponding position on the stitched image. The mobile phone uses the content of the corresponding position on the reference wide-angle image to fill the hole in the stitched image, and the hole position is the area where the image of the deleted moving object on the target telephoto image 2 is located.

When the mobile phone retains the complete image of the moving object on the target telephoto image 2 on the spliced image of the target telephoto image 2 and the target telephoto image 1, the mobile phone can delete the image of the moving object on the target telephoto image 1, and Fill the deleted area with the content of the corresponding position on the wide-angle image.

When the mobile phone retains the complete image of the moving object on the wide-angle image on the spliced image of the target telephoto image 2 and the target telephoto image 1, the mobile phone can delete the moving object on the target telephoto image 1 and the target telephoto image 2. image, and fill the deleted area with the content of the corresponding position on the wide-angle image.

Exemplarily, reference to the wide-angle image may refer to (a) in FIG. 12C . The target telephoto image 1 can be seen in (b) of FIG. 12C , including an image 1221 of a moving object. The target telephoto image 2 can be seen in (c) of FIG. 12C, including an image 1222 of a moving object. The mobile phone deletes the image of the moving object on the target telephoto image 1 and the target telephoto image 2, and fills the deleted area with the content of the corresponding position on the reference wide-angle image, so as to obtain the image shown in (d) in Figure 12C stitched image.

(2), fast moving objects

When the moving speed of the moving object is fast, the following situations may exist:

Case a. No ghosts of moving objects appear on the reference wide-angle image. The mobile phone can use the same method as the above-mentioned slow-moving object to perform ghost removal processing, and use the content of the corresponding position on the reference wide-angle image to fill the ghost area deleted on the stitched image.

Case b. The ghost of a moving object appears on the reference wide-angle image, and the ghost of a moving object appears on the multi-frame wide-angle images collected in real time by the mobile phone. In this case, the mobile phone can perform motion vector estimation or optical flow estimation in combination with the multi-frame wide-angle images collected during the photographing process, so as to determine the motion trajectory of the moving object, and remove the connected area of the moving object on the wide-angle image to obtain a complete Images of moving objects without ghosting. That is to say, the mobile phone can obtain a wide-angle image without ghost images of moving objects directly or after processing.

When the moving speed of the moving object is fast, the moving object appears on both the adjacent target telephoto image 2 and the target telephoto image 1, and the images of the moving object on different target telephoto images are connected with each other, which easily leads to the splicing of images. Ghosting of moving objects appears.

In this case, in some embodiments, the mobile phone can attach the target telephoto image 1 and the target telephoto image 2 to the ghost-free wide-angle image, thereby marking the target telephoto image 1 and the target telephoto image 2 The connected area of the moving object image on the fitted overall image, that is, the ghost area. Wherein, the wide-angle image without ghost image may be a wide-angle image without ghost image in a certain frame, or a wide-angle image obtained after performing ghost removal processing. When the ghost-free wide-angle image is the reference wide-angle image, the mobile phone can fit the target telephoto image 1 and the target telephoto image 2 to the ghost-free wide-angle image during the registration process, and mark the connected area.

In other embodiments, the mobile phone can perform motion vector estimation and other processing in combination with multiple frames of wide-angle images collected during the photographing process, thereby determining the connected area of the moving object image on the target telephoto image 2 and the target telephoto image 1 . The mobile phone can delete the image area of the moving object on the spliced image of the target telephoto image 2 and the target telephoto image 1, and fill the deleted area according to the content of the corresponding position on the ghost-free wide-angle image to avoid the spliced image. Ghosting of moving objects appears.

Exemplarily, reference to the wide-angle image may refer to (a) in FIG. 12D . The target telephoto image 1 can be seen in (b) of FIG. 12D , including the ghost image 1231 of the moving object. The target telephoto image 2 can be seen in (c) of FIG. 12D , including the ghost image 1232 of the moving object. The mobile phone deletes the ghost image of the moving object on the target telephoto image 1 and the target telephoto image 2, and fills the deleted area with the content of the corresponding position on the reference wide-angle image, so as to obtain (d) in Figure 12D stitched image.

In this way, the mobile phone can eliminate the ghosting of moving objects on the stitched image of the target telephoto image during the photographing process, and present a clear, ghost-free stitched image to the user on the shooting interface, thereby improving the image display effect during the photographing process.

The above description is given by taking the ghost image appearing on the target telephoto image 1 and the target telephoto image 2 as an example. It is understandable that ghost images may also appear on consecutive multi-frame target telephoto images, for example, the ghost area on consecutive multi-frame target telephoto images is a cat whose body is dragged for a long time.

The above description is given by taking the splicing of the target telephoto image 2 and the target telephoto image 1 as an example to obtain a spliced image. Similarly, the mobile phone can use the same stitching method to stitch the subsequent target telephoto image k (k is an integer greater than 2) with the previously generated stitched image, thereby generating a new stitched image. For example, after obtaining the stitched image according to the target telephoto image 2 and the target telephoto image 1, the mobile phone can stitch the target telephoto image 3 with the previously generated stitched image to generate a new stitched image. The splicing of the target telephoto image 3 with the previous spliced image can also be understood as the splicing of the target telephoto image 3 and the target telephoto image 2 adjacent to the target telephoto image 3 in the previous spliced image. The stitching process of the subsequent target telephoto images will not be described in detail here.

Wherein, the non-first frame target telephoto image may be spliced with one or more adjacent target telephoto images. For example, for the target telephoto image corresponding to the grid in the first row and first column of the guide frame, registration and splicing can be performed only with the target telephoto image corresponding to the grid in the first row and second column, or it can be registered with all the target telephoto images corresponding to the grid in the first row and second column. Registration and stitching with its adjacent target telephoto images.

In some embodiments of the present application, after obtaining the target telephoto image 1 that matches the grid 1 (ie, the first grid to be matched), the mobile phone can use the thumbnail of the target telephoto image 1 as the background After the wide-angle images of the images are matched and fused, a thumbnail of the target telephoto image 1 is displayed on the shooting interface, so that the user can see the real picture of the target telephoto image 1 that matches the grid 1 . The thumbnail of the target telephoto image 1 is attached to an area on the wide-angle image that has the same image content as the target telephoto image 1 . Wherein, the position and size of the thumbnail of the target telephoto image 1 may have some deviations from the matched grid 1, or may be exactly the same as the position and size of the matched grid 1. Illustratively, referring to the photographing interface shown in (b) of FIG. 12A , the thumbnail image of the target telephoto image 1 is overlaid on the area of the same image content of the wide-angle image.

The thumbnail of the target telephoto image 1 is an image obtained by down-sampling the target telephoto image 1 collected by the telephoto camera. The wide-angle image displayed by the mobile phone on the preview interface and the shooting interface is usually an image obtained by down-sampling the image collected by the wide-angle camera. Since the thumbnail of the target telephoto image and the wide-angle image displayed on the interface come from different cameras and data sources, the display effect of the interface may be different before and after the thumbnail of the target telephoto image is attached.

In some embodiments, during the photographing process, the mobile phone may display a thumbnail of the spliced image on the photographing interface, so as to facilitate the user to know the current real-time photographing progress. For example, after matching and fusing the stitched image thumbnail with the wide-angle image as the background image, the mobile phone can fit the stitched image thumbnail on the area where the wide-angle image and the stitched image thumbnail have the same image content.

Wherein, the stitched image thumbnail can be obtained in different ways. In one implementation, the mobile phone registers and splices the down-sampled adjacent target telephoto images to obtain spliced image thumbnails, and attaches the spliced image thumbnails to the wide-angle image and the spliced image thumbnails that have the same image above the content area. In this implementation, based on the target telephoto image with lower resolution after downsampling, the processing process of obtaining the stitched image thumbnail by the mobile phone is relatively simple, and the processing time is short, and the stitched image thumbnail can be obtained in real time and displayed on the shooting interface in time to avoid stuttering. The process of obtaining the stitched image thumbnail by the mobile phone is processed in parallel with the process of obtaining the stitched image by the mobile phone.

It should be noted that, according to the above description, the stitched images can be obtained by stitching in real time during the photographing process; they can also be obtained by uniform stitching after obtaining the target telephoto images matching all grids or after shooting. In the solution of displaying stitched image thumbnails on the shooting interface, in order to facilitate the user to know the shooting progress, the mobile phone can obtain and display the stitched image thumbnails in real time according to the target telephoto image obtained each time during the photographing process.

In another implementation manner, the mobile phone downsamples the stitched image to obtain a stitched image thumbnail, and fits the stitched image thumbnail on the wide-angle image on an area having the same image content as the stitched image thumbnail. On the shooting interface, the position and size of the thumbnails of the spliced images may have some deviations from the positions and sizes of the multiple grids that have been matched, or may be exactly the same as the positions and sizes of the multiple grids that have been matched. The mobile phone fits the stitched image thumbnail on the wide-angle image of the shooting interface, allowing users to feel the real, real-time stitching process and stitching progress.

The following describes the display situations of the spliced image thumbnails on the shooting interface, respectively, in the two cases that the background image is fixed and not fixed.

Case 1. The background image is fixed as the reference wide-angle image

In the embodiments of the present application, the user can move the telephoto camera by moving the mobile phone, or the user can directly move the telephoto camera, or the mobile phone can automatically control the telephoto camera to move. In case 1, the background image on the shooting interface is fixed as the reference wide-angle image, the relative position of the guide frame and the background image remains unchanged, the content of the background image corresponding to each grid also remains unchanged, and the telephoto frame is relative to the background. The position of the image can be changed in real time.

Exemplarily, according to the photographing sequence shown in (c) in FIG. 11 , during the photographing process, the mobile phone first matches the grid in the middle row of the guide frame, then matches the grid in the row above the guide frame, and then matches the row below the guide frame. grid. After the mobile phone obtains the target telephoto image corresponding to the middle grid in the middle row, the photographing interface displaying the thumbnails of the spliced images can be referred to (c) in FIG. 12A . Then, the telephoto camera continues to move. As shown in (c) of FIG. 12A , the mobile phone displays a real-time telephoto frame corresponding to the shooting range of the telephoto camera on the shooting interface. After the mobile phone obtains the target telephoto image that matches the grid on the rightmost row above the guide frame, the photographing interface displaying the thumbnails of the stitched images can be seen in (d) of FIG. 12A . After the mobile phone obtains the target telephoto image that matches the leftmost grid in the row below the guide frame, the photographing interface displaying the thumbnails of the spliced images can be referred to (e) in FIG. 12A . After the mobile phone obtains the target telephoto image that matches the last grid in the guide frame, the shooting interface showing the thumbnails of the stitched images can be seen in (f) in Figure 12A, and the mobile phone generates a stitching corresponding to the 3*3 grid Image thumbnail. For another example, in the case where the photographing interface displays the photographing sequence prompt information, the schematic diagram of the photographing interface may refer to (a)-(e) in FIG. 14 .

Case 2. The background image is not fixed and is a wide-angle image collected in real time by a mobile phone

In this case, when the user moves the mobile phone to move the wide-angle camera and the telephoto camera synchronously, the content of the background image corresponding to each grid remains unchanged, but since the background image is not fixed and changes in real time, the guide frame and the background The relative position of the image changes in real time as the content of the background image changes. It is understandable that since the position of the telephoto frame is determined by the lens center of the wide-angle camera and the telephoto camera, the object distance and the size of the field of view, when the object distance is basically unchanged, in view of the wide-angle camera. The relative position of the lens center of the telephoto camera remains unchanged, and the relative position of the telephoto frame and the background image is basically unchanged. For example, when taking a photo, the telephoto frame is always positioned near the middle of the background image.

During the photographing process, when the mobile phone moves, the wide-angle camera also moves, and the content of the wide-angle image as the background image also changes, but the content of the background image corresponding to each grid remains unchanged. It can also be understood that during the photographing process, the corresponding relationship between the grid and the content of the wide-angle image in the grid remains unchanged, and the grid and the content of the wide-angle image serving as the background image are bound.

Mobile phones can bind grids to wide-angle image content in a number of ways. For example, the mobile phone can record the content of the wide-angle image corresponding to each grid after the photo-taking operation is detected; in the subsequent photo-taking process, the mobile phone can use the image matching method to match the grid with the content of the wide-angle image acquired in real time, In this way, the binding of the grid and the content of the wide-angle image is realized. For another example, the mobile phone can record the coordinate position of each grid and the corresponding wide-angle image content after the photo-taking operation is detected; in the subsequent photo-taking process, the mobile phone can determine the translation amount and /Rotation amount, so as to calculate the new coordinate position of each grid and the corresponding wide-angle image content according to the translation amount and /rotation amount, so as to realize the binding of the grid and the wide-angle image content.

For example, when the phone moves to the right, the wide-angle camera and telephoto camera move to the right at the same time, the field of view and image content of the wide-angle image move to the right, and the position of the guide frame on the screen moves to the right with the content of the wide-angle image. Shifting to the left, the position of the telephoto frame relative to the wide-angle image remains largely unchanged. Exemplarily, in the case where the background image is a wide-angle image collected in real time by a mobile phone, after the leftmost grid in the middle row is matched, a schematic diagram of the shooting interface can be seen in (a) in FIG. 13C , and a target is displayed on the shooting interface. Thumbnail of telephoto image 1; after the grids in the middle row are matched, the schematic diagram of the shooting interface can be seen in (b) in Figure 13C, and the stitched image thumbnails are displayed on the shooting interface; After grid matching, the schematic diagram of the photographing interface can be seen in (c) in FIG. 13C , and the stitched image thumbnails are displayed on the photographing interface. As shown in (a)-(c) of FIG. 13C , the wide-angle image is shifted to the right, the guide frame is shifted to the left, and the telephoto frame is substantially located in the middle of the wide-angle image.

It should be noted that, FIG. 12A is described by taking an example that the thumbnail of the target telephoto image is basically aligned with the edge of the matched grid. During the photographing process, as shown in (a)-(c) in Fig. 13C, when the thumbnail of the target telephoto image is deviated from the position of the grid, the edges of the stitched image may be jagged, and Not substantially aligned with the edges of the matched grid.

In addition, in the case where the user directly moves the telephoto camera or the mobile phone automatically controls the movement of the telephoto camera, the telephoto camera and the wide-angle camera may not move synchronously. When the background image is not fixed and is a wide-angle image collected in real time by the mobile phone, for example, when the background image changes in real time due to the shaking of the mobile phone, the relative position of the guide frame and the background image will change in real time with the content of the background image. The content of the background image corresponding to each grid remains basically unchanged, and the relative position of the telephoto frame and the background image can change in real time as the telephoto camera moves.

In the embodiment of the present application, the size, position, and content of the image covered by the stitched image thumbnail and the wide-angle image are basically the same. The size of the stitched image thumbnail is small, which is inconvenient for users to view the image details. In some embodiments, in response to a user's click or long press and other preset operations, the mobile phone can zoom in and display a thumbnail of the stitched image on the shooting interface, so that the user can clearly see the target telephoto stitched image captured by the telephoto camera at any time. The specific details of the object.

Exemplarily, as shown in (a) of FIG. 15A , after the mobile phone detects that the user clicks the thumbnail of the stitched image on the shooting interface, as shown in (b) of FIG. 15A , the stitched image is enlarged and displayed on the shooting interface. Thumbnail. After the mobile phone detects the user's operation of clicking the stitched image again or detects the user's return operation, the mobile phone resumes displaying the stitched image thumbnail as shown in (a) of FIG. 15A .

In other embodiments, in order to avoid the size of the stitched image thumbnails being small (especially when the equivalent focal lengths of the telephoto camera and the wide-angle camera are quite different, for example, the ratio of the equivalent focal lengths is greater than a certain preset value), resulting in It is inconvenient for users to view the details of the images, and the mobile phone can automatically zoom in and display the thumbnails of the stitched images on the shooting interface. Similarly, the mobile phone can also enlarge and display the thumbnail of the target telephoto image 1 on the shooting interface. The magnified display multiple may be a default value, or may be related to the ratio of the equivalent focal length of the wide-angle camera to the telephoto camera, or may be a value set by the user, which is not limited.

In order to conveniently guide the user to move the mobile phone or the telephoto camera, the thumbnail of the target telephoto image 1 after enlarged display or the thumbnail of the spliced image cannot block the grid to be matched. Exemplarily, referring to (a)-(b) in FIG. 15B , when shooting in a grid order from left to right, the thumbnail of the zoomed-in target telephoto image 1 and the thumbnail of the stitched image on the shooting interface. The right side is aligned with the right side of the recently matched grid, that is, aligned with the left side of the grid to be matched, so as to avoid the thumbnail of the target telephoto image 1 and the thumbnail of the spliced image from blocking the grid to be matched on the right side. When shooting in grid order from top to bottom, the lower side of the thumbnail of the zoomed-in target telephoto image 1 or the thumbnail of the stitched image is aligned with the lower side of the grid that has been matched recently, that is, the grid to be matched align the upper side of the target telephoto image 1 and the thumbnail of the spliced image to block the grid to be matched on the lower side. For another example, the grid in the middle row of the guide frame usually corresponds to the image content that the user most wants to capture, so the thumbnail of the target telephoto image 1 and the thumbnail of the spliced image try not to block the unmatched grid in the middle row.

In some other embodiments, the mobile phone displays the target area image corresponding to the guide frame on the wide-angle image on the shooting interface, instead of displaying the complete wide-angle image. The ratio of the size of the target area image to the size of the guide frame is r, and r≥1. The target area image can be obtained by cropping and zooming in on the full wide-angle image. In this way, the size of the guide frame and the grid displayed on the shooting interface is larger, which is convenient for the mobile phone to perform mobile shooting and matching according to the larger size grid.

In particular, when the equivalent focal length of the telephoto camera is quite different from the equivalent focal length of the wide-angle camera, if a complete wide-angle image and a corresponding guide frame are displayed on the shooting interface, the guide frame, grid, and target telephoto image 1 Thumbnails/stitched image thumbnails are small in size and inconvenient for users to view. The mobile phone can enlarge the target area image, the guide frame, the thumbnail of the target telephoto image 1/stitched image thumbnail, etc., and display it on the shooting interface, so that the user can view the thumbnail of the larger target telephoto image 1/ The specific content of the stitched image thumbnail.

Exemplarily, the complete wide-angle image on the shooting interface shown in (d) of FIG. 12A may be replaced with the target area image 1500 in the wide-angle image on the shooting interface shown in FIG. 15C . Wherein, compared with the photographing interface shown in (d) in FIG. 12A , on the photographing interface shown in FIG. 15C , the target area image, the guide frame and the stitched image thumbnails are enlarged in equal proportions, and the target area is enlarged in equal proportions. The ratio r of the size of the image to the size of the guide frame is greater than 1.

In the solution described in the above embodiment, during the photographing process, the mobile phone displays the thumbnail of the target telephoto image 1 and the thumbnail of the spliced image on the photographing interface. In some other embodiments, during the photographing process, the mobile phone may not display the thumbnail of the target telephoto image 1 and the thumbnail of the spliced image, but only display a splicing frame on the wide-angle image, and the splicing frame is the target telephoto image 1 , or the border of a stitched image thumbnail. In this way, the mobile phone can remind the user of the current shooting progress through the splicing box, and does not need to acquire and display the thumbnail of the target telephoto image 1 and the thumbnail of the spliced image, which can reduce the processing load of the mobile phone. In other embodiments, during the photographing process, the mobile phone may not display the thumbnail of the target telephoto image 1 and the thumbnail of the stitched image, but only highlight (for example, highlight or bold) the matched grid or the stitched image. Matches the border of the grid. In this way, the mobile phone can remind the user of the current shooting progress by highlighting the stitched grid, and does not need to acquire and display the thumbnail of the target telephoto image 1 and the stitched image thumbnail, which can reduce the processing load of the mobile phone.

In addition, in some embodiments, during the photographing process, the mobile phone can also give a corresponding prompt to the user according to the current real-time photographing situation. For example, when the phone is moving too fast for the telephoto image to match the grid, the phone can prompt the user to move the phone slowly. For another example, when the moving direction of the mobile phone is opposite to the direction indicated by the shooting sequence, or when the mobile phone moves in the direction of the matched grid, the mobile phone may prompt the user to move the mobile phone in the indicated direction. Or, the mobile phone directly terminates the shooting process, and generates the final target image according to the acquired target telephoto image. For another example, when the telephoto frame is far away from the top of the grid to be matched. The mobile phone can prompt the user to move the mobile phone downwards; when the telephoto frame deviates far below the grid to be matched, the mobile phone can prompt the user to move the mobile phone upwards.

It is worth noting that, in the process of taking pictures, the wide-angle image collected by the mobile phone in real time can be used to display as a background image when the background image is not fixed; on the other hand, it can also be used to move moving objects. Vector estimation to determine the connected regions of moving objects and assist in ghost removal. In the process of taking pictures, the reference wide-angle image with better quality can be used as a reference to perform AE, AWB and DRC configuration on the target telephoto image, used as a reference to register and fuse multiple target telephoto images, and use It is used as a reference to perform processing such as hole filling.

204. After the mobile phone determines that the shooting is completed, a target image is generated according to the stitched image.

After the mobile phone determines that the shooting is completed, the target image can be generated according to the stitched images obtained from the multiple frames of target telephoto images.

In some embodiments, after the shooting, the mobile phone can also perform processing such as ghost removal, dynamic range enhancement, or hole filling on the stitched image before generating the target image according to the stitched image, so as to improve the quality of the stitched image. The spliced image of the target image is generated and the quality of the target image is improved.

Go ghosting:

If the mobile phone does not perform ghost removal processing on moving objects during the photographing process, the mobile phone can perform ghost removal processing on the stitched image after the shooting ends and before generating the target image according to the stitched image, so that according to the stitching after ghost removal image to generate the target image. For example, when there are moving objects on adjacent multiple frames of target images, the mobile phone can process the ghosts on the stitched images obtained after shooting together. In a possible implementation solution, in order to make the position of the moving object on the final target image closer to what the user sees last, the mobile phone can retain the complete image of the moving object on the last frame of the target telephoto image. For another example, when the mobile phone generates a stitched image after the shooting, the mobile phone can process ghost images on the stitched image together.

In this case, similar to the above-mentioned method for removing ghost images in the photographing process, for slow-moving objects, the mobile phone can delete the image of the moving object on the spliced image, and keep the target telephoto image of a certain frame or the wide-angle image of a certain frame of the image. A complete image of a moving object, and fill the hole with the content of the corresponding location on the wide-angle image. For fast-moving objects, the mobile phone can perform motion vector estimation based on multiple frames of wide-angle images, so as to remove the connected areas of moving objects on the wide-angle images to obtain a complete image of moving objects without ghosts. The mobile phone deletes the image area of the moving object on the spliced image, and fills the deleted area with the content of the corresponding position on the wide-angle image without ghosting.

This embodiment of the present application does not limit whether the ghost removal process is performed during the shooting process or after the shooting ends.

Dynamic range enhancement:

In the embodiment of the present application, the stitched image of the multi-frame target telephoto images may have a smaller dynamic range, fewer bright layers, a smaller brightness range, insufficient contrast between bright and dark, and insufficient details of dark and bright parts. In some embodiments, the mobile phone can enhance the dynamic range of the entire stitched image after the shooting ends and before generating the target image according to the stitched image, for example, a method of enhancing the dynamic range of the photographed photo (HDRnet) can be used to make the stitched image bright The layers are richer and the range of light and dark is larger, and then the target image is generated according to the stitched image with enhanced dynamic range.

For example, in the process of enhancing the dynamic range, the mobile phone can process the stitched images through the convolutional neural network (AI network) to directly obtain the effect of high dynamic range, thereby obtaining the target image with high dynamic range.

In the process of enhancing the dynamic range, the mobile phone can also adjust the brightness distribution of the spliced image according to the brightness distribution of the reference wide-angle image, so as to increase the brightness range of the spliced image, thereby enhancing the dynamic range of the spliced image. For example, according to the luminance histograms of the reference wide-angle image and the stitched image, the luminance values in the corresponding regions of the reference wide-angle image and the stitched image range from 30 to 250, and the luminance value of the stitched image ranges from 180 to 230. The mobile phone can adjust the brightness value of some pixels whose brightness value is close to 180 on the spliced image (such as randomly selected pixels or pixels in the edge part, etc.) to 30-180; The brightness value of the pixel is adjusted to between 230-250; thereby increasing the brightness range of the spliced image and enhancing the dynamic range of the spliced image.

In addition, the mobile phone can also extract the high-frequency component image of the reference wide-angle image, and fuse the high-frequency component image and the stitched image to increase the high-frequency detail of the stitched image (also called high-frequency component fusion). For example, the high-frequency component image includes pixels in the upper edge portion of the reference wide-angle image or pixels in the transition portion of the edges of different objects, and the like. The mobile phone can use wavelet transform or deep learning to extract the high-frequency component image of the reference wide-angle image, so as to extract high-frequency details with large frequency and texture changes and superimpose them on the stitched image to enhance the high-frequency details of the stitched image and enhance the stitching. Image and target image clarity and detail.

In the process of enhancing the dynamic range, the mobile phone can also synthesize multiple frames of wide-angle images with high dynamic range, and apply the synthesized high dynamic range effect to the stitched image through the algorithm of uniform light and color. The dynamic range information performs uniform light and color processing on the stitched image, thereby enhancing the dynamic range of the stitched image. In addition, the mobile phone can also superimpose the details of the stitched image onto the wide-angle image with high dynamic range effect, and finally obtain the target image with high dynamic range and rich texture details.

For example, in some embodiments, the mobile phone can use different exposures of multiple frames of wide-angle images to synthesize a wide-angle image with high dynamic range, and then perform uniform light and color processing on the stitched image based on the high dynamic range of the wide-angle image, or The processing of style transfer, so that the stitched image also has the effect of high dynamic range.

For example, in some embodiments, a cell phone may utilize different exposures of multiple frames of wide-angle images to synthesize a wide-angle image with high dynamic range. The phone can then extract details and textures from the stitched image, overlay it onto a wide-angle image with high dynamic range, and fuse details, color, and brightness, and finally obtain a target image with high dynamic range and high-definition detail.

Wherein, the algorithm for enhancing the dynamic range is not limited to the traditional algorithm or the algorithm of the convolutional neural network. The algorithm for synthesizing a wide-angle image with high dynamic range from multiple frames of wide-angle images is not limited to traditional algorithms or convolutional neural network algorithms.

Hole filling:

In the embodiments of the present application, there may be some holes between different target telephoto images on the stitched image. In some embodiments, the mobile phone can also perform hole filling on the stitched image, and then generate the target image according to the stitched image after the holes are filled. Since there may be misalignment between the target telephoto images, the edges of the stitched images may not be smooth enough, and there may be empty edges between image parts from different target telephoto images. Therefore, the mobile phone can also fill in the empty edges according to the reference wide-angle image. For neat, beautiful rectangular or square stitched images. In this way, the mobile phone fills the empty edge according to the corresponding content of the reference wide-angle image, and does not need to crop the target telephoto image according to the empty edge, which leads to the loss of the image field angle and image resolution during splicing, which can make the spliced image more efficient. Large field of view and image resolution, so that the target image generated from the spliced image has a large field of view and image resolution. In addition, for the filling of holes and empty edges, the mobile phone can use the corresponding areas of multiple frames of wide-angle images for high-resolution synthesis, or use the image super-resolution algorithm to process the corresponding areas of a single frame of reference wide-angle images. The filling of holes and empty edges can have a higher resolution and improve the user experience.

Exemplarily, the stitched image with empty edges obtained by the mobile phone according to multiple frames of target telephoto images can refer to the image framed by the solid line shown in (a) in FIG. 15D . After the mobile phone performs hole filling processing according to the reference wide-angle image , the obtained stitched image can refer to the image framed by the solid line shown in (b) of FIG. 15D .

It should be noted that in the process of taking pictures, the reference wide-angle image with better quality is used as a reference to perform AE, AWB adjustment, and DRC configuration on the target telephoto image, which is used as a reference to register and fuse multi-frame target telephoto. The image is used as a reference for hole filling, and is used as a reference to enhance the dynamic range of the entire stitched image after shooting. In addition, in the whole photographing process, each frame of target telephoto image is processed according to the same reference wide-angle image, which can make the effect of each target telephoto image and the reference wide-angle image as consistent as possible, so that the final stitched image and The overall effect of the target image is more consistent with the image effect of the reference wide-angle image, so that the final stitched image and the target image as a whole are more natural, with smooth transition and better quality.

In addition, since the mobile phone generates the target image from the stitched image, it may take a certain amount of time to perform processing such as dynamic range enhancement. Therefore, in some embodiments, the mobile phone may prompt the user that the target image is being generated during the processing, so as to prevent the user from mistakenly thinking that the mobile phone is stuck or other abnormal conditions have occurred. For example, the mobile phone can prompt the user through text prompts, voice prompts, or a rotating circle logo. Exemplarily, referring to FIG. 16 , the mobile phone can prompt the user that the target image is currently being generated through text information: processing, please wait; and a rotating circle mark.

Among them, there may be various methods for the mobile phone to determine the end of shooting, and there may also be various methods for the mobile phone to generate the target image according to the spliced image.

In some cases, before the grids in the guide frame have not been photographed (that is, the grids in the guide frame have not been matched), the mobile phone determines that the photographing is ended after detecting the user's operation of stopping photographing. Exemplarily, after the mobile phone detects that the user clicks the operation of the stop photographing control 1200 as shown in (e) in FIG. 12A , it determines that the photographing is ended. It can be understood that the operation to stop taking pictures may also be other gesture operations or user voice instruction operations, and the embodiment of the present application does not limit the operations that trigger the mobile phone to end the taking pictures process. In other cases, before the grid in the guide frame is photographed, if the moving direction of the mobile phone seriously deviates from the indicated direction of the guide frame (for example, the deviation range between the moving direction and the indicated direction of the guide frame is greater than or equal to a certain preset value) ) or move out of the guide frame, the shooting ends automatically.

In some embodiments, if the mobile phone determines that the shooting is completed before the grid in the guide frame is completed, the target image is generated by intercepting the part corresponding to the guide frame from the reference wide-angle image.

In other embodiments, if the mobile phone determines that the shooting is completed before the grid in the guide frame is completed, and the shooting process is performed according to the rows of the grid, the mobile phone discards the entire row of grids that are not completed on the spliced image. For the part of grid shooting, a target image is generated according to the stitched images corresponding to the entire row of grids that have been captured, so that the target image is an image corresponding to the entire row of grids. Exemplarily, in the case shown in (e) in FIG. 12A , if the mobile phone detects the user’s operation to stop taking pictures, the stitched images corresponding to the grids in the middle row and the upper row of the guide frame are generated as shown in FIG. 17 . The target image shown in (a).

Similarly, if the mobile phone determines that the shooting is over before the grid in the guide frame has been shot, and the photo is taken according to the columns of the grid, the mobile phone removes the part of the stitched image that has not been photographed in the entire column, according to the A target image is generated from the stitched images corresponding to the entire column of grids that have been captured, so that the target image is an image corresponding to the entire column of grids.

In other embodiments, if the mobile phone determines that the shooting is completed before the grid in the guide frame is completed, and the shooting process is performed according to the rows of the grid, the mobile phone supplements the stitched image according to the reference wide-angle image. The image corresponding to the latest row of grids, thereby generating the target image. Similarly, if the mobile phone determines that the shooting is over before the grid in the guide frame has been shot, and the shooting process is carried out according to the columns of the grid, the mobile phone will supplement the latest grid column on the stitched image according to the reference wide-angle image. corresponding image to generate the target image.

In some other embodiments, if the mobile phone determines that the shooting is completed before the grid in the guide frame is completed, the photographing process is abnormal, and the mobile phone does not generate the target image. In addition, the mobile phone can also prompt the user that the shooting is stopped or the shooting is abnormal.

In other cases, after all the grids in the guide frame are captured, the phone automatically ends capturing and generates a target image based on the stitched image. In some embodiments, the size of the target image may be the same as the size of the guide frame. If the size of the stitched image is inconsistent with the size of the guide frame due to the dislocation between the target telephoto images during stitching, the mobile phone can crop or fill the stitched image according to the size of the guide frame (fill according to the reference wide-angle image). ), so that the size of the stitched image is consistent with the size of the guide frame. Exemplarily, the target image generated by the mobile phone can be referred to (b) in Fig. 17 . In other embodiments, if the edges of the spliced images are not aligned due to the misalignment between the target telephoto images during splicing, the mobile phone can fill the spliced images according to the reference wide-angle image, so as to obtain rules such as rectangles or squares. The shape of the target image, the position and size of the target image and the position and size of the guide frame may differ.

Among them, the resolution of the wide-angle image collected by the wide-angle camera and the resolution of the telephoto image collected by the telephoto camera are both larger. Since the field of view of the wide-angle camera is much larger than that of the telephoto camera, the number of pixels corresponding to the telephoto image per unit field of view is much larger than the number of pixels corresponding to the wide-angle image per unit field of view. For example, the resolution of the wide-angle image collected by the wide-angle camera is 4000*3000, that is, 12 million pixels, and the resolution of the telephoto image collected by the telephoto camera is 3264*2448, that is, 8 million pixels. If the field of view of the wide-angle camera is 5 times that of the telephoto camera, the number of pixels corresponding to the telephoto image per unit field of view is equal to the number of pixels corresponding to the wide-angle image per unit field of view. 5 times the number. That is, the resolution of the telephoto image is 800/(1200/5)=3.33 times the resolution of the wide-angle image within a unit angle of view. That is to say, the resolution of the telephoto image is larger than that of the wide-angle image within a unit field of view, and compared with the wide-angle image, the telephoto image is clearer and the details are clearer.

Since the resolution of the telephoto image is greater than the resolution of the wide-angle image within a unit field of view, the target image generated by the mobile phone based on the spliced image of the multi-frame target telephoto image has higher definition and clearer details than the wide-angle image. Shoot better. In addition, the mobile phone can obtain a high-definition target image with a large field of view by stitching multiple frames of target telephoto images.

It should also be noted that the telephoto image in the foregoing embodiment is a single-frame image collected by the telephoto camera. In some other embodiments, the telephoto images involved in the foregoing embodiments may also be multiple frames (for example, 2 frames or 3 frames, etc.) of telephoto images collected by a telephoto camera and generated after registration and fusion of better quality. a frame of image.

In some embodiments of the present application, the target image obtained by splicing the target telephoto image and obtained by splicing the target telephoto image stored in the mobile phone can be specially identified from other images, so as to facilitate the user to intuitively know this type of image. For example, referring to (a) in FIG. 18 , the target image obtained by the mobile phone has a character identification 1801 of "cp" displayed. For another example, referring to (b) in FIG. 18 , a specific symbol 1802 is displayed on the target image obtained by the mobile phone.

Scenarios that do not display the bootstrap frame:

The photographing method described in the above embodiments may be referred to as a solution for displaying a guide frame. Some other embodiments of the present application provide another shooting method. Different from the above-mentioned embodiments, the mobile phone does not display a guide frame on the preview interface and the shooting interface.

The differences from the above solution of displaying a guide frame are mainly described in detail below, and the similarities are not described in detail. As shown in Figure 19, the shooting method may include:

1900. The mobile phone starts the camera function.

In some embodiments, after the mobile phone starts the photographing function, the mobile phone can take a photograph through the solution provided by the embodiments of the present application without displaying the guide frame.

In other embodiments, after the mobile phone activates the photographing function and enters the target photographing mode, the mobile phone can take photographs by using the solution provided by the embodiments of the present application without displaying the guide frame. For example, the target photographing mode is the aforementioned wide-view mode.

For other relevant descriptions of step 1900, reference may be made to the description in the above-mentioned step 200, which will not be repeated here.

1901. The mobile phone displays a wide-angle image and a telephoto frame on the preview interface.

In the scheme where the guide frame is not displayed, the mobile phone displays a wide-angle image on the preview interface. Different from the above-mentioned scheme of displaying the guide frame, in the scheme of not displaying the guide frame, the mobile phone displays a telephoto frame on the preview interface to facilitate the user to know the real-time shooting range of the telephoto camera; and the mobile phone does not display the guide on the preview interface. frame. And, in the solution where the guide frame is not displayed, the mobile phone is on the preview interface. Exemplarily, see (a) in FIG. 20 , the preview interface includes a wide-angle image and a telephoto frame.

For other related descriptions in step 1901, reference may be made to the description in the above-mentioned step 201, and details are not repeated here.

1902. After detecting the user's photographing operation, the mobile phone displays a wide-angle image and a telephoto frame superimposed on the wide-angle image on the photographing interface.

Different from the above solution of displaying the guide frame, in step 1902, after the mobile phone detects the user's photographing operation, it does not display the guide frame on the shooting interface, and the mobile phone displays the wide-angle image and the telephoto frame on the shooting interface.

For other related descriptions in step 1902, reference may be made to the descriptions in the foregoing step 202, and details are not repeated here.

1903. The mobile phone generates a stitched image according to the acquired target telephoto image, and displays a thumbnail of the stitched image on the shooting interface.

In the solution where the guide frame is not displayed, when the user wants to shoot the target image, he can refer to the wide-angle image on the preview interface to move the mobile phone or telephoto camera to compose the picture, so that the shooting range of the telephoto camera and the telephoto frame are located in the position where the user wants to The starting position of the captured area. Then, the user can trigger a photographing operation. After the mobile phone detects the user's photographing operation, it can determine the configuration parameters of the telephoto camera according to the image blocks of the reference wide-angle image corresponding to the shooting range of the telephoto camera. For example, the configuration parameters may include configuration parameters such as AE, AWB, or DRC. The mobile phone collects the telephoto image through the telephoto camera according to the configuration parameters such as AE, AWB or DRC, and obtains the first frame of the target telephoto image, that is, the target telephoto image 1. For the process of the mobile phone determining configuration parameters such as AE, AWB, and DRC, reference may be made to the relevant description in the solution for displaying a guide frame above, which will not be repeated here.

It should be noted that, different from the scheme in which the guide frame is displayed, the target telephoto image needs to match the grid. In the scheme without the guide frame, since the guide frame is not displayed on the shooting interface, the target telephoto image does not match the guide frame. box to match the grid. For example, when a certain frame of telephoto image collected by the mobile phone matches the target telephoto image 1 , the frame of telephoto image may be the target telephoto image 2 . The matching of the target telephoto image 2 with the target telephoto image 1 includes: the deviation between the abscissa range of the target telephoto image 2 and the abscissa range of the target telephoto image 1 is less than or equal to the preset threshold 1, or the target length The deviation between the ordinate range of the focus image 2 and the ordinate range of the target telephoto image 1 is less than or equal to the preset threshold 2 . That is to say, the target telephoto image 2 and the target telephoto image 1 are basically distributed side-by-side in left and right, or basically distributed in up and down.

The matching of the target telephoto image 2 and the target telephoto image 1 also includes: the overlapping area between the target telephoto image 2 and the target telephoto image 1 is greater than or equal to a preset value of 8, or the target telephoto image 2 and the target telephoto image 1. The gap between 1 is less than or equal to the preset value of 9, etc. For example, when the mobile phone determines that the matching content between the collected telephoto image and the reference wide-angle image and the matching content of the target telephoto image 1 on the reference wide-angle image are greater than or equal to the preset value of 8, the mobile phone determines that the matching content between the target telephoto image 1 and the reference wide-angle image is greater than or equal to the preset value of 8. The target telephoto image 2 is matched with the focal image 1.

In the solution where the guide frame is not displayed, since the guide frame is not displayed on the shooting interface, the user cannot move the mobile phone or the telephoto camera according to the instructions of the guide frame. In this case, the user can move the mobile phone or directly move the telephoto camera according to the position of the obtained image content of the target telephoto image 1 relative to the global wide-angle image, as well as his own needs, habits or wishes. During the movement of the telephoto camera, the mobile phone can adjust the configuration parameters of the telephoto camera in real time according to the image blocks of the reference wide-angle image corresponding to the shooting range of the telephoto camera, so as to collect the target telephoto image according to the configuration parameters. Image-guided target telephoto images for configurations such as AE, AWB, and DRC.

In some embodiments, during the photographing process, the mobile phone may perform splicing with the previously obtained spliced image every time a new target telephoto image is obtained, thereby generating a new spliced image. For example, the mobile phone can stitch the target telephoto image 1 and the target telephoto image 2 to obtain a stitched image. Subsequently, after a certain frame of telephoto image matches the obtained target telephoto image, the mobile phone obtains a new target telephoto image, and the new target telephoto image can be stitched with the previously obtained stitched image to obtain a new stitched image. . In other embodiments, the mobile phone generates a stitched image according to the target telephoto image after the shooting is completed. This embodiment of the present application does not limit the stitching timing of the stitched images. For the image splicing process, reference may be made to the description in the solution for displaying the guide frame above, which will not be repeated here.

In some embodiments of the present application, similar to the solution for displaying the guide frame, after obtaining the target telephoto image 1, the mobile phone can display the thumbnail of the target telephoto image 1 on the shooting interface. Exemplarily, for a photographing interface displaying a thumbnail image of the target telephoto image 1, reference may be made to (b) in FIG. 20 . During the photographing process, the mobile phone can display the stitched image thumbnails on the photographing interface to prompt the user of the current real-time photographing progress. Exemplarily, reference may be made to (c)-(d) of FIG. 20 for the photographing interface displaying the thumbnails of the stitched images. In some other embodiments, during the photographing process, the mobile phone displays a splicing frame on the photographing interface. In other embodiments, in order to avoid the small size of the spliced image thumbnails, which is inconvenient for the user to view the image details, the mobile phone can automatically enlarge and display the thumbnail of the target telephoto image 1 and the spliced image thumbnail on the shooting interface. Exemplarily, the photographing interface for zooming in and displaying the thumbnail of the target telephoto image 1 may refer to (a) in FIG. 21 , and the photographing interface for zooming and displaying the thumbnail of the spliced image may refer to (b)-(c) in FIG. 21 . In some other embodiments, the mobile phone displays the target area image corresponding to the guide frame on the wide-angle image on the shooting interface, instead of displaying the complete wide-angle image. The ratio of the size of the target area image to the size of the guide frame is r, and r≥1. The target area image can be obtained by cropping and zooming in on the full wide-angle image.

1904. After the mobile phone determines that the shooting is over, a target image is generated according to the stitched image.

Similar to the above step 204, in step 1904, after the shooting is completed, the mobile phone can also perform processing such as ghost removal, dynamic range enhancement or hole filling on the stitched image before generating the target image according to the stitched image, so as to improve the quality of the stitched image. , so as to generate a target image according to the processed spliced image and improve the quality of the target image, which will not be repeated here.

In the solution in which the guide frame is not displayed, the mobile phone ends the photographing after detecting the user's operation of stopping the photographing. Alternatively, the mobile phone can preset a maximum number of shooting frames, and the mobile phone automatically ends the shooting after acquiring the target telephoto image with the preset number of frames. After the mobile phone finishes shooting, the target image is generated according to the stitched image of the target telephoto image.

In some embodiments, if the edges of the stitched image are not aligned, the mobile phone can minimize the edges of the stitched image to obtain a stitched image with a regular shape such as a rectangle or a square, so as to generate a target image according to the stitched image. Exemplarily, the schematic diagram of the stitched image thumbnails obtained by the mobile phone according to the target telephoto camera can refer to (a) in Figure 22, and the schematic diagram of the new stitched image thumbnails obtained by the mobile phone after cropping can refer to Figure 22 (b). ).

In other embodiments, if the edges of the stitched images are not aligned, for the vacant part, the mobile phone can fill the vacant part according to the content of the corresponding position on the reference wide-angle image, so as to obtain a stitched image of a regular shape such as a rectangle or a square, so as to obtain a stitched image with a regular shape such as a rectangle or a square. Stitch the images to generate the target image. Exemplarily, the schematic diagram of the stitched image thumbnails obtained by the mobile phone according to the target telephoto camera can refer to (a) in FIG. (b) in 23A.

In other embodiments, regardless of whether the edges of the stitched image are aligned, the mobile phone generates a target image according to the stitched image, and the size of the target image is consistent with the size of the stitched image.

In some other embodiments, in the solution where the guide frame is not displayed, the size of the target image obtained by the mobile phone may not be the size desired by the user. Therefore, in a possible technical solution, the user can also edit the generated target image in the gallery to obtain the target image of the desired size.

In another possible technical solution, the user can also edit the stitched image to obtain a stitched image of an ideal size, thereby generating a target image of an ideal size. Exemplarily, see (a) in FIG. 23B , the mobile phone may prompt the user on the shooting interface after the shooting is completed, please set the range of the target image on the stitched image. After the mobile phone detects the user's operation of setting the dotted frame 2300 and clicks the OK control, the stitched image is cropped according to the dotted frame 2300, thereby generating the target image within the range indicated by the user as shown in (b) in FIG. 23B .

In the scheme of not displaying the guide frame, since the resolution of the telephoto image is larger than that of the wide-angle image within the unit field of view, the target image generated by the mobile phone based on the spliced image of the multi-frame target telephoto images is smaller than that of the wide-angle image. The sharpness is higher, the details are clearer, and the shooting effect is better. In addition, the mobile phone can obtain a high-definition target image with a large field of view by stitching multiple frames of target telephoto images.

In addition, in the solution where the guide frame is not displayed, the user can move the mobile phone or telephoto camera to collect telephoto images according to their own needs, so as to obtain the target telephoto image of the user's desired position, and then according to the user's desired position. Image Get a stitched image of any size or shape the user wants. The mobile phone can obtain the target image of the corresponding size or shape according to the stitched image of any specification, which can facilitate the user to shoot and obtain the target image of various sizes in wide format, square format or panoramic format.

Hybrid zoom scheme:

Some other embodiments of the present application also provide a shooting method, which can obtain a stitched image with a larger and clearer field of view by stitching a telephoto image with a smaller field of view, and then crop the stitched image to obtain different A clear target image corresponding to the target zoom magnification. In this solution, the mobile phone does not need to use digital zoom to enlarge the image, so the high resolution of the telephoto camera and the high definition of the telephoto image can be retained, and the zoom effect of optical zoom can be realized. This scheme combines stitching and cropping of telephoto images, and can also be called a hybrid zoom scheme.

The differences from the above solution of displaying a guide frame are mainly described in detail below, and the similarities are not described in detail. As shown in Figure 24, the hybrid zoom scheme may include:

2400. The mobile phone starts the camera function.

In some embodiments, after the mobile phone starts the photographing function, the hybrid zoom method provided by the embodiments of the present application can be used for processing.

In other embodiments, after the mobile phone starts the photographing function and enters the target photographing mode, it can be processed by the hybrid zoom method provided by the embodiments of the present application.

In the following embodiments, the target photographing mode is taken as an example of the hybrid zoom mode for description.

For other related descriptions of step 2400, reference may be made to the description in the foregoing step 200, and details are not repeated here.

2401. The mobile phone displays a wide-angle image on the preview interface.

Different from the above solution of displaying the guide frame, in step 2401 of the hybrid zoom mode, the mobile phone may not display the guide frame on the preview interface first, and then display the guide corresponding to the target zoom magnification on the preview interface after obtaining the target zoom magnification subsequently. frame.

In some embodiments, the phone does not display a telephoto frame on the preview interface. In other embodiments, the mobile phone continuously displays the telephoto frame on the preview interface. In some other embodiments, the mobile phone does not display the telephoto frame on the preview interface first, and displays the telephoto frame on the preview interface after obtaining the target zoom magnification subsequently.

2402. The mobile phone obtains the target zoom ratio.

Wherein, the target zoom ratio is greater than the zoom ratio of the wide-angle camera (ie, the first camera), and is smaller than the zoom ratio of the telephoto camera (ie, the second camera). The target zoom ratio is the zoom ratio of the target image obtained based on the stitched image of the target telephoto image, the zoom ratio of the final image the user wants to capture, not the zoom ratio of the wide-angle image as the background image. Before and after the user sets the target zoom magnification, the zoom magnification of the wide-angle image as the background image does not change.

In some embodiments, the target zoom magnification is the zoom magnification set by the user. In a possible implementation manner, the mobile phone may prompt the user to set the target zoom magnification by displaying information or playing a voice. Exemplarily, see (a)-(c) in Figure 25, the mobile phone prompts the user on the preview interface: In this mode, you can set the zoom ratio to capture a high-definition image corresponding to the zoom ratio.

In the embodiments of the present application, there may be various manners for the user to set the target zoom magnification. Exemplarily, see (a) in FIG. 25 , the preview interface of the mobile phone includes multiple optional zoom magnification controls, such as 1.5X controls, 2X controls, 2.5X controls, 3X controls, 3.5X controls, 4X controls and 4.5X controls, etc., the mobile phone determines the corresponding target zoom magnification according to the zoom magnification control selected by the user.

For another example, after the mobile phone detects that the user clicks on the zoom magnification setting control on the preview interface, a setting interface can be displayed, and the user can set the target zoom magnification based on the setting page.

For another example, as shown in (b) of FIG. 25 , after detecting the zooming/zooming operation of the user on the preview interface, the mobile phone obtains the corresponding target zoom ratio after the zooming/zooming operation.

For another example, as shown in (c) of FIG. 25 , after detecting the user's drag operation on the zoom magnification adjustment lever, the mobile phone obtains the corresponding target zoom magnification after the drag operation.

For another example, after the mobile phone detects the user's voice instruction to set the zoom ratio, the mobile phone obtains the target zoom ratio set by the user's voice.

It should be noted that, when the target zoom ratio is larger than that of the wide-angle camera and smaller than that of the telephoto camera, the mobile phone can use the hybrid zoom method provided by the embodiment of the present application for processing. If the target zoom ratio is less than or equal to the zoom ratio of the wide-angle camera, the mobile phone may not use the hybrid zoom scheme, but directly generate the target image corresponding to the target zoom ratio according to the image captured by the wide-angle camera or the ultra-wide-angle camera. If the target zoom ratio is greater than or equal to the zoom ratio of the long-angle camera, the mobile phone may not use the hybrid zoom scheme, but directly generate the target image corresponding to the target zoom ratio according to the image captured by the telephoto camera or the ultra-telephoto camera.

In some other embodiments, the target zoom ratio may also be the default zoom ratio (for example, the default zoom ratio of the wide-angle camera) or the last used zoom ratio. The mobile phone can also modify the target zoom ratio according to the user's operation.

In some embodiments of the present application, the mobile phone superimposes and displays a guide frame corresponding to the target zoom magnification on the wide-angle image of the preview interface. Different from the guide frame displayed by the mobile phone in the above scheme of displaying the guide frame, the guide frame in the hybrid zoom scheme corresponds to the target zoom magnification, and is a guide frame of the minimum specification including the image area size corresponding to the field angle of the target zoom magnification.

In the embodiment of the present application, the image area corresponding to the field of view of the target zoom magnification is located in the middle of the wide-angle image by default. For example, the zoom ratio of the wide-angle camera is 1X, the zoom ratio of the telephoto camera is 5X, and the guide frame can include up to 5*5 grids. If the target zoom magnification is 2.5X, the field of view of the target zoom magnification corresponds to the image area 2601; for the guide frame corresponding to the target zoom magnification 2.5X, please refer to the dashed guide frame 2602 shown in (a) in FIG. 26A . Block 2602 includes 3*3 grids, and the size of the grid corresponds to the field angle of the telephoto camera; the size of the image area corresponding to the field angle of the target zoom magnification is 2.5 times the grid size. Exemplarily, when the target zoom magnification is 2.5X, for a schematic diagram of the preview interface, see (c) in FIG. 26A , including a guide frame 2604 corresponding to the target zoom magnification. If the target zoom magnification is 3X, the image area corresponding to the target zoom magnification 3X is the same size as the guide frame. For the guide frame corresponding to the target zoom magnification 3X, please refer to the dotted guide frame 2603 shown in (b) in FIG. 26A . The guide frame 2603 includes 3*3 grids, and the image size corresponding to the field angle of the target zoom magnification is 3 times the grid size.

In some embodiments, in the preview state, the mobile phone can prompt the user by displaying information, voice broadcast, etc., during the photographing process, please press the guide frame to shoot, so as to crop the generated stitched image and obtain a zoom ratio that meets the target zoom ratio. target image. Exemplarily, see (d) in FIG. 26A , the mobile phone can prompt the user by displaying information on the preview interface: Please press the dotted guide frame to capture a stitched image during the photographing process, and the stitched image is used for cropping to obtain the zoom you instructed. The image corresponding to the magnification!

In some embodiments, the mobile phone can also continuously display the target frame corresponding to the target zoom ratio on the wide-angle image of the preview interface. The position and size of the target frame are consistent with the position and size of the image area corresponding to the field angle of the target zoom magnification. After obtaining the target zoom magnification, the mobile phone can display the target frame corresponding to the target zoom magnification on the preview interface, so as to prompt the user with the position and size of the image area size corresponding to the field of view of the current target zoom magnification, so that the user can know the zoom according to the current target zoom ratio. The size of the target image that can be obtained by the magnification. The guide frame is the smallest size guide frame including the target frame. The size of the guide frame is greater than or equal to the size of the target frame, that is, greater than or equal to the image size corresponding to the field angle of the target zoom magnification.

In some embodiments of the present application, the target frame is located in the middle of the wide-angle image by default. For example, the zoom ratio of the wide-angle camera is 1X, the zoom ratio of the telephoto camera is 5X, and the guide frame can include up to 5*5 grids. If the target zoom magnification is 2.5X, the target frame corresponding to the target zoom magnification of 2.5X may refer to the solid-line rectangular frame 2601 shown in (b) of FIG. 26A . If the target zoom magnification is 3X, the size of the target frame corresponding to the target zoom magnification 3X is consistent with the size of the frame of the guide frame. Exemplarily, when the target zoom magnification is 2.5X, see FIG. 26B for a schematic diagram of the preview interface, including a guide frame 2604 corresponding to the target zoom ratio and a target frame 2605 corresponding to the target zoom ratio.

In the embodiment of the present application, the target frame on the preview interface is located in the middle of the wide-angle image by default. In some technical solutions, the user can also move the position of the target frame, and the position of the guide frame also changes correspondingly with the change of the position of the target frame. Exemplarily, on the preview interface shown in (a) in FIG. 27 , if the mobile phone detects that the user drags the target frame to the right, as shown in (b) in FIG. 27 , the target frame and the guide frame are move to the right. Alternatively, the user can move the position of the guide frame, and the position of the target frame changes accordingly.

In some other embodiments, after acquiring the target zoom ratio, the mobile phone briefly displays the target frame on the preview interface to prompt the user the size of the image corresponding to the field of view of the current target zoom ratio, so as to facilitate the user to determine the current target Check whether the zoom ratio is appropriate, and then stop displaying the target frame. In some technical solutions, the mobile phone can record the position and size of the target frame. After the mobile phone determines that the shooting is over, the stitched image can be cropped according to the recorded position and size of the target frame to obtain the target image.

2403. After detecting the user's photographing operation, the mobile phone displays a wide-angle image on the photographing interface, and a guide frame superimposed on the wide-angle image, where the guide frame corresponds to the target zoom magnification.

On the shooting interface, the phone can display a wide-angle image and a guide frame superimposed on the wide-angle image.

In some embodiments, the mobile phone can also display a telephoto frame on the shooting interface.

Optionally, the mobile phone can also display a target frame on the shooting interface.

In some embodiments, during the photographing process, the mobile phone can temporarily or continuously prompt the user by displaying information, voice broadcast, etc., during the photographing process, please press the guide frame to photograph, so as to crop the stitched image obtained by photographing, Get the target image that matches the target zoom ratio.

In some embodiments, as described above, in the hybrid zoom scheme, the mobile phone displays the captured complete wide-angle image on the preview interface and the shooting interface. In some other embodiments, similar to the above solution of displaying the guide frame, the mobile phone can replace the complete wide-angle image displayed on the preview interface and the shooting interface with the target area image corresponding to the guide frame on the wide-angle image.

2404. The mobile phone generates a stitched image according to the acquired target telephoto image, and displays a thumbnail of the stitched image on the shooting interface.

For the relevant description of step 2404, reference may be made to the description in step 203, which will not be repeated here. Exemplarily, see (a) in FIG. 28 , a thumbnail image of the target telephoto image 1 , as well as a wide-angle image, a guide frame, and a telephoto frame are displayed on the shooting interface. Referring to (b)-(c) in FIG. 28 , the stitched image thumbnails, as well as a wide-angle image, a guide frame, and a telephoto frame are displayed on the shooting interface.

2405. After the mobile phone determines that the shooting is over, the stitched image is cropped to generate a target image.

Similar to the above step 204, in step 2405, after the shooting is completed, the mobile phone can also perform processing such as ghost removal, dynamic range enhancement or hole filling on the stitched image before generating the target image according to the stitched image, so as to improve the quality of the stitched image. , so as to generate a target image according to the processed spliced image and improve the quality of the target image, which will not be repeated here.

Among them, there are various situations in which the mobile phone determines that the shooting ends. For example, in the first case, after the grids in the guide frame are all photographed, the mobile phone automatically ends the photographing, and the stitched image is cropped to the size of the target frame to generate the target image. Wherein, when the size of the guide frame is the same as the size of the target frame, the mobile phone does not need to crop the stitched image, but can directly generate the target image according to the stitched image. Exemplarily, in the case shown in (c) of FIG. 28 , the target image generated by the mobile phone may refer to (d) of FIG. 28 .

In the second case, before the grid in the guide frame has not been photographed, the mobile phone determines that the photographing is ended after detecting the user's operation of stopping photographing. In this case, the mobile phone crops and enlarges the reference wide-angle image or the wide-angle image, thereby generating an image of the target zoom magnification through digital zooming. If the target frame is not displayed on the shooting interface, after the mobile phone determines that the shooting is over, the position and size of the target frame can be determined according to the target zoom magnification, so that the target image can be obtained by cropping the reference wide-angle image or wide-angle image according to the target frame.

In the case where the target frame is not displayed on the shooting interface, the mobile phone can determine the position and size of the image area corresponding to the field of view of the target zoom magnification after the shooting is completed, so as to determine the position and size of the stitched image according to the position and size of the image area. Crop to get the target image. Alternatively, when the target frame is not displayed on the shooting interface, the mobile phone can determine the position and size of the target frame according to the target zoom ratio after the shooting is completed, so as to obtain the target image by cropping the stitched image according to the position and size of the target frame. Alternatively, if the target frame is not displayed on the shooting interface, after obtaining the target zoom ratio, the mobile phone can determine the position and size of the target frame according to the target zoom ratio, and record the position and size of the target frame, so that after the shooting, according to the The position and size of the target frame are used to crop the stitched image to obtain the target image. The position and size of the image area corresponding to the field of view of the target zoom magnification are the position and size of the target frame.

Compared with the target image corresponding to the target zoom magnification achieved through the wide-angle image and digital zoom, the solution corresponding to the above-mentioned first case is obtained by stitching and cropping a telephoto image with a higher resolution and a clearer small field of view. For a target image with a large field of view corresponding to the target zoom magnification, the resolution and clarity of the target image are also higher, the image quality is better, and the zoom effect of optical zoom can be realized.

At present, there is another zoom scheme, that is, the images captured by two cameras with different FOV sizes are fused and then cropped to realize zooming. Among them, the image with small FOV is located in the middle area, which can improve the definition in the middle of the target image, but the definition at the edge of the picture is poor. However, the solution corresponding to the above-mentioned first case in the embodiment of the present application can make the definition of the entire target image higher, and achieve the zoom effect of optical zoom.

In addition, in the shooting methods described in the above embodiments, the telephoto image collected in real time is not displayed in the telephoto frame, but is only used to indicate the real-time shooting range corresponding to the telephoto camera. In some other embodiments, a down-sampling image of the telephoto image collected by the telephoto camera in real time is displayed in the telephoto frame, so as to present the telephoto image collected by the telephoto camera in real time to the user. And, the position of the telephoto image is opposite to the position of the same content on the wide-angle image. In other embodiments, a telephoto image captured in real time by the telephoto camera is displayed in the telephoto frame. In other embodiments, the telephoto frame is located at a preset position on the interface, for example, at the lower left corner or the lower right corner of the interface. Exemplarily, referring to FIG. 29 , the telephoto frame 2900 is located at the lower left corner of the interface.

The above description is given by taking the first camera as a wide-angle camera and the second camera as a telephoto camera as an example. When the first camera and/or the second camera are other cameras, the shooting method provided in the above embodiment can still be used to obtain the The target image is not repeated in this embodiment of the present application.

The above description takes the electronic device as a mobile phone as an example. When the electronic device is other devices such as a tablet computer or a smart watch, the target image can still be obtained by using the shooting method provided in the above embodiment, which is not repeated in this embodiment of the present application.

It can be understood that, in order to realize the above-mentioned functions, the electronic device includes corresponding hardware and/or software modules for executing each function. The present application can be implemented in hardware or in the form of a combination of hardware and computer software in conjunction with the algorithm steps of each example described in conjunction with the embodiments disclosed herein. Whether a function is performed by hardware or computer software driving hardware depends on the specific application and design constraints of the technical solution. Those skilled in the art may use different methods to implement the described functionality for each particular application in conjunction with the embodiments, but such implementations should not be considered beyond the scope of this application.

In this embodiment, the electronic device can be divided into functional modules according to the above method examples. For example, each functional module can be divided corresponding to each function, or two or more functions can be integrated into one processing module. The above-mentioned integrated modules can be implemented in the form of hardware. It should be noted that, the division of modules in this embodiment is schematic, and is only a logical function division, and there may be other division manners in actual implementation.

Embodiments of the present application further provide an electronic device, including one or more processors and one or more memories. The one or more memories are coupled to the one or more processors for storing computer program code, the computer program code comprising computer instructions that, when executed by the one or more processors, cause the electronic device to perform The above-mentioned related method steps implement the shooting method in the above-mentioned embodiment.

An embodiment of the present application also provides an electronic device, as shown in FIG. 30 , including: a display screen (or screen) 3001, one or more processors 3002, multiple cameras 3003, a memory 3004, and one or more computers In procedure 3005, the above-mentioned devices may be connected through one or more communication buses 3006. wherein the one or more computer programs 3005 are stored in the aforementioned memory 3004 and configured to be executed by the one or more processors 3002, the one or more computer programs 3005 comprising instructions that may be used to perform the aforementioned implementations the steps in the example. Wherein, all relevant contents of the steps involved in the foregoing method embodiments can be cited in the functional description of the corresponding physical device, and details are not described herein again.

Exemplarily, the processor 3002 may be the processor 110 shown in FIG. 1, the memory 3004 may be the internal memory 121 shown in FIG. 1, the camera 3003 may be the camera 193 shown in FIG. The display screen 3001 may specifically be the display screen 194 shown in FIG. 1 .

Embodiments of the present application further provide a computer-readable storage medium, where computer instructions are stored in the computer-readable storage medium, and when the computer instructions are executed on an electronic device, the electronic device executes the above-mentioned related method steps to realize the above-mentioned embodiments shooting method in .

Embodiments of the present application also provide a computer program product, which, when the computer program product runs on a computer, causes the computer to execute the above-mentioned relevant steps, so as to realize the photographing method executed by the electronic device in the above-mentioned embodiment.

In addition, the embodiments of the present application also provide an apparatus, which may specifically be a chip, a component or a module, and the apparatus may include a connected processor and a memory; wherein, the memory is used for storing computer execution instructions, and when the apparatus is running, The processor can execute the computer-executed instructions stored in the memory, so that the chip executes the photographing method executed by the electronic device in the foregoing method embodiments.

Wherein, the electronic device, computer-readable storage medium, computer program product or chip provided in this embodiment are all used to execute the corresponding method provided above. Therefore, for the beneficial effects that can be achieved, reference may be made to the above-provided method. The beneficial effects in the corresponding method will not be repeated here.

From the description of the above embodiments, those skilled in the art can understand that for the convenience and brevity of the description, only the division of the above functional modules is used as an example for illustration. In practical applications, the above functions can be allocated by different The function module is completed, that is, the internal structure of the device is divided into different function modules, so as to complete all or part of the functions described above.

In the several embodiments provided in this application, it should be understood that the disclosed apparatus and method may be implemented in other manners. For example, the device embodiments described above are only illustrative. For example, the division of the modules or units is only a logical function division. In actual implementation, there may be other division methods. For example, multiple units or components may be Incorporation may either be integrated into another device, or some features may be omitted, or not implemented. On the other hand, the shown or discussed mutual coupling or direct coupling or communication connection may be through some interfaces, indirect coupling or communication connection of devices or units, and may be in electrical, mechanical or other forms.

The units described as separate components may or may not be physically separated, and the components shown as units may be one physical unit or multiple physical units, that is, they may be located in one place, or may be distributed to multiple different places . Some or all of the units may be selected according to actual needs to achieve the purpose of the solution in this embodiment.

In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically alone, or two or more units may be integrated into one unit. The above-mentioned integrated units can be realized in the form of hardware, and can also be realized in the form of software functional units.

If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it may be stored in a readable storage medium. Based on such understanding, the technical solutions of the embodiments of the present application can be embodied in the form of software products in essence, or the parts that contribute to the prior art, or all or part of the technical solutions, which are stored in a storage medium , including several instructions to make a device (may be a single chip microcomputer, a chip, etc.) or a processor (processor) to execute all or part of the steps of the methods described in the various embodiments of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read only memory (ROM), random access memory (random access memory, RAM), magnetic disk or optical disk and other media that can store program codes.

The above content is only a specific embodiment of the present application, but the protection scope of the present application is not limited to this. Covered within the scope of protection of this application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

Claims

A shooting method, applied to an electronic device, the electronic device comprising a first camera and a second camera, the equivalent focal length of the second camera is greater than the equivalent focal length of the first camera, wherein the method is include:

Start the camera function;

Displaying the image and the guide frame collected by the first camera on the preview interface;

After detecting the user's photographing operation, a first image is displayed on the photographing interface, and the guide frame is superimposed on the first image; wherein, the first image is obtained according to the image collected by the first camera , the guide frame includes a plurality of grids, and a single said grid corresponds to the size of the field of view of the second camera;

The stitching information is displayed on the shooting interface, the stitching information is used to indicate the shooting progress, and the stitching information corresponds to the multi-frame target shooting images matched with the grids in the guide frame. The image is acquired by the second camera;

generating a stitched image according to the multi-frame target captured images;

After the shooting is completed, a target image is generated according to the stitched image.
The method according to claim 1, wherein the first image is a first frame image collected by the first camera after detecting a user's photographing operation;

Alternatively, the first image is an image obtained by merging Q frames of images collected by the first camera after detecting the user's photographing operation, where Q is an integer greater than 1;

Alternatively, the first image is an image acquired by the first camera during the photographing process after detecting the photographing operation of the user.
The method according to claim 1 or 2, wherein, in the target captured image and the matched grid, the overlap ratio of the same content of the first image is greater than or equal to a first preset value;

Or, in the target captured image and the matched grid, the similarity of the histogram of the feature parameter of the first image is greater than or equal to a second preset value;

Or, in the target captured image and the matched grid, the similarity of the first image in the same transformation domain is greater than or equal to a third preset value;

Alternatively, in the target captured image and the matched grid, the feature matching degree of the first image is greater than or equal to a fourth preset value.
The method according to any one of claims 1-3, wherein the method further comprises:

According to the image blocks corresponding to the grid on the reference image, determine the configuration parameters of the image captured by the second camera to capture the target;

The captured image of the target matched with the grid is acquired according to the configuration parameter.
The method according to claim 4, wherein the configuration parameters include at least one or more of the following: automatic exposure AE configuration parameters, automatic white balance AWB configuration parameters or dynamic range correction DRC configuration parameters.
The method according to any one of claims 1-5, wherein the multi-frame target captured images include a first target captured image and a second target captured image, and the stitching is generated according to the multi-frame target captured images images, including:

A mosaic image is generated after first image processing is performed on the first target captured image and the second target captured image, and the first image processing includes image registration.
The method according to claim 6, wherein performing image registration on the captured image of the first target and the captured image of the second target comprises:

According to a preset image registration algorithm, features are respectively extracted from the first target captured image and the second target captured image;

According to the obtained feature matching pairs, the first captured image of the target and the second captured image of the target are registered.
The method according to claim 7, wherein the method further comprises:

The registered image is corrected according to the reference image.
The method according to claim 6, wherein performing image registration on the captured image of the first target and the captured image of the second target comprises:

Extracting features from the first target captured image and the reference image respectively, and registering the first target captured image and the reference image according to the obtained feature matching pair;

Features are extracted from the second target captured image and the reference image, respectively, and the second target captured image and the reference image are registered according to the obtained feature matching pair.
The method according to claim 9, wherein the method further comprises:

Extracting features from the first target captured image and the second target captured image, respectively, and registering the first target captured image and the second target captured image according to the obtained feature matching pair;

Calculate a homography matrix according to feature matching pairs of overlapping regions of the first target captured image, the second target captured image and the reference image;

The second target captured image is deformed according to the homography matrix.
The method according to claim 6, wherein performing image registration on the captured image of the first target and the captured image of the second target comprises:

Attaching the first target captured image to a coordinate position with the same image content on the reference image;

Attaching the second target captured image to the reference image at a coordinate position with the same image content;

If a hole exists between the first target captured image and the second target captured image, filling the hole is performed according to the image content at the position corresponding to the hole on the reference image.
The method according to any one of claims 6-11, wherein the first image processing further comprises ghost removal processing, and the first target captured image and the second target captured image are processed After the first image is processed, a stitched image is generated, including:

Retain the ghost-free image of the moving object on the first target captured image;

deleting the image of the moving object on the second target captured image;

generating a stitched image after performing image registration on the first target captured image and the second target captured image;

Fill the holes on the stitched image according to the reference image.
The method according to any one of claims 6-11, wherein the first image processing further comprises ghost removal processing, and the first target captured image and the second target captured image are processed After the first image is processed, a stitched image is generated, including:

performing image registration on the captured image of the first target and the captured image of the second target;

retaining a ghost-free image of a moving object on the registered image, the ghost-free image from the first target captured image, the second target captured image, or the first image;

delete the images of the moving object in other regions on the registered image;

The holes caused by the deletion are filled according to the reference image to generate a stitched image.
The method according to any one of claims 6-11, wherein the first image processing further comprises ghost removal processing, and the first target captured image and the second target captured image are processed After the first image is processed, a stitched image is generated, including:

deleting the images of moving objects on the first target captured image and the second target captured image;

Determine the motion trajectory of the moving object according to the multi-frame images collected by the first camera, so as to delete the image of the moving object on the first image, and obtain a first image without ghost images;

Attaching the first target captured image and the second target captured image after deleting the image of the moving object to the first image without ghost images;

The holes caused by the deletion are filled according to the reference image to generate a stitched image.
The method according to any one of claims 6-14, wherein the first image processing further includes light and color leveling processing and image fusion processing.
The method according to any one of claims 1-15, wherein the generating a target image according to the spliced image comprises:

After the second image processing is performed on the stitched image, the target image is generated, and the second image processing includes at least one or more of the following: anti-ghosting processing, dynamic range enhancement, high-frequency component fusion, uniform light and uniform color treatment or hole filling.
The method according to claim 16, characterized in that, performing high-frequency component fusion on the stitched image, comprising:

Extracting a high-frequency component image of the reference image, and fusing the high-frequency component image with the spliced image.
The method according to claim 16, wherein performing uniform light and color processing on the stitched image, comprising:

performing high dynamic range synthesis on the images collected by the multiple frames of the first camera;

According to the obtained high dynamic range information, uniform light and color processing is performed on the spliced image.
The method according to claim 16, wherein filling holes in the stitched image comprises:

Perform super-resolution image synthesis on the regions corresponding to the holes of the spliced images on the images collected by the first camera in multiple frames;

According to the obtained image information, the stitched image is filled with holes.
The method according to claim 16, wherein filling holes in the stitched image comprises:

Perform image super-resolution processing on the region corresponding to the hole in the spliced image on the reference image;

According to the obtained image information, the stitched image is filled with holes.
The method according to any one of claims 4-20, wherein the reference image is a first frame image collected by the first camera after detecting a user's photographing operation;

Alternatively, the first image is an image obtained by merging Q frames of images collected by the first camera after detecting the user's photographing operation, where Q is an integer greater than 1.
The method according to any one of claims 1-21, wherein the determining that the shooting is ended comprises:

After the grids in the guide frame are matched, it is determined that the shooting ends.
The method according to claim 22, wherein the generating a target image according to the spliced image comprises:

The stitched image is the target image;

Or, if the edges of the spliced images are not aligned, then the spliced images are cropped to obtain the target image with aligned edges;

Or, if the edges of the spliced images are not aligned, fill the edge regions of the spliced images according to the first image to obtain the target image with aligned edges.
The method according to any one of claims 1-21, wherein the determining that the shooting is ended comprises:

Before the matching of the grids in the guide frame is completed, if it is detected that the user stops the photographing operation, it is determined that the photographing is ended;

Or, if the moving direction of the electronic device moves out of the guide frame, it is determined that the shooting ends;

Alternatively, if the deviation range between the moving direction of the electronic device and the indicated direction of the guide frame is greater than or equal to the fifth preset value, it is determined that the shooting ends.
The method according to claim 24, wherein the generating a target image according to the spliced image comprises:

generating the target image according to the stitched image corresponding to the matched whole row/column grid;

Alternatively, the target image is generated according to the stitched image corresponding to the matched grid and the image area corresponding to the unmatched grid on the first image.
The method according to any one of claims 1-25, wherein the method further comprises:

acquiring a target zoom magnification, the guide frame corresponds to the target zoom magnification, and the target zoom magnification is greater than the zoom magnification of the first camera and smaller than the zoom magnification of the second camera;

The generating a target image according to the spliced image includes:

The stitched image is cropped to generate the target image, and the target image corresponds to the target zoom magnification.
A shooting method, applied to an electronic device, the electronic device comprising a first camera and a second camera, the equivalent focal length of the second camera is greater than the equivalent focal length of the first camera, wherein the method is include:

Start the camera function;

A third image and an image frame are displayed on the preview interface, the third image is an image captured by the first camera, and the screen range of the third image in the image frame corresponds to the screen of the second image, so The second image is an image collected by the second camera;

After detecting the user's photographing operation, displaying a first image and the image frame on the photographing interface, where the first image is obtained according to an image collected by the first camera;

Displaying splicing information on the shooting interface, the splicing information is used to indicate the shooting progress, the splicing information corresponds to the multi-frame target shooting images obtained during the photographing process, and the adjacent target shooting images match each other;

generating a stitched image according to the multi-frame target captured images;

After the shooting is completed, a target image is generated according to the stitched image.
The method according to claim 27, wherein, in the multi-frame target captured images, the matched third target captured image and the fourth target captured image satisfy:

The deviation between the abscissa range of the captured image of the third target and the abscissa range of the captured image of the fourth target is less than or equal to a first preset threshold;

Alternatively, the deviation between the ordinate range of the third target captured image and the ordinate range of the fourth target captured image is less than or equal to a second preset threshold.
The method according to claim 28, wherein the matched third target captured image and the fourth target captured image also satisfy:

The overlapping area between the third target captured image and the fourth target captured image is greater than or equal to a sixth preset value;

Alternatively, the gap between the third target captured image and the fourth target captured image is less than or equal to a seventh preset value.
An electronic device, comprising:

a first camera and a second camera for capturing images;

screen, used to display the interface;

one or more processors;

memory;

and one or more computer programs, wherein the one or more computer programs are stored in the memory, the one or more computer programs comprising instructions that, when executed by the electronic device, cause all The electronic device performs the shooting method according to any one of claims 1-29.
A computer-readable storage medium, characterized by comprising computer instructions, when the computer instructions are executed on a computer, causing the computer to execute the photographing method according to any one of claims 1-29.
A computer program product, characterized in that, when the computer program product runs on a computer, the computer is caused to execute the shooting method according to any one of claims 1-29.