WO2021238317A1

WO2021238317A1 - Panoramic image capture method and device

Info

Publication number: WO2021238317A1
Application number: PCT/CN2021/078666
Authority: WO
Inventors: 漆思远; 李伟
Original assignee: 华为技术有限公司
Priority date: 2020-05-29
Filing date: 2021-03-02
Publication date: 2021-12-02
Also published as: CN113747044B; CN113747044A

Abstract

Embodiments of the present application relate to the field of electronic technology, and provided therein are a panoramic image capture method and device, which can splice images, acquired from different angles, in two directions perpendicular to one another so as to generate a panoramic image, so that the field of view of the panoramic image can be expanded in the two directions, improving the panoramic image capture experience of a user. The specific solution being: an electronic device enters a panoramic image capture mode of a camera application; the electronic device displays first guidance information on a preview interface, the first guidance information comprises a first guide path, the first guide path comprises at least two sub-paths that are disposed along a first direction and that are parallel to one another, the first direction is parallel to a side edge of the electronic device, and the first guide path is used to guide a user to rotate the electronic device along the first guide path during an image capture process. The embodiments of the present invention are used for image capturing panoramic images.

Description

Method and equipment for panoramic shooting

This application claims the priority of a Chinese patent application filed with the State Intellectual Property Office on May 29, 2020, the application number is 202010478652.1, and the application name is "a method and equipment for panoramic shooting", the entire content of which is incorporated into this application by reference middle.

Technical field

The embodiments of the present application relate to the field of electronic technology, and in particular to a panoramic shooting method and device.

Background technique

With the development of electronic technology, the camera functions of mobile phones and other electronic devices have become more and more powerful. The electronic device can support multiple shooting modes and functions. For example, the electronic device can support night scene shooting, skin beautification shooting, time-lapse photography shooting, or panoramic shooting. Among them, in the panoramic shooting, the mobile phone can guide the user to turn the mobile phone according to the guide line 01 shown in Figure 1, so that the mobile phone collects multiple frames of images with different angles, and stitches the images with different angles in the extension direction of the guide line 01 , So as to form a panoramic image with a wide field of view. However, this panoramic shooting method lacks novelty and cannot meet the growing diverse shooting needs of users.

Summary of the invention

The embodiments of the present application provide a panoramic shooting method and device, which can respectively stitch images collected from different angles in two mutually perpendicular directions to generate a panoramic image, so that the angle of view of the panoramic image can be expanded in two directions , Improve the user's panoramic shooting experience.

In order to achieve the foregoing objectives, the following technical solutions are adopted in the embodiments of the present application:

On the one hand, an embodiment of the present application provides a panoramic shooting method, including: an electronic device enters a panoramic shooting mode of a camera application. The electronic device displays the first guide information on the preview interface, and the first guide information includes the first guide path. Wherein, the first guide path includes at least two sub-paths that are arranged along a first direction and are parallel to each other, the first direction is parallel to a side of the electronic device, and the first guide path is used to guide the user along the first guide path during the shooting process. Turn the electronic device.

In this solution, a guide path is displayed on the preview interface, and the guide path includes at least two sub-paths arranged in a first direction and parallel to each other, so as to guide the user to rotate the electronic device along the guide path during the shooting process. In this way, in the process of shooting a panoramic image, when the electronic device collects images along a certain sub-path set along the first direction and performs image splicing, the angle of view of the panoramic image in the first direction can be expanded; When multiple sub-paths set along the first direction collect images and perform image stitching, the panoramic image can be expanded in a second direction perpendicular to the first direction while expanding the angle of view of the panoramic image in the first direction. The angle of view.

In a possible design, the coordinate ranges corresponding to the different sub-paths set along the first direction overlap in the first direction.

In other words, the different sub-paths that are arranged along the first direction and are parallel to each other are not completely staggered.

In another possible design, different sub-paths set along the first direction correspond to the same coordinate range in the first direction.

That is, the two ends of the different sub-paths arranged along the first direction are respectively aligned, and the different sub-paths have the same length.

In another possible design, the preview interface also includes a splicing preview window for displaying thumbnails of the images collected by the electronic device. Moreover, the splicing preview window is located at the beginning of the first guide path. In this way, in some cases, the stitching preview window may be blocked and occupy a part of the guide path.

In another possible design, the first guide path further includes at least one sub-path arranged along the second direction, and the at least one sub-path is used to connect at least two sub-paths arranged along the first direction, and the second direction is connected to the first direction. The direction is vertical.

In this way, the entire guide path is a continuous complete path. The sub-paths set along the second direction can guide the user to take the sequence of different sub-paths set along the first direction when shooting.

In another possible design, the method further includes: after the electronic device detects the shooting operation, displaying second guide information on the shooting interface. The second guide information includes a splicing preview window, a second guide path, and a deviation indicator. The deviation indicator is used to indicate the position of the center line of the image collected by the electronic device, and the deviation indicator moves along the second guide path during the shooting. The second guide path includes a portion of the first guide path that does not deviate from the indicator mark. The electronic device displays the target image obtained by splicing the images collected by the electronic device when the deviation indicator mark moves along the sub-path in the first direction in the stitching preview window; the electronic device stops shooting after the deviation indicator mark reaches the end of the second guide path, and the electronic device The target image obtained by the splicing of the equipment is the panoramic image.

That is to say, during the shooting process, the electronic device can rotate along the guide path, and stitch the images collected when the sub path rotates along the first direction to generate a panoramic image. In addition, in order to better guide the user to turn the electronic device, the shooting interface only displays a guide path that deviates from the unfinished shooting part where the indicating arrow has not passed.

In another possible design, when the deviation indicator moves along any sub-path in the second guide path, the second guide information further includes that it is located on both sides of the any sub-path and is parallel to the any sub-path. The deviation range indicator line. The method further includes: if the deviation indicator mark exceeds the range indicated by the deviation range indicator line, the electronic device stops shooting.

Wherein, the deviation range indicator line is used to indicate the maximum range within which the center line of the image collected by the electronic device can deviate from the guide path. If the deviation indicator mark exceeds the range indicated by the deviation range indicator line, the center of the image collected by the electronic device exceeds the maximum range that can be deviated. At this time, the image may not include the image within the cropping range required for stitching, and thus cannot be performed. Image splicing, so the electronic device can stop the panoramic image shooting process.

In another possible design, the first guide path includes a first sub-path and a third sub-path that are arranged along the first direction and are parallel to each other, and the first sub-path is the initial sub-path. The first guide path further includes a second sub-path arranged along the second direction, and the second sub-path is used to connect the first sub-path and the third sub-path. During the shooting, the deviation indicator mark moves along the first sub-path, the second sub-path, and the third sub-path in sequence.

In this solution, the guide path includes two sub-paths arranged along the first direction and one sub-path arranged along the second direction.

In another possible design, the first guide path further includes a fifth sub-path arranged along the first direction and parallel to the first sub-path, and the third sub-path and the fifth sub-path are located on both sides of the first sub-path . The first guide path further includes a fourth sub-path arranged along the second direction and parallel to the second sub-path, and the fourth sub-path is used to connect the third sub-path and the fifth sub-path. During the shooting process, the deviation indicator mark moves along the first sub-path, the second sub-path, the third sub-path, the fourth sub-path, and the fifth sub-path in sequence.

In this solution, the guide path includes three sub-paths arranged along the first direction and two sub-paths arranged along the second direction.

In another possible design, during the shooting process, when the deviation indicator moves along the first sub-path, the second guide path includes the portion of the first sub-path that the deviation indicator does not pass and the second sub-path to the first sub-path. Five sub-paths, and deviation range indication lines are displayed on both sides of the first sub-path; the stitching preview window displays the target image obtained by splicing the images collected by the electronic device when the deviation indicating mark moves along the first sub-path. When the deviation indicator moves along the second sub-path, the second guide path includes the portion of the second sub-path that the deviation indicator does not pass and the third to fifth sub-paths, and deviations are displayed on both sides of the second sub-path Range indicator line; the target image displayed in the splicing preview window is the splicing result corresponding to the first path obtained by splicing the images collected by the electronic device after the deviation indicator mark moves to the end of the first sub-path. When the deviation indicator moves along the third sub-path, the second guide path includes the part of the third sub-path that the deviation indicator does not pass, the fourth and fifth sub-paths, and deviations are displayed on both sides of the third sub-path Range indicator line; the target image displayed in the splicing preview window is an image obtained by splicing the image collected by the electronic device to the splicing result corresponding to the first sub-path when the deviation indicator mark moves along the second sub-path.

That is, on the shooting interface, the electronic device displays the guide path of the unfinished shooting and the deviation range indication line of the sub-path along which the electronic device is currently moving. In addition, the target image in the preview splicing window is an image generated by splicing the electronic device along the sub-paths in the first direction, rather than the image generated by splicing the electronic device along all the sub-paths.

In another possible design, when the deviation indicator moves along the first sub-path, the electronic device displays in the mosaic preview window when the deviation indicator moves along the first sub-path, the image collected by the electronic device is spliced The target image of includes: the electronic device _{maps the i-th frame image I i of the} first sub-path to the cylindrical surface to obtain an image

i is an integer greater than 1. Electronic equipment extraction

with

The feature points F _I,i and F _I,i-1 ,

It is the image obtained after the i-1th frame image I _i-1 of the first subpath is mapped to the cylindrical surface. The electronic device calculates the matching result of _{F I,i} and F _I,i-1. According to the matching result of _{F I,i} and F _{I,i-1, the electronic equipment will}

Towards

Map it. The electronic device will be mapped

The part within the preset first cropping range is spliced with the spliced image RI(i-1) of the first sub-path, so as to obtain the spliced image RIi of the first sub-path. Wherein, the first clipping range includes a clipping line corresponding to the deviation range indication line of the first sub-path and a range defined by a left boundary line and a right boundary line preset by the electronic device.

In this solution, when the deviation indicator moves along the first sub-path, the electronic device rotates along the first sub-path, and the electronic device performs cylindrical mapping on two adjacent frames of images collected during the rotation and extracts the feature points, and The matching result is calculated according to the feature points, and the homography matrix is calculated according to the matching result, so that the next frame image is mapped to the previous frame image according to the homography matrix, and the next frame image after the mapping is cropped to match the previous frame image. The splicing results of the splicing are spliced, and a new splicing result is obtained.

In addition, the cylindrical mapping of images from different angles taken by the electronic device in different postures can match the mapped size and imaging characteristics of the same object on the images taken at different angles, and then perform registration and stitching to generate a panoramic image , In order to meet the visual effect of the same image size of each part of the panorama.

In another possible design, when the deviation indicator moves along the first sub-path, the method further includes: the electronic device obtains multiple key frames from the image frames collected while rotating along the sub-path in the first direction.

In this way, when the deviation indicator moves along the third sub-path, the electronic device can display the target image obtained by splicing the images collected by the electronic device when the deviation indicator moves along the third sub-path in the mosaic preview window according to the key frame. When the deviation indicator moves along the fifth sub-path, the electronic device may display the target image obtained by splicing the images collected by the electronic device when the deviation indicator moves along the fifth sub-path in the mosaic preview window according to the key frame.

In another possible design, when the deviation indicator moves along the third sub-path, the electronic device displays in the stitching preview window the image obtained by splicing the image collected by the electronic device when the deviation indicator moves along the sub-path in the first direction. The target image includes: the electronic device obtains a target key frame G _{f1 that} _{matches the first frame image A 1 of the} third sub-path from a plurality of key frames. The electronic device _{maps G f1} to the cylindrical surface to get the image

The electronic device _{maps A 1} to the cylindrical surface to get the image

Electronic equipment extraction

with

The feature points F _A,1 and F _A,f1 . The electronic device calculates the matching result of _{F A,1} and F _A,f1. According to the matching results of _{F A,1} and F _{A,f1, the electronic device will}

Towards

Map it. The electronic device will be mapped

The part within the preset second cropping range is spliced with the corresponding splicing result of the first sub-path to obtain the spliced image RA1 of the second sub-path. Wherein, the second clipping range includes a clipping line corresponding to the deviation range indication line of the third subpath and a range defined by a left boundary line and a right boundary line preset by the electronic device.

In this solution, when the deviation indicator moves along the third sub-path, the electronic device rotates along the third sub-path, the electronic device determines the target key frame corresponding to the first frame of image After the image and the target key frame are mapped to the cylindrical surface, the feature points are extracted, and the matching results are calculated according to the feature points, and the homography matrix is calculated according to the matching results, so that the first frame image is mapped to the target key frame image according to the homography matrix. The first frame image after the mapping is cropped, so that the splicing result corresponding to the first sub-path is spliced, and a new splicing result is obtained.

In addition, the electronic device registers the image of the third sub-path with the key frame in the image of the first sub-path, and can timely correct the misalignment error between the image of the third sub-path and the image of the first sub-path during the splicing of the images of the third sub-path, so that the first sub-path The images of the three sub-paths are accurately registered with the splicing result of the first sub-path, thereby realizing global registration. The splicing result of the third sub-path and the splicing result of the first sub-path can form a smooth and natural transition overall image.

In another possible design, in the xyz coordinate system, the lens of the electronic device camera is located on the xy plane of the xyz three-dimensional coordinate system; the interval of the rotation angle around the y axis corresponding to the key frame is greater than or equal to the preset value Δβ; Among the multiple key frames, the difference between the rotation angle around the y axis corresponding to the _{target key frame G f1 and} the rotation angle around the y axis corresponding _{to A 1 is the smallest.}

In this way, there may be partial overlap between different key frames, and they are basically evenly distributed on the first sub-path. The image misalignment error of the image of the target key frame and the third sub-path is the smallest when stitching, and the image after the two are mapped to the cylindrical surface is easier to be registered.

In another possible design, when the deviation indicator moves along the third sub-path, the electronic device displays in the stitching preview window the image obtained by splicing the image collected by the electronic device when the deviation indicator moves along the sub-path in the first direction. target image, further comprising: an electronic device to acquire the i-th third sub-frame image a _i path matches the i-th keyframe G _fi target from the plurality of keyframes. The electronic device _{maps the G fi} to the cylindrical surface to obtain the image

The electronic device to obtain an image A _i is mapped to the cylindrical surface

Electronic equipment extraction

with

Feature points F _A,i , F _A,i-1 and F _A,fi ,

It is the image obtained after the i-1th frame image _Ai-1 of the second subpath is mapped to the cylindrical surface. The electronic device calculates the matching results of _{F A,i} , F _A,i-1 and F _A,fi. According to the matching results of _{F A,i} , F _A,i-1 and F _{A,fi, the electronic equipment will}

Towards

with

Map it. The electronic device will be mapped

For the part within the preset second cropping range, the stitching result corresponding to the first sub-path and the stitched image RA(i-1) of the second sub-path are stitched to obtain the stitched image RAi of the second sub-path.

In this solution, when the electronic device rotates along the third sub-path, the electronic device can perform registration, mapping, and splicing of images other than the first frame image according to the previous frame image and the determined target key frame.

In another possible design, when the deviation indicator moves along the fifth sub-path, the electronic device displays in the stitching preview window the image obtained by splicing the image collected by the electronic device when the deviation indicator moves along the sub-path in the first direction. The target image includes: the electronic device splicing the image collected while rotating along the fifth sub-path with the splicing result corresponding to the first sub-path according to a plurality of key frames.

It is understandable that when the electronic device splices the image collected while rotating along the fifth sub-path with the splicing result corresponding to the first sub-path according to a plurality of key frames, the splicing result corresponding to the first sub-path may already be the same as the splicing result of the first sub-path. The splicing results of the three sub-paths are spliced. That is, the electronic device splices the image collected while rotating along the fifth sub-path with the splicing result of the first sub-path and the splicing result of the third sub-path, thereby forming a spliced image with a larger field of view.

In another possible design, in the xyz coordinate system, the lens of the electronic device camera is located on the xy plane of the xyz three-dimensional coordinate system. The electronic device is preset with a left boundary line, a left baseline, a left clipping line, a right clipping line, a right baseline and a right boundary line; the left baseline corresponds to the fourth sub-path, the left baseline corresponds to the rotation angle around the y axis is 0, and the right baseline corresponds to The second subpath corresponds to the rotation angle around the y axis corresponding to the right baseline is β _r , the rotation angle around the y axis corresponding to the left clipping line is β ₂ , and the rotation angle around the y axis corresponding to the right clipping line is β ₃ , The rotation angle around the y-axis corresponding to the left boundary line is β ₁ , and the rotation angle around the y-axis corresponding to the right boundary line is β ₄ . The electronic device is also preset with upper, middle, and lower baselines, as well as the first cutting line, the second cutting line, the third cutting line, and the fourth cutting line; among them, the upper, middle, and lower baselines are the same as the third The sub-path, the first sub-path and the fifth sub-path correspond to each other, the first and second cutting lines correspond to the deviation range indication lines of the third sub-path, and the second and third cutting lines correspond to the deviation of the first sub-path. The deviation range indicator line corresponds to the third cutting line and the fourth cutting line correspond to the deviation range indicator line of the fifth subpath; the first cutting line, the upper baseline, the second cutting line, the middle line, the third cutting line, the lower baseline The rotation angles around the x-axis corresponding to the fourth cutting line are α ₁ , α _t , α ₂ , 0, α ₃ , α _b , and α _{4, respectively} . The method further includes: when the deviation indicator moves along the first sub-path, if the rotation angle of the electronic device around the x axis is α>α ₁ or α<α ₂ , the electronic device stops shooting. When the electronic device about the y-axis rotational angle β> β _r, departing from the indicating mark is moved by a first switch to a second sub-path is traversed along a sub-path; beta] the rotation angle around the y axis when the electronic device is> β ₄ or β<β ₃ , the electronic device stops shooting. When the rotation angle of the electronic device around the x-axis α>α _t , the deviation indicator switch is switched from moving along the second sub-path to moving along the third sub-path; when the rotation angle of the electronic device around the x-axis α>α ₁ or α <when α _2, the electronic device to stop recording. When the rotation angle β<0 of the electronic device around the y-axis, the deviation indicator will switch from moving along the third sub-path to moving along the fourth sub-path; when the rotation angle β>β ₂ or β of the electronic device around the y-axis <when β _1, the electronic device to stop recording. When the electronic device about the x axis rotation angle α <When α _b, departing from the indicating mark is moved by the switching path is traversed along fourth sub fifth sub path; when the electronic device about the x axis rotation angle α> α ₃ or When α<α ₄ , the electronic device stops shooting. When the electronic device about the y-axis rotational angle β> β _r, the electronic device to stop recording.

In other words, the electronic device can determine whether to switch the sub-path along which it rotates, and determine whether the center line of the image collected by the electronic device exceeds the cropping range, etc., according to the size of the rotation angle around the x-axis or the y-axis.

On the other hand, an embodiment of the present application provides a photographing device, which is included in an electronic device. The device has the function of realizing the behavior of the electronic device in any method in the foregoing aspects and possible designs, so that the electronic device executes the panoramic shooting method executed by the electronic device in any of the foregoing aspects in a possible design. This function can be realized by hardware, or by hardware executing corresponding software. The hardware or software includes at least one module or unit corresponding to the above-mentioned functions. For example, the device may include a processing unit, a display unit, a detection unit, and so on.

In yet another aspect, an embodiment of the present application provides an electronic device, including: a camera for capturing images; a screen for displaying an interface; one or more processors; and a memory in which codes are stored. When the code is executed by the electronic device, the electronic device is caused to execute the panoramic photography method executed by the electronic device in any one of the possible designs in the foregoing aspects.

In another aspect, an embodiment of the present application provides an electronic device, including: one or more processors; and a memory, in which code is stored. When the code is executed by the electronic device, the electronic device is caused to execute the panoramic photography method executed by the electronic device in any one of the possible designs in the foregoing aspects.

On the other hand, an embodiment of the present application provides a computer-readable storage medium, including computer instructions, which, when the computer instructions run on an electronic device, cause the electronic device to execute the panoramic photography method in any one of the possible designs in the foregoing aspects.

In another aspect, an embodiment of the present application provides a computer program product, which when the computer program product runs on a computer, causes the computer to execute the panoramic shooting method executed by the electronic device in any one of the above-mentioned possible designs.

On the other hand, an embodiment of the present application provides a chip system, which is applied to an electronic device. The chip system includes one or more interface circuits and one or more processors; the interface circuit and the processor are interconnected by wires; the interface circuit is used to receive signals from the memory of the electronic device and send signals to the processor. The signals include the memory Stored computer instructions; when the processor executes the computer instructions, it causes the electronic device to execute any one of the above-mentioned aspects of the possible design of the panoramic photography method.

For the corresponding beneficial effects of the other aspects mentioned above, please refer to the description of the beneficial effects of the method, which will not be repeated here.

Description of the drawings

FIG. 1 is a schematic diagram of a preview interface for panoramic shooting in the prior art;

2 is a schematic diagram of the hardware structure of an electronic device provided by an embodiment of the application;

FIG. 3 is a flowchart of a panoramic shooting provided by an embodiment of the application;

4A is a schematic diagram of a set of interfaces provided by an embodiment of the application;

4B is a schematic diagram of a set of guide paths provided by an embodiment of the application;

Figure 5 is a schematic diagram of a set of guidance information provided by an embodiment of the application;

FIG. 6 is a schematic diagram of another set of interfaces provided by an embodiment of the application;

FIG. 7 is a schematic diagram of a set of rule lines provided by an embodiment of the application;

8 is a schematic diagram of a coordinate system provided by an embodiment of the application, and a schematic diagram of the quantitative relationship between image frames and gyroscope data;

FIG. 9 is a schematic diagram of a set of cylindrical surface mapping effects provided by an embodiment of the application;

FIG. 10 is a schematic diagram of a splicing effect provided by an embodiment of this application;

FIG. 11 is a schematic diagram of a comparison of feature points provided by an embodiment of this application;

FIG. 12 is a schematic diagram of a set of splicing effects provided by an embodiment of the application;

FIG. 13 is a schematic diagram of a set of key frames provided by an embodiment of the application;

14A is a schematic diagram of another set of interfaces provided by an embodiment of the application;

14B is a schematic diagram of a set of guide paths on a shooting interface provided by an embodiment of the application;

FIG. 15 is a schematic diagram of another interface provided by an embodiment of the application;

16A is a schematic diagram of another interface provided by an embodiment of the application;

16B is a schematic diagram of another splicing effect provided by an embodiment of the application;

16C is a schematic diagram of comparison between a set of upper baseline images and target key frames provided by an embodiment of the application;

16D is a schematic diagram of another splicing effect provided by an embodiment of the application;

16E is a schematic diagram of another splicing effect provided by an embodiment of the application;

FIG. 17 is a schematic diagram of another splicing effect provided by an embodiment of the application;

FIG. 18 is a schematic diagram of another set of splicing effects provided by an embodiment of the application;

FIG. 19 is a schematic diagram of another set of interfaces provided by an embodiment of the application;

FIG. 20 is a schematic diagram of another set of guide paths provided by an embodiment of this application;

FIG. 21 is a schematic diagram of another interface provided by an embodiment of this application;

22A is a schematic diagram of another interface provided by an embodiment of the application;

22B is a schematic diagram of another set of guide paths provided by an embodiment of the application;

22C is a schematic diagram of another splicing effect provided by an embodiment of the application;

FIG. 23 is a schematic diagram of a set of guide paths and panoramic images obtained by stitching provided by an embodiment of the application; FIG.

24 is a schematic diagram of another set of guide paths and panoramic images obtained by stitching provided by an embodiment of the application;

25 is a schematic diagram of another set of guide paths and panoramic images obtained by stitching provided by an embodiment of the application;

FIG. 26 is a schematic structural diagram of an electronic device provided by an embodiment of this application.

Detailed ways

The technical solutions in the embodiments of the present application will be described below in conjunction with the drawings in the embodiments of the present application. Among them, in the description of the embodiments of the present application, unless otherwise specified, "/" means or, for example, A/B can mean A or B; "and/or" in this document is only a description of related objects The association relationship of indicates that there can be three kinds of relationships, for example, A and/or B, which can indicate: A alone exists, A and B exist at the same time, and B exists alone. In addition, in the description of the embodiments of the present application, "plurality" refers to two or more than two.

Hereinafter, the terms "first" and "second" are only used for descriptive purposes, and cannot be understood as indicating or implying relative importance or implicitly indicating the number of indicated technical features. Therefore, the features defined with "first" and "second" may explicitly or implicitly include one or more of these features. In the description of the present embodiment, unless otherwise specified, "plurality" means two or more.

The embodiment of the present application provides a panoramic shooting method, which can stitch images collected from different angles in two mutually perpendicular directions to generate a panoramic image, so that the view of the stitched and synthesized image can be expanded in two directions. Field angle, to obtain panoramic images that can cover a larger viewing angle in both directions, and improve the user's shooting experience.

The panoramic shooting method provided in the embodiments of the present application can be applied to electronic devices. For example, the electronic device may be a mobile phone, a tablet computer, a wearable device, a vehicle-mounted device, an augmented reality (AR)/virtual reality (VR) device, a notebook computer, an ultra-mobile personal computer (ultra-mobile personal computer). Computers, UMPCs), netbooks, personal digital assistants (personal digital assistants, PDAs) and other devices can also be professional cameras and other devices. The embodiments of the present application do not impose any restrictions on the specific types of electronic devices.

Exemplarily, FIG. 2 shows a schematic structural diagram of the electronic device 100. The electronic device 100 may include a processor 110, an external memory interface 120, an internal memory 121, a universal serial bus (USB) interface 130, a charging management module 140, a power management module 141, a battery 142, an antenna 1, and an antenna 2. , Mobile communication module 150, wireless communication module 160, audio module 170, speaker 170A, receiver 170B, microphone 170C, earphone jack 170D, sensor module 180, buttons 190, motor 191, indicator 192, camera 193, display screen 194, and Subscriber identification module (subscriber identification module, SIM) card interface 195, etc. The sensor module 180 may include a pressure sensor 180A, a gyroscope sensor 180B, an air pressure sensor 180C, a magnetic sensor 180D, an acceleration sensor 180E, a distance sensor 180F, a proximity light sensor 180G, a fingerprint sensor 180H, a temperature sensor 180J, a touch sensor 180K, and ambient light Sensor 180L, bone conduction sensor 180M, etc.

The processor 110 may include one or more processing units. For example, the processor 110 may include an application processor (AP), a modem processor, a graphics processing unit (GPU), and an image signal processor. (image signal processor, ISP), controller, memory, video codec, digital signal processor (digital signal processor, DSP), baseband processor, and/or neural-network processing unit (NPU) Wait. Among them, the different processing units may be independent devices or integrated in one or more processors.

The controller may be the nerve center and command center of the electronic device 100. The controller can generate operation control signals according to the instruction operation code and timing signals to complete the control of fetching and executing instructions.

A memory may also be provided in the processor 110 to store instructions and data. In some embodiments, the memory in the processor 110 is a cache memory. The memory can store instructions or data that have just been used or recycled by the processor 110. If the processor 110 needs to use the instruction or data again, it can be directly called from the memory. Repeated accesses are avoided, the waiting time of the processor 110 is reduced, and the efficiency of the system is improved.

The electronic device 100 implements a display function through a GPU, a display screen 194, an application processor, and the like. The GPU is a microprocessor for image processing, connected to the display 194 and the application processor. The GPU is used to perform mathematical and geometric calculations and is used for graphics rendering. The processor 110 may include one or more GPUs that execute program instructions to generate or change display information.

The display screen 194 is used to display images, videos, and the like. For example, the display screen 194 may display a preview interface and a shooting interface in the panoramic shooting mode. The display screen 194 includes a display panel. The display panel can adopt liquid crystal display (LCD), organic light-emitting diode (OLED), active matrix organic light-emitting diode or active-matrix organic light-emitting diode (active-matrix organic light-emitting diode). AMOLED, flexible light-emitting diode (FLED), Miniled, MicroLed, Micro-oLed, quantum dot light-emitting diode (QLED), etc. In some embodiments, the electronic device 100 may include one or N display screens 194, and N is a positive integer greater than one.

The electronic device 100 can realize a shooting function through an ISP, a camera 193, a video codec, a GPU, a display screen 194, and an application processor.

The ISP is used to process the data fed back from the camera 193. For example, when taking a picture, the shutter is opened, and the light is transmitted to the photosensitive element of the camera through the lens, the light signal is converted into an electrical signal, and the photosensitive element of the camera transmits the electrical signal to the ISP for processing, which is converted into an image visible to the naked eye. ISP can also optimize the image noise, brightness, and skin color. ISP can also optimize the exposure, color temperature and other parameters of the shooting scene. In some embodiments, the ISP may be provided in the camera 193.

The camera 193 is used to capture still images or videos. The object generates an optical image through the lens and is projected to the photosensitive element. The photosensitive element may be a charge coupled device (CCD) or a complementary metal-oxide-semiconductor (CMOS) phototransistor. The photosensitive element converts the optical signal into an electrical signal, and then transfers the electrical signal to the ISP to convert it into a digital image signal. ISP outputs digital image signals to DSP for processing. DSP converts digital image signals into standard RGB, YUV and other formats of image signals. In some embodiments, the electronic device 100 may include one or N cameras 193, and N is a positive integer greater than one.

The camera 193 may include a front camera and/or a rear camera. The camera 193 may also include multiple types. For example, the camera 193 may include a telephoto camera, a wide-angle camera, an ultra-wide-angle camera, etc. whose field of view angle varies from small to large. In the embodiment of the present application, in the panoramic shooting mode, the electronic device 100 may use a camera with a larger field of view (for example, an ultra-wide-angle camera or a wide-angle camera) to collect multiple frames of images at different angles, so as to have a larger view of the captured images. After cropping multiple frames of images in the viewing angle range, they are stitched into a panoramic image with a larger viewing angle range.

Digital signal processors are used to process digital signals. In addition to digital image signals, they can also process other digital signals. For example, when the electronic device 100 selects a frequency point, the digital signal processor is used to perform Fourier transform on the energy of the frequency point.

Video codecs are used to compress or decompress digital video. The electronic device 100 may support one or more video codecs. In this way, the electronic device 100 can play or record videos in multiple encoding formats, such as: moving picture experts group (MPEG) 1, MPEG2, MPEG3, MPEG4, and so on.

NPU is a neural-network (NN) computing processor. By drawing on the structure of biological neural networks, for example, the transfer mode between human brain neurons, it can quickly process input information, and it can also continuously self-learn. The NPU can realize applications such as intelligent cognition of the electronic device 100, such as image recognition, face recognition, voice recognition, text understanding, and so on.

The internal memory 121 may be used to store computer executable program code, and the executable program code includes instructions. The processor 110 executes various functional applications and data processing of the electronic device 100 by running instructions stored in the internal memory 121. The internal memory 121 may include a storage program area and a storage data area. Among them, the storage program area can store an operating system, at least one application program (such as a sound playback function, an image playback function, etc.) required by at least one function. The data storage area can store data (such as audio data, phone book, etc.) created during the use of the electronic device 100. In addition, the internal memory 121 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one magnetic disk storage device, a flash memory device, a universal flash storage (UFS), and the like.

In the embodiment of the present application, the processor 110 runs the instructions stored in the internal memory 121 to stitch images collected from different angles by the camera 193 in two mutually perpendicular directions to generate a panoramic image. Expand the angle of view of the panoramic image in two directions.

The electronic device 100 can implement audio functions through the audio module 170, the speaker 170A, the receiver 170B, the microphone 170C, the earphone interface 170D, and the application processor. For example, music playback, recording, etc.

The gyro sensor 180B may be used to determine the movement posture of the electronic device 100. In some embodiments, the angular velocity of the electronic device 100 around three axes (ie, x, y, and z axes) can be determined by the gyroscope sensor 180B. The gyro sensor 180B can be used for image stabilization. Exemplarily, when the shutter is pressed, the gyro sensor 180B detects the shake angle of the electronic device 100, calculates the distance that the lens module needs to compensate according to the angle, and allows the lens to counteract the shake of the electronic device 100 through reverse movement to achieve anti-shake. The gyro sensor 180B can also be used for navigation and somatosensory game scenes.

In the embodiment of the present application, the gyroscope sensor 180B may be used to calculate the rotation angle α of the mobile phone around the x-axis and the rotation angle β around the y-axis in the panoramic shooting process. The rotation angle α and the rotation angle β can be used to determine the shooting stage of the panoramic shooting process, determine the position of the deviation indicator corresponding to the center of the image frame, and determine whether the deviation range of the current image frame exceeds the maximum deviation range, and so on.

Touch sensor 180K, also called "touch panel". The touch sensor 180K may be provided on the display screen 194, and the touch screen is composed of the touch sensor 180K and the display screen 194, which is also called a “touch screen”. The touch sensor 180K is used to detect a touch operation acting on or near it, for example, a touch operation used to instruct to take a panoramic image. The touch sensor can pass the detected touch operation to the application processor to determine the type of touch event. The visual output related to the touch operation can be provided through the display screen 194. In other embodiments, the touch sensor 180K may also be disposed on the surface of the electronic device 100, which is different from the position of the display screen 194.

It can be understood that the structure illustrated in the embodiment of the present application does not constitute a specific limitation on the electronic device 100. In other embodiments of the present application, the electronic device 100 may include more or fewer components than shown, or combine certain components, or split certain components, or arrange different components. The illustrated components can be implemented in hardware, software, or a combination of software and hardware.

In the embodiment of the present application, in the panoramic shooting mode, the camera 193 can collect multiple frames of images at different angles. The display screen 194 can display the preview interface and the shooting interface in the panoramic shooting mode. The processor 110, by running the instructions stored in the internal memory 121, stitches the images collected by the camera 193 from different angles in two mutually perpendicular directions to generate a panoramic image, so that the panoramic image can be expanded in two directions. Angle of view. The gyro sensor 180B can be used to calculate the rotation angle α of the mobile phone around the x-axis and the rotation angle β around the y-axis in the panoramic shooting process. The rotation angle α and the rotation angle β can be used to determine the shooting stage of the panoramic shooting process, determine the position of the deviation indicator corresponding to the center of the image frame, and determine whether the deviation range of the current image frame exceeds the maximum deviation range, and so on.

The following will take the electronic device as a mobile phone and the screen of the mobile phone as a touch screen as an example to describe the panoramic shooting method provided in the embodiment of the present application. Referring to Figure 3, the method may include:

301. After the mobile phone enters the panoramic shooting mode, a guide path is displayed on a preview interface, and the guide path includes at least two parallel sub-paths set along a first direction.

For example, after the mobile phone detects that the user has clicked the camera icon 401 shown in (a) in FIG. 4A, it starts the camera application and enters the photographing mode shown in (b) in FIG. 4A. Exemplarily, after the mobile phone detects that the user has clicked the operation of the panoramic shooting control 402 shown in (b) in FIG. 4A, it enters the panoramic shooting mode and displays the preview interface as shown in (c) in FIG. 4A. For another example, after the mobile phone detects that the user clicks on the control 403 shown in (b) in FIG. 4A, it displays the interface shown in (d) in FIG. 4A; after the mobile phone detects that the user clicks on the control 404, , Enter the panoramic shooting mode, and display the preview interface as shown in (c) in Figure 4A.

It should be noted that the mobile phone can also enter the panoramic shooting mode in response to other touch operations, voice commands, or shortcut gestures of the user. The embodiment of the present application does not limit the operation of triggering the mobile phone to enter the panoramic shooting mode.

Among them, the preview interface in the panoramic shooting mode includes guide information for panoramic shooting. The guidance information includes the guidance path. The guide path is used to guide the user to rotate (and/or move) the mobile phone along the guide path during the shooting process, so that the center line of the image collected by the mobile phone moves along the guide path to complete panoramic shooting. The guide path includes at least two sub-paths arranged along the first direction and parallel to each other. The mobile phone can collect images from different angles along the sub-paths in the first direction, and stitch them along each sub-path to generate a panoramic image. When the mobile phone stitches images along a sub-path in the first direction, the angle of view of the panoramic image in the first direction can be expanded; when the sub-path in the first direction includes multiple sub-paths, the field of view in the first direction is expanded At the same time, the mobile phone can also expand the angle of view of the panoramic image in the second direction perpendicular to the first direction.

In the embodiment of the present application, the different sub-paths set along the first direction are used to guide the user to take multiple shots back and forth along the first direction to expand the angle of view of the panoramic image in the second direction perpendicular to the first direction. This kind of guidance can be precise guidance or general trend guidance.

For example, different sub-paths set along the first direction correspond to the same coordinate range in the first direction. The two ends of the different sub-paths arranged along the first direction are respectively aligned, and the different sub-paths have the same length, so that the user's shooting direction is accurately guided through each sub-path. Exemplarily, referring to (a) in FIG. 4B, the guide path includes a middle sub-path 41, an upper sub-path 42 and a lower sub-path 43 arranged along the first direction, and the middle sub-path 41, the upper sub-path 42 and the lower sub-path The coordinate ranges corresponding to the path 43 in the first direction are the same.

For another example, different sub-paths arranged along the first direction of the guide path overlap between corresponding coordinate ranges in the first direction, and the different sub-paths arranged along the first direction and parallel to each other are not completely staggered. The two ends of the different sub-paths are not aligned, and the lengths of the different sub-paths are not equal, so that the user's shooting direction is guided by the guide path. Exemplarily, referring to (b) in FIG. 4B, the guide path includes a middle sub-path 41, an upper sub-path 42 and a lower sub-path 43 arranged along the first direction, and the middle sub-path 41, the upper sub-path 42 and the lower sub-path There is a partial overlap between the corresponding coordinate ranges of the path 43 in the first direction.

In addition, as shown in (a) of FIG. 4B, the sub-path set along the first direction may be provided with a direction indicator (for example, an arrow on the sub-path) to guide the direction of rotation when the mobile phone rotates along the sub-path. Of course, as shown in (b) of FIG. 4B, the sub-path set along the first direction may not be provided with a direction indication mark, and the user only needs to complete the rotation along the sub-path, and the specific rotation direction of the user is not limited.

In addition, the mobile phone can remind the user of the sequence of shooting along different sub-paths in the first direction by means of display information or voice broadcast. For example, the mobile phone can prompt the user through text messages to first rotate along the middle sub-path during the shooting, and then rotate along the upper sub-path and the lower sub-path to shoot. For another example, the mobile phone can remind the user of the sequence of shooting along different sub-paths by indicating arrows or other indicating methods. Exemplarily, referring to (c) in Figure 4B, the arrow 44 is used to instruct the user to first turn and shoot along the middle sub-path, and then to rotate along the upper sub-path to shoot; the arrow 45 is used to instruct the user to rotate and shoot along the upper sub-path , And then rotate and shoot along the sub-path below.

In some embodiments, the guide path may further include at least one sub-path arranged along the second direction. The sub-paths set along the second direction are used to remind the user of the sequence of shooting of different sub-paths along the first direction. In a possible implementation, as shown in (d) in FIG. 4B, the guide path further includes a right sub-path 46 and a left sub-path 47. The right sub-path 46 is used to instruct the user to first follow the middle sub-path. The path rotates and shoots, and then rotates along the upper sub-path to shoot; the left sub-path 47 is used to instruct the user to rotate along the upper sub-path for shooting, and then rotate along the lower sub-path to shoot.

In another possible implementation, the sub-path in the second direction is used to connect the sub-path in the first direction, and the entire guide path is a continuous path. When the sub-paths arranged along the second direction include multiple sub-paths, the multiple sub-paths are parallel to each other. Exemplarily, the guide path may refer to the path 405 shown in (c) in FIG. 4A.

It is understandable that the mobile phone can also remind the user of the sequence of shooting along different sub-paths in the first direction in various other ways, which is not limited in the embodiment of the present application.

Wherein, the first direction and the second direction may be two directions respectively parallel to two adjacent sides of the mobile phone, and the second direction is perpendicular to the first direction.

When the user holds the mobile phone for panoramic shooting, one side of the mobile phone is usually parallel to the horizontal direction (or the angle between one side of the mobile phone and the horizontal direction is less than or equal to the preset value, that is, one side of the mobile phone is basically parallel to the horizontal direction. Parallel), the other side adjacent to the side is usually parallel to the vertical direction (or the angle between the other side of the mobile phone and the vertical direction is less than or equal to the preset value, that is, the other side of the mobile phone is The vertical direction is basically parallel). That is, the first direction and the second direction may be a horizontal direction and a vertical direction.

The following describes the different directions of the first direction and the second direction respectively:

Case 1. The first direction is the horizontal direction (or horizontal direction), and the second direction is the vertical direction (or vertical direction).

When the first direction is the horizontal direction and the second direction is the vertical direction, the guide path includes horizontal and vertical sub-paths, and the horizontal sub-path may include at least two, and the vertical sub-path is used for Connect the sub-paths in the horizontal direction. Exemplarily, referring to (a) in FIG. 5, the guide path includes three sub-paths: the middle sub-path 501, the upper sub-path 502, and the lower sub-path 503 in the horizontal direction, and the left sub-path 504 and the lower sub-path 503 in the vertical direction. The right sub-path 505 is two sub-paths. The guide path on the preview interface is used to remind the user of the complete path that needs to be taken during the entire shooting process.

In the panoramic shooting mode, the guide information on the preview interface may also include a deviation indicator mark, which is used to indicate the position of the center line (ie, the horizontal center line) of the image currently collected by the camera in real time in the first direction. In some embodiments, the deviation indicator arrow may also point to the direction to be moved of the deviation indicator arrow, that is, to the end direction of the initial sub-path, that is, to the direction to be rotated of the mobile phone. On the preview interface, the deviation indicator is located at the beginning of the starting sub-path. Exemplarily, in the case shown in (b) in FIG. 5, the deviation indicator mark may be a deviation indicator arrow 506, the starting sub-path is the middle sub-path 501, and the deviation indicator arrow 506 is located on the middle sub-path 501 on the preview interface. The left end of and points to the right end of the middle sub-path 501. It is understandable that the deviation indicator mark may also be a deviation indicator line or other forms, which is not limited in the embodiment of the present application. The following will take the deviation indicator as the deviation indicator arrow as an example for description.

In some embodiments, the guidance information on the preview interface may also include a deviation range indication line of the sub-path. Since the user is prone to shaking when holding the mobile phone, the position of the horizontal center line of the image collected by the mobile phone will usually change, and the position of the deviation indicator arrow will also change. The deviation range indicator line is located on both sides of the sub-path, and is parallel to the sub-path, and is used to indicate the maximum allowable range of the deviation indicator arrow from the sub-path. On the preview interface, the mobile phone can only display the deviation range indicator line of the starting sub-path. Exemplarily, in the case shown in (b) of FIG. 5, the starting sub-path is the middle sub-path, and the deviation range indication lines 501a-501b may be located on both sides of the middle sub-path 501, and are in line with the middle sub-path. 501 Parallel dotted lines.

In some embodiments, in the preview state, the mobile phone may prompt the user by displaying information or voice broadcast, etc., that the deviation indicator arrow overlaps with the guide path as much as possible, and should not exceed the deviation range indicator line. Exemplarily, referring to (c) in FIG. 4A, the mobile phone can prompt the user through text messages on the preview interface: After starting to shoot, slowly turn the mobile phone so that the arrow moves along the guide path and does not exceed the range of the dotted line.

In some embodiments, referring to (b) in FIG. 5, the preview interface may further include a splicing preview window 507. On the preview interface, the splicing preview window is located at the beginning of the guide path, and the deviation indicator arrow can be located beside the splicing preview window and on the side where the end of the starting sub-path is located. For example, as shown in FIG. 5(b), the splicing preview window is located at the beginning of the starting sub-path 501, and the deviation indicating arrow is located on the right side of the splicing preview window. As shown in (c) of FIG. 4A, in the preview state, the stitching preview window is used to display the thumbnail of the preview image currently displayed on the preview interface (or called the preview image of the preview image).

In some embodiments, the stitching preview window will occupy or block a part of the guide path. It can be seen from Fig. 5(b) that the lengths of the sub-path 501, the sub-path 502 and the sub-path 503 are equal, but because the splicing preview window will block a part of the guide path, as shown in Fig. 5(b), the sub-path 502 and sub-path 503 are indented compared to the left end of sub-path 501, and the right ends of the three sub-paths are aligned. In other implementations, since the splicing preview window occupies a certain area, the length of the sub-path 501 itself is smaller than the sub-path 502 and the sub-path 503, and compared with the sub-path 502 and the sub-path 503, the left end of the sub-path 501 is reduced. Then, the left ends of the sub-path 502 and the sub-path 503 are aligned, and the right ends of the three sub-paths are also aligned.

302. After the mobile phone detects the user's shooting operation, it displays an image obtained by splicing images collected according to the guide path on the shooting interface.

Exemplarily, after the mobile phone detects that the user has clicked the operation of the shooting control 406 shown in (c) of FIG. 4A, it starts to shoot a panoramic image and displays the shooting interface. The mobile phone displays the images collected by the camera on the shooting interface in real time. It is understandable that the mobile phone can also start panoramic image shooting in response to operations such as voice instructions or shortcut gestures of the user. The embodiment of the present application does not limit the operation of triggering the mobile phone to start panoramic image shooting.

The mobile phone displays guide information on the shooting interface, and the guide information includes a guide path, a deviation indicator arrow, and a deviation range indicator line. Exemplarily, after the mobile phone detects the shooting operation, the displayed shooting interface may refer to (a) in FIG. 6. Wherein, the deviation indicator arrow is used to remind the user of the position of the center line of the image currently collected by the camera and the degree of deviation between the position of the center line and the target sub-path. In some embodiments, the deviation indicator arrow may also point to the direction to be moved of the deviation indicator arrow, that is, to the end direction of the target sub-path, that is, to the direction to be rotated of the mobile phone. Wherein, the target sub-path is a sub-path that deviates from the indicated arrow currently moving along it. After the shooting starts, the target sub-path is the starting sub-path (for example, the intermediate sub-path described above), and as the shooting process proceeds, the target sub-path will switch. Wherein, when the target sub-path is a horizontal sub-path (that is, a sub-path set in the horizontal direction), the deviation indicating arrow moves along the horizontal sub-path, and the deviation indicating arrow is used to indicate the position of the horizontal center line of the image collected by the mobile phone. The horizontal center lines of the different images move along the horizontal sub-path. When the target sub-path is a vertical sub-path (that is, a sub-path set in the vertical direction), the deviation indicating arrow moves along the vertical sub-path, and the deviation indicating arrow is used to indicate the position of the vertical center line of the image collected by the mobile phone. The vertical center lines of the different images collected by the mobile phone move along the vertical sub-path.

During the shooting process, the mobile phone can prompt the user by displaying information or voice broadcast, etc., slowly turning the mobile phone along the guide path, and the deviation indicator arrow should coincide with the guide path as much as possible, and do not exceed the deviation range indicator line. Exemplarily, referring to (a) in FIG. 6, the mobile phone can prompt the user through a text message: Please turn the mobile phone slowly so that the arrow moves along the guide path and does not exceed the range of the dotted line. When the deviation indicator arrow deviates from the target sub-path, the mobile phone can prompt the user to move the mobile phone to coincide the deviation indicator arrow with the target sub-path. Exemplarily, referring to (b) in FIG. 6, when the deviation indicating arrow is located below the middle sub-path, the mobile phone may prompt the user: Please move up so that the arrow coincides with the guide path.

Just after entering the shooting process, the guide path displayed on the shooting interface is a complete guide path. Then, the mobile phone rotates to the right so that the deviation indicator arrow moves to the right along the horizontal sub-path. During the shooting process, the phone keeps turning to collect images from different angles, so that the deviation indicator arrow moves to the right along the middle sub-path, upwards along the right sub-path, and to the left along the upper sub-path, along the The left sub-path moves down, and moves to the right along the lower sub-path, until the entire guide path is taken. During the shooting process, the sub-path that deviates from the current movement of the indicating arrow is the target sub-path. For example, when the deviation indicating arrow moves along the middle sub-path, the target sub-path is the middle sub-path; when the deviation indicating arrow moves along the right sub-path, the target sub-path is the right sub-path.

In some embodiments, in order to better guide the user to shoot, in different shooting stages, the guide path displayed on the shooting interface will change with the user's shooting process, and the shooting interface may only display the guide path of the unfinished shooting. , That is, the part of the guide path displayed on the preview interface that deviates from the indicated arrow. Among them, because in the shooting process, the deviation indicator arrow can also fluctuate within the deviation range indicator line when it moves along the guide path. Therefore, it can be understood that deviation from the guide path that the indicating arrow passes includes, but is not limited to, the part that the deviation indicating arrow passes through when the indicating arrow coincides with the guide path, and also includes the deviation indicating arrow that does not coincide with the guide path and is along the deviation range indicating line. The part of the guide path when it moves. In other embodiments, the mobile phone only displays the current target sub-path. In other embodiments, the mobile phone displays the complete guide path on the shooting interface, and the display mode of the guide path for the completed shooting and the guide path for the incomplete shooting is different. For example, the guide path for unfinished shooting is a solid line, and the guide path for completed shooting is a dotted line. In other embodiments, the mobile phone displays the complete guide path on the shooting interface until the shooting is completed.

In addition, the shooting interface also includes a stitching preview window, which is used to display thumbnails (or called previews of stitched images) of images obtained by stitching by mobile phones during shooting. On the shooting interface, the deviation indicator arrow can be located beside the stitching preview window and on the side where the end of the target subpath is located. On the shooting interface, the size, position, and shape of the stitching preview window correspond to the guide path of the completed shooting. For example, the stitching preview window on the shooting interface can cover the guide path of the completed shooting. After the start of shooting, the mobile phone collects image 1, the mobile phone displays image 1 on the shooting interface, and displays image 1 in the stitching preview window. During the shooting process, when the phone rotates along the guide path, the shooting angle and shooting range of the camera also change, so that images from different angles can be collected. The phone can stitch images from different angles and display them in the stitching preview window. The image displayed by the mobile phone in the splicing preview window can be specifically the target image obtained by splicing the collected images when the mobile phone rotates along the horizontal sub-path in the first direction, that is, when the deviation indicator moves along the horizontal sub-path in the first direction The target image obtained by splicing the collected images by the mobile phone.

During the shooting process, images of different angles collected by the mobile phone can be cropped and stitched according to preset rule lines. As shown in FIG. 7, the rule line includes an upper baseline, a middle baseline, and a lower baseline, which correspond to the upper sub-path, the middle sub-path, and the lower sub-path in the horizontal direction, respectively. The rule line also includes the first cutting line, the second cutting line, the third cutting line, and the fourth cutting line. Wherein, the first cutting line and the second cutting line are located on both sides of the upper baseline, and respectively correspond to the deviation range indication lines on both sides of the upper sub-path in the horizontal direction. The first cropping line and the second cropping line constitute the upper and lower cropping ranges of the upper baseline image (that is, the image collected when the mobile phone rotates along the upper sub-path), which limits the images collected during the process of the mobile phone rotating along the upper sub-path. The maximum deviation range of the horizontal centerline. The second cutting line and the third cutting line are located on both sides of the middle baseline, and respectively correspond to the deviation range indication lines on both sides of the middle sub-path in the horizontal direction. The second cropping line and the third cropping line constitute the upper and lower cropping ranges of the mid-baseline image (that is, the image collected when the mobile phone rotates along the middle sub-path), which limits the image collected during the shooting of the mobile phone while rotating along the middle sub-path The maximum deviation range of the horizontal centerline. The 3rd cutting line and the 4th cutting line are located on both sides of the lower baseline, and respectively correspond to the deviation range indicator lines on both sides of the upper and lower subpaths in the horizontal direction. The 3rd and 4th cutting lines form the upper and lower cutting ranges of the lower baseline image (that is, the image collected when the mobile phone rotates along the lower sub-path), which limits the images collected during the process of the mobile phone rotating along the lower sub-path. The maximum deviation range of the horizontal centerline.

In addition, the ruled line also includes a left baseline and a right baseline, which respectively correspond to the left sub-path and the right sub-path in the vertical direction. And, the rule line also includes a left boundary, a left clipping line, a right clipping line, and a right boundary. Among them, the left border and the left clipping line are located on both sides of the left baseline, and respectively correspond to the deviation range indication lines on both sides of the left sub-path in the vertical direction. The left border and the left crop line define the left and right maximum deviation range of the vertical center line of the left baseline image (ie, the image collected when the mobile phone rotates along the left sub-path). In addition, the left border also defines the left border when stitching the panoramic image, and the part of the image beyond the left border can be directly cut off when stitching.

The right border and the right clipping line are located on both sides of the right baseline, and respectively correspond to the deviation range indication lines on both sides of the right sub-path in the vertical direction. The right border and the right crop line define the left and right maximum deviation range of the vertical center line of the right baseline image (that is, the image collected when the mobile phone rotates along the right sub-path). In addition, the right border also defines the right border when stitching the panoramic image, and the part of the image beyond the right border can be directly cut off when stitching.

Wherein, the size of the deviation indication range corresponding to different baselines may be the same or different. It should be noted that, for ease of illustration, each of the regular lines shown in FIG. 7 is a straight line on a plane, and in fact, each regular line is a curve on a cylindrical surface.

In the panoramic image shooting process, the posture of the mobile phone when the image is collected and the image collection time can be used to mark the position deviating from the indicator arrow, to switch the target sub-path, and to determine whether the center line of the image exceeds the cropping range, etc. Exemplarily, in a three-dimensional coordinate system shown in (a) of FIG. 8, the three-dimensional coordinate axes include x-axis, y-axis, and z-axis. The plane where the mobile phone lens is located is on the xy plane or parallel to the xy plane, and the z axis is perpendicular to the plane where the mobile phone lens is located. The plane where the lens of the mobile phone is located is also parallel to the screen where the screen of the mobile phone is located. In this three-dimensional coordinate system, the posture of the mobile phone when collecting images can be represented by the rotation angle of the mobile phone around the x-axis, y-axis, and z-axis. When the mobile phone rotates along the horizontal sub-path, the mobile phone rotates around the y-axis. When the mobile phone rotates along the vertical sub-path, the mobile phone rotates around the x-axis.

The mobile phone gyroscope outputs the three-axis rotational angular velocity ρ _α , ρ _β , ρ _{γ of the} mobile phone at the corresponding time according to the preset frequency. At the same time, the mobile phone adds a corresponding time stamp t to the gyroscope data. In addition, the mobile phone will stamp a corresponding timestamp T at the moment when the image frame is generated. Generally, referring to (b) in Figure 8, the sampling frequency of the gyroscope is much higher than the video frame, so there may be multiple gyroscope data between two adjacent frames.

Among them, the rotation angle α'of the mobile phone around the x-axis between the collection moments of two adjacent frames of images can be obtained by Equation 1:

In formula 1, T ₀ represents the time corresponding to the previous frame, and T ₁ represents the time corresponding to the next frame,

Represents the rotational angular velocity of the k-th gyroscope between _{T 0} and T _1,

Represents the rotational angular velocity of the k-1th gyroscope between _{T 0} and T ₁ _{, t k} represents the time corresponding to the k-th gyroscope data, t _k-1 represents the time corresponding to the k-1th gyroscope data,

Represents the rotational angular velocity of the 0th gyroscope between _{T 0} and T ₁ _{, t 0} represents the time corresponding to the 0th gyroscope data,

It represents the rotational angular velocity of the Nth gyroscope between _{T 0} and T ₁ _{, and t N} represents the time corresponding to the Nth gyroscope data. Among them, there are 0-N gyroscope data between _{T 0} and T _1.

Similarly, the mobile phone can also obtain the rotation angle of the mobile phone around the y-axis between the collection moments of two adjacent frames of images. By accumulating the rotation angle of the phone between frames, the rotation angle α of the phone around the x-axis and the rotation angle β around the y-axis at the time of collection of any frame of image can be obtained (also called the rotation angle α corresponding to any frame of image And the rotation angle β).

Among them, the middle baseline corresponds to the horizontal center line of the first frame image collected by the mobile phone after the start of shooting, that is, α=0. The horizontal centerline of the first frame of image mapped to the cylindrical surface coincides with the center baseline on the cylindrical surface. The upper and lower baselines are rotated by a certain angle around the x-axis relative to the middle baseline, and the rotation angles are denoted by α _t and α _{b respectively} . The rotation angles of the horizontal cutting lines 1-4 around the x-axis can also be denoted as α ₁ , α ₂ , α ₃ , and α _{4, respectively} . The values of these rotation angles can be used to adjust the extension range of the vertical view angle when stitching panoramic images, and the maximum range within which the horizontal center line of the image can deviate from the baseline.

Similarly, the left baseline corresponds to the vertical center line of the first frame image collected by the mobile phone after the start of shooting, that is, β=0. The right baseline rotates a certain angle around the y axis relative to the left baseline, which is recorded as β _r . The rotation angles of the left border, the left clipping line, the right clipping line, and the right border around the y-axis can be denoted as β ₁ , β ₂ , β ₃ , β _{4, respectively} . Among them, the corresponding relationship between each ruled line and the rotation angle can be seen in FIG. 7.

The mobile phone can determine the horizontal centerline of the current image by the rotation angle α of the mobile phone when each frame of image is collected, thereby drawing a deviation indicator arrow corresponding to the rotation angle α on the shooting interface. The mobile phone can determine whether the deviation indicator arrow exceeds the deviation range indicator line of the horizontal sub-path according to the rotation angle α, and the mobile phone can also determine whether the deviation indicator arrow exceeds the deviation range indicator line of the vertical sub-path according to the rotation angle β. The mobile phone can also determine whether to switch the target sub-path according to the rotation angle α and the rotation angle β. Moreover, the target sub-path is different, the image stitching method is also different. Hereinafter, the guide path shown in (a) of FIG. 4A will be used as an example for description.

(1) The target sub-path is the middle sub-path, and the baseline image is spliced

After shooting, the phone first rotates to the right along the middle sub-path. The image collected by the phone can be called the middle baseline image or the image of the middle sub-path. The target sub-path is the middle sub-path, and the rotation angle β Constantly changing from 0.

After the mobile phone collects the baseline image in the first frame (that is, the first frame image of the intermediate sub-path), it performs cylindrical mapping on the first frame image according to formula 2-4 to obtain the baseline image' in the first frame on the cylindrical surface.

Among them, α represents the rotation angle of the mobile phone around the x axis, f represents the focal length parameter of the mobile phone lens, (c _x , c _y ) is the coordinate of the image center point, (x, y) and (x', y') respectively represent the mapping Corresponding pixels in the front and back images. Exemplarily, for the image frame collected by the camera, when the rotation angle α=0, α>0 and α<0, the corresponding cylindrical surface mapping results can be seen in (a), (b) and ( c). Among them, when α=0, the scale changes of the upper and lower boundaries of the image after mapping are symmetrical. When α>0, the phone rotates upward by a certain angle, and the scale change of the upper boundary of the image after mapping is greater than the scale change of the lower boundary. When α<0, the phone rotates downward by a certain angle, and the scale change of the upper boundary of the image after mapping is smaller than the scale change of the lower boundary.

Among them, since the position of the middle baseline is defined according to the horizontal center line of the baseline image in the first frame, the rotation angle α = 0 corresponding to the baseline image in the first frame (that is, the rotation of the mobile phone when the mobile phone collects the baseline image in the first frame) Angle α=0), the mapping result of the baseline image in the first frame is symmetric with respect to the middle baseline level. The left edge of the baseline image in the first frame is the left edge of the entire panoramic image. See Figure 10. The part of the baseline image in the first frame that lies within the cropping range is the initial mid-baseline image stitching result RI1, that is, the deviation indicating arrow edge The initial target image RI1 obtained by stitching when the middle sub-path moves. Exemplarily, referring to (a) in Figure 6, the mobile phone displays the middle baseline image stitching result RI1 in the stitching preview window.

It is understandable that when the mobile phone shoots images in different postures, the angles of the captured images are different, and the size and imaging characteristics of the same object on the images shot at different angles are also different, so it is impossible to directly register the images shot at different angles. And stitching to generate a panoramic image. In the embodiment of the present application, images with different angles taken in different poses can be first mapped to a cylindrical surface, so that the size and imaging characteristics of the same object on the images taken at different angles can be matched, so that registration and stitching can be performed. Generate a panoramic image, which can meet the visual effect of basically the same size of each part of the panoramic image.

Then, the user rotates the mobile phone along the middle sub-path, the mobile phone collects the baseline image in the i-th frame (i is an integer greater than 1), denoted as I _i , the mobile phone performs a cylindrical surface on the baseline image in the i-th frame After mapping, get the baseline image in the i-th frame'

The previous frame image of the baseline image I _i in the i-th frame is the baseline image in the i-1th frame, denoted as I _i-1 . The cylindrical surface mapping result of I _{i-1 is recorded as}

As shown in Figure 11, the mobile phone extracts according to the preset algorithm

with

Feature point F _{I, i} and F _{I, i-1} and then calculate F _{I, i} and F _{I, i-1} the matching result. The matching result is used to indicate the corresponding relationship between the feature points that match each other in _{F I,i} and F _I,i-1. For example, the matching result may be a combination, including multiple matching feature point pairs.

The mobile phone further calculates according to the matching result

with

The homography matrix H. See (a) in Figure 12, the mobile phone will

Towards

Map it. The phone will be mapped

The rectangular part within the cropping range (that is, the rectangular part filled with horizontal lines in (a) in Figure 12) is cropped and spliced to the middle baseline image stitching result RI(i-1) (that is, in (a) in Figure 12). The right side of the rectangular part filled with stripes); thereby forming the middle baseline image stitching result RIi, that is, the target image RIi obtained by stitching when the deviation indicator arrow moves along the middle sub-path. Among them, the cropping range corresponding to the middle baseline image is the range defined by the second cropping line, the third cropping line, the left boundary line, and the right boundary line. Exemplarily, the mid-baseline image stitching result RIi displayed in the stitching preview window of the mobile phone can be seen in (b) in FIG. 12.

Then, the user continues to turn the phone along the middle sub-path. The mobile phone repeats the above collection, mapping and splicing process until β>β _r . At this time, all the mid-baseline image frames collected by the mobile phone along the middle sub-path are stitched, thereby generating a middle-baseline image stitching result RI, that is, the stitching result RI corresponding to the middle sub-path. Note that, in the final baseline image for a case in β≤β _r mapped to image the cylindrical surface, cut lines may be used as the right boundary of the right side thereof, beyond the right boundary portion may be cut.

It should be noted that during the stitching process of the mid-baseline image, the mobile phone can select key frames according to the preset algorithm and combined with the rotation angle α corresponding to the mid-baseline image, so that these key frames overlap partially and are basically evenly distributed. On the midline. These key frames can make the upper and lower baseline image stitching results and the middle baseline image stitching results better match and merge. Among them, the interval Δβ of the rotation angle around the y-axis between adjacent key frames can be flexibly set according to factors such as the size of the buffer or actual requirements. Exemplarily, refer to FIG. 13 for a schematic diagram of the key frames acquired in the mid-baseline image.

Wherein, in the process of the deviation indicating arrow moving to the right along the middle sub-path, the rotation angle β changes continuously with the rotation of the mobile phone. In some technical solutions, referring to (b) in FIG. 12, the mobile phone displays the middle sub-path, the right sub-path, the upper sub-path, the left sub-path, and the lower sub-path that have not been photographed. In other technical solutions, see (a) in Figure 14A, the mobile phone only displays the currently targeted middle sub-path; see (b) in Figure 14A, and then display the right sub-path when it is about to switch to the right sub-path . In some other technical solutions, the mobile phone displays the complete shooting path on the shooting interface, and the guide path for the completed shooting is a dotted line, and the guide path for the unfinished shooting is a solid line. In other embodiments, the mobile phone still displays the complete guidance path.

During the shooting process, when the guide path does not include the sub-path in the second direction, the mobile phone can prompt the user to take the sequence of different sub-paths in the first direction through the sub-path in the second direction. During the shooting process, when the guide path does not include the sub-paths in the second direction, the mobile phone can prompt the user to take pictures along the different sub-paths by means of display information or voice broadcast. A description will be given by taking the guide path as the path shown in (a) in FIG. 4B as an example. Exemplarily, referring to (a) in Figure 14B, the mobile phone displays the middle sub-path (that is, the part of the middle sub-path that deviates from the indicating arrow that has not passed), the upper sub-path, and the arrow pointing to the upper sub-path. It is used to remind the user to continue shooting along the upper sub-path after the shooting along the middle sub-path is completed. For another example, referring to (b) in Figure 14B, the mobile phone displays the middle sub-path, the upper sub-path, the lower sub-path and the arrow pointing to the upper sub-path of the unfinished shooting. The arrow is used to prompt the user to move along the middle sub-path. After the path shooting is completed, continue shooting along the upper sub-path. For another example, the mobile phone displays the middle sub-path, upper sub-path and lower sub-path that have not been photographed. When the deviating indicator arrow reaches the end of the middle sub-path, the mobile phone highlights the upper sub-path to prompt the user to follow the upper sub-path next. The path turns the phone to shoot.

On the shooting interface, deviation range indication lines are displayed on both sides of the middle sub-path. During the shooting along the middle sub-path, that is, when the target sub-path is the middle sub-path, if α ₃ <α<α ₂ , then the deviation indication The arrow is located within the maximum deviation range of the middle sub-path; if α>α ₂ , or α<α ₃ , the deviation indicator arrow exceeds the maximum deviation range of the middle sub-path, and the mobile phone stops shooting.

Among them, when the deviation indicator arrow exceeds the maximum deviation range of the middle sub-path, the image collected by the mobile phone may not include the complete cropping range. The size of the image retained by the mobile phone after cropping according to the preset cropping range is smaller than the cropping range, that is, less than The other images captured by the camera are cropped and retained. Therefore, after the retained image is spliced to the panoramic image, the panoramic image will have blank parts, which will result in poor splicing effect. Therefore, the mobile phone can stop the shooting process. In some embodiments, the mobile phone may also prompt the user on the shooting interface or by voice or other means that the camera has automatically stopped shooting if the maximum deviation range is exceeded.

(2), the target sub-path is the right sub-path

When the deviating indicator arrow reaches the end of the middle sub-path, and the rotation angle of the phone β> β _r or β ₃ <β< β ₄ , the image on the middle sub-path has been taken, and the user is guided to turn the phone upwards along the right sub-path. Make the deviation indicator arrow move upward along the right sub-path. In the process of deviating the indicating arrow and moving upward along the right sub-path, the rotation angle α changes continuously with the rotation of the mobile phone. In some technical solutions, referring to FIG. 15, the mobile phone displays the unfinished right sub-path 1501, the upper sub-path, the left sub-path, and the lower sub-path 1501. Among them, the dotted lines on both sides of the right sub-path 1501 represent the deviation range indicator line, and the arrow 1502 represents the deviation indicator arrow. In some other technical solutions, the mobile phone only displays the currently targeted right sub-path. In some other technical solutions, the mobile phone displays the complete shooting path on the shooting interface, and the guide path for the completed shooting is a dotted line, and the guide path for the unfinished shooting is a solid line. In other embodiments, the mobile phone still displays the complete guidance path.

Among them, the deviation range indicator lines are displayed on both sides of the right sub-path. During the shooting along the right sub-path, if the rotation angle β ₃ <β <β ₄ , the deviation indicates the maximum deviation of the arrow on the right sub-path Within the range, the last frame image taken by the mobile phone along the right sub-path, that is, the baseline image on the first frame, can be cropped and spliced; if the rotation angle β<β ₃ or β>β ₄ , then If the deviation indicator arrow exceeds the maximum deviation range of the right sub-path, it is easy to make the last frame of the image taken by the mobile phone along the right sub-path, that is, the baseline image on the first frame, exceeds the cropping range, so the mobile phone can stop shooting. For example, when β>β ₄ , the mobile phone can also prompt the user on the shooting interface or through voice and other means: beyond the right boundary, the shooting has been automatically stopped.

It should be noted that during the shooting process where the target sub-path is the right sub-path, the mobile phone may not perform image stitching.

(3) The target sub-path is the upper sub-path, and the baseline image is spliced

When the deviation indicator arrow reaches the end of the right sub-path, and the rotation angle of the mobile phone is α>α _t or α ₂ <α<α ₁ , the user is guided to turn the mobile phone to the left along the upper sub-path, so that the deviation indicator arrow follows the upper sub-path move to the left. When the mobile phone rotates along the upper sub-path, the image collected by the mobile phone can be called the upper baseline image or the image of the upper sub-path, and the target sub-path is the upper sub-path. It is understandable that the first frame of the upper baseline image taken by the mobile phone along the upper sub-path is also the last frame of the image taken along the right sub-path. In the process of deviating the indicating arrow and moving to the left along the upper sub-path, the rotation angle β changes continuously with the rotation of the mobile phone. In some technical solutions, referring to FIG. 16A, the mobile phone displays the upper sub-path 1601, the left sub-path, and the lower sub-path that have not been photographed. Among them, the dotted lines on both sides of the upper sub-path 1601 represent the deviation range indicator line, and the arrow 1602 represents the deviation indicator arrow. In some other technical solutions, the mobile phone only displays the currently targeted upper sub-path. In some other technical solutions, the mobile phone displays the complete shooting path on the shooting interface, and the guide path for the completed shooting is a dotted line, and the guide path for the unfinished shooting is a solid line. In other embodiments, the mobile phone still displays the complete guidance path.

Among them, the deviation range indicator lines are displayed on both sides of the upper sub-path. During the shooting along the upper sub-path, if α ₂ <α<α ₁ , the deviation indicator arrow is located within the maximum deviation range of the upper sub-path; if α >α ₁ , or α<α ₂ , the deviation indicator arrow exceeds the maximum deviation range of the upper sub-path, and the mobile phone stops shooting. In some embodiments, the mobile phone may also prompt the user on the shooting interface or by voice or other means that the camera has automatically stopped shooting if the maximum deviation range is exceeded.

The mobile phone stitches the upper baseline image, which specifically includes processes such as cylindrical surface mapping, feature extraction, feature matching, and image stitching. Among them, for the baseline image A ₁ _{on the first frame, the mobile phone determines a target key frame whose rotation angle β is closest to the rotation angle β 1} corresponding to the baseline image on the first frame from the above key frames according to a preset algorithm as the reference frame G _f1 , that is, G _f1 is a key frame matching _{A 1. In the} above key frames, the difference between the rotation angles around the y axis corresponding to _{G f1} and A _{1 is the smallest.} In this way, _{the image misalignment error of the G f1} and A ₁ images during stitching is the smallest, and the images after the two are mapped to the cylindrical surface are easier to register. Then, the mobile phone extracts the image after _{G f1 cylindrical surface mapping}

The feature points F _{A, f1} . _{The mobile phone maps the upper baseline image A 1 in the} first frame to the cylindrical surface according to the above formula 2-4 to obtain the upper baseline image in the first frame.

And get the baseline image on the first frame'

The feature points F _{A, f1} . The mobile phone calculates _{the matching result of F A,1} and F _A,f1 , and calculates the homography matrix H according to the matching result. The H matrix obtained according to the target key frame is more accurate, and can be better registered with the key frame in the mid-baseline image, so that it can be matched with the mid-baseline splicing result. Phone will

Towards

Map it. Refer to Figure 16B, the mobile phone will map the

The rectangular part within the cropping range is cropped and stitched to the upper right of the middle baseline image stitching result RI to form the upper baseline image stitching result RA1 (that is, the horizontal line filling part), that is, stitching when the deviation indicator arrow moves along the upper sub-path The obtained target image RA1. Wherein, the cropping range corresponding to the upper baseline image is the range defined by the first cropping line, the second cropping line, the left boundary line, and the right boundary line. Exemplarily, for a schematic diagram of the upper baseline image splicing result RA1 spliced above the middle baseline image splicing result RI displayed on the shooting interface of the mobile phone, please refer to FIG. 16A.

The first I (an integer greater than 1) a baseline frame of image A _i, the mobile phone according to a preset algorithm, determining a rotation angle beta] and A _i β _i closest to the rotational angle of the target from said key frames in the key frame as a reference frame G _fi . That is, G _fi is a key frame that matches _{A i.} Refer to Figure 16C, the mobile phone extracts the image after the _{G fi cylindrical surface mapping}

The feature point F _{A,fi of} _{, the mobile phone maps A i} to the cylindrical surface according to the above formula 2-4, so as to obtain the baseline image on the i-th frame.

The feature points F _A,fi of the mobile phone can also extract the baseline image A _i-1 on the i-1th frame after mapping it to the cylindrical surface, and the baseline image on the i-1th frame is obtained.

The feature point F _A,i-1 . The mobile phone calculates _{the matching results of F A,i} , F _A,i-1 , and F _A,fi to calculate the homography matrix H. The H matrix obtained in this way is more accurate, and can be better registered with the key frame while registering with the baseline image on the previous frame. Phone will

Towards

with

Map it. See Figure 16D, the phone will map

The rectangular part within the cropping range is cropped and stitched to the left of the upper baseline image stitching result RA(i-1) and the upper side of the middle baseline image stitching result RI, thereby forming the upper baseline image stitching result RAi (i.e. horizontal Line filling part), that is, the target image RAi obtained by stitching when the deviation indicator arrow moves along the upper sub-path. Exemplarily, referring to FIG. 16E, the mobile phone displays the upper baseline image stitching result RAi that is stitched above the middle baseline image stitching result RI in the stitching preview window.

Then, the user continues to turn the phone along the upper sub-path. The mobile phone repeats the above collection, mapping and splicing process until β<0. At this time, all the upper baseline image frames collected by the mobile phone along the middle sub-path are spliced, thereby generating the upper baseline image splicing result RA, that is, the splicing result RA corresponding to the upper sub-path. It should be noted that for the image where the baseline image on the last frame is mapped to the cylindrical surface under the condition of β≤0, the left border can be used as the left side of the crop line, and the part beyond the left border can be cut off.

It should be noted that if the upper baseline image is not configured according to the key frame, the registration of the middle baseline image and the registration of the upper baseline are performed separately, and the two processes are separate, the upper baseline image stitching result and the middle baseline image The stitching results cannot correspond, and the stitching errors of the two baseline image stitching results are easy to accumulate, which causes a large misalignment in the stitching results of the two baseline images. In the embodiment of the present application, the mobile phone registers the key frames in the upper baseline image and the middle baseline image, which can correct the misalignment error between the upper baseline image and the middle baseline splicing result in time, so that the upper baseline image and the middle baseline image are spliced The results are accurately registered to achieve global registration. The upper baseline image stitching result and the middle baseline image stitching result can form a smooth, natural transition overall image.

(4), the target sub-path is the left sub-path

When the deviating indicator arrow reaches the end of the upper sub-path, and the rotation angle of the phone β<0 or β ₁ <β<β ₂ , the image on the upper sub-path has been taken, and the user is guided to turn the phone down along the left sub-path. Make the deviation indicator arrow move down along the left sub-path. In the process of deviating the indicating arrow and moving down the left sub-path, the rotation angle α changes continuously with the rotation of the mobile phone. In some technical solutions, referring to FIG. 17, the mobile phone displays the left sub-path 1701 and the lower sub-path that have not been photographed. Among them, the dashed lines on both sides of the left sub-path 1701 represent the deviation range indicator line, and the arrow 1702 represents the deviation indicator arrow. In some other technical solutions, the mobile phone only displays the currently targeted left sub-path. In some other technical solutions, the mobile phone displays the complete shooting path on the shooting interface, and the guide path for the completed shooting is a dotted line, and the guide path for the unfinished shooting is a solid line. In other embodiments, the mobile phone still displays the complete guidance path.

Among them, the deviation range indicator line is displayed on both sides of the left sub-path. During the shooting along the left sub-path, if the rotation angle β ₁ <β <β ₂ , the deviation indicates the maximum deviation of the arrow on the left sub-path Within the range; if the rotation angle β<β ₁ or β>β ₂ , the deviation indicator arrow exceeds the maximum deviation range of the left sub-path, and the phone stops shooting.

It should be noted that in the shooting process where the target sub-path is the left sub-path, the mobile phone may not perform image stitching.

(5) The target sub-path is the lower sub-path, and the baseline image is stitched together

When the deviating arrow reaches the end of the left sub-path, and the rotation angle of the phone is α<α _b or α ₃ <α<α ₄ , guide the user to turn the phone to the right along the lower sub-path, so that the deviating arrow follows the lower sub-path move to the right. When the mobile phone rotates along the lower sub-path, the image collected by the mobile phone can be called the lower baseline image or the image of the lower sub-path, and the target sub-path is the lower sub-path. It is understandable that the first frame of the lower baseline image taken by the mobile phone along the lower sub-path is also the last frame of the image taken along the left sub-path. In the process of deviating the indicating arrow and moving to the left along the lower sub-path, the rotation angle β changes continuously with the rotation of the mobile phone. In some technical solutions, referring to (b) in FIG. 18, the mobile phone displays the lower sub-path 1801 that has not been photographed. Among them, the dashed lines on both sides of the lower sub-path 1801 represent the deviation range indicator line, and the arrow 1802 represents the deviation indicator arrow. In some other technical solutions, the mobile phone displays the complete upper sub-path that is currently targeted. In some other technical solutions, the mobile phone displays the complete shooting path on the shooting interface, and the guide path for the completed shooting is a dotted line, and the guide path for the unfinished shooting is a solid line. In other embodiments, the mobile phone still displays the complete guidance path.

Among them, the deviation range indicator lines are displayed on both sides of the lower sub-path. During the shooting along the lower sub-path, if α ₃ <α<α ₄ , the deviation indicator arrow is located within the maximum deviation range of the lower sub-path; if α >α ₄ , or α<α ₃ , the deviation indicator arrow exceeds the maximum deviation range of the lower sub-path, and the mobile phone stops shooting.

The method of stitching the lower baseline image by the mobile phone is the same as that of the lower baseline image. Among them, for the first frame of the baseline image B ₁ (that is, the last frame of image collected when the mobile phone rotates along the right sub-path), the mobile phone determines a rotation angle β and the first frame from the above key frames according to a preset algorithm. _{The target key frame with the closest rotation angle β 1} corresponding to the baseline image under the frame is used as the reference frame C _f1 , and the image after the cylindrical surface mapping of _{C f1 is extracted}

Feature points F _C,f1 . _{The mobile phone maps the lower baseline image B 1 in the} first frame to the cylindrical surface according to the above formula 2-4 to obtain the lower baseline image in the first frame.

And get the baseline image under the first frame'

The characteristic point F _B,1 . The mobile phone calculates _{the matching result of F B,1} and F _C,f1 , thereby calculating the homography matrix H according to the matching result. Then, according to the matrix H, the mobile phone will

Towards

Map it. The H matrix obtained by combining the target key frame is more accurate, which enables the lower baseline image to be better registered with the key frame, so that the registration can be replaced with the middle baseline splicing result. See (a) in Figure 18, the phone will map

The rectangular part within the cropping range is cropped and stitched to the lower left of the middle baseline image stitching result RI to form the lower baseline image stitching result RB1, that is, the target image RB1 obtained by stitching when the deviation indicator arrow moves along the lower sub-path. Among them, the cropping range corresponding to the lower baseline image is the range defined by the third cropping line, the fourth cropping line, the left boundary line, and the right boundary line. Exemplarily, the mobile phone displays the lower baseline image stitching result RB1 stitched below the middle baseline stitching result RI on the shooting interface, see (b) in FIG. 18. It should be noted that the upper baseline splicing result RA is also spliced above the middle baseline splicing result RI.

_{For the baseline image B i in the i-th} frame (which is an integer greater than 1), the mobile phone determines a target key frame whose rotation angle β is _{closest to the rotation angle β i of} _{I i} from the above key frames according to a preset algorithm as the reference frame C _fi , and extract the image after _{C fi cylindrical surface mapping}

Feature points F _c,fi . _{The mobile phone maps B i} to the cylindrical surface according to the above formula 2-4 to obtain the baseline image under the i-th frame'

The feature point F _B,i . The mobile phone calculates F _B,i , the image _{after the baseline image B i-1 is} mapped to the cylindrical surface in the i-1th frame

Feature points F _B,i-1 , and _{the matching results of F c,fi} to calculate the homography matrix H. The H matrix obtained in this way is more accurate, and can be better registered with the key frame while registering with the baseline image of the previous frame. Phone will

Towards

with

Map it. The phone will be mapped

The rectangular part within the cropping range is cropped and stitched to the right of the lower baseline image stitching result RB(i-1) to form the lower baseline image stitching result RBi, which is obtained by stitching when the deviation indicator arrow moves along the lower sub-path The target image RBi.

Then, the user continues to turn the phone along the sub-path below. The mobile phone repeats the above collection, mapping and splicing process until β>β _r . At this time, all the lower baseline image frames collected by the mobile phone along the middle sub-path are stitched together, and the lower baseline image stitching result RB is generated, that is, the stitching result RB corresponding to the lower sub-path, the entire panoramic image stitching is completed, and the entire shooting process ends. Incidentally, for the last one in the baseline image β≥β _r case where the image mapped to the cylindrical surface, cut lines may be used as the right boundary of the right side thereof, beyond the right boundary portion may be cut. Exemplarily, for a schematic diagram of the stitching result of the baseline image in the last frame, see (c) in FIG. 18.

Similarly, the mobile phone registers the key frames in the lower baseline image and the middle baseline image, which can correct the misalignment error between the lower baseline image and the middle baseline splicing result in time, so that the lower baseline image and the middle baseline image splicing result can be accurately matched. In order to achieve global registration, the stitching result of the lower baseline image and the middle baseline image can form a smooth, natural transition overall image.

303. After the mobile phone stops shooting, it generates a panoramic image.

When the user turns the phone along the downward sub-path, an indicator arrow departing from reaching the end of the downward sub-path, the rotation angle β> β _r, the end of the phone guide the user, stops displaying the guide route, after the panoramic image is generated at a baseline image stitching is completed. Exemplarily, the panoramic image generated by the mobile phone may refer to the thumbnail 1901 shown in (a) in FIG. 19 and the thumbnail 1902 in the gallery shown in (b) in FIG. 19.

Or, during the shooting process, if the mobile phone detects the user's operation to stop shooting (for example, the user clicks the operation of the stop shooting control 1803 shown in (b) in FIG. 18), the shooting of the panoramic image is stopped. Or, as described in the above embodiment, if the deviation indicator arrow exceeds the maximum deviation range of the target sub-path in the previous shooting process, the mobile phone stops shooting. Or, during the shooting process, if the mobile phone detects the user's operation to stop shooting, it will stop shooting. Wherein, if the target sub-path is the middle sub-path when the shooting is suspended, the obtained panoramic image is the result of the middle baseline image stitching when the shooting is suspended. In addition, if the target sub-path is the right sub-path or the upper sub-path when the shooting is stopped, the panoramic image obtained by the mobile phone is the complete mid-baseline image stitching result. If the target sub-path is the left sub-path or the lower sub-path when the shooting is stopped, the panoramic image obtained by the mobile phone is the complete middle baseline image and the upper baseline image stitching result. In this way, it can be avoided that in the panoramic image captured by the mobile phone, the field angle of the middle baseline image stitching result is larger and the field angle of the upper baseline image or the lower baseline image stitching result is smaller, resulting in irregular and irregular panoramic images obtained. whole.

In addition, in some embodiments, the mobile phone can also generate a video after stopping shooting, and the video image is the spliced image displayed in the splicing preview window during the shooting process. When the user instructs to play the video, the mobile phone can dynamically present the image splicing process during the shooting process to the user.

In other embodiments, the mobile phone may also save the image sequence collected according to the guide path during the shooting process, and in response to the user's editing operation of the image sequence, the mobile phone may generate a panoramic image.

In the embodiment of the present application, the mobile phone can expand the field of view of the panoramic image in the horizontal direction by splicing the upper baseline image, upper baseline image, or lower baseline image in the horizontal direction; by combining the upper baseline image and the middle baseline image Perform registration and stitching, and register and stitch the lower baseline image and the middle baseline image. It can also expand the angle of view of the panoramic image in the vertical direction, thereby making the entire angle of view of the panoramic image larger and improving the user Shooting experience.

In addition, in the real-time application of this application, the use of cylindrical mapping can match the size and imaging characteristics of the same object after mapping on the image taken by the mobile phone from different angles, so that the images taken from different angles can be registered and stitched. In turn, a panoramic image is generated, which can conform to the visual effect that the image size of each part of the panoramic image is basically the same.

In addition, the panoramic image is not a simple combination of the splicing results of the upper, middle, and lower baseline images. Each kind of baseline image stitching result is obtained by registration and stitching based on multiple homography matrices H, and each small part corresponds to a different homography matrix H, so different types of baseline image stitching results cannot be calculated between A homography matrix H enables good correspondence and matching between the parts of the two baseline image stitching results, so it is impossible to simply directly stitch the upper, middle, and lower baseline image stitching results. In the embodiment of the present application, the upper baseline image and the lower baseline image are also registered and spliced according to the key frame of the middle baseline. The misalignment error of the splicing result of the upper and lower baseline images and the middle baseline can be corrected in time, so that the upper baseline The image, the lower baseline image and the middle baseline image stitching results are accurately registered to achieve global registration. The upper baseline image stitching results, the middle baseline image stitching results, and the lower baseline image stitching results can be better integrated to form An overall image with a smooth, natural transition.

In the above embodiment, the beginning of the guide path is the left end of the middle sub-path, and the end of the guide path is the end of the lower sub-path. It is understandable that in the scenario where the first direction is the horizontal direction and the second direction is the vertical direction shown in case 1, depending on the beginning, end, or changing direction (or running direction), there can be more guide paths. Different forms. Exemplarily, the guide paths of different start and end can refer to (a)-(f) in FIG. 20. It is understandable that in case 1, other guiding paths other than the example shown in FIG. 20 may also be included, which are not limited in the embodiment of the present application.

In some embodiments of the present application, when the mobile phone is in a different state such as a vertical screen or a horizontal screen, the guide path will change accordingly with the state of the mobile phone. Exemplarily, in the scenario shown in (b) of FIG. 4A, if the mobile phone is switched to the vertical screen state, the preview interface displayed by the mobile phone can be seen in FIG. 21.

Case 2. The first direction is the vertical direction, and the second direction is the horizontal direction.

In case 2, the first direction is the vertical direction, and the second direction is the horizontal direction. Compared with case 1, the first direction is a vertical direction, the sub-path in the vertical direction is longer, and the guide path includes at least two sub-paths in the vertical direction. The guide path may further include at least one horizontal sub-path for connecting the vertical sub-path. Similarly, in a guidance path corresponding to case 2, a schematic diagram of a preview interface displayed by the mobile phone can be seen in FIG. 22A. In case 2, the guide path can also take many different forms according to the beginning, end, or running direction. Exemplarily, the different start and end guide paths can be referred to (a)-(c) in FIG. 22B. It is understandable that in case 1, other guiding paths other than the example illustrated in FIG. 22B may be included, which are not limited in the embodiment of the present application. The panoramic shooting method corresponding to case 2 is similar to case 1, and will not be repeated here. The difference is: in case 1, the mobile phone stitches the images according to the three baselines in the horizontal direction; and in case 2, the mobile phone uses the vertical direction The three baselines of the image are stitched together.

It can be understood that case 1 is suitable for shooting scenes with a large field of view in the horizontal direction; case 2 is suitable for shooting scenes with a large field of view in the vertical direction (for example, shooting high-rise buildings). Exemplarily, when the shooting corresponding to Case 2 is about to end, the schematic diagram of the image splicing result in the splicing preview window can be seen in FIG. 22C.

In case 2, the mobile phone can expand the angle of view of the panoramic image in the vertical direction by stitching the images collected along a certain vertical sub-path; The splicing of the images collected by the sub-path can also expand the angle of view of the panoramic image in the horizontal direction, thereby making the entire angle of view of the panoramic image larger and improving the user's shooting experience.

In the embodiment of the present application, the user can select or switch the guide path mode corresponding to the situation 1 or the situation 2 to take the panoramic image according to the actual needs of the shooting scene.

The above embodiments are described by taking the first direction including three sub-paths as an example. In some other embodiments, the first direction may also include only two sub-paths, and the second direction may include multiple sub-paths for connecting the sub-paths of the first sub-direction. Exemplarily, when the first direction is the horizontal direction, the second direction is the horizontal direction, and the first direction includes two sub-paths, the guide path may refer to (a) in FIG. 23, and the panoramic image obtained by shooting may refer to FIG. 23 In (b). It is understandable that when the start, end, or running direction is different, the guide path can also take many different forms.

As another example, when the first direction is the horizontal direction, the second direction is the horizontal direction, and the first direction includes two sub-paths, the guide path may refer to (a) in FIG. 24, and the panoramic image obtained by shooting may refer to FIG. (B) in 24. It is understandable that when the start, end, or running direction is different, the guide path can also take many different forms.

Compared with the first direction including three sub-paths, when the guide path includes two sub-paths, although the guide path reduces one sub-path in the first direction, it can still expand the field of view in the second direction and simplify the panoramic view. The shooting process of the image.

In some other embodiments, the first direction may also include more than three sub-paths, and the second direction may include multiple sub-paths for connecting the sub-paths in the first direction. Exemplarily, in case 1, the guide path including 4 sub-paths in the horizontal direction can be seen in (a)-(b) in FIG. 25, and the panoramic image obtained by shooting can be seen in (c) in FIG. 25. When the number of sub-paths in the first direction is greater, the mobile phone can expand the range of the field of view in the second direction perpendicular to the first direction.

The above is an example of rear panoramic shooting. For front panoramic shooting, the mobile phone can also use the method described in the above embodiment to shoot panoramic images, which will not be repeated here.

The above description is based on an example in which the electronic device is a mobile phone. When the electronic device is a device other than a mobile phone, the method described in the above embodiment can also be used to perform panoramic shooting, which will not be repeated here.

It can be understood that, in order to realize the above-mentioned functions, the electronic device includes hardware and/or software modules corresponding to each function. In combination with the algorithm steps of the examples described in the embodiments disclosed herein, the present application can be implemented in the form of hardware or a combination of hardware and computer software. Whether a certain function is executed by hardware or computer software-driven hardware depends on the specific application and design constraint conditions of the technical solution. Those skilled in the art can use different methods for each specific application in combination with the embodiments to implement the described functions, but such implementation should not be considered as going beyond the scope of the present application.

In this embodiment, the electronic device can be divided into functional modules according to the foregoing method examples. For example, each functional module can be divided corresponding to each function, or two or more functions can be integrated into one processing module. The above-mentioned integrated modules can be implemented in the form of hardware. It should be noted that the division of modules in this embodiment is illustrative, and is only a logical function division, and there may be other division methods in actual implementation.

For example, in a division method, referring to FIG. 26, the electronic device 2600 may include a camera 2601, an ISP 2602, an input buffer unit 2603, a calculation processing unit 2604, an inertial measurement unit (IMU) 2605, and an output buffer unit 2606 , Encoder 2607, display unit 2608, and other units/modules.

Among them, the ISP processes the image frames collected by the camera and outputs them to the input buffer unit. The calculation processing unit performs corresponding cropping and splicing processing on the image data in the input buffer unit according to the data of the IMU, and outputs the processing result to the output buffer unit. The display unit displays the interface and guide information according to the processing result in the output buffer unit. After the splicing process is completed, the encoder encodes the image data in the processing result and outputs it to a gallery or other applications.

Exemplarily, the case including three horizontal sub-paths shown in case 1 is taken as an example for description. When the user enters the panoramic shooting mode, the calculation processing unit initializes various parameters in the scene, including the rotation angle α _t around the x axis corresponding to the upper and lower baselines, α _b (the middle baseline rotation angle α is 0), first, 2,3,4 The rotation angle corresponding to the clipping boundary α ₁ , α ₂ , α ₃ , α ₄ _{, the rotation angle β r} around the y-axis corresponding to the right baseline (the left baseline rotation angle β is 0), the left boundary and the left clipping _{The rotation angles β 1} , β ₂ , β ₃ , β ₄ corresponding to the line, the right clipping line, and the right border, and the key frame interval Δβ, etc. The calculation processing unit determines the position of the guidance information according to the rotation angle. The display screen can display a preview interface as shown in (b) of FIG. 4A, and the preview interface includes guidance information. When the electronic device detects the user's shooting operation, the ISP processes the image frame received by the camera and sends it to the calculation processing unit. The calculation processing unit refreshes the interface in real time according to the shooting process, and the display screen displays the shooting interface as shown in (a) in FIG. 6. The calculation and processing unit stitches the mid-baseline images collected by the camera. The output buffer unit buffers the key frames and the splicing result, and the display screen displays the splicing result. When the rotation angle of the electronic device beta]> β _r, the display shows the screen shown in FIG shot 15, to guide the user rotates the electronic device upwards. When the rotation angle of the electronic device α>α _t , the display screen displays the shooting interface as shown in Fig. 16A to guide the user to turn the electronic device to the left, and the calculation processing unit stitches the upper baseline image collected by the camera according to the key frame . The output buffer unit buffers the splicing result, and the display screen displays the splicing result. When the rotation angle β of the electronic device is less than 0, the display screen displays the shooting interface as shown in FIG. 17 to guide the user to turn the electronic device downward. When the rotation angle α <α _b of the electronic device, the screen display shown in FIG. 18 (b) capturing interface shown, to guide the user rotates the electronic device to the right, the calculation processing unit according to key frames captured by the camera to The lower baseline image is stitched. The output buffer unit buffers the splicing result, and the display screen displays the splicing result. When the rotation angle of the electronic device beta]> β _r, panoramic image stitching is completed.

An embodiment of the present application also provides an electronic device, including: a camera for collecting images; a display screen for displaying an interface; one or more processors and one or more memories. The one or more memories are coupled with one or more processors, and the one or more memories are used to store computer program codes. The computer program codes include computer instructions. When the one or more processors execute the computer instructions, the electronic device executes The above-mentioned related method steps implement the panoramic shooting method in the above-mentioned embodiment.

An embodiment of the present application also provides an electronic device including one or more processors and one or more memories. The one or more memories are coupled with one or more processors, and the one or more memories are used to store computer program codes. The computer program codes include computer instructions. When the one or more processors execute the computer instructions, the electronic device executes The above-mentioned related method steps implement the panoramic shooting method in the above-mentioned embodiment.

The embodiments of the present application also provide a computer-readable storage medium that stores computer instructions in the computer-readable storage medium. When the computer instructions run on an electronic device, the electronic device executes the above-mentioned related method steps to implement the above-mentioned embodiments. Panorama shooting method in.

The embodiments of the present application also provide a computer program product, which when the computer program product runs on a computer, causes the computer to execute the above-mentioned related steps, so as to realize the panoramic photography method executed by the electronic device in the above-mentioned embodiment.

In addition, the embodiments of the present application also provide a device. The device may specifically be a chip, component, or module. The device may include a processor and a memory connected to each other. The memory is used to store computer execution instructions. When the device is running, The processor can execute the computer-executable instructions stored in the memory, so that the chip executes the panoramic shooting method executed by the electronic device in the foregoing method embodiments.

Among them, the electronic device, computer readable storage medium, computer program product, or chip provided in this embodiment are all used to execute the corresponding method provided above. Therefore, the beneficial effects that can be achieved can refer to the above provided The beneficial effects of the corresponding method will not be repeated here.

Through the description of the above embodiments, those skilled in the art can understand that for the convenience and brevity of the description, only the division of the above-mentioned functional modules is used as an example for illustration. In practical applications, the above-mentioned functions can be allocated to different functions as required. The function module is completed, that is, the internal structure of the device is divided into different function modules to complete all or part of the functions described above.

In the several embodiments provided in this application, it should be understood that the disclosed device and method can be implemented in other ways. For example, the device embodiments described above are merely illustrative. For example, the division of the modules or units is only a logical function division. In actual implementation, there may be other division methods, for example, multiple units or components may be It can be combined or integrated into another device, or some features can be omitted or not implemented. In addition, the displayed or discussed mutual coupling or direct coupling or communication connection may be indirect coupling or communication connection through some interfaces, devices or units, and may be in electrical, mechanical or other forms.

The units described as separate parts may or may not be physically separate. The parts displayed as units may be one physical unit or multiple physical units, that is, they may be located in one place, or they may be distributed to multiple different places. . Some or all of the units may be selected according to actual needs to achieve the objectives of the solutions of the embodiments.

In addition, the functional units in the various embodiments of the present application may be integrated into one processing unit, or each unit may exist alone physically, or two or more units may be integrated into one unit. The above-mentioned integrated unit can be implemented in the form of hardware or software functional unit.

If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a readable storage medium. Based on this understanding, the technical solutions of the embodiments of the present application are essentially or the part that contributes to the prior art, or all or part of the technical solutions can be embodied in the form of a software product, and the software product is stored in a storage medium. It includes several instructions to make a device (may be a single-chip microcomputer, a chip, etc.) or a processor (processor) execute all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage media include: U disk, mobile hard disk, read only memory (read only memory, ROM), random access memory (random access memory, RAM), magnetic disk or optical disk and other media that can store program codes.

The above content is only the specific implementation manners of this application, but the protection scope of this application is not limited to this. Any person skilled in the art can easily think of changes or substitutions within the technical scope disclosed in this application. Covered in the scope of protection of this application. Therefore, the protection scope of this application should be subject to the protection scope of the claims.

Claims

A panoramic shooting method, characterized in that it comprises:

The electronic device enters the panoramic shooting mode of the camera application;

The electronic device displays first guide information on the preview interface, the first guide information includes a first guide path; the first guide path includes at least two sub-paths that are arranged along a first direction and are parallel to each other, the first A direction is parallel to a side of the electronic device, and the first guide path is used to guide the user to rotate the electronic device along the first guide path during the shooting process.
The method according to claim 1, wherein the coordinate ranges corresponding to the different sub-paths set along the first direction overlap in the first direction.
The method according to claim 2, wherein the different sub-paths set along the first direction correspond to the same coordinate range in the first direction.
The method according to any one of claims 1 to 3, wherein the first guide path further comprises at least one sub-path arranged along the second direction, and the at least one sub-path is used to connect along the first guide path. At least two sub-paths are arranged in one direction, and the second direction is perpendicular to the first direction.
The method according to any one of claims 1-4, wherein the method further comprises:

After the electronic device detects the shooting operation, the second guide information is displayed on the shooting interface. The second guide information includes a splicing preview window, a second guide path, and a deviation indicator; wherein, the deviation indicator is used to indicate The position of the center line of the image collected by the electronic device, the deviation indication mark moves along the second guide path during the shooting process; the second guide path includes the deviation indication on the first guide path Mark the unpassed part;

The electronic device displays the target image obtained by splicing images collected by the electronic device when the deviation indicator mark moves along the sub-path in the first direction in the splicing preview window;

The electronic device stops shooting after the deviation indication mark reaches the end of the second guide path, and the target image obtained by splicing by the electronic device is a panoramic image.
The method according to claim 5, wherein when the deviation indication mark moves along any sub-path of the second guide path, the second guide information further includes the The deviation range indicator lines on both sides of the path and parallel to any one of the sub-paths; the method further includes:

If the deviation indicator mark exceeds the range indicated by the deviation range indicator line, the electronic device stops shooting.
The method according to claim 6, wherein the first guide path includes a first sub-path and a third sub-path that are arranged along a first direction and are parallel to each other, and the first sub-path is the initial sub-path. path;

The first guide path further includes a second sub-path arranged along a second direction, and the second sub-path is used to connect the first sub-path and the third sub-path;

During the shooting process, the deviation indicator mark sequentially moves along the first sub-path, the second sub-path, and the third sub-path.
The method according to claim 7, wherein the first guide path further comprises a fifth sub-path arranged along the first direction and parallel to the first sub-path, the third sub-path and The fifth sub-path is located on both sides of the first sub-path;

The first guide path further includes a fourth sub-path arranged along the second direction and parallel to the second sub-path, and the fourth sub-path is used to connect the third sub-path and the fifth sub-path. Subpath

During the shooting process, the deviation indicator mark moves along the first sub-path, the second sub-path, the third sub-path, the fourth sub-path, and the fifth sub-path in sequence.
8. The method according to claim 8, characterized in that, during the shooting process, when the deviation indicator mark moves along the first sub-path, the second guide path includes the first sub-path. The part that the deviation indicator does not pass and the second sub-path to the fifth sub-path, and the deviation range indicator line is displayed on both sides of the first sub-path; the deviation indicator is displayed in the splicing preview window When moving along the first sub-path, a target image obtained by splicing images collected by the electronic device;

When the deviation indicator moves along the second sub-path, the second guide path includes the portion of the second sub-path that the deviation indicator does not pass and the third sub-path to the first sub-path. Five sub-paths, and deviation range indicator lines are displayed on both sides of the second sub-path; the target image displayed in the stitching preview window is after the deviation indicator mark moves to the end of the first sub-path, The splicing result corresponding to the first path obtained by splicing the images collected by the electronic device;

When the deviation indicator moves along the third sub-path, the second guide path includes the portion of the third sub-path that the deviation indicator does not pass, and the fourth sub-path and the first Five sub-paths, and deviation range indication lines are displayed on both sides of the third sub-path; the target image displayed in the stitching preview window is when the deviation indication mark moves along the second sub-path An image obtained by splicing the image collected by the electronic device with the splicing result corresponding to the first sub-path.
The method according to claim 9, wherein when the deviation indicator mark moves along the first sub-path, the electronic device displays the deviation indicator mark along the When the sub-path in the first direction moves, the target image obtained by splicing the images collected by the electronic device includes:

The electronic device maps the i-th frame image I i of the first sub-path to a cylindrical surface to obtain an image
i is an integer greater than 1;

The electronic device extracts the
with
Feature points F I,i and F I,i-1 , the
Is an image obtained after the i-1th frame image I i-1 of the first subpath is mapped to a cylindrical surface;

The electronic device calculates the matching result of the F I,i and the F I,i-1;

According to the matching result of the F I,i and the F I,i-1, the electronic device converts the
To said
To map;

The electronic device will be mapped
The part within the preset first cropping range is spliced with the spliced image RI(i-1) of the first sub-path, so as to obtain the spliced image RIi of the first sub-path;

Wherein, the first clipping range includes a clipping line corresponding to the deviation range indication line of the first sub-path and a range defined by a left boundary line and a right boundary line preset by the electronic device.
The method according to claim 10, wherein when the deviation indicator moves along the first sub-path, the method further comprises:

The electronic device obtains a plurality of key frames from image frames collected while rotating along the sub-path in the first direction.
The method according to claim 11, wherein when the deviation indication mark moves along the third sub-path, the electronic device displays the deviation indication mark along the The target image obtained by splicing the images collected by the electronic device when the subpath in the first direction moves includes:

The electronic device obtains, from the plurality of key frames, a target key frame G f1 that matches the first frame image A 1 of the third sub-path;

The electronic device maps the G f1 to a cylindrical surface to obtain an image

The electronic device maps the A 1 to a cylindrical surface to obtain an image

The electronic device extracts the
And said
Feature points F A,1 and F A,f1 ;

Calculating, by the electronic device, a matching result of the FA,1 and the FA ,f1;

According to the matching result of the FA,1 and the FA ,f1, the electronic device converts the
To said
To map;

The electronic device will be mapped
The part within the preset second cropping range is spliced with the splicing result corresponding to the first sub-path to obtain the spliced image RA1 of the second sub-path;

Wherein, the second clipping range includes a clipping line corresponding to the deviation range indication line of the third sub-path and a range defined by a left boundary line and a right boundary line preset by the electronic device.
The method according to claim 12, wherein, in the xyz coordinate system, the lens of the electronic device camera is located on the xy plane of the xyz three-dimensional coordinate system; the interval of the rotation angle around the y axis corresponding to the key frame is greater than Or equal to the preset value Δβ; wherein, among the multiple key frames, the difference between the rotation angle around the y axis corresponding to the target key frame G f1 and the rotation angle around the y axis corresponding to the A 1 is the smallest .
The method according to claim 12 or 13, wherein when the deviation indication mark moves along the third sub-path, the electronic device displays the deviation indication mark along the splicing preview window. The target image obtained by splicing the images collected by the electronic device when the sub-path in the first direction moves, further includes:

The electronic device acquires the i-th frame image A of the third sub-path of the i-i matches the target keyframe G fi from the plurality of key frames;

The electronic device maps the G fi to a cylindrical surface to obtain an image

The electronic device, the cylindrical surface A i to obtain an image map

The electronic device extracts the
And said
The feature points F A,i , F A,i-1 and F A,fi , the
Is an image obtained after the i-1th frame image Ai-1 of the second subpath is mapped to a cylindrical surface;

The electronic device calculates the matching result of the FA,i , the FA ,i-1 and the FA ,fi;

According to the matching result of the F A,i , the F A,i-1 and the F A,fi, the electronic device converts the
To said
And said
To map;

The electronic device will be mapped
For the part within the preset second cropping range, the splicing result corresponding to the first sub-path and the spliced image RA(i-1) of the second sub-path are spliced to obtain the second sub-path The stitched image RAi of the path.
The method according to any one of claims 11-14, wherein when the deviation indication mark moves along the fifth sub-path, the electronic device displays the deviation indication in the splicing preview window The target image obtained by splicing the images collected by the electronic device when the marker moves along the sub-path in the first direction includes:

According to the plurality of key frames, the electronic device splices the image collected while rotating along the fifth sub-path with the splicing result corresponding to the first sub-path.
The method according to any one of claims 8-15, wherein in the xyz coordinate system, the lens of the electronic device camera is located on the xy plane of the xyz three-dimensional coordinate system;

The electronic device is preset with a left boundary line, a left baseline, a left clipping line, a right clipping line, a right baseline, and a right boundary line; the left baseline corresponds to the fourth sub-path, and the left baseline corresponds to a line around the y-axis The rotation angle is 0, the right baseline corresponds to the second subpath, the rotation angle about the y-axis corresponding to the right baseline is β r , and the rotation angle about the y-axis corresponding to the left clipping line is β 2 , The rotation angle around the y-axis corresponding to the right clipping line is β 3 , the rotation angle around the y-axis corresponding to the left boundary line is β 1 , and the rotation angle around the y-axis corresponding to the right boundary line is β 4 ;

The electronic device is also preset with an upper baseline, a middle baseline, and a lower baseline, as well as a first cutting line, a second cutting line, a third cutting line, and a fourth cutting line; wherein, the upper baseline, the middle baseline, and the The lower baseline corresponds to the third sub-path, the first sub-path, and the fifth sub-path respectively, and the deviation of the first cutting line and the second cutting line from the third sub-path The range indication line corresponds, the second cutting line and the third cutting line correspond to the deviation range indication line of the first sub-path, and the third cutting line and the fourth cutting line correspond to the fifth cutting line. The deviation range indication line of the sub-path corresponds to; the first cutting line, the upper baseline, the second cutting line, the middle line, the third cutting line, the lower baseline, and the fourth cutting The rotation angles around the x-axis corresponding to the lines are α 1 , α t , α 2 , 0, α 3 , α b , and α 4 respectively ; the method further includes:

When the deviation indication mark moves along the first sub-path, if the rotation angle of the electronic device around the x axis is α>α 1 or α<α 2 , the electronic device stops shooting;

When the electronic device about the y-axis rotational angle β> β r, is moved by the deviation indicator is switched to the first sub-movement along a path along the second sub-path; if the electronic device about If the rotation angle of the y axis β>β 4 or β<β 3 , the electronic device stops shooting;

When the rotation angle of the electronic device around the x axis is α>α t , the deviation indication mark is switched from moving along the second sub-path to moving along the third sub-path; When the x-axis rotation angle α>α 1 or α<α 2 , the electronic device stops shooting;

When the rotation angle β of the electronic device around the y-axis is less than 0, the deviation indication mark is switched from moving along the third sub-path to moving along the fourth sub-path; when the electronic device is moving around the y-axis rotation angle β> β 2, or when β <β 1, the electronic device to stop recording;

When the electronic device about the x axis rotation angle α <α b, the deviation indicator is moved by the switching path of movement along the fourth sub-sub-path along the fifth; about when the electronic device When the x-axis rotation angle α>α 3 or α<α 4 , the electronic device stops shooting;

When the electronic device about the y-axis rotational angle β> β r, the electronic device to stop recording.
An electronic device, characterized in that it comprises:

Camera, used to collect images;

Screen, used to display the interface;

One or more processors;

Memory

And one or more computer programs, wherein the one or more computer programs are stored in the memory, and the one or more computer programs include instructions; when the instructions are executed by the electronic device, the The electronic device executes the panoramic shooting method according to any one of claims 1-16.
A computer-readable storage medium, characterized by comprising computer instructions, when the computer instructions are run on a computer, the computer is caused to execute the panoramic shooting method according to any one of claims 1-16.
A computer program product, characterized in that, when the computer program product runs on a computer, the computer is caused to execute the panoramic shooting method according to any one of claims 1-16.