CN110012209B - Panoramic image generation method, device, storage medium and electronic device - Google Patents

Abstract

The application discloses a panoramic image generation method, a panoramic image generation device, a storage medium and electronic equipment, wherein the panoramic image generation method comprises the following steps: acquiring a panoramic shooting instruction; starting a camera according to the panoramic shooting instruction to shoot a video, and displaying the shot content in a preview frame; in the video shooting process, performing snake-shaped scanning on the preview frame along a second direction at a preset first direction overlapping rate to obtain a plurality of groups of scanning images, wherein the first direction is vertical to the second direction; generating corresponding local images in the second direction according to each group of scanning images; the panoramic image is generated according to the local image, so that the method is suitable for image splicing of various complex scenes, and is simple, high in flexibility and good in splicing effect.

Description

Panoramic image generation method, device, storage medium and electronic device

technical field

The present application relates to the field of computer technology, and in particular, to a method, device, storage medium and electronic device for generating panoramic images.

Background technique

With the continuous development of mobile communication technology, a mobile terminal has become one of the necessary tools in people's daily life, which can not only provide functions of calling and short messages, but also have multimedia playback and photographing functions.

Most of the current mobile terminals have a panoramic shooting mode. In the panoramic shooting mode, a series of small-view images with partially overlapping borders can be matched and aligned using different processing algorithms according to the characteristics of different images, so as to form a wide-angle image. However, the existing panoramic shooting technology can only stitch some relatively simple scene images, and it is difficult to adapt to complex scenes, such as close-up shooting of buildings, murals, etc., with large limitations and poor stitching effect.

SUMMARY OF THE INVENTION

The embodiments of the present application provide a method, device, storage medium and electronic device for generating a panoramic image, which can adapt to various shooting scenarios and have a good stitching effect.

The embodiment of the present application provides a panoramic image generation method, including:

Get panoramic shooting instructions;

Start the camera to shoot video according to the panoramic shooting instruction, and display the shooting content in the preview frame;

During the video shooting process, serpentine scanning is performed on the shooting content with a preset first direction overlap rate along a second direction to obtain multiple sets of scanned images, and the first direction is perpendicular to the second direction;

generating corresponding partial images in the second direction according to each group of scanned images;

A panoramic image is generated from the partial images.

The embodiment of the present application also provides a panoramic image generating apparatus, which is applied to electronic equipment, including:

an acquisition module for acquiring panoramic shooting instructions;

a shooting module, configured to start the camera to shoot video according to the panoramic shooting instruction, and display the shooting content in the preview frame;

The scanning module is used for serpentine scanning of the shooting content along the second direction with a preset first direction overlap ratio during the video shooting process to obtain multiple sets of scanned images, the first direction is perpendicular to the second direction ;

a first generating module, configured to generate corresponding partial images in the second direction according to each group of scanned images;

The second generating module is configured to generate a panoramic image according to the partial image.

Further, the first generation module specifically includes:

A determination submodule for determining a corresponding key frame image group according to each group of scanned images;

Extraction sub-module for extracting feature points of key frame images in each key frame image group;

A stitching sub-module, configured to stitch the key frame image group according to the feature points, so as to generate corresponding partial images in the second direction.

Further, the splicing submodule is used for:

Calculate the corresponding first homography matrix according to the feature points of two adjacent frames of key frame images in the key frame image group;

Calculate a first global transformation matrix according to the first homography matrix;

The corresponding key frame image groups are stitched by using the first global transformation matrix.

Further, the splicing submodule is specifically used for:

Matching the feature points of two adjacent frames of key frame images in the key frame image group;

Obtain the location information of the successfully matched feature points;

A first homography matrix is calculated according to the position information.

Further, the determining submodule is specifically used for:

In each group of scanned images, the first frame of scanned images is used as a key frame image;

Obtain the next frame of scanned image adjacent to the key frame image as the target frame image;

According to the feature points of the key frame image and the feature points of the target frame image, determine whether the target frame image satisfies the preset condition;

If so, take the target frame image as the key frame image, and return to perform the operation of acquiring the next frame scan image adjacent to the key frame image;

If not, take the next scanned image adjacent to the target frame image as the target frame image, and return to execute the judgment of the target according to the feature points of the key frame image and the feature points of the target frame image The operation of whether the frame image meets the preset condition.

Further, the determining submodule is specifically used for:

matching the feature points of the key frame image with the feature points of the target frame image;

Count the number of successfully matched feature points in the key frame image and the total number of feature points in the key frame image;

Whether the target frame image satisfies a preset condition is determined according to the number of successfully matched feature points and the total number.

Further, the determining submodule is specifically used for:

Calculate similarity according to the number of the successfully matched feature points and the total number;

When the similarity is less than a first preset threshold, and the number of successfully matched feature points is greater than a second preset threshold, it is determined that the target frame image satisfies a preset condition.

Further, the second generation module is specifically used for:

Calculate the corresponding second homography matrix according to the two adjacent local images;

calculating a second global transformation matrix according to the second homography matrix;

The partial images are stitched in the first direction by using the second global transformation matrix to obtain a panoramic image.

An embodiment of the present application further provides a storage medium, where a plurality of instructions are stored in the storage medium, and the instructions are adapted to be loaded by a processor to execute any one of the above-mentioned panoramic image generation methods.

An embodiment of the present application further provides an electronic device, including a processor and a memory, the processor is electrically connected to the memory, the memory is used for storing instructions and data, and the processor is used for any of the above The steps in the panoramic image generation method described above.

The panoramic image generation method, device, storage medium and electronic device provided by the present application obtain a panoramic shooting instruction, then start the camera to shoot video according to the panoramic shooting instruction, and display the shooting content in the preview frame, while in the video shooting During the process, serpentine scanning is performed on the shooting content along the second direction with a preset first direction overlap rate to obtain multiple sets of scanned images, the first direction is perpendicular to the second direction, and according to each set of scanned images in the first Corresponding partial images are generated in two directions, and then a panoramic image is generated according to the partial images, so that it can be applied to image stitching of various complex scenes. The method is simple, the flexibility is high, and the stitching effect is good.

Description of drawings

The technical solutions and other beneficial effects of the present application will be apparent through the detailed description of the specific embodiments of the present application in conjunction with the accompanying drawings.

FIG. 1 is a schematic flowchart of a panoramic image generation method provided by an embodiment of the present application.

FIG. 2 is a schematic diagram of a serpentine scan in a preview frame provided by an embodiment of the present application.

FIG. 3 is another schematic flowchart of the panorama image generating method provided by the embodiment of the present application.

FIG. 4 is a schematic flowchart of step 203 provided in this embodiment of the present application.

FIG. 5 is a schematic structural diagram of a panoramic image generating apparatus provided by an embodiment of the present application.

FIG. 6 is a schematic structural diagram of a first generation module provided by an embodiment of the present application.

FIG. 7 is a schematic structural diagram of an electronic device provided by an embodiment of the present application.

Detailed ways

The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, but not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative work fall within the protection scope of the present application.

Embodiments of the present application provide a method, apparatus, storage medium, and electronic device for generating a panoramic image.

A panorama image generation method, applied to electronic equipment, includes: acquiring a panorama shooting instruction; starting a camera to shoot video according to the panorama shooting instruction, and displaying the shooting content in a preview frame; during the video shooting process, along a second direction Perform serpentine scanning on the shooting content with a preset overlap ratio in the first direction to obtain multiple sets of scanned images, the first direction being perpendicular to the second direction; and generating corresponding partial images in the second direction according to each set of scanned images ; Generate a panoramic image based on the partial image.

As shown in FIG. 1, FIG. 1 is a schematic flowchart of a panoramic image generation method provided by an embodiment of the present application, which is applied to an electronic device, and the specific process may be as follows:

101. Obtain a panoramic shooting instruction.

In this embodiment, when the user clicks a certain button of the photographing application, the electronic device may generate a panoramic photographing instruction.

102. Start the camera to shoot video according to the panoramic shooting instruction, and display the shooting content in the preview frame.

In this embodiment, the size of the preview frame may be set manually, or may be determined according to the size of the display screen of the electronic device, for example, three quarters of the size of the display screen.

103. During the video shooting process, perform a serpentine scan on the shooting content along a second direction with a preset overlap ratio in a first direction to obtain multiple sets of scanned images, the first direction being perpendicular to the second direction.

In this embodiment, the second direction and the first direction are usually set in advance, which may be the direction in which the length and width of the preview frame are located, or may be manually defined horizontal and vertical directions, and of course other settings may also be used. Way. The first direction overlap rate refers to the scan track overlap rate in the first direction, which can be set manually, for example, not less than 50%. After each scan, a set of corresponding scan images will be generated. The scan length of the serpentine scan can be a fixed value or a variable. The overlap rate of the scan track depends on the starting point position of two adjacent serpentine scans, such as Please refer to Figure 2. For two adjacent serpentine scans with a fixed scan length, if the scan starting points are a1 and a2 respectively, and the scan length is b, the scan track overlap rate is [b-(a2-a1)]/b .

104. Generate a corresponding partial image in the second direction according to each group of scanned images.

In this embodiment, the partial image is formed by splicing along the second direction. Since each group of scanned images is not scanned for the entire preview frame, but scans a certain vertical area or horizontal area of the preview frame, correspondingly formed The stitched image is also not a panoramic image, but a partial image.

For example, the above step 104 may specifically include:

1-1. Determine the corresponding key frame image group according to each group of scanned images.

For example, the above step 1-1 may specifically include:

In each group of scanned images, the first frame of scanned images is used as a key frame image;

Obtain the next frame scanned image adjacent to the key frame image as the target frame image;

According to the feature points of the key frame image and the feature points of the target frame image, determine whether the target frame image satisfies the preset condition;

If so, take the target frame image as a key frame image, and return to perform the operation of acquiring the next frame scan image adjacent to the key frame image;

If not, take the next frame of scanned image adjacent to the target frame image as the target frame image, and return to execute whether the target frame image meets the requirements of the feature points of the key frame image and the feature points of the target frame image. Actions with preset conditions.

In this embodiment, in the image processing, the feature points refer to the points where the gray value of the image changes drastically or the points with large curvature on the edge of the image (that is, the intersection of two edges), and each feature point may include texture information such as features and color features. Before stitching the scanned images, it is necessary to filter out the scanned images that do not need to be used for stitching. For example, only one of the images with high similarity needs to be stitched, while the images with extremely low similarity can be discarded. , etc., so that the number of stitched images can be reduced and the stitching efficiency can be improved under the condition of ensuring the integrity of the image information.

Further, the above-mentioned step "judging whether the target frame image satisfies the preset conditions according to the feature points of the key frame image and the feature points of the target frame image" may specifically include:

matching the feature points of the key frame image with the feature points of the target frame image;

Count the number of successfully matched feature points in the key frame image and the total number of feature points in the key frame image;

Whether the target frame image satisfies the preset condition is determined according to the number of successfully matched feature points and the total number.

In this embodiment, the preset condition may be set manually, for example, the similarity between the key frame image and the target frame image may be within a certain range. Generally, the more feature points that are successfully matched, the higher the similarity.

Further, the above-mentioned steps of "judging whether the target frame image meets the preset condition according to the number of successfully matched feature points and the total number" may specifically include:

Calculate the similarity according to the number and the total number of successfully matched feature points;

When the similarity is less than the first preset threshold and the number of successfully matched feature points is greater than the second preset threshold, it is determined that the target frame image satisfies the preset condition.

In this embodiment, the ratio between the number of successfully matched feature points and the total number may be calculated, and the ratio is used as the similarity. The first preset threshold and the second preset threshold may be set manually, for example, the first preset threshold may be 70%, and the second preset threshold may be 20.

1-2. Extract feature points of key frame images in each key frame image group.

1-3. Splicing the key frame image group according to the feature point to generate a corresponding partial image in the second direction.

For example, the above steps 1-3 may specifically include:

1-3-1. Calculate a corresponding first homography matrix according to the feature points of two adjacent frames of key frame images in the key frame image group.

For example, the above step 1-3-1 may specifically include:

matching the feature points of two adjacent key frame images in the key frame image group;

Obtain the location information of the successfully matched feature points;

A first homography matrix is calculated according to the position information.

In this embodiment, the homography (the first homography matrix) is a conversion matrix of the mapping relationship from one image to another image. Specifically, the key of the two adjacent frames may be determined first. What are the successfully matched feature points in the frame image, and then select a certain number of successfully matched feature points to calculate the corresponding homography matrix, so that after the feature points in one frame of key frame image are transformed by the homography matrix, you can It is located in the same spatial coordinate system as the feature points in another keyframe image.

1-3-2. Calculate a first global transformation matrix according to the first homography matrix.

In this embodiment, the first homography matrix of key frame images of adjacent frames may be calculated in order. For example, for a certain key frame image group M1 to Mn, between M1 and M2, between M2 and M3, and between M3 and M3 may be calculated. The first homography matrix between M4 and Mn-1 and Mn, each homography matrix is equivalent to the local transformation matrix of the key frame image group, and finally, all the first homography matrix Unify to the same coordinate system to get the first global transformation matrix.

1-3-3. Use the first global transformation matrix to splicing the corresponding key frame image groups.

In this embodiment, each key frame image in the key frame image group can be projected into the same coordinate space system through the global transformation matrix, and the obtained image is the local image obtained by splicing.

105. Generate a panoramic image according to the partial image.

For example, the above step 105 may specifically include:

Calculate the corresponding second homography matrix according to the two adjacent local images;

Calculate a second global transformation matrix according to the second homography matrix;

The local image is stitched in the first direction by using the second global transformation matrix to obtain a panoramic image.

In this embodiment, since the serpentine scanning is an overlapping scanning in the first direction, that is, the scanning trajectories of two adjacent scanning operations overlap, when the scanned images are stitched in the second direction, it is necessary to Further splicing is performed in the first direction, that is, the partial images spliced by each column (row) of scanned images are spliced again along the direction of the row (column) to realize two-dimensional grid splicing. The image information is fully utilized, the method is simple, the splicing effect is good, and it can be applied to video image splicing of various complex scenes, with strong flexibility.

It can be seen from the above that the panorama image generation method provided in this embodiment is applied to an electronic device, by acquiring a panorama shooting instruction, then starting the camera to perform video shooting according to the panorama shooting instruction, and displaying the shooting content in the preview frame, and then in the preview frame. During the video shooting process, serpentine scanning is performed on the shooting content along the second direction with a preset first direction overlap rate to obtain multiple sets of scanned images, the first direction is perpendicular to the second direction, and according to each set of scanned images A corresponding partial image is generated in the second direction, and then a panoramic image is generated according to the partial image, so that it can be applied to image stitching of various complex scenes, the method is simple, the flexibility is high, and the stitching effect is good.

In this embodiment, the description will be made from the perspective of the panoramic image generating apparatus, and specifically, the panoramic image generating apparatus will be integrated into an electronic device as an example for detailed description.

Please refer to FIG. 3, a panoramic image generation method, applied to electronic equipment, the specific process can be as follows:

201. The electronic device acquires a panoramic shooting instruction, starts a camera to shoot video according to the panoramic shooting instruction, and then displays the shooting content in a preview frame.

For example, when the electronic device enters the shooting interface, you can click the panorama shooting button of the shooting application. At this time, the electronic device will start the camera to shoot video, and display the shooting content in the preview box in real time.

202. During the video shooting process, the electronic device performs a serpentine scan on the shooting content along a second direction with a preset first direction overlap ratio to obtain multiple sets of scanned images, the first direction being perpendicular to the second direction.

For example, the first direction may be horizontal, and the second direction may be vertical, that is, the electronic device will perform serpentine scanning on the content displayed in the preview frame along the vertical direction, please refer to FIG. For the second serpentine scan, if the scan start points are a1 and a2 respectively, and the scan length is b, the horizontal scan overlap ratio is [b-(a2-a1)]/b, for example, b/a may not be less than 50%.

203. The electronic device determines a corresponding key frame image group according to each group of scanned images.

For example, referring to FIG. 4 , the above-mentioned step 203 may specifically include:

2031. In each group of scanned images, the electronic device uses the first frame of scanned images as a key frame image;

2032. The electronic device acquires the next frame of scanned image adjacent to the key frame image as the target frame image;

2033. The electronic device determines whether the target frame image satisfies the preset conditions according to the feature points of the key frame image and the feature points of the target frame image, and if so, then uses the target frame image as the key frame image, and returns to perform the above steps 2032 , if not, take the next frame scanned image adjacent to the target frame image as the target frame image, and return to performing the above step 2033 .

In this embodiment, before splicing the scanned images, it is necessary to filter out the scanned images that do not need to be used for splicing. For example, only one of the images with high similarity needs to be used for splicing, while for images with extremely low similarity The image can be discarded, etc., so that the number of spliced images can be reduced and the splicing efficiency can be improved under the condition of ensuring the integrity of the image information. That is, the above step 2033 may further include:

matching the feature points of the key frame image with the feature points of the target frame image;

Count the number of successfully matched feature points in the key frame image and the total number of feature points in the key frame image;

Calculate the similarity according to the number and the total number of successfully matched feature points;

When the similarity is less than the first preset threshold and the number of successfully matched feature points is greater than the second preset threshold, it is determined that the target frame image satisfies the preset condition.

In this embodiment, since the more feature points that are successfully matched, the higher the similarity between the two is, and vice versa, the lower the similarity is. Therefore, when the number of successfully matched feature points is greater than the second preset threshold, such as 20 , it can be considered that the similarity between the two is not lower than the minimum. When the ratio of the number of successfully matched feature points to the total number is less than the first preset threshold, such as 70%, it can be considered that the similarity between the two is not higher than the maximum , only the target frame image that satisfies these two conditions at the same time can be considered to meet the preset conditions and can be used as a key frame image.

204. The electronic device extracts the feature points of the key frame images in each key frame image group, and matches the feature points of two adjacent key frame images in the key frame image group.

205. The electronic device acquires the position information of the successfully matched feature points, and calculates a first homography matrix according to the position information.

206. The electronic device calculates a first global transformation matrix according to the first homography matrix, and uses the first global transformation matrix to stitch the corresponding key frame image groups to obtain a partial image.

For example, since each key frame image group is obtained by a single vertical scan, when image splicing is used to process the key frame image group, the partial image spliced by each key frame image group corresponds to the vertical scan. The image information contained in the area, the specific splicing method can be to calculate the first homography matrix of adjacent key frame images in pairs, and then normalize all the first homography matrices to obtain a global transformation matrix, and use the global transformation matrix. The transformation matrix projects each key frame image in the key frame image group into the same coordinate space to complete the vertical splicing of each key frame image group.

207. The electronic device calculates a corresponding second homography matrix according to the two adjacent partial images, calculates a second global transformation matrix according to the second homography matrix, and then uses the second global transformation matrix for the partial image. Stitching is performed in the first direction to obtain a panoramic image.

For example, since the serpentine scan has a certain horizontal overlap rate, that is, there will be the same part in two adjacent partial images, the partial images can be further stitched horizontally, and the stitching method can be to generate the second two first. homography matrix, and then normalize the second homography matrix to obtain a second global transformation matrix, and then use the second global transformation matrix to project all local images into a coordinate space to complete the horizontal direction of the local images stitching.

It can be seen from the above that the method for generating a panoramic image provided in this embodiment is applied to an electronic device, wherein the electronic device can obtain a panoramic shooting instruction, start the camera to shoot video according to the panoramic shooting instruction, and then display the shooting content in the preview frame, Then, during the video shooting process, serpentine scanning is performed on the shooting content along the second direction with a preset first direction overlap ratio to obtain multiple sets of scanned images, the first direction is perpendicular to the second direction, and then, according to each The group scan images determine the corresponding key frame image group, then, extract the feature points of the key frame images in each key frame image group, and match the feature points of the two adjacent key frame images in the key frame image group, and then , obtain the position information of the successfully matched feature points, and calculate the first homography matrix according to the position information, then calculate the first global transformation matrix according to the first homography matrix, and use the first global transformation matrix to The corresponding key frame image groups are spliced to obtain a partial image, then, a corresponding second homography matrix is calculated according to the two adjacent partial images, and a second global transformation matrix is calculated according to the second homography matrix, and then The second global transformation matrix is used to stitch the local image in the first direction to obtain a panoramic image, so that the panorama can be generated in the form of a two-dimensional grid, which is suitable for image stitching of various complex scenes, and the method is simple and flexible High performance, good splicing effect.

According to the methods described in the foregoing embodiments, this embodiment will further describe from the perspective of a panoramic image generating apparatus, which may be implemented as an independent entity or integrated in an electronic device, such as a terminal. , the terminal may include a mobile phone, a tablet computer, a personal computer, and the like.

Please refer to FIG. 5 . FIG. 5 specifically describes the panoramic image generating apparatus provided by the embodiment of the present application, which is applied to electronic equipment. The panoramic image generating apparatus may include: an acquisition module 10 , a photographing module 20 , a scanning module 30 , and a first generating module 40 and a second generation module 50, wherein:

(1) Acquisition module 10

The obtaining module 10 is used for obtaining a panoramic shooting instruction.

In this embodiment, when the user clicks a certain button of the photographing application, the electronic device may generate a panoramic photographing instruction.

(2) Shooting module 20

The shooting module 20 is configured to start the camera to shoot video according to the panoramic shooting instruction, and display the shooting content in the preview frame.

In this embodiment, the size of the preview frame may be set manually, or may be determined according to the size of the display screen of the electronic device, for example, three quarters of the size of the display screen.

(3) Scanning module 30

The scanning module 30 is used for serpentine scanning of the shooting content along a second direction with a preset first direction overlap rate during the video shooting process to obtain multiple sets of scanned images, the first direction being perpendicular to the second direction.

In this embodiment, the second direction and the first direction are usually set in advance, which may be the direction in which the length and width of the preview frame are located, or may be manually defined horizontal and vertical directions, and of course other settings may also be used. Way. The first direction overlap rate refers to the scan track overlap rate in the first direction, which can be set manually, for example, not less than 50%. After each scan, a set of corresponding scan images will be generated. The scan length of the serpentine scan can be a fixed value or a variable. The overlap rate of the scan track depends on the starting point position of two adjacent serpentine scans, such as Referring to FIG. 2 , for two adjacent serpentine scans with a fixed scan length, if the scan length is a and the overlap length in the first direction is b, the scan track overlap ratio is b/a.

(4) The first generation module 40

The first generating module 40 is configured to generate corresponding partial images in the second direction according to each group of scanned images.

In this embodiment, the partial image is formed by splicing along the second direction. Since each group of scanned images is not scanned for the entire preview frame, but scans a certain vertical area or horizontal area of the preview frame, correspondingly formed The stitched image is also not a panoramic image, but a partial image.

For example, referring to FIG. 6, the first generation module 40 may specifically include a determination sub-module 41, an extraction sub-module 42 and a splicing sub-module 43, wherein:

The determination sub-module 41 is configured to determine a corresponding key frame image group according to each group of scanned images.

For example, the determination sub-module 41 can be specifically used for:

In each group of scanned images, the first frame of scanned images is used as a key frame image;

Obtain the next frame scanned image adjacent to the key frame image as the target frame image;

According to the feature points of the key frame image and the feature points of the target frame image, determine whether the target frame image satisfies the preset condition;

If so, take the target frame image as a key frame image, and return to perform the operation of acquiring the next frame scan image adjacent to the key frame image;

If not, take the next frame of scanned image adjacent to the target frame image as the target frame image, and return to execute whether the target frame image meets the requirements of the feature points of the key frame image and the feature points of the target frame image. Actions with preset conditions.

In this embodiment, in the image processing, the feature points refer to the points where the gray value of the image changes drastically or the points with large curvature on the edge of the image (that is, the intersection of two edges), and each feature point may include texture information such as features and color features. Before stitching the scanned images, it is necessary to filter out the scanned images that do not need to be used for stitching. For example, only one of the images with high similarity needs to be stitched, while the images with extremely low similarity can be discarded. , etc., so that the number of stitched images can be reduced and the stitching efficiency can be improved under the condition of ensuring the integrity of the image information.

For example, the determination sub-module 41 can be specifically used for:

matching the feature points of the key frame image with the feature points of the target frame image;

Count the number of successfully matched feature points in the key frame image and the total number of feature points in the key frame image;

Whether the target frame image satisfies the preset condition is determined according to the number of successfully matched feature points and the total number.

In this embodiment, the preset condition may be set manually, for example, the similarity between the key frame image and the target frame image may be within a certain range. Generally, the more feature points that are successfully matched, the higher the similarity.

Further, the determination sub-module 41 is specifically used for:

Calculate the similarity according to the number and the total number of successfully matched feature points;

When the similarity is less than the first preset threshold and the number of successfully matched feature points is greater than the second preset threshold, it is determined that the target frame image satisfies the preset condition.

In this embodiment, the ratio between the number of successfully matched feature points and the total number may be calculated, and the ratio is used as the similarity. The first preset threshold and the second preset threshold may be set manually, for example, the first preset threshold may be 70%, and the second preset threshold may be 20.

The extraction sub-module 42 is used for extracting the feature points of the key frame images in each key frame image group.

The splicing sub-module 43 is used for splicing the key frame image group according to the feature point, so as to generate a corresponding partial image in the second direction.

For example, the splicing submodule 43 can be specifically used for:

1-3-1. Calculate a corresponding first homography matrix according to the feature points of two adjacent frames of key frame images in the key frame image group.

At this time, the splicing submodule 43 can be specifically used for:

matching the feature points of two adjacent key frame images in the key frame image group;

Obtain the location information of the successfully matched feature points;

A first homography matrix is calculated according to the position information.

In this embodiment, the homography (the first homography matrix) is a conversion matrix of the mapping relationship from one image to another image. Specifically, the key of the two adjacent frames may be determined first. What are the successfully matched feature points in the frame image, and then select a certain number of successfully matched feature points to calculate the corresponding homography matrix, so that after the feature points in one frame of key frame image are transformed by the homography matrix, you can It is located in the same spatial coordinate system as the feature points in another keyframe image.

1-3-2. Calculate a first global transformation matrix according to the first homography matrix.

In this embodiment, the first homography matrix of key frame images of adjacent frames may be calculated in order. For example, for a certain key frame image group M1 to Mn, between M1 and M2, between M2 and M3, and between M3 and M3 may be calculated. The first homography matrix between M4 and Mn-1 and Mn, each homography matrix is equivalent to the local transformation matrix of the key frame image group, and finally, all the first homography matrix Unify to the same coordinate system to get the first global transformation matrix.

1-3-3. Use the first global transformation matrix to splicing the corresponding key frame image groups.

In this embodiment, each key frame image in the key frame image group can be projected into the same coordinate space system through the global transformation matrix, and the obtained image is the local image obtained by splicing.

(5) The second generation module 50

The second generating module 50 is configured to generate a panoramic image according to the partial image.

For example, the second generation module 50 can be specifically used for:

Calculate the corresponding second homography matrix according to the two adjacent local images;

Calculate a second global transformation matrix according to the second homography matrix;

The local image is stitched in the first direction by using the second global transformation matrix to obtain a panoramic image.

In this embodiment, since the serpentine scanning is an overlapping scanning in the first direction, that is, the scanning trajectories of two adjacent scanning operations overlap, when the scanned images are stitched in the second direction, it is necessary to Further splicing is performed in the first direction, that is, the partial images spliced by each column (row) of scanned images are spliced again along the direction of the row (column) to realize two-dimensional grid splicing. The image information is fully utilized, the method is simple, the splicing effect is good, and it can be applied to video image splicing of various complex scenes, with strong flexibility.

During specific implementation, the above units can be implemented as independent entities, or can be arbitrarily combined to be implemented as the same or several entities. The specific implementation of the above units can refer to the previous method embodiments, which will not be repeated here.

It can be seen from the above that the panorama image generating method provided in this embodiment is applied to electronic equipment, and the panorama shooting instruction is acquired through the acquisition module 10, and then the shooting module 20 starts the camera to perform video shooting according to the panorama shooting instruction, and displays the shooting content on the In the preview frame, during the video shooting process, the scanning module 30 performs serpentine scanning on the shooting content along the second direction with a preset first direction overlap rate to obtain multiple sets of scanned images, the first direction is perpendicular to the second direction direction, the first generation module 40 generates a corresponding partial image in the second direction according to each group of scanned images, and then the second generation module 50 generates a panoramic image according to the partial image, so as to be suitable for image stitching of various complex scenes, The method is simple, the flexibility is high, and the splicing effect is good.

In addition, an embodiment of the present application further provides an electronic device, where the electronic device may be a device such as a smart phone, a tablet computer, or the like. As shown in FIG. 7 , the electronic device 900 includes a processor 901 , a memory 902 , a display screen 903 and a control circuit 904 . The processor 901 is electrically connected to the memory 902 , the display screen 903 and the control circuit 904 respectively.

The processor 901 is the control center of the electronic device 900, uses various interfaces and lines to connect various parts of the entire electronic device, executes the electronic Various functions of the device and processing data, so as to carry out the overall monitoring of the electronic device.

In this embodiment, the processor 901 in the electronic device 900 loads the instructions corresponding to the processes of one or more application programs into the memory 902 according to the following steps, and the processor 901 executes the instructions stored in the memory 902 . applications in , so as to achieve various functions:

Get panoramic shooting instructions;

Start the camera to shoot video according to the panoramic shooting instruction, and display the shooting content in the preview box;

During the video shooting process, serpentine scanning is performed on the shooting content along a second direction with a preset first direction overlap rate to obtain multiple sets of scanned images, the first direction being perpendicular to the second direction;

generating corresponding partial images in the second direction according to each group of scanned images;

A panoramic image is generated from the partial image.

Memory 902 may be used to store applications and data. The application program stored in the memory 902 contains instructions executable in the processor. Applications can be composed of various functional modules. The processor 901 executes various functional applications and data processing by executing application programs stored in the memory 902 .

The display screen 903 may be used to display information input by the user or information provided to the user and various graphical user interfaces of the terminal, which may be composed of images, texts, icons, videos, and any combination thereof.

The control circuit 904 is electrically connected to the display screen 903 for controlling the display screen 903 to display information.

In some embodiments, as shown in FIG. 7 , the electronic device 900 further includes: a radio frequency circuit 905 , an input unit 906 , an audio circuit 907 , a sensor 908 and a power supply 909 . The processor 901 is electrically connected to the radio frequency circuit 905 , the input unit 906 , the audio circuit 907 , the sensor 908 and the power supply 909 respectively.

The radio frequency circuit 905 is used for transmitting and receiving radio frequency signals, so as to establish wireless communication with the network equipment or other electronic equipment through wireless communication, and send and receive signals with the network equipment or other electronic equipment.

The input unit 906 may be used to receive input numbers, character information, or user characteristic information (eg, fingerprints), and to generate keyboard, mouse, joystick, optical, or trackball signal input related to user settings and function control. Wherein, the input unit 906 may include a fingerprint identification module.

The audio circuit 907 can provide an audio interface between the user and the terminal through speakers and microphones.

The electronic device 900 may also include at least one sensor 908, such as a light sensor, a motion sensor, and other sensors. Specifically, the light sensor may include an ambient light sensor and a proximity sensor, wherein the ambient light sensor may adjust the brightness of the display panel according to the brightness of the ambient light, and the proximity sensor may turn off the display panel and/or the backlight when the terminal is moved to the ear . As a kind of motion sensor, the gravitational acceleration sensor can detect the magnitude of acceleration in all directions (usually three axes), and can detect the magnitude and direction of gravity when stationary, and can be used for applications that recognize the attitude of mobile phones (such as horizontal and vertical screen switching, related games, magnetometer attitude calibration), vibration recognition related functions (such as pedometer, tapping), etc.; as for other sensors such as gyroscopes, barometers, hygrometers, thermometers, infrared sensors, etc. Repeat.

Power supply 909 is used to power various components of electronic device 900 . In some embodiments, the power supply 909 may be logically connected to the processor 901 through a power management system, so as to implement functions such as managing charging, discharging, and power consumption through the power management system.

Although not shown in FIG. 7 , the electronic device 900 may further include a camera, a Bluetooth module, and the like, which will not be repeated here.

Those of ordinary skill in the art can understand that all or part of the steps in the various methods of the above-mentioned embodiments can be completed by instructions, or by instructions that control relevant hardware, and the instructions can be stored in a computer-readable storage medium, and loaded and executed by the processor. To this end, the embodiments of the present invention provide a storage medium in which a plurality of instructions are stored, and the instructions can be loaded by a processor to execute steps in any of the panoramic image generating methods provided by the embodiments of the present invention.

Wherein, the storage medium may include: a read only memory (ROM, Read Only Memory), a random access memory (RAM, Random Access Memory), a magnetic disk or an optical disk, and the like.

Because the instructions stored in the storage medium can execute the steps in any of the panoramic image generation methods provided by the embodiments of the present invention, it is possible to achieve the capabilities of any panoramic image generation methods provided by the embodiments of the present invention. For the beneficial effects achieved, see the foregoing embodiments for details, and details are not repeated here.

For the specific implementation of the above operations, reference may be made to the foregoing embodiments, and details are not described herein again.

To sum up, although the present application has been disclosed above with preferred embodiments, the above preferred embodiments are not intended to limit the present application. Those of ordinary skill in the art can make various Therefore, the scope of protection of the present application is subject to the scope defined by the claims.

Claims (11)

1. A panoramic image generation method is applied to electronic equipment and is characterized by comprising the following steps:

acquiring a panoramic shooting instruction;

starting a camera according to the panoramic shooting instruction to shoot a video, and displaying the shot content in a preview frame;

in the video shooting process, carrying out snake-shaped scanning on the shot content along a second direction at a preset first direction overlapping rate to obtain a plurality of groups of scanning images, wherein the first direction is vertical to the second direction;

generating corresponding local images in the second direction according to each group of scanning images;

and generating a panoramic image according to the local image.

2. The panoramic image generation method according to claim 1, wherein the generating the corresponding partial image in the second direction from each set of the scanned images includes:

determining a corresponding key frame image group according to each group of scanning images;

extracting the characteristic points of the key frame images in each key frame image group;

and splicing the key frame image group according to the characteristic points to generate a corresponding local image in the second direction.

3. The method of generating a panoramic image according to claim 2, wherein the stitching the key frame image group according to the feature points comprises:

calculating a corresponding first homography matrix according to the feature points of two adjacent key frame images in the key frame image group;

calculating a first global transformation matrix according to the first homography matrix;

and splicing the corresponding key frame image group by using the first global transformation matrix.

4. The method of claim 3, wherein the calculating a corresponding first homography matrix according to the feature points of two adjacent key frame images in the key frame image group comprises:

matching the feature points of two adjacent key frame images in the key frame image group;

acquiring the position information of the feature points successfully matched;

and calculating a first homography matrix according to the position information.

5. The method of claim 2, wherein determining the corresponding set of key frame images from each set of scan images comprises:

in each group of scanning images, taking a first frame scanning image as a key frame image;

acquiring a next frame scanning image adjacent to the key frame image as a target frame image;

judging whether the target frame image meets a preset condition or not according to the feature points of the key frame image and the feature points of the target frame image;

if so, taking the target frame image as a key frame image, and returning to execute the operation of acquiring the next frame scanning image adjacent to the key frame image;

and if not, taking the next frame of scanned image adjacent to the target frame image as the target frame image, and returning to execute the operation of judging whether the target frame image meets the preset condition according to the feature points of the key frame image and the feature points of the target frame image.

6. The method for generating the panoramic image according to claim 5, wherein the determining whether the target frame image satisfies a preset condition according to the feature points of the key frame image and the feature points of the target frame image includes:

matching the characteristic points of the key frame images with the characteristic points of the target frame images;

counting the number of the feature points successfully matched in the key frame image and the total number of the feature points in the key frame image;

and judging whether the target frame image meets a preset condition or not according to the number and the total number of the successfully matched feature points.

7. The method for generating the panoramic image according to claim 6, wherein the determining whether the target frame image meets a preset condition according to the number and the total number of the feature points successfully matched comprises:

calculating similarity according to the number and the total number of the feature points successfully matched;

and when the similarity is smaller than a first preset threshold and the number of the successfully matched feature points is larger than a second preset threshold, judging that the target frame image meets a preset condition.

8. The panoramic image generation method according to any one of claims 1 to 7, wherein the generating a panoramic image from the partial image includes:

calculating a corresponding second homography matrix according to the two adjacent local images;

calculating a second global transformation matrix according to the second homography matrix;

and splicing the local images in the first direction by using the second global transformation matrix to obtain a panoramic image.

9. A panoramic image generation device applied to an electronic device, comprising:

the acquisition module is used for acquiring a panoramic shooting instruction;

the shooting module is used for starting a camera to shoot videos according to the panoramic shooting instruction and displaying shot contents in a preview frame;

the scanning module is used for performing snake-shaped scanning on the shot content along a second direction at a preset first direction overlapping rate in the video shooting process to obtain a plurality of groups of scanning images, wherein the first direction is vertical to the second direction;

the first generation module is used for generating corresponding local images in the second direction according to each group of scanning images;

and the second generation module is used for generating a panoramic image according to the local image.

10. A storage medium having stored therein a plurality of instructions adapted to be loaded by a processor to perform the panoramic image generation method of any one of claims 1 to 8.

11. An electronic device comprising a processor and a memory, the processor being electrically connected to the memory, the memory being configured to store instructions and data, the processor being configured to perform the steps of the panoramic image generation method of any one of claims 1 to 8.

Images