WO2019127269A1

WO2019127269A1 - Image stitching method, image stitching device and electronic device

Info

Publication number: WO2019127269A1
Application number: PCT/CN2017/119565
Authority: WO
Inventors: 程俊; 郝洛莹
Original assignee: 中国科学院深圳先进技术研究院
Priority date: 2017-12-26
Filing date: 2017-12-28
Publication date: 2019-07-04
Also published as: CN108230245A; CN108230245B

Abstract

Provided in the present application are an image stitching method, an image stitching device, an electronic device, and a computer-readable storage medium, the image stitching method comprising: acquiring a sequence of images obtained by continuous photographing; detecting a moving target within the sequence of images in a dynamic background on the basis of an optical flow method; extracting from the sequence of images a first image and second image to be stitched; processing the first image and the second image to obtain a first processed image and a second processed image, wherein the image processing comprises: image registration; and fusing the first processed image and the second processed image on the basis of the result of detection, so as to obtain a stitched image. The technical solution according to the present application is beneficial to improving the effect of eliminating ghost images.

Description

Image stitching method, image stitching device and electronic device

Technical field

The present application belongs to the field of image processing technologies, and in particular, to an image stitching method, an image stitching apparatus, an electronic device, and a computer readable storage medium.

Background technique

The acquisition of panoramic images is an emerging research field and hotspot of computer vision. At present, the panoramic image is obtained mainly by the following two methods: 1. directly using a dedicated wide-angle imaging device (such as a fisheye optical lens, a convex optical lens such as a convex optical lens) to take a sufficiently large horizontal angle image at a time, but The cost is high, and the resolution and viewing angle are difficult to balance, and the image will be severely distorted. 2. By image stitching technology, a set of low-resolution or small-view images with overlapping regions are stitched into a high-resolution, wide-view image. New image. Since the second method has low requirements on the device and can retain the detailed information of the original captured image, the image stitching technique is very important for the acquisition of the panoramic image.

However, in general, images used for image stitching have moving objects in addition to static objects, and the misalignment and superposition of moving objects are the main causes of ghosting in the stitched images. How to eliminate ghosting in image stitching is one of the most difficult problems in the industry.

In the prior art, in the image splicing process, the brightness, color or texture structure of the corresponding pixel in the overlapping area of the image to be spliced is used to determine the position of the moving object, and the moving object is selectively shielded during the image splicing process. This reduces the occurrence of ghosting. However, this method is susceptible to exposure differences, interference pixel points, etc., and only relies on pixel difference for ghost elimination. For most slightly complicated scenes, misoperations are apt to occur, resulting in ghost effect elimination.

technical problem

In view of this, the present application provides an image splicing method, an image splicing device, an electronic device, and a computer readable storage medium, which are beneficial to improving ghost elimination effects.

Technical solution

A first aspect of the embodiment of the present application provides an image mosaic method, including:

Obtaining a sequence of images obtained by continuous shooting;

Detecting a moving target in a dynamic background in the image sequence based on an optical flow method;

Extracting a first image and a second image to be stitched from the image sequence;

Performing image processing on the first image and the second image to obtain a first processed image and a second processed image, wherein the image processing includes: image registration;

And based on the result of the detecting, the first processed image and the second processed image are image-fused to obtain a stitched image.

Based on the first aspect of the present application, in a first possible implementation, a moving target exists in a dynamic background in the image sequence;

The extracting the first image and the second image to be spliced from the image sequence includes:

Extracting the first image and the second image to be spliced from the sequence of images, and causing a complete image of the moving object to be included in an area overlapping the background portion of the second image in the first image.

According to a first possible implementation manner of the first aspect of the present application, in a second possible implementation manner, the image fusion of the first processed image and the second processed image is performed based on a result of the detecting :

Determining, in a result of the detecting, a first region in the first processed image and a second region in the second processed image, wherein a background portion of the first region and the second region overlap;

When an image of the moving object does not exist in the second area, replacing an image portion corresponding to the moving object in the first area with a background portion of a corresponding position in the second area, and a portion of the first processed image other than the image portion corresponding to the moving object, and a portion of the second processed image other than the background portion of the corresponding position and the entire image of the moving object

When a partial image of the moving target exists in the second region, replacing an image portion corresponding to the moving target in the first region with a background portion of a corresponding position in the second region, and And a portion other than the image portion corresponding to the moving object in the first processed image, and image fusion with the background portion of the second processed image and the portion other than the partial image of the moving target ;

When a complete image of the moving object exists in the second region, replacing an image portion corresponding to the moving target in the first region with a background portion of a corresponding position in the second region, and Image fusion of the portion of the first processed image other than the image portion corresponding to the moving object, and other portions of the second processed image other than the background portion of the corresponding position and the complete image of the moving object Or replacing the image portion corresponding to the moving object in the second region with the background portion of the corresponding position in the first region, and dividing the image portion corresponding to the moving target in the second processed image The other portion is image-fused with the portion of the first processed image other than the background portion of the corresponding position and the entire image of the moving object.

Based on the first aspect of the present application, or the first possible implementation manner of the first aspect of the application, or the second possible implementation manner of the first aspect of the present application, in a third possible implementation, the image Processing also includes: morphological processing;

The performing image processing on the first image and the second image includes:

Performing image registration on the first image and the second image;

Based on the result of the detecting, the image-registered first image and the second image are subjected to morphological processing to refine the moving object in the first image and the second image.

The second aspect of the present application provides an image splicing apparatus, including:

An acquiring unit, configured to obtain a sequence of images obtained by continuous shooting;

An optical flow detecting unit, configured to detect a moving target in a dynamic background in the image sequence based on an optical flow method;

Extracting unit, configured to extract a first image and a second image to be stitched from the image sequence;

An image processing unit, configured to perform image processing on the first image and the second image to obtain a first processed image and a second processed image, where the image processing includes: image registration;

And an image fusion unit configured to perform image fusion on the first processed image and the second processed image based on a result of the detection by the optical flow detecting unit to obtain a stitched image.

Based on the second aspect of the present application, in a first possible implementation manner, a moving target exists in a dynamic background in the image sequence;

The extracting unit is specifically configured to: extract a first image and a second image to be spliced from the image sequence, and include an area in the first image that overlaps with a background portion of the second image A complete image of the target.

Based on the first possible implementation manner of the second aspect of the present application, in a second possible implementation manner, the image fusion unit specifically includes:

a determining unit, configured to determine a first region in the first processed image and a second region in the second processed image based on a result of the detecting, wherein the first region and the second region The background overlaps partially;

a sub-fusion unit, configured to replace an image portion corresponding to the moving object in the first region with a background portion of a corresponding position in the second region when there is no image of the moving target in the second region, And performing image fusion with other portions of the first processed image other than the image portion corresponding to the moving object, and other portions of the second processed image other than the background portion of the corresponding position; When there is a partial image of the moving target in the second region, the image portion corresponding to the moving target in the first region is replaced with the background portion of the corresponding position in the second region, and the first processed image is Performing image fusion with other portions of the second processed image other than the background portion of the corresponding position and the partial image of the moving target; When a complete image of the moving object exists in the two regions, replacing an image portion corresponding to the moving target in the first region with a corresponding one in the second region a background portion, and other portions of the first processed image other than the image portion corresponding to the moving object, and a background portion of the second processed image other than the corresponding position and the moving target Other portions of the entire image are image-fused, or when a complete image of the moving object exists in the second region, the image portion corresponding to the moving target in the second region is replaced with the first a background portion of a corresponding position in the region, and a portion of the second processed image other than the image portion corresponding to the moving object, and a background portion of the first processed image other than the corresponding position Image fusion is performed on other parts of the moving image beyond the complete image.

Based on the second aspect of the present application, or the first possible implementation manner of the second aspect of the present application, or the second possible implementation manner of the second aspect of the present application, in a third possible implementation manner, the image Processing also includes: morphological processing;

The image processing unit is specifically configured to: perform image registration on the first image and the second image; and perform morphology on the image-registered first image and the second image based on the detection result Processing to refine the moving objects in the first image and the second image.

A third aspect of the present application provides an electronic device including a memory, a processor, and a computer program stored on the memory and operable on the processor, the processor implementing the computer program to implement the first aspect or the first aspect An image stitching method mentioned in any of the possible implementations.

A fourth aspect of the present application provides a computer readable storage medium having stored thereon a computer program, wherein the computer program is executed by a processor to implement the first aspect or any of the possible implementations of the first aspect The image stitching method mentioned in the paper.

A fifth aspect of the present application provides a computer program product comprising a computer program, the computer program being executed by one or more processors to implement any of the above first aspects or any of the above first aspects The image stitching method mentioned in the way.

Beneficial effect

It can be seen from the above that the solution of the present application acquires the image sequence obtained by continuous shooting, and detects the moving target in the dynamic background in the image sequence based on the optical flow method. And then extracting the first image and the second image to be spliced from the image sequence, performing image processing on the first image and the second image to obtain the first processed image and the second processed image, and based on the result of the detecting, The image is processed and the second processed image is image-fused to obtain a stitched image. Since the ghost image is mostly caused by the moving target in the image to be stitched, and the scheme for judging the moving target in the image based on the pixel difference, the optical flow method can effectively detect the image in a complicated scene. The moving target, therefore, the solution of the present application detects the moving target in the dynamic background in the image sequence by the optical flow method, and then performs image fusion on the first processed image and the second processed image based on the detected result, thereby effectively improving the ghost image. Elimination effect.

DRAWINGS

In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings used in the embodiments or the prior art description will be briefly described below. Obviously, the drawings in the following description are only the present application. For some embodiments, other drawings may be obtained from those of ordinary skill in the art in light of the inventive workability.

FIG. 1 is a schematic flowchart of an embodiment of an image splicing method provided by the present application;

FIG. 1-b is a schematic diagram of an image fusion process that can be applied to the embodiment shown in FIG. 1-a according to the present application;

FIG. 2-a is a schematic diagram of a first processing image of an application scenario provided by the present application;

FIG. 2-b is a schematic diagram of a second processing image of an application scenario provided by the present application;

Figure 2-c is a schematic diagram of a first area determined based on the first processed image and the second processed image shown in Figures 2-a and 2-b;

Figure 2-d is a schematic diagram of a second region determined based on the first processed image and the second processed image illustrated in Figures 2-a and 2-b;

FIG. 2-e is a schematic diagram of an image obtained by image fusion based on the first region and the second region illustrated in FIG. 2-c and FIG. 2-d; FIG.

3 is a schematic structural diagram of an embodiment of an image splicing device provided by the present application;

FIG. 4 is a schematic structural diagram of an embodiment of an electronic device provided by the present application.

Embodiments of the invention

In the following description, for purposes of illustration and description However, it will be apparent to those skilled in the art that the present invention may be practiced in other embodiments without these specific details. In other instances, detailed descriptions of well-known systems, devices, circuits, and methods are omitted so as not to obscure the description of the application.

It should be understood that the size of the serial number of each step in the following method embodiments does not mean the order of execution sequence, and the execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of each embodiment. .

In order to explain the technical solutions described in the present application, the following description will be made by way of specific embodiments.

Embodiment 1

The image splicing method can be applied to an image splicing device, and the image splicing device can be an independent device, or the image splicing device can also be integrated in an electronic device (for example, a smart phone or a tablet). In computers, computers, wearables, etc.). Optionally, the operating system of the device or the electronic device integrated with the image splicing device may be an ios system, an android system, a windows system, or other operating system, which is not limited herein.

Referring to FIG. 1-a, the image splicing method in the embodiment of the present application may include:

Step 101: Acquire an image sequence obtained by continuous shooting;

In the embodiment of the present application, the image sequence may be a set of consecutive frame images taken by a single camera. In order to ensure that the resolution of each image in the image sequence and the field of view of the spliced image are expanded, in the embodiment of the present application, the camera can be slowly rotated along a vertical axis (ie, the angular velocity of the camera rotation is less than an angular velocity threshold (for example, 50 degrees). / sec)) Take a picture of the image to get the sequence of images.

In step 101, the image sequence may be captured in real time, or a pre-stored image sequence may be obtained from a database, which is not limited herein.

Step 102: Detecting a moving target in a dynamic background in the image sequence based on an optical flow method;

The optical flow field is used to represent the instantaneous velocity field of the pixel point change trend in the image, and the principle of detecting the moving target in the dynamic background in the above image sequence based on the optical flow method is: based on the image of each adjacent frame in the image sequence The change in the distribution of the pixel points calculates the relevant motion information (for example, the moving speed and the moving direction) of the moving object in each image. In the embodiment of the present application, if there is a moving target in the image sequence, the motion in the image sequence is jointly generated by the motion of the moving target itself and the motion of the camera. Since there is relative motion between the moving target and the background, the moving target is located. The motion vector of the pixel is different from the motion vector of the background, so that the moving target in the dynamic background in the image sequence can be detected based on the optical flow method. For example, the background threshold and the moving target can be distinguished by setting the corresponding threshold based on the motion vector of the background.

The optical flow method can be divided into a sparse optical flow method (such as LK optical flow method (ie, Lucas-Kanade method) and dense optical flow method (such as Gunnar Farneback optical flow method). For the sparse optical flow method, calculation needs to be Specifying a number of feature points (such as corner points) before detecting a moving target, for a less textured moving target part (such as a human hand), the sparse optical flow method is easier to lose. The dense optical flow method is different from the sparse optical flow. Only for a number of feature points on the image, the dense optical flow method calculates the offset of all the points on the image, thus forming a dense optical flow field. Therefore, in order to better improve the detection effect of the moving target, the follow-up ghost The effect of eliminating the shadow is better. In the embodiment of the present application, the moving object in the dynamic background in the image sequence may be detected based on the dense optical flow method.

Step 103: Extract a first image and a second image to be spliced from the image sequence.

In step 103, the first image and the second image to be spliced may be automatically extracted from the image sequence, or the first image and the second image to be spliced may be manually selected by the user from the image sequence. In step 103, the first image and the second image are extracted based on the user's selection.

For a scene in which there is a moving object in the dynamic background of the image sequence, the displacement of the moving target may cause ghosting in the image after the stitching. Therefore, in this scenario, step 103 may specifically be: extracting from the image sequence. The first image and the second image to be spliced, and such that a region overlapping the background portion of the second image in the first image includes a complete image of the moving object. That is, the area where the first image overlaps the background portion of the second image is the area S1 (the area S1 belongs to the image area in the first image), and the area where the second image overlaps the background portion of the first image is the area S2 ( The area S2 belongs to the image area in the first image, and the area S1 contains a complete image of the moving object, and the area S2 may contain a complete image or a partial image of the moving object, or may not contain the moving object. Further, on the basis of ensuring that the first image overlaps with the background image of the second image, the entire image of the moving object is included, and the second image is also ensured as much as possible with respect to the background of the first image (or The first image is extended relative to the background of the second image such that the field of view of the subsequently obtained stitched image is expanded.

Optionally, the step 103 may include: extracting, according to the detection result of step 102, an image including a complete image of the moving target from the image sequence as a first image; and extracting, from the image sequence, the preset frame from the first image interval The number of images is used as the second image, thereby achieving automatic extraction of the first image and the second image. For example, if the image sequence includes 10 images obtained by continuous shooting, and the preset number of frames is 3, and the first image includes a complete image of the moving object, the first image in the image sequence may be extracted as the first image. Then, the fifth image in the image sequence is extracted as the second image (the number of frames in which the fifth image is separated from the first image is 3).

Step 104: Perform image processing on the first image and the second image to obtain a first processed image and a second processed image.

Considering that the first image and the second image may have translation, rotation, scaling, etc., the image processing is performed on the first image and the second image in step 104, wherein step 104 is performed on the first image and the second image. Performing image processing on the image includes performing image registration on the first image and the second image. The process of performing image registration on the first image and the second image may be as follows: performing feature extraction and matching on the first image and the second image to find matching feature point pairs; determining the first based on the matched feature point pairs And a coordinate transformation parameter of the image and the second image, and finally performing image registration on the first image and the second image based on the coordinate transformation parameter. Optionally, the step 104 may perform image registration on the first image and the second image based on the SURF algorithm, and the specific process of performing image registration on the two images may be implemented by referring to the prior art, and details are not described herein again.

In order to eliminate the influence of the interfering pixel on the moving object in the image, in step 104, the first image and the second image may be further subjected to morphological processing (for example, etching, expansion, etc.). The step 104 may specifically include: performing image registration on the first image and the second image; and performing morphological processing on the image-registered first image and the second image based on the result of the detecting to accurately a moving target in the first image and the second image described above. Taking the first image as a moving target as an example for description, based on the result of step 102, the contour of the moving target in the first image may be extracted, and the first image is subjected to morphological processing, and the image is etched to remove the miscellaneous in the contour. Point, fill the broken part of the outline with image expansion, and then fill the inside of the outline to obtain the area where the moving target in the first image is located. Specifically, the process of performing morphological processing on an image may be implemented by referring to the prior art, and details are not described herein again.

Further, after the morphological processing is performed on the first image and the second image, the moving object in the image may also be locked by using a connectivity area (for example, a rectangular frame) to determine an image portion corresponding to the moving target. For example, if a moving object in an image is locked by a rectangular frame, the portion framed by the rectangular frame is the image portion corresponding to the moving target in the image.

Step 105: Perform image fusion on the first processed image and the second processed image based on the result of the detecting, to obtain a stitched image;

In the embodiment of the present application, image fusion refers to a process of processing a plurality of images through a computer such as image processing, extracting favorable information in each image to the utmost, and finally integrating into a high-quality image. In step 105, based on the result of step 102, the moving object included in each image in the image sequence may be extracted, and after the image processing in step 104, the first processed image and the second processed image are image-fused. Get the stitched image.

When there is a moving target in the dynamic background in the image sequence, in order to avoid that the image of the first processed image and the second processed image both contain the moving target, the stitched image has a moving object in the region, where In the scenario, as shown in FIG. 1-b, step 105 may include:

Step 1051: Determine, according to a result of the foregoing detection, a first area in the first processed image and a second area in the second processed image;

The background regions of the first region and the second region overlap.

In step 1051, based on the result of step 102, the image portion corresponding to the background portion and the moving target in the first processed image and the second processed image may be distinguished, and the first processed image may be further determined based on the similarity measure. An area in which the second processed image partially overlaps the background (ie, the first area), and an area in the second processed image that overlaps with the first processed image in the background portion (ie, the second area).

Step 1052: When there is no image of the moving target in the second region, replace the image portion corresponding to the moving target in the first region with the background portion of the corresponding position in the second region, and remove the motion in the first processed image. The other part except the image part corresponding to the target is image-fused with the other part of the second processed image except the background part of the corresponding position;

In step 1052, the first area includes a complete image of the moving object, and for the second area, there are the following three situations: 1. the image of the moving target does not exist in the second area; 2. the second area There is a partial image of the above moving target; 3. The completed image of the moving target exists in the second region.

For the first case described above, in step 1052, the image portion corresponding to the moving object in the first region is replaced with the background portion of the corresponding position in the second region, and the image portion corresponding to the moving target in the first processed image is The other portion is image-fused with other portions of the second processed image than the background portion of the corresponding position.

Step 1053: When there is a partial image of the moving target in the second region, replace the image portion corresponding to the moving target in the first region with the background portion of the corresponding position in the second region, and remove the motion in the first processed image. And the other portion except the image portion corresponding to the target is image-fused with the background portion of the second processed image except the partial portion of the moving target and the portion of the moving target;

For the second case mentioned in step 1052, in step 1053, the image portion corresponding to the moving object in the first region may be replaced with the background portion of the corresponding position in the second region, and the first processed image is divided into the above. The other portion except the image portion corresponding to the moving object is image-fused with the portion of the second processed image other than the background portion of the corresponding position and the partial image of the moving target.

For example, as shown in FIG. 2-a and FIG. 2-b, the first processed image and the second processed image are respectively, wherein the artificial moving target in FIG. 2-a and FIG. 2-b is other than human. For the background portion, based on step 1051, it can be determined that Figure 2-c is the first region described above, and Figure 2-d is the second region. Since there is a partial image of the moving object in the first region shown in FIG. 2-c, in step 1053, the image portion corresponding to the moving object in FIG. 2-c is replaced with the background portion of the corresponding position in FIG. 2-d. And merging the other portion of the image in FIG. 2-a except the image portion corresponding to the moving object with the background portion of the corresponding position in FIG. 2-c and the portion other than the partial image of the moving object, An image as shown in FIG. 2-e (ie, the stitched image) can be obtained. As can be seen from FIG. 2-e, the image portion corresponding to the moving object in the second region shown in FIG. 2-d is illustrated by FIG. 2-c. The background portion in the first region shown is replaced, and Figure 2-e has a wider field of view relative to Figures 2-c and 2-d. Step 1054: When there is a complete image of the moving target in the second region, replace the image portion corresponding to the moving target in the first region with the background portion of the corresponding position in the second region, and save the first processed image The other part of the image corresponding to the moving object is image-fused with the background portion of the second processed image except the background portion of the corresponding position and the entire image of the moving object;

For the third case mentioned in step 1052, in step 1054, the image portion corresponding to the moving target in the first region may be replaced with the background portion of the corresponding position in the second region, and the first processed image may be divided. The other portion except the image portion corresponding to the moving object is image-fused with the background portion of the second processed image except the background portion of the corresponding position and the entire image of the moving object; or, in other embodiments, when When there is a complete image of the moving object in the second region, the image portion corresponding to the moving target in the second region may be replaced with the background portion of the corresponding position in the first region, and the second processed image may be divided into the above. The other portion except the image portion corresponding to the moving object is image-fused with the portion of the first processed image except the background portion of the corresponding position and the entire image of the moving object.

It can be seen from the above that in the embodiment of the present application, the image sequence obtained by continuous shooting is acquired, and the moving target in the dynamic background in the image sequence is detected based on the optical flow method. And then extracting the first image and the second image to be spliced from the image sequence, performing image processing on the first image and the second image to obtain the first processed image and the second processed image, and based on the result of the detecting, The image is processed and the second processed image is image-fused to obtain a stitched image. Since the ghost image is mostly caused by the moving target in the image to be stitched, and the scheme for judging the moving target in the image based on the pixel difference, the optical flow method can effectively detect the image in a complicated scene. The moving target, therefore, the solution of the present application detects the moving target in the dynamic background in the image sequence by the optical flow method, and then performs image fusion on the first processed image and the second processed image based on the detected result, thereby effectively improving the ghost image. Elimination effect.

Embodiment 2

The embodiment of the present application provides an image splicing device. As shown in FIG. 3, the image splicing device 300 in the embodiment of the present application includes:

The acquiring unit 301 is configured to acquire a sequence of images obtained by continuous shooting;

The optical flow detecting unit 302 is configured to detect a moving target in a dynamic background in the image sequence based on an optical flow method;

The extracting unit 303 is configured to extract, from the image sequence, a first image and a second image to be stitched;

The image processing unit 304 is configured to perform image processing on the first image and the second image to obtain a first processed image and a second processed image, where the image processing includes: image registration;

The image fusion unit 305 is configured to perform image fusion on the first processed image and the second processed image based on the result detected by the optical flow detecting unit 302 to obtain a stitched image.

Optionally, there is a moving target in the dynamic background in the image sequence. The extracting unit 303 is specifically configured to: extract a first image and a second image to be spliced from the image sequence, and include a moving target in an area overlapping the background portion of the second image in the first image Full image.

Optionally, the image fusion unit 305 specifically includes:

Optionally, the image processing further includes: morphological processing. The image processing unit 304 is specifically configured to: perform image registration on the first image and the second image; perform morphological processing on the first image and the second image after image registration based on the result of the detecting To refine the moving objects in the first image and the second image.

It should be noted that the image splicing device in the embodiment of the present application may be an independent device, or alternatively, the image splicing device may be integrated into an electronic device (such as a smart phone, a tablet computer, a computer, a wearable device, etc.). . Optionally, the operating system of the device or the electronic device integrated with the image splicing device may be an ios system, an android system, a windows system, or other operating system, which is not limited herein.

Embodiment 3

The embodiment of the present application provides an electronic device. Referring to FIG. 4, the electronic device in the embodiment of the present application includes: a memory 401, one or more processors 402 (only one is shown in FIG. 4), and stored in the memory 401. A computer program that can be run on a processor. Wherein: the memory 401 is used to store software programs and modules, and the processor 402 executes various functional applications and data processing by running software programs and units stored in the memory 401. Specifically, the processor 402 implements the following steps by running the above computer program stored in the memory 401:

Obtaining a sequence of images obtained by continuous shooting;

Assuming that the foregoing is a first possible implementation manner, in a second possible implementation manner provided by the first possible implementation manner, a moving target exists in a dynamic background in the image sequence;

Extracting the first image and the second image to be spliced from the image sequence is:

In a third possible implementation manner provided by the foregoing second possible implementation manner, the performing image fusion on the first processed image and the second processed image based on the result of the detecting includes:

When there is no image of the moving target in the second area, replacing an image portion corresponding to the moving target in the first area with a background portion of a corresponding position in the second area, and the first Processing other portions of the image other than the image portion corresponding to the moving object, performing image fusion with other portions of the second processed image other than the background portion of the corresponding position; when there is motion in the second region a partial image of the target, replacing an image portion corresponding to the moving object in the first region with a background portion of a corresponding position in the second region, and corresponding to the moving target in the first processed image Other portions of the image portion other than the portion of the second processed image other than the background portion of the corresponding position and the partial image of the moving object; when the second region is present When a complete image of the moving target is used, the image portion corresponding to the moving target in the first region is replaced with the background portion of the corresponding position in the second region, And other portions of the first processed image other than the image portion corresponding to the moving object, and other portions of the second processed image other than the background portion of the corresponding position and the complete image of the moving target Performing image fusion, or, when a complete image of the moving object exists in the second region, replacing an image portion corresponding to the moving target in the second region with a background of a corresponding position in the first region a portion, and the other portion of the second processed image except the image portion corresponding to the moving object, and the background portion of the first processed image except the background portion of the corresponding position and the complete image of the moving target The other part of the image is fused. In a fourth possible implementation manner provided by the foregoing first possible implementation manner, or the foregoing second possible implementation manner or the foregoing third possible implementation manner, the image processing further includes: Learning

Performing image registration on the first image and the second image;

Optionally, as shown in FIG. 4, the foregoing electronic device may further include: one or more input devices 403 (only one is shown in FIG. 4) and one or more output devices 404 (only one is shown in FIG. 4). . The memory 401, the processor 402, the input device 403, and the output device 404 are connected by a bus 405.

It should be understood that in the embodiment of the present application, the so-called processor 402 may be a central processing unit (Central) Processing Unit (CPU), which can also be other general-purpose processors, digital signal processors (DSPs), and application specific integrated circuits (Application). Specific Integrated Circuit (ASIC), Field-Programmable Gate Array (FPGA) or other programmable logic device, discrete gate or transistor logic device, discrete hardware components, etc. The general purpose processor may be a microprocessor or the processor or any conventional processor or the like.

The input device 403 can include a keyboard, a touchpad, a fingerprint sensor (for collecting fingerprint information of the user and direction information of the fingerprint), a microphone, etc., and the output device 404 can include a display, a speaker, and the like.

Memory 404 can include read only memory and random access memory and provides instructions and data to processor 401. Some or all of memory 404 may also include non-volatile random access memory. For example, the memory 404 can also store information of the device type.

It will be apparent to those skilled in the art that, for convenience and brevity of description, only the division of each functional unit and module described above is exemplified. In practical applications, the above functions may be assigned to different functional units as needed. The module is completed by dividing the internal structure of the above device into different functional units or modules to perform all or part of the functions described above. Each functional unit and module in the embodiment may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit, and the integrated unit may be hardware. Formal implementation can also be implemented in the form of software functional units. In addition, the specific names of the respective functional units and modules are only for the purpose of facilitating mutual differentiation, and are not intended to limit the scope of protection of the present application. For the specific working process of the unit and the module in the foregoing system, reference may be made to the corresponding process in the foregoing method embodiment, and details are not described herein again.

In the above embodiments, the descriptions of the various embodiments are different, and the parts that are not detailed or described in a certain embodiment can be referred to the related descriptions of other embodiments.

Those of ordinary skill in the art will appreciate that the elements and algorithm steps of the various examples described in connection with the embodiments disclosed herein can be implemented in electronic hardware or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the solution. A person skilled in the art can use different methods to implement the described functions for each particular application, but such implementation should not be considered to be beyond the scope of the present application.

In the embodiments provided by the present application, it should be understood that the disclosed apparatus and method may be implemented in other manners. For example, the system embodiments described above are merely illustrative. For example, the division of the above modules or units is only a logical function division. In actual implementation, there may be another division manner, for example, multiple units or components may be combined. Or it can be integrated into another system, or some features can be ignored or not executed. In addition, the mutual coupling or direct coupling or communication connection shown or discussed may be an indirect coupling or communication connection through some interface, device or unit, and may be in electrical, mechanical or other form.

The units described above as separate components may or may not be physically separated. The components displayed as units may or may not be physical units, that is, may be located in one place, or may be distributed to multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of the embodiment.

The above-described integrated unit, if implemented in the form of a software functional unit and sold or used as a stand-alone product, may be stored in a computer readable storage medium. Based on such understanding, the present application implements all or part of the processes in the above embodiments, and may also be completed by a computer program to instruct related hardware. The computer program may be stored in a computer readable storage medium. The steps of the various method embodiments described above may be implemented when executed by a processor. Wherein, the above computer program comprises computer program code, and the computer program code may be in the form of source code, object code form, executable file or some intermediate form. The computer readable medium may include any entity or device capable of carrying the above computer program code, a recording medium, a USB flash drive, a removable hard disk, a magnetic disk, an optical disk, a computer memory, a read only memory (ROM, Read-Only). Memory), random access memory (RAM, Random) Access Memory), electrical carrier signals, telecommunications signals, and software distribution media. It should be noted that the contents of the above computer readable medium may be appropriately increased or decreased according to the requirements of legislation and patent practice in jurisdictions. For example, in some jurisdictions, according to legislation and patent practice, computer readable media are not It includes electrical carrier signals and telecommunication signals.

The above embodiments are only used to explain the technical solutions of the present application, and are not limited thereto; although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that the foregoing embodiments can still be The technical solutions are modified, or some of the technical features are replaced by equivalents; and the modifications or substitutions do not deviate from the spirit and scope of the technical solutions of the embodiments of the present application, and should be included in the present disclosure. Within the scope of protection of the application.

Claims

An image mosaic method, comprising:

Obtaining a sequence of images obtained by continuous shooting;

Detecting a moving target in a dynamic background in the image sequence based on an optical flow method;

Extracting a first image and a second image to be stitched from the image sequence;

Performing image processing on the first image and the second image to obtain a first processed image and a second processed image, wherein the image processing includes: image registration;

And based on the result of the detecting, the first processed image and the second processed image are image-fused to obtain a stitched image.
The image stitching method according to claim 1, wherein a moving object exists in a dynamic background in the image sequence;

Extracting the first image and the second image to be spliced from the image sequence is:

Extracting the first image and the second image to be spliced from the sequence of images, and causing a complete image of the moving object to be included in an area overlapping the background portion of the second image in the first image.
The image splicing method according to claim 2, wherein the image fusion of the first processed image and the second processed image based on the result of the detecting comprises:

Determining, in a result of the detecting, a first region in the first processed image and a second region in the second processed image, wherein a background portion of the first region and the second region overlap;

When an image of the moving object does not exist in the second area, replacing an image portion corresponding to the moving object in the first area with a background portion of a corresponding position in the second area, and And a portion other than the image portion corresponding to the moving object in the first processed image is image-fused with other portions of the second processed image except the background portion of the corresponding position;

When a partial image of the moving target exists in the second region, replacing an image portion corresponding to the moving target in the first region with a background portion of a corresponding position in the second region, and And a portion other than the image portion corresponding to the moving object in the first processed image, and image fusion with the background portion of the second processed image and the portion other than the partial image of the moving target When a complete image of the moving object exists in the second area, replacing an image portion corresponding to the moving object in the first area with a background portion of a corresponding position in the second area, and An image of the first processed image other than the image portion corresponding to the moving object, and an image of the second processed image other than the background portion of the corresponding position and the entire image of the moving object Or merging; or replacing an image portion corresponding to the moving object in the second region with a background portion of a corresponding position in the first region, and Other portions of the second processed image other than the image portion corresponding to the moving object, and image fusion with the background portion of the first processed image other than the complete image of the moving target .
The image splicing method according to any one of claims 1 to 3, wherein the image processing further comprises: morphological processing;

The performing image processing on the first image and the second image includes:

Performing image registration on the first image and the second image;

Based on the result of the detecting, the image-registered first image and the second image are subjected to morphological processing to refine the moving object in the first image and the second image.
An image splicing device, comprising:

An acquiring unit, configured to obtain a sequence of images obtained by continuous shooting;

An optical flow detecting unit, configured to detect a moving target in a dynamic background in the image sequence based on an optical flow method;

Extracting unit, configured to extract a first image and a second image to be stitched from the image sequence;

An image processing unit, configured to perform image processing on the first image and the second image to obtain a first processed image and a second processed image, where the image processing includes: image registration;

And an image fusion unit configured to perform image fusion on the first processed image and the second processed image based on a result of the detection by the optical flow detecting unit to obtain a stitched image.
The image splicing apparatus according to claim 5, wherein a moving target exists in a dynamic background in the image sequence;

The extracting unit is specifically configured to: extract a first image and a second image to be spliced from the image sequence, and include an area in the first image that overlaps with a background portion of the second image A complete image of the target.
The image splicing apparatus according to claim 6, wherein the image merging unit specifically comprises:

a determining unit, configured to determine a first region in the first processed image and a second region in the second processed image based on a result of the detecting, wherein the first region and the second region The background overlaps partially;

a sub-fusion unit, configured to replace an image portion corresponding to the moving object in the first region with a background portion of a corresponding position in the second region when there is no image of the moving target in the second region, And performing image fusion with other portions of the first processed image other than the image portion corresponding to the moving object, and other portions of the second processed image other than the background portion of the corresponding position; When there is a partial image of the moving target in the second region, the image portion corresponding to the moving target in the first region is replaced with the background portion of the corresponding position in the second region, and the first processed image is Performing image fusion with other portions of the second processed image other than the background portion of the corresponding position and the partial image of the moving target; When a complete image of the moving object exists in the two regions, replacing an image portion corresponding to the moving target in the first region with a corresponding one in the second region a background portion, and other portions of the first processed image other than the image portion corresponding to the moving object, and a background portion of the second processed image other than the corresponding position and the moving target Other portions of the entire image are image-fused, or when a complete image of the moving object exists in the second region, the image portion corresponding to the moving target in the second region is replaced with the first a background portion of a corresponding position in the region, and a portion of the second processed image other than the image portion corresponding to the moving object, and a background portion of the first processed image other than the corresponding position Image fusion is performed on other parts of the moving image beyond the complete image.
The image splicing apparatus according to any one of claims 5 to 7, wherein the image processing further comprises: morphological processing;

The image processing unit is specifically configured to: perform image registration on the first image and the second image; and perform morphology on the image-registered first image and the second image based on the detection result Processing to refine the moving objects in the first image and the second image.
An electronic device comprising a memory, a processor, and a computer program stored in the memory and operable on the processor, wherein the processor executes the computer program as claimed in claim 1 4 The steps of any of the methods described.
A computer readable storage medium storing a computer program, wherein the computer program, when executed by a processor, implements the steps of the method of any one of claims 1 to 4.