WO2019000427A1

WO2019000427A1 - Image processing method and apparatus, and electronic device

Info

Publication number: WO2019000427A1
Application number: PCT/CN2017/091245
Authority: WO
Inventors: 苏冠华; 刘昂; 毛曙源; 胡骁
Original assignee: 深圳市大疆创新科技有限公司
Priority date: 2017-06-30
Filing date: 2017-06-30
Publication date: 2019-01-03
Also published as: CN108521823A

Abstract

An image processing method and apparatus, and an electronic device. The method comprises: acquiring a set of target environment images collected while an aircraft is flying along a specific trajectory, wherein the set of images comprises a plurality of images; extracting, from the set of images, a foreground portion of a target environment and the respective position of the foreground portion in corresponding images; splicing background portions of the target environment in at least some images from the set of images to generate a panoramic image of the background portion of the target environment; and generating a micro-motion image, wherein each frame of image in the micro-motion image is synthesized according to the panoramic image of the background portion and the respective position of the foreground portion in the corresponding images. By means of the embodiments of the present invention, a micro-motion image can be automatically generated according to a set of images collected by an aircraft, so that the synthesis efficiency of the micro-motion image is improved, and the automation and intelligence of the synthesis of the micro-motion image is realized.

Description

Image processing method, device and electronic device

Technical field

The present invention relates to the field of image processing technologies, and in particular, to an image processing method, apparatus, and electronic device.

Background technique

In order to meet the individual needs of users and enrich the way images are represented, Cinema Graph came into being. Micro-motion photography is a kind of photography between photos and videos. The pictures obtained by micro-motion photography are presented in the form of moving pictures. The local area in the picture is dynamic, while the other parts are static, which can achieve outstanding dynamics. The artistic effect of the area. For example, the picture of a micro-motion diagram is that on a road, only the protagonist walks along the road, and other people and things are still, which can highlight the protagonist.

In the prior art, the micro-motion photography is to manually record a video and then manually apply an area to the recorded video. The area to be smeared is dynamic, the other areas are still, and the dynamic area is after editing. And the static area is saved as an image file, which is a micro image. However, the shooting procedure in the above manner is complicated, and the micro-motion map cannot be generated after the shooting is completed, and the user needs to manually process the captured video. The manual processing wastes time and the accuracy is poor, and the micro-motion drawing synthesis efficiency is low.

Summary of the invention

The embodiment of the invention discloses an image processing method, device and electronic device, which can automatically generate a micro-motion map and improve the synthesis efficiency of the micro-motion map.

The first aspect of the embodiment of the present invention discloses an image processing method, including:

Acquiring a collection of images acquired for the target environment when the aircraft is flying along a particular trajectory, the collection of images comprising a plurality of images;

Extracting a foreground portion of the target environment and a position of the foreground portion in the corresponding image from the image set;

Splicing a background portion of the target environment in at least a portion of the image set, a panoramic view of the background portion of the target environment;

A micro-motion map is generated, wherein each frame image in the micro-motion map is synthesized according to a panoramic view of the background portion and a position of the foreground portion in a corresponding image, respectively.

The second aspect of the embodiment of the present invention discloses an image processing apparatus, including:

An acquisition module, configured to acquire a collection of images acquired by the aircraft for a target environment when flying along a specific trajectory, the image collection comprising a plurality of images;

An extracting module, configured to extract, from the image set, a foreground portion of the target environment and a position of the foreground portion in a corresponding image;

a splicing module, configured to splicing a background portion of the target environment in at least part of the image of the image set to generate a panoramic view of a background portion of the target environment;

And a synthesizing module, configured to generate a micro-motion image, wherein each frame image in the micro-motion map is synthesized according to a panoramic view of the background portion and a position of the foreground portion in a corresponding image.

A third aspect of the embodiments of the present invention discloses an electronic device, including: a processor and a memory,

The memory is configured to store program instructions;

The processor is configured to execute the program instructions stored by the memory, when the program instructions are executed, the processor is configured to:

Splicing a background portion of the target environment in at least a portion of the image set to generate a panoramic view of a background portion of the target environment;

A fourth aspect of an embodiment of the present invention discloses a computer program product, wherein the image processing method is executed when an instruction in the computer program product is executed by a processor.

A fifth aspect of an embodiment of the present invention discloses a storage medium, wherein when an instruction in the storage medium is executed by a processor of the electronic device, the electronic device is enabled to execute the image processing method described above.

In the embodiment of the present invention, first, acquiring an image set collected by the aircraft for the target environment, and then Extracting the foreground part of the target environment and the position of the foreground part in the corresponding image, generating a panoramic view of the background part of the target environment according to the image included in the image set, and finally generating a micro-motion map, which may be based on the image collection collected by the aircraft The micro-motion map is automatically generated, which can improve the synthesis efficiency of the micro-motion map and realize the automation and intelligence of the micro-motion map synthesis.

DRAWINGS

In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings to be used in the embodiments will be briefly described below. It is obvious that the drawings in the following description are only some embodiments of the present invention. Those skilled in the art can also obtain other drawings based on these drawings without paying for creative labor.

1 is a schematic flow chart of a first embodiment of an image processing method according to an embodiment of the present invention;

2 is a schematic diagram of an image processing process disclosed in an embodiment of the present invention;

3 is a schematic diagram of another image processing process disclosed in an embodiment of the present invention;

4 is a schematic diagram of still another image processing process disclosed in an embodiment of the present invention;

FIG. 5 is a schematic flowchart diagram of a second embodiment of an image processing method according to an embodiment of the present invention; FIG.

FIG. 6 is a schematic structural diagram of an image processing apparatus according to an embodiment of the present invention; FIG.

FIG. 7 is a schematic structural diagram of an electronic device according to an embodiment of the present invention.

Detailed ways

The technical solutions in the embodiments of the present invention will be clearly and completely described in the following with reference to the accompanying drawings.

FIG. 1 is a schematic flowchart diagram of a first embodiment of an image processing method according to an embodiment of the present invention. The image processing method described in this embodiment includes but is not limited to the following steps:

S101. The terminal acquires a collection of images collected by the aircraft for the target environment when the aircraft is flying along a specific trajectory.

In the embodiment of the present invention, the target environment is a user-specified shooting area for collecting image material for making a micro-motion image. The image collection includes images of a plurality of target environments, and the images of the plurality of target environments have a certain correlation, that is, the image collection includes a plurality of images continuously acquired by the aircraft for the target environment, and the aircraft is Adjacent locations, or maps acquired at adjacent pan/tilt angles There is a partial overlap between the images, and a panoramic view of the target environment, that is, an image of the entire area of the target environment, can be obtained from the plurality of images in the image set. The plurality of images in the image set may be a plurality of photos collected by the aircraft for the target environment, or may be a plurality of frames in the video captured by the aircraft for the target environment, which is not limited in the embodiment of the present invention.

In this embodiment of the present invention, the image set may include two image subsets, and the two image subsets may be a first image set and a second image set. The first image set and the second image set may be a set of images collected by the aircraft for the target environment when the aircraft performs two flight routes. Specifically, the first time the aircraft flies along the first trajectory for the target environment. Image collection; the second image collection is acquired for the target environment when the aircraft flies along the second trajectory for the second time.

In the embodiment of the present invention, the image collected by the aircraft for the target environment includes a foreground portion and a background portion of the target environment. The foreground part of the target environment refers to the target object used to reflect the subject of the shot when the image is taken for the target environment, that is, the moving target corresponding to the dynamic area that needs to be highlighted in the micro-motion map when making the micro-motion map; the background part of the target environment It is the part of the target environment other than the foreground part. Wherein each of the first image sets does not include a foreground portion of the target environment, and at least a portion of the images in the second image set include a foreground portion of the target environment, which may be a partial image of the second image set Including the foreground portion of the target environment, it is also possible that all of the images in the second image set include the foreground portion of the target environment. It should be noted that, in the content in the second image set compared to the image in the first image set, the excess content may be the foreground portion of the target environment; or may be the excess content. For the foreground part of the target environment, part of the content of the foreground that is not part of the target environment may also be the background part of the target environment.

In some possible implementations, the first trajectory is the same as the second trajectory, that is, the flight path when the aircraft acquires the first image set is consistent with the flight path when the aircraft acquires the second image set. Further, the first image set and the second image set are taken by the aircraft using the same shooting strategy during the execution of the flight, and the shooting strategy includes at least one of a shooting position, a pan/tilt shooting angle, and a shooting frequency. The aircraft collects photos or videos through the shooting equipment carried on the gimbal. The collection of images collected by the aircraft can be collected by a shooting device carried by the aircraft or by multiple shooting devices carried by the aircraft. It should be noted that the first trajectory and the second trajectory may also be different, for example, allowing the first trajectory and the second trajectory to have a certain deviation; the aircraft twice The shooting strategy for acquiring images can also be different. The aircraft may fly twice according to different trajectories, but the same image acquisition strategy is used to obtain the first image set and the second image set; or the aircraft may fly twice according to the same trajectory, but adopt different shooting strategies to obtain the first The image set and the second image set; or the aircraft may fly twice according to different trajectories, and the first image set and the second image set are acquired by using different shooting strategies. Although the image in the first image set collected in the above manner has a certain error with the image in the second image set, the image recognition technology can be used to obtain useful information in each image. The aircraft is traversed by the same trajectory twice, and the first image set and the second image set are collected by the same shooting strategy. The embodiment of the present invention will be used for the image collection collected by the preferred scheme. description.

In some possible implementations, the first image set may be a set of images acquired for a background portion of the target environment according to a preset shooting strategy when the aircraft performs flight according to the preset route, such that the first image set includes only the first image set. a background portion of the target environment; the second image set may be a set of images acquired for the target environment according to the preset shooting strategy when the aircraft performs flight again according to the preset route, and the second image set includes a foreground of the target environment Part and background section. For example, suppose the scene captured by the aircraft, that is, the target environment, is that a car is driving slowly on the road, while other objects around the car are stationary, and the generated micro-motion map needs to reflect the picture of the car. The car is the foreground part of the target environment, and the other objects around the car are the background part of the target environment. For the first time, the aircraft photographs the environment of the road and its surroundings when the vehicle is not driving on the road. The obtained image in the first image set does not include the car, and the first image set only includes the background portion of the target environment. The second time the aircraft is photographed while the vehicle is driving on the road, the vehicle and its surroundings are photographed, and at least part of the obtained second image set includes a car, and the second image set includes a background portion of the target environment and a foreground portion.

The first image set may also be a collection of images acquired for the background portion in the target environment according to the photographing instruction input by the user when the aircraft performs the flight task according to the flight control instruction manually input by the user. In the process of collecting the first image set, the aircraft records parameters such as the trajectory of the flight of the aircraft, the position of the captured image, and the shooting angle of the gimbal. The second image set is a set of images acquired by the aircraft for the target environment according to the above-mentioned recording parameters. Specifically, the second image set is when the aircraft performs flight according to the flight trajectory when collecting the first image set, and collects the first image set. Same time cloud A collection of images acquired at a target location for the target environment at the same shooting location.

In the above manner, since the aircraft captures the shooting position of each image in the first image set and the pan/tilt shooting angle and the like, the parameters are the same as the shooting position of the image captured by the aircraft in the second image and the pan/tilt shooting angle. The first image in the second image set may be consistent with the corresponding image capturing area of the corresponding image (eg, the second image) of the first image in the first image set, which may be understood as the first image and the first image. The two images are identical, except that the first image may have more foreground portions of the target environment relative to the second image, and the second image may have more background for a portion of the target environment relative to the first image. The parameters for capturing the first image and the position at which the second image is captured and the panning angle are the same, so the first image corresponds to the second image.

S102. The terminal extracts, from the image set, a foreground portion of the target environment and a position of the foreground portion in a corresponding image.

In the embodiment of the present invention, since the first image set includes only the background portion of the target environment, and the second image set includes the foreground portion of the target environment, the first image in the second image set and the first image may be A second image corresponding to the first image in the set is poor to extract a foreground portion of the target environment in the first image. Specifically, the difference portion of the first image relative to the second image may be extracted by subtracting the corresponding pixels of the first image and the second image; further, the connected portion is filtered by the connected domain, and filtered. The portion of the difference that is not part of the foreground of the target environment is obtained, and the foreground portion of the target environment in the first image is obtained. In the process of performing the difference between the first image and the second image, the position of the foreground portion of the target environment in the first image in the first image may be determined according to the position of the pixel corresponding to the different portion, and further determined The foreground portion of the target environment in the first image is at a corresponding location in the second image.

S103. The terminal splices a background portion of the target environment in at least part of the image of the image set to generate a panoramic view of a background portion of the target environment.

In the embodiment of the present invention, since the first image set includes only the background portion of the target environment, only at least part of the images in the first image set may be spliced to obtain a panoramic view of the background portion of the target environment. Specifically, all the images in the first image set are spliced by using an image tiling method preset by the terminal, and the overlapping regions in the spliced panoramic image are removed to obtain a panoramic view of the background portion of the target environment. Alternatively, the partial image in the first image set is spliced by using an image tiling method preset by the terminal, and the overlapping area in the spliced panoramic image is removed to obtain a target environment. Panorama of the background section. In the above manner, it is not necessary to splicing each image of the first image set, and the efficiency of image stitching can be improved to some extent; although the obtained panorama of the background portion may miss part of the background, as long as the missing portion The background is not limited to the effect of the panoramic view affecting the background portion, but is within the allowable range of the embodiment of the present invention.

In some feasible implementation manners, the pan-tilt shooting attitude when the aircraft captures each image in the first image set may be combined, and at least part of the image in the first image set is stitched by using a preset image stitching method of the terminal, Generate a panorama of the background portion of the target environment. Among them, the pan/tilt shooting posture includes a lateral angle, a longitudinal angle, a deflection angle, and the like. In the above manner, the position of each image in the first image set in the panoramic view of the generated background portion may be determined according to the pan/tilt shooting posture when the aircraft collects each image in the first image set, thereby avoiding Each image in the first image set is parsed to determine the position of each image in the first image set in the panoramic view of the generated background portion, which can further improve the efficiency of image stitching.

S104. The terminal generates a micromotion map.

In the embodiment of the present invention, each frame image in the micro-motion map is synthesized according to the panoramic view of the background portion and the position of the foreground portion of the target environment in the corresponding image. Specifically, for each first image in the second image set, the first image is an image including a foreground portion of the target environment in the second image set: first according to the foreground portion of the target environment in the first image a position in an image, determining a corresponding position of a foreground portion of the target environment in the first image in a second image corresponding to the first image; and then according to a foreground portion of the target environment in the first image in the second image Corresponding to the location, determining a location area in the panoramic view of the background portion corresponding to the foreground portion of the target environment in the first image; finally inserting the foreground portion of the target environment in the first image into a panorama of the background portion, In the location area corresponding to the foreground portion of the target environment in the first image, a first target image is obtained, which is one of the image frames in the micro-motion map to be generated. By analogy, at least part of the foreground image in the image of the foreground portion of the second image set including the foreground portion of the target environment may be inserted into the panorama of the background portion, thereby obtaining each frame in the micro-motion map to be generated. image. Thus, only one region in each frame image of the obtained micro-motion map corresponds to the foreground portion of the target environment, and the background portion of each frame image is a panoramic view of the target environment.

For example, taking the above-mentioned one car to drive slowly on the road, and other objects around the car are all stationary target environments, the first image is an image including the car, assuming that the car is located in the first image. At the crossroads of the road, and the car covers the lower part of the mailbox, and then inserts the car into the panorama of the background part of the target environment according to the position of the car in the first image, which is to insert the car into the panorama. The position corresponding to the intersection, and the position of the car just covers the lower half of the mailbox, and obtains a frame image of the micro-motion map. The position corresponding to the foreground portion of the target environment in the first image in the panorama of the background portion is the position in the second image of the background portion corresponding to the foreground portion of the target environment in the first image. After each frame of the micro-motion image is obtained, each frame image of the micro-motion image is synthesized in a moving image format to obtain a micro-motion image, and the foreground portion of the target environment in the obtained micro-motion image is moved, and the target environment is The background portion remains stationary. The animation format includes but is not limited to a Graphics Interchange Format (GIF).

For example, the aircraft automatically flies along a specific trajectory for the first time, and uses a specific shooting strategy to acquire multiple images of the background portion of the target environment; or when the aircraft performs flight according to the flight control command input by the user, according to the user input The instruction captures multiple images of the background portion of the target environment. If the aircraft performs flight according to the flight control command input by the user, the flight trajectory during the flight of the aircraft, the position of the image acquired by the aircraft, and the pan/tilt shooting angle when the image is acquired are recorded. The plurality of images of the background portion of the collected target environment are the first image set {I _i }, _{i=1 to N} , where i is a positive integer.

The second flight of the aircraft automatically flies along the trajectory of the first flight. At the same time, the foreground part of the target environment moves during the process of capturing images by the aircraft. The aircraft collects the target environment according to parameters such as the first shooting position and the pan-tilt shooting angle. Multiple images. The plurality of images of the collected target environment are the second image set {I' _i }, _i=1～N , and at least part of the images in the second image set include the foreground portion of the target environment. Further, each image in the first image set is arranged in the order of shooting time. Similarly, each image in the second image set is arranged in the order of shooting time, so that the image I _i and image I _'i corresponds, i.e., the image I _i and image I' _i corresponding to the same imaging region. For example, the effect of the generated micro-motion map is that one person walks on one path while the other objects are still. For the first time, the aircraft only photographs the background portion of the target environment, that is, the aircraft photographs the path of the protagonist who is not walking on the trail and the surrounding environment, and obtains a first image set that does not include the foreground portion of the target environment. For the second time, the aircraft photographed the path, the hero, and the surrounding environment when the hero walked on the trail, and obtained a second image set including the foreground portion of the target environment. Among them, the protagonist is the foreground part of the target environment, that is, the dynamic target in the micro-motion map after generating the micro-motion map. Since the shooting position of the first image set and the second image set is the same, the images acquired by the aircraft are basically the same, and the number of images is the same, except that at least part of the images in the second image set include the target. The foreground part of the environment.

After acquiring the first image set and the second image set acquired by the aircraft for the target environment, the foreground portion of the target environment is first extracted by subtracting the image acquired twice by the aircraft. Referring to FIG. 2 together, the i-th image in the second image set is compared with the pixel corresponding to the i-th image in the first image set to obtain a difference map {K _i =I _i -I' _i } , _i=1～N . The region where the pixel is not zero in the difference graph is the foreground portion of the target environment. Because there may be errors between the images, the connected region may be filtered to filter out the small dynamic points in the difference map to obtain the target environment. The foreground portion {S _i }, _{i=1 to N} , that is, the foreground portion of the i-th image in the second image set is S _i . Further, the position of S _i in the ith image may be determined according to the position of the pixel point corresponding to the foreground portion S _i of the ith image in the second image set.

Please refer to FIG. 3 together, and then splicing at least part of the image in the first image set collected by the aircraft according to the preset image splicing algorithm, and removing the overlapping area in the spliced large image to obtain the background of the target environment. Part of the panorama P. Next, insert the foreground portion S _i of the i-th image in the second image set into a corresponding position in the panorama of the background portion, please refer to FIG. 4 together, because the image I′ _i in the second image set Corresponding to the image I _i in the first image set, and the panorama of the background portion of the target environment is stitched according to the image I _i in the first image set, so S _{i is} inserted into the position of I _i in P, The i-th frame image C _{i of the} micro-motion map is obtained. By analogy, each frame image {C _i }, _{i=1 to N of the} micro-motion map can be obtained. Finally, each frame image {C _i }, _{i=1~N of the} micro-motion map is synthesized in GIF format to obtain a micro-motion map.

In the embodiment of the present invention, the image set collected by the aircraft for the target environment is first acquired, and then the foreground portion of the target environment and the position of the foreground portion in the corresponding image are extracted from the image set, and the target environment is generated according to the image included in the image set. The panorama of the background part finally generates the micro-motion map, which can automatically generate the micro-motion map according to the image collection collected by the aircraft, thereby improving the synthesis efficiency of the micro-motion diagram and realizing the automation and intelligence of the micro-motion pattern synthesis.

FIG. 5 is a schematic flowchart diagram of a second embodiment of an image processing method according to an embodiment of the present invention. The image processing method described in this embodiment includes but is not limited to the following steps:

S501. The terminal acquires a set of images acquired by the aircraft along a specific trajectory for the target environment.

In the embodiment of the present invention, the aircraft flies once along a specific trajectory, and at the same time, the foreground portion of the target environment, that is, the photographic subject moves during the process of acquiring images by the aircraft. The image collection may be a collection of images acquired by the aircraft according to a preset shooting strategy for the target environment when the aircraft automatically performs the flight according to the preset specific trajectory, and the image collection may also be performed by the aircraft according to a specific flight trajectory corresponding to the flight control instruction manually input by the user. When flying, the collection of images acquired for the target environment according to the camera instruction input by the user. The collection of images includes a plurality of images and includes a foreground portion of the target environment and a background portion.

S502. The terminal extracts, from the image set, a foreground portion of the target environment and a position of the foreground portion in a corresponding image.

In the embodiment of the present invention, all the images in the image set may include the foreground part of the target environment, or only part of the image may include the foreground part of the target environment. First, an image recognition technology may be used to determine a third image of the foreground portion of the image set including the target environment, and then for each third image, a foreground portion of the target environment in the third image is extracted, and corresponding to the foreground portion The position of the pixel determines the position of the foreground portion in the third image.

S503. The terminal splices a background portion of the target environment in at least part of the image of the image set to generate a panoramic view of a background portion of the target environment.

In the embodiment of the present invention, since at least part of the image of the first image set includes the foreground portion of the target environment, the foreground portion of the target environment in each image of the foreground portion of the image set including the foreground environment may be removed first. That is, the foreground part of the target environment is extracted by the mapping technique in the image processing technology, and an image including only the background portion of the target environment is obtained, but at least part of the image in the background portion of the target environment has the foreground portion of the target environment removed. The resulting blank area; then splicing the image of the background portion of each target environment to obtain a panoramic view of the background portion of the target environment including the blank area. It should be noted that each image of the foreground portion of the target environment is not included in the image set, that is, an image including only the background portion of the target environment.

In some feasible implementation manners, the background portion of the target environment in the plurality of images included in the image set may be used to perform the blank region generated after removing the foreground portion of the target environment in the image of the background portion of each target environment. Fill to get the background portion of the target environment that does not include white space Image. The images of the background portions of the target environments that do not include the blank areas are then spliced to obtain a panoramic view of the background portion of the target environment that does not include the blank areas. It should be noted that each image of the foreground portion of the target environment is not included in the image collection, that is, an image of a background portion of the target environment that does not include the blank region.

In some feasible implementation manners, after obtaining the panoramic view of the background portion of the target environment including the blank area, the partial blank area of the panoramic part of the background part of the target environment including the blank area may be filled to obtain a micro-motion map. A background panorama of each frame image. Specifically, the background portion of the target environment in the plurality of images included in the image set is used, and the blank in the background portion of the background portion of the target environment including the blank region is excluded from the position region corresponding to the ith image in the image set A background fill is performed on other areas outside the area to obtain a background panorama of the i-th frame image of the micro-motion map.

In some feasible implementation manners, the foreground portion of the target environment in each image of the foreground portion of the image set including the target environment may not be removed, and the target environment in the plurality of images included in the image set may be directly utilized. The background portion covers the foreground portion of the target environment in each image, and an image of the background portion of the target environment that does not include the blank region is obtained. The images of the background portions of the target environments that do not include the blank areas are then spliced to obtain a panoramic view of the background portion of the target environment that does not include the blank areas.

For example, for example, the effect of the micro-motion we need to generate is for a person to walk on a small road while other objects are still. Since at least part of the images in the collected image set includes the protagonist, we can remove the region corresponding to the hero in the image including the protagonist, and obtain an image including only the background portion including the blank region, and then include the blank. The image of the area including only the background portion and other images not including the hero are spliced into a panorama including a background portion having a blank area. Further, the panorama of the background portion including the blank area may be filled with the background portion of the plurality of images in the image set to obtain a panoramic view of the background portion not including the blank area. Further, a partial blank area in the panoramic view including the background portion of the blank area may be filled, for example, the image i includes a character, and the image of the background portion including the blank area is included in the area corresponding to the image i. The area other than the blank area (ie, the area corresponding to the hero in the image i) is filled to obtain a background panorama of the i-th frame image of the micro-motion map.

S504. The terminal generates a micromotion map.

In the embodiment of the present invention, the foreground portions according to the target environment are respectively in the respective images of the image collection. The position of the foreground environment of the target environment is inserted into the panorama of the background portion, and the panoramic view of the background portion of each foreground portion inserted into the target environment is a frame image in the micro-motion map. Specifically, the foreground portion of the target environment in the ith picture in the image set is inserted into a blank area in the location area corresponding to the i-th picture in the panoramic view of the background portion of the target environment including the blank area, An ith frame image including a blank area of the fretting map, and then performing background filling on the blank area of the ith frame image including the blank area by using a background portion of at least two images in the image set to obtain a micro-motion map The i-th frame image. After each frame image of the micro-motion map is obtained, each frame image of the micro-motion map is synthesized in an animation format to obtain a micro-motion map. The animation format includes but is not limited to the GIF format.

In some feasible implementation manners, if the background panorama of the i-th frame image of the micro-motion map is obtained, the foreground portion of the target environment in the i-th picture in the image set is inserted into the ith frame of the micro-motion map. The blank area in the background panorama of the image gives the image of the ith frame of the micromotion map. If the obtained panorama of the background portion of the target environment that does not include the blank area is obtained, the foreground portion of the target environment in the i-th picture in the image set is inserted into the panorama of the background portion of the target environment not including the blank area The position corresponding to the i-th picture in the figure is the image of the i-th frame of the micro-motion picture.

In some possible implementations, before extracting the foreground portion of the target environment and the location of the foreground portion in the corresponding image from the image set, the terminal may determine the foreground portion of the target environment according to the user's operation. When acquiring an image for a target environment, there may be multiple moving objects, and only one of the moving objects needs to be highlighted in the generated micro-motion map. At this time, the terminal may use the image collection collected by the aircraft for the target environment. The image recognition technology recognizes all moving objects whose positions are changed in the image set, and only uses the selected target moving object of the user as the foreground part of the target environment, and statically processes the other moving objects as the background part of the target environment. In the panorama of the background portion of the generated target environment, a plurality of regions of other moving objects may be included, and at this time, all regions or partial regions of other moving objects may be performed by using the background portion of the plurality of images in the image collection. Fill to make the panorama of the background section more beautiful.

In some feasible implementation manners, a plurality of moving target objects may also be allowed in the generated micro-motion map, and the plurality of moving target objects are all used as foreground parts of the target environment. The plurality of moving targets may be processed as a foreground part of the target environment, and the plurality of moving targets may be processed as different foreground parts of the target environment. For the specific processing, refer to the method in the foregoing method embodiment. Description, no longer repeat here.

For example, the scene we shot was a person walking on the road, while a car next to it slowly followed the hero, and other objects were still. Then, after acquiring the image collection acquired for the shooting scene, analyzing the image in the image collection, it can be determined that the person and the car are changing positions. If the user's choice is received to make people move, and the car does not move, only the person is regarded as the foreground part of the target environment, and the car is treated as the background part of the target environment. A plurality of vehicles appear in the panorama of the background portion of the target environment thus spliced, but the plurality of vehicles are the same vehicle, and the terminal can arbitrarily select a vehicle or a part of the vehicle corresponding to the area, and according to the image collection The background portion of the plurality of images in the background fills the background of other vehicles, and only one or a few positions of the stationary vehicle are present in the generated micro-motion map, and only the person is walking. If the user's choice is received to both move and leave the car, both the person and the car are treated as foreground parts of the target environment.

In some feasible implementation manners, the image processing method and apparatus provided by the embodiments of the present invention may be applied to an intelligent terminal installed with an APP (application software), and the smart terminal may be selected as a smart phone, a tablet computer, or the like. In order to obtain a real-time video stream of the drone aerial photography, the smart terminal is communicably connected to at least one of a drone, a pan-tilt mounted on the drone, and an imaging device mounted on the pan-tilt. In a specific implementation, the drone is communicatively coupled to the smart terminal. Specifically, the image including the target object captured by the imaging device can be transmitted back to the smart terminal through the wireless link.

In some possible implementations, the APP is provided with a micro-motion shooting mode in which the drone has a flight path and/or shooting strategy that can be selected by the user, or has a fixed flight path and/or shooting. The strategy, or the flight trajectory and/or shooting strategy input by the user may be received through the APP. When the user selects the mode, the drone performs flight according to the flight trajectory in the mode, and/or controls the imaging device to perform imaging according to the shooting strategy in the mode.

FIG. 6 is a schematic structural diagram of an image processing apparatus according to an embodiment of the present invention. Ben The image processing apparatus described in the embodiment includes:

The obtaining module 601 is configured to acquire a collection of images acquired by the aircraft for the target environment when flying along a specific trajectory, the image collection comprising a plurality of images.

The extracting module 602 is configured to extract a foreground portion of the target environment and a position of the foreground portion in the corresponding image from the image set.

The splicing module 603 is configured to splicing a background portion of the target environment in at least part of the image of the image set to generate a panoramic view of a background portion of the target environment.

The synthesizing module 604 is configured to generate a micro-motion image, wherein each frame image in the micro-motion map is synthesized according to a panoramic view of the background portion and a position of the foreground portion in a corresponding image.

In the embodiment of the present invention, the multiple images in the image set are multiple photos collected by the aircraft, or multiple frames in the video captured by the aircraft.

In some possible implementations, the image set includes a first image set and a second image set; the first image set is a set of images acquired for the target environment when the aircraft flies along the first trajectory, The second set of images is a set of images acquired for the target environment when the aircraft is flying along a second trajectory; the first set of images does not include the foreground portion, and the second set of images includes the foreground portion.

In some possible implementations, the first trajectory and the second trajectory are the same. The first set of images and the second set of images are taken by the aircraft using the same shooting strategy in flight. The shooting strategy includes at least one of the following: a shooting position, a pan/tilt shooting angle, and a shooting frequency.

The extracting module 602 is configured to perform a difference between the first image in the second image set and the corresponding image in the first image set to obtain a foreground of the first image. a portion and a location of the foreground portion in the first image.

The splicing module 603 is specifically configured to splicing at least part of the images in the first image set to generate a panoramic view of a background portion of the target environment.

The synthesizing module 604 is specifically configured to insert a foreground portion of the first image into a panoramic view of the background portion according to a position of a foreground portion of the first image in the first image, to obtain a first a target image, the first target image being a frame image in the fine motion map.

In some possible implementations, the set of images is for the aircraft to fly along the particular trajectory A collection of images acquired for the target environment is taken once.

The splicing module 603 is further configured to:

Removing a foreground portion of the target environment in each image of the image collection to obtain an image of a background portion of the target environment;

A splicing process is performed on the images of the respective background portions to obtain a panoramic view of the background portion of the target environment.

In some possible implementations, the apparatus further includes: a determining module 605, configured to determine a foreground portion of the target environment according to a user operation.

The splicing module 603 is further configured to: use a background portion of the target environment in the plurality of images included in the image set to remove a blank generated by removing a foreground portion of the target environment from an image of each of the background portions The area is filled.

The synthesizing module 604 is further configured to insert the foreground portion into the panoramic view of the background portion according to a position of each of the foreground portions in each image of the image set, each insertion A panoramic view of the background portion of the foreground portion is a frame of image in the micro-motion map.

It can be understood that the functions of the functional modules of the image processing apparatus provided by the embodiments of the present invention may be specifically implemented according to the first embodiment and the second embodiment of the image processing method, and the specific implementation process may refer to The related description of the foregoing method embodiments is not described herein again.

In the embodiment of the present invention, the acquiring module 601 first acquires an image set collected by the aircraft for the target environment, and then the extracting module 602 extracts a foreground portion of the target environment and a position of the foreground portion in the corresponding image from the image set, and the splicing module 603 is configured according to the splicing module 603. The image set includes an image to generate a panoramic view of the background portion of the target environment, and finally the synthesis module 604 generates a micro-motion map, which can automatically generate a micro-motion map according to the image collection collected by the aircraft, thereby improving the synthesis efficiency of the micro-motion image and implementing the micro-motion Automation and intelligence of graph synthesis.

FIG. 7 is a schematic structural diagram of an electronic device according to an embodiment of the present invention. The electronic device described in this embodiment includes: a processor 701, a communication interface 702, and a memory 703. The processor 701, the communication interface 702, and the memory 703 can be connected by using a bus or other means. The embodiment of the present application is exemplified by a bus connection.

The processor 701 can be a central processing unit (CPU), network processing Network processor (NP), graphics processing unit (GPU), or a combination of CPU, GPU, and NP. The processor 701 can also be a core for implementing communication identity binding in a multi-core CPU, a multi-core GPU, or a multi-core NP.

The processor 701 described above may be a hardware chip. The hardware chip may be an application-specific integrated circuit (ASIC), a programmable logic device (PLD), or a combination thereof. The PLD may be a complex programmable logic device (CPLD), a field-programmable gate array (FPGA), a general array logic (GAL), or any combination thereof.

The above communication interface 702 can be used for transceiving information or signaling interactions, as well as receiving and transmitting signals. The memory 703 may mainly include a storage program area and a storage data area, wherein the storage program area may store an operating system, a storage program required for at least one function (such as a text storage function, a location storage function, etc.); the storage data area may be stored according to The data created by the use of the device (such as image data, text data), etc., and may include an application storage program or the like. Further, the memory 703 may include a high speed random access memory, and may also include a nonvolatile memory such as at least one magnetic disk storage device, flash memory device, or other volatile solid state storage device.

The above memory 703 is also used to store program instructions. When the processor 701 is a processor other than the hardware chip, the program instructions stored in the memory 703 can be invoked to implement the image processing method as shown in the embodiment of the present application.

Specifically, the processor 701 calls the program instructions stored in the memory 703 to perform the following steps:

The method performed by the processor 701 in the embodiment of the present application is described from the perspective of the processor 701. It can be understood that the processor 701 in the embodiment of the present application needs to perform other hardware structures in order to execute the foregoing method. Hehe. The specific implementation process is not described and limited in detail in the embodiments of the present application.

In some possible implementations, the processor 701 is specifically configured to: compare a first image in the second image set with a corresponding image in the first image set, to obtain a foreground portion of the first image and a location of the foreground portion in the first image.

The processor 701 is specifically configured to splicing at least part of the images in the first image set to generate a panoramic view of a background portion of the target environment.

The processor 701 is specifically configured to insert a foreground portion of the first image into a panoramic view of the background portion according to a position of a foreground portion of the first image in the first image to obtain a first target. An image, the first target image being a frame image in the micro-motion map.

In some possible implementations, the set of images is a collection of images acquired by the aircraft for the target environment along the particular trajectory.

The processor 701 is further configured to:

The processor 701 is further configured to determine a foreground portion of the target environment according to a user operation.

The processor 701 is further configured to utilize the target in multiple images included in the image set The background portion of the environment fills a blank area created after removing the foreground portion of the target environment in the image of each of the background portions.

The processor 701 is further configured to insert the foreground portion into a panoramic view of the background portion according to a position of each of the foreground portions in each image of the image set, and insert the A panoramic view of the background portion of the foreground portion is a frame of image in the micro-motion map.

In a specific implementation, the processor 701, the communication device 702, and the memory 703, which are described in the embodiments of the present invention, may be implemented in the first embodiment and the second embodiment of the image processing method provided by the embodiment of the present invention. The implementation manner described in the embodiment of the image processing apparatus provided by the embodiment of the present invention may also be implemented, and details are not described herein again.

The present invention also provides a computer readable storage medium having instructions stored therein that, when run on a computer, cause the computer to perform the image processing method described in the above method embodiments.

The present invention also provides a computer program product comprising instructions which, when run on a computer, cause the computer to perform the image processing method described in the above method embodiments.

It should be noted that, for the foregoing various method embodiments, for the sake of simple description, they are all expressed as a series of action combinations, but those skilled in the art should understand that the present invention is not limited by the described action sequence. Because certain steps may be performed in other sequences or concurrently in accordance with the present invention. In the following, those skilled in the art should also understand that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required by the present invention.

A person skilled in the art can understand that all or part of the steps of the foregoing embodiments can be completed by a program to instruct related hardware. The program can be stored in a computer readable storage medium, and the storage medium can include: Flash disk, Read-Only Memory (ROM), Random Access Memory (RAM), disk or optical disk.

The image processing method, device and electronic device provided by the embodiments of the present invention are described in detail above. The principles and embodiments of the present invention are described in the following. The description of the above embodiments is only for helping. The method of the present invention and its core idea are understood; at the same time, for those skilled in the art, according to the idea of the present invention, there are changes in the specific embodiments and application scopes. It should be understood that the invention is limited.

Claims

An image processing method, the method comprising:

Acquiring a collection of images acquired for the target environment when the aircraft is flying along a particular trajectory, the collection of images comprising a plurality of images;

Extracting a foreground portion of the target environment and a position of the foreground portion in the corresponding image from the image set;

Splicing a background portion of the target environment in at least a portion of the image set to generate a panoramic view of a background portion of the target environment;

A micro-motion map is generated, wherein each frame image in the micro-motion map is synthesized according to a panoramic view of the background portion and a position of the foreground portion in a corresponding image, respectively.
The method of claim 1 wherein the plurality of images in the set of images are multiple photos captured by the aircraft or multi-frame images in the video captured by the aircraft.
The method of claim 1 wherein the set of images comprises a first set of images and a second set of images;

The first set of images is a set of images acquired for the target environment when the aircraft flies along a first trajectory, and the second set of images is an image acquired for the target environment when the aircraft flies along a second trajectory set;

The first set of images does not include the foreground portion, and the second set of images includes the foreground portion.
The method of claim 3 wherein said first trajectory and said second trajectory are the same.
The method of claim 4 wherein said first set of images and said second set of images are taken by said aircraft in flight using the same shooting strategy.
The method of claim 5 wherein said photographing strategy comprises at least the following One: shooting position, pan/tilt shooting angle, shooting frequency.
The method according to claim 3, wherein the extracting the foreground portion of the target environment and the location of the foreground portion in the corresponding image from the image set comprises:

Placing a first image in the second image set with a corresponding image of the first image in the first image set to obtain a foreground portion of the first image and the foreground portion in the The position in the first image.
The method according to claim 3, wherein the splicing the background portion of the target environment in at least part of the image of the image set to generate a panoramic view of the background portion of the target environment comprises:

Splicing at least a portion of the images in the first set of images to generate a panoramic view of a background portion of the target environment.
The method according to claim 7, wherein said generating a micromotion map comprises:

Inserting a foreground portion of the first image into a panoramic view of the background portion according to a position of a foreground portion of the first image in the first image to obtain a first target image, the first target image Is a frame of image in the micro-motion map.
The method of claim 1 wherein said set of images is a collection of images acquired by said aircraft along said particular trajectory for said target environment.
The method according to claim 1 or 10, wherein the framing the background portion of the target environment in at least part of the image of the image set to generate a panoramic view of the background portion of the target environment, including :

Removing a foreground portion of the target environment in each image of the image collection to obtain an image of a background portion of the target environment;

Splicing processing images of each of the background portions to obtain a background portion of the target environment Panorama.
The method according to claim 11, wherein the method further comprises: extracting a foreground portion of the target environment and a position of the foreground portion in a corresponding image from the image set, the method further comprising:

The foreground portion of the target environment is determined based on user operations.
The method according to claim 11, wherein the splicing processing of the images of the respective background portions comprises:

Filling a blank area generated after removing the foreground portion of the target environment in the image of each of the background portions by using a background portion of the target environment in the plurality of images included in the image set.
The method according to claim 11, wherein said generating a micromotion map comprises:

Inserting the foreground portions into a panoramic view of the background portion according to positions of the foreground portions in respective images of the image set, each of which is inserted into the background portion of the foreground portion The panorama is an image of one frame in the micro-motion map.
An image processing apparatus, characterized in that the apparatus comprises:

An acquisition module, configured to acquire a collection of images acquired by the aircraft for a target environment when flying along a specific trajectory, the image collection comprising a plurality of images;

An extracting module, configured to extract, from the image set, a foreground portion of the target environment and a position of the foreground portion in a corresponding image;

a splicing module, configured to splicing a background portion of the target environment in at least part of the image of the image set to generate a panoramic view of a background portion of the target environment;

And a synthesizing module, configured to generate a micro-motion image, wherein each frame image in the micro-motion map is synthesized according to a panoramic view of the background portion and a position of the foreground portion in a corresponding image.
The apparatus of claim 15 wherein said plurality of images in said set of images Like multiple photos captured for the aircraft, or multiple frames of images in the video captured by the aircraft.
The apparatus of claim 15 wherein said set of images comprises a first set of images and a second set of images;

The first set of images is a set of images acquired for the target environment when the aircraft flies along a first trajectory, and the second set of images is an image acquired for the target environment when the aircraft flies along a second trajectory set;

The first set of images does not include the foreground portion, and the second set of images includes the foreground portion.
The apparatus of claim 17 wherein said first trajectory and said second trajectory are the same.
The apparatus of claim 18 wherein said first set of images and said second set of images are taken by said aircraft in flight using the same shooting strategy.
The apparatus according to claim 19, wherein the shooting strategy comprises at least one of the following: a shooting position, a pan/tilt shooting angle, and a shooting frequency.
The device of claim 17 wherein:

The extracting module is configured to perform a difference between a first image in the second image set and a corresponding image in the first image set to obtain a foreground portion of the first image. And a location of the foreground portion in the first image.
The device of claim 17 wherein:

The splicing module is specifically configured to splicing at least part of the images in the first image set to generate a panoramic view of a background portion of the target environment.
The device according to claim 21, wherein

The synthesizing module is configured to insert a foreground portion of the first image into a panoramic view of the background portion according to a position of a foreground portion of the first image in the first image to obtain a first target An image, the first target image being a frame image in the micro-motion map.
The apparatus of claim 15 wherein said set of images is a collection of images acquired by said aircraft along said particular trajectory for said target environment.
The device according to claim 15 or 24, wherein the splicing module is further configured to:

Removing a foreground portion of the target environment in each image of the image collection to obtain an image of a background portion of the target environment;

A splicing process is performed on the images of the respective background portions to obtain a panoramic view of the background portion of the target environment.
The device of claim 25, wherein the device further comprises:

A determining module is configured to determine a foreground portion of the target environment according to a user operation.
The device according to claim 25, wherein

The splicing module is further configured to use a background portion of the target environment in the plurality of images included in the image set to remove a blank region generated by removing a foreground portion of the target environment from an image of each of the background portions Fill it up.
The device according to claim 25, wherein

The synthesizing module is further configured to insert the foreground portion into a panoramic view of the background portion according to a position of each of the foreground portions in each image of the image set, and insert the A panoramic view of the background portion of the foreground portion is a frame of image in the micro-motion map.
An electronic device, comprising: a processor and a memory,

The memory is configured to store program instructions;

The processor is configured to execute the program instructions stored by the memory, when the program instructions are executed, the processor is configured to:

Acquiring a collection of images acquired for the target environment when the aircraft is flying along a particular trajectory, the collection of images comprising a plurality of images;

Extracting a foreground portion of the target environment and a position of the foreground portion in the corresponding image from the image set;

Splicing a background portion of the target environment in at least a portion of the image set to generate a panoramic view of a background portion of the target environment;

A micro-motion map is generated, wherein each frame image in the micro-motion map is synthesized according to a panoramic view of the background portion and a position of the foreground portion in a corresponding image, respectively.
The electronic device according to claim 29, wherein the plurality of images in the image set are a plurality of photos collected by the aircraft or a multi-frame image in a video captured by the aircraft.
The electronic device of claim 29, wherein the set of images comprises a first set of images and a second set of images;

The first set of images is a set of images acquired for the target environment when the aircraft flies along a first trajectory, and the second set of images is an image acquired for the target environment when the aircraft flies along a second trajectory set;

The first set of images does not include the foreground portion, and the second set of images includes the foreground portion.
The electronic device of claim 31, wherein the first trajectory and the second trajectory are the same.
38. The electronic device of claim 32, wherein the first set of images and the second set of images are taken by the aircraft using the same shooting strategy in flight.
The electronic device according to claim 33, wherein the shooting strategy comprises at least one of the following: a shooting position, a pan/tilt shooting angle, and a shooting frequency.
The electronic device according to claim 31, wherein the processor extracts, from the image set, a foreground portion of the target environment and a position of the foreground portion in a corresponding image, specifically for :

Placing a first image in the second image set with a corresponding image of the first image in the first image set to obtain a foreground portion of the first image and the foreground portion in the The position in the first image.
The electronic device according to claim 31, wherein said processor splices a background portion of said target environment in at least part of said image of said image set to generate a panorama of said background portion of said target environment Specifically for:

Splicing at least a portion of the images in the first set of images to generate a panoramic view of a background portion of the target environment.
The electronic device according to claim 35, wherein when the processor generates the micro-motion map, the method is specifically configured to:

Inserting a foreground portion of the first image into a panoramic view of the background portion according to a position of a foreground portion of the first image in the first image to obtain a first target image, the first target image Is a frame of image in the micro-motion map.
The electronic device of claim 29, wherein the set of images is a collection of images acquired by the aircraft for the target environment along the particular trajectory.
The electronic device according to claim 29 or claim 38, wherein the processor splices a background portion of the target environment in at least part of the image of the image set to generate a panorama of a background portion of the target environment When used in the figure, it is specifically used to:

Removing a foreground portion of the target environment in each image of the image collection to obtain an image of a background portion of the target environment;

A splicing process is performed on the images of the respective background portions to obtain a panoramic view of the background portion of the target environment.
The electronic device according to claim 39, wherein said processor extracts, from said image set, a foreground portion of said target environment and said foreground portion respectively before a position in said corresponding image, :

The foreground portion of the target environment is determined based on user operations.
The electronic device according to claim 39, wherein when the processor performs splicing processing on the images of the respective background portions, the processor is further configured to:

Filling a blank area generated after removing the foreground portion of the target environment in the image of each of the background portions by using a background portion of the target environment in the plurality of images included in the image set.
The electronic device according to claim 39, wherein when the processor generates the micro-motion map, the method is specifically configured to:

Inserting the foreground portions into a panoramic view of the background portion according to positions of the foreground portions in respective images of the image set, each of which is inserted into the background portion of the foreground portion The panorama is an image of one frame in the micro-motion map.