WO2022170824A1 - 图像拼接的处理方法、装置、电子系统和设备、可读介质 - Google Patents

图像拼接的处理方法、装置、电子系统和设备、可读介质 Download PDF

Info

Publication number
WO2022170824A1
WO2022170824A1 PCT/CN2021/134126 CN2021134126W WO2022170824A1 WO 2022170824 A1 WO2022170824 A1 WO 2022170824A1 CN 2021134126 W CN2021134126 W CN 2021134126W WO 2022170824 A1 WO2022170824 A1 WO 2022170824A1
Authority
WO
WIPO (PCT)
Prior art keywords
image
energy
mask
map
overlapping
Prior art date
Application number
PCT/CN2021/134126
Other languages
English (en)
French (fr)
Inventor
刘伟舟
胡晨
周舒畅
Original Assignee
北京迈格威科技有限公司
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by 北京迈格威科技有限公司 filed Critical 北京迈格威科技有限公司
Publication of WO2022170824A1 publication Critical patent/WO2022170824A1/zh

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T3/00—Geometric image transformations in the plane of the image
    • G06T3/40—Scaling of whole images or parts thereof, e.g. expanding or contracting
    • G06T3/4038—Image mosaicing, e.g. composing plane images from plane sub-images
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T5/00—Image enhancement or restoration
    • G06T5/50—Image enhancement or restoration using two or more images, e.g. averaging or subtraction
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/10—Segmentation; Edge detection
    • G06T7/11—Region-based segmentation
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/10—Segmentation; Edge detection
    • G06T7/13—Edge detection
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/20—Special algorithmic details
    • G06T2207/20081—Training; Learning
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/20—Special algorithmic details
    • G06T2207/20212—Image combination
    • G06T2207/20221—Image fusion; Image merging

Definitions

  • the present application relates to the technical field of image processing, and in particular, to a processing method, apparatus, electronic system, electronic device, computer-readable storage medium, and computer program product for image stitching.
  • the task of image stitching is to combine multiple images with overlapping fields of view to obtain high-resolution pictures with a large field of view.
  • the effect of the image content in the overlapping area on the stitching caused by the misalignment can be alleviated to a certain extent through the seam search.
  • the current mainstream seam search method is based on the graph cut algorithm. This algorithm searches for the minimum energy path of the local area of the image through the energy function defined by the difference of image pixel values. However, the energy of the local area is minimized. For Important areas with insignificant differences (areas with saliency targets, etc.) may also belong to the position where the energy is minimized, and are then determined as seams, which affects the effect of subsequent image stitching.
  • the present application provides a processing method, apparatus, electronic system, electronic device, computer-readable storage medium, and computer program product for image stitching, so as to improve the effect of image stitching.
  • Some embodiments of the present application provide an image stitching processing method.
  • the method can be applied to an electronic device.
  • the method can include: acquiring first images corresponding to overlapping regions of the first image and the second image to be stitched, respectively. overlapping images and second overlapping images; determining fusion energy maps corresponding to the first overlapping images and the second overlapping images; wherein the fusion energy maps correspond to the first overlapping images and the second overlapping images.
  • the pixel energy value of the pixel at the target position in the image may be higher than the pixel energy value of other positions; the target position may include the edge position and the position corresponding to the salient area; determine the first edge to the second edge of the fusion energy map.
  • the embodiment of the present application provides a possible implementation manner in some embodiments of the present application, wherein the fused energy map corresponding to the first overlapping image and the second overlapping image is determined.
  • the steps include: determining a first edge energy map corresponding to the first overlapping image and a second edge energy map corresponding to the second overlapping image; determining a first saliency energy map corresponding to the first overlapping image and the second edge energy map corresponding to the first overlapping image.
  • the embodiments of the present application provide a possible implementation of some embodiments of the present application, wherein according to the saliency energy combination map, the first edge energy map and the second
  • the step of determining the fusion energy map corresponding to the first overlapping image and the second overlapping image by the edge energy map includes: determining the first edge energy map according to the splicing mode of the first image and the second image The corresponding enhanced energy area; wherein, the splicing method may include left-right splicing or up-down splicing; multiply each pixel in the enhanced energy area by the first set value to obtain a first edge enhanced energy map; wherein, the The first set value may be greater than 1; according to the saliency energy combination map, the first edge enhancement energy map and the second edge energy map Fusion energy diagram.
  • the embodiment of the present application provides a possible implementation of some embodiments of the present application, wherein, when the stitching method is left-right stitching and the first image is located on the left side, the A part of the region near the left side of the first edge energy map is determined as an enhanced energy region corresponding to the first edge energy map.
  • the embodiments of the present application provide a possible implementation of some embodiments of the present application, wherein, according to the saliency energy combination map, the first edge-enhanced energy map, and the second
  • the step of determining the fusion energy map corresponding to the first overlapping image and the second overlapping image by the edge energy map includes: combining the saliency energy map, the first edge enhancement energy map and the second edge energy map Weighted summation is performed on each corresponding pixel in to obtain a fusion energy map corresponding to the first overlapping image and the second overlapping image.
  • the embodiments of the present application provide a possible implementation of some embodiments of the present application, wherein, according to the saliency energy combination map, the first edge-enhanced energy map, and the first edge-enhanced energy map
  • the step of determining the fusion energy map corresponding to the first overlapped image and the second overlapped image by the two-edge energy map includes: if the first overlapped image contains a black edge region, acquiring the black edge region of the first overlapped image The corresponding black border area energy map; if the second overlapping image contains a black border area, obtain the black border area energy map corresponding to the black border area of the second overlapping image; the obtained black border area energy map
  • Each pixel in is multiplied by a second set value to obtain an energy enhancement map in the black border area, where the second set value can be greater than 1; , and each corresponding pixel in the second edge energy map and the black border region energy enhancement map is weighted and summed to obtain a fusion energy map corresponding to
  • the embodiments of the present application provide a possible implementation of some embodiments of the present application, wherein a salient energy is obtained by combining the first saliency energy map and the second saliency energy map
  • the step of merging the maps includes: for each corresponding pixel in the first saliency energy map and the second saliency energy map, obtaining a new pixel value by directly summing pixel values or weighted summing of pixel values, and obtaining The new pixel value of is the pixel value of the saliency energy combination map.
  • the embodiments of the present application provide a possible implementation of some embodiments of the present application, wherein a salient energy is obtained by combining the first saliency energy map and the second saliency energy map
  • an image stitching processing method the method can be applied to an electronic device, and the method may include: acquiring the first image to be stitched and the second image respectively corresponding to the overlapping area of the second image. an overlapping image and a second overlapping image; inputting the first overlapping image and the second overlapping image into a trained image segmentation network to obtain an image segmentation result output by the image segmentation network; wherein, the image segmentation network
  • the image sample pair in the training process may correspond to a mask truth value pair, and the mask truth value pair may be determined based on the minimum energy path corresponding to the fusion energy map corresponding to the image sample pair, wherein the fusion energy
  • the pixel energy value of the pixel at the target position corresponding to the image sample pair may be higher than the pixel energy value of the pixel at other positions; the target position may include edge positions and positions corresponding to salient regions; based on the image segmentation As a result, a first mask corresponding to the first overlapping image and a second
  • the embodiment of the present application provides a possible implementation manner of the other embodiments of the present application, wherein the training process of the image segmentation network includes: acquiring an image sample pair and the image The corresponding mask truth value pairs of the sample pairs; input the image sample pairs into the initial image segmentation network to obtain the mask prediction pairs corresponding to the image sample pairs; according to the corresponding mask truth value pairs of the image sample pairs Calculate the training loss value with the mask prediction pair; according to the training loss value, determine whether the training stop condition is met; if not, adjust the initial network parameters of the image segmentation according to the training loss value, and continue training; if so, stop the training , taking the current image segmentation initial network as the image segmentation network.
  • the embodiment of the present application provides a possible implementation manner of the other embodiments of the present application, wherein the step of acquiring the image sample pair includes: performing target processing on the initial image sample, Obtaining an image transformation sample corresponding to the initial image sample; wherein, the target processing may include at least one of the following: rotation, translation, and affine transformation; adding black borders to the initial image sample and/or the image transformation sample region to obtain the image sample pair.
  • the embodiment of the present application provides a possible implementation manner of the other embodiments of the present application, wherein the step of acquiring the mask truth value pair corresponding to the image sample pair includes: by The edge detection method obtains the first edge energy map corresponding to the initial image sample in the pair of image samples and the second edge energy map corresponding to the image transformation sample; obtains the first edge energy map corresponding to the initial image sample through a saliency target detection model.
  • a saliency energy map and a second saliency energy map corresponding to the image transformation sample combine the first saliency energy map and the second saliency energy map to obtain a saliency energy combination map; according to the saliency energy combination map, the The first edge energy map and the second edge energy map determine the fusion energy map corresponding to the image sample pair, and determine the mask truth value pair corresponding to the image sample pair based on the fusion energy map and the minimum energy path.
  • the embodiment of the present application provides a possible implementation manner of the other embodiments of the present application, wherein the determining the first overlapping image corresponding to the first overlapping image based on the image segmentation result
  • the step of a mask and a second mask corresponding to the second overlapping image includes: if the image segmentation result is the first mask corresponding to the first overlapping image, applying the same size as the first mask And the image whose pixel value is all 1 is compared with the first mask to obtain a second mask corresponding to the second overlapping image.
  • the embodiment of the present application provides a possible implementation manner of the other embodiments of the present application, wherein the determining the first overlapping image corresponding to the first overlapping image based on the image segmentation result
  • the step of a mask and a second mask corresponding to the second overlapping image includes: if the image segmentation result is a second mask corresponding to the second overlapping image, applying the same size as the second mask And the image whose pixel value is all 1 is compared with the second mask to obtain the first mask corresponding to the first overlapping image.
  • an image stitching processing apparatus the apparatus can be applied to electronic equipment, and the apparatus can include: an overlapping image acquisition module, and the overlapping image acquisition module can be configured to acquire an image to be acquired. A first overlapping image and a second overlapping image corresponding to the overlapping regions of the spliced first image and the second image respectively; a fusion image determination module, the fusion image determination module may be configured to determine the first overlapping image and the second overlapping image; The fusion energy map corresponding to the second overlapping image; wherein, the pixel energy value of the pixel corresponding to the pixel at the target position in the first overlapping image and the second overlapping image in the fusion energy map may be higher than that of the other The pixel energy value of the pixel at the position, the target position may include the edge position and the position corresponding to the salient area; the energy path determination module, the energy path determination module may be configured to determine the first edge of the fusion energy map.
  • the sum of the pixel values of the pixels may all be 1;
  • a first stitching module the first stitching module may be configured to stitch the first image and the first image based on the first mask and the second mask. The second image, the stitched image is obtained.
  • an image stitching processing apparatus the apparatus can be applied to electronic equipment, the apparatus can include: an image acquisition module, and the image acquisition module can be configured to acquire images to be stitched. a first overlapping image and a second overlapping image respectively corresponding to the overlapping regions of the first image and the second image; an image segmentation module, the image segmentation module can be configured to be used to separate the first overlapping image and the second overlapping image
  • the image is input into the trained image segmentation network, and the image segmentation result output by the image segmentation network is obtained; wherein, the image sample pairs in the training process of the image segmentation network may correspond to a mask truth value pair, and the mask truth value The pair may be determined based on the minimum energy path corresponding to the fusion energy map corresponding to the image sample pair, wherein the pixel energy value of the pixel at the target position corresponding to the image sample pair in the fusion energy map may be higher than other a pixel energy value of a pixel at a position, the target position including an edge
  • the electronic system may include: an image acquisition device, a processing device, and a storage device; the image acquisition device may be configured to acquire an image; A computer program may be stored that, when executed by the processing device, executes the processing method for image stitching as described in any one of the preceding embodiments.
  • the electronic device may include a processor and a memory, the processor may be configured to perform the image stitching according to any one of the preceding embodiments at runtime processing method.
  • Still other embodiments of the present application provide a computer program product, and the computer program product may include a computer program that, when executed by a processor, implements the image stitching described in any one of the preceding embodiments. steps of the processing method.
  • the above-mentioned image stitching processing method, device, electronic system, electronic device, computer-readable storage medium, and computer program product provided by the embodiments of the present application are based on the fusion energy map corresponding to the first overlapping image and the second overlapping image of the images to be stitched.
  • the pixel energy value of determines the area corresponding to the target position (edge position and salient area) in the first overlapping image and the second overlapping image, and determines the minimum energy path between the relative edges of the fusion energy map based on this area, and then obtains
  • the masks corresponding to the first overlapping image and the second overlapping image are stitched based on the determined masks.
  • the area corresponding to the target position can be determined according to the fusion energy map corresponding to the first overlapped image and the second overlapped image, and the patchwork searched by the minimum energy path based on the area corresponding to the target position can minimize as much as possible.
  • the seam is more conducive to image splicing, and the effect of image splicing is effectively improved.
  • FIG. 1 is a schematic structural diagram of an electronic system according to an embodiment of the present application.
  • FIG. 2 is a flowchart of a processing method for image stitching provided by an embodiment of the present application
  • FIG. 3 is a flowchart of another image stitching processing method provided by an embodiment of the present application.
  • FIG. 4 is a flowchart of another image stitching processing method provided by an embodiment of the present application.
  • FIG. 5 is a schematic diagram of a black border area provided by an embodiment of the present application.
  • FIG. 6 is a schematic diagram of another image stitching processing method provided by an embodiment of the present application.
  • FIG. 7 is a flowchart of still another image stitching processing method provided by an embodiment of the present application.
  • FIG. 8 is a schematic structural diagram of an image segmentation network provided by an embodiment of the present application.
  • FIG. 9 is a schematic flowchart of a training method for an image segmentation network provided by an embodiment of the present application.
  • FIG. 10 is a schematic structural diagram of a processing device for image stitching provided by an embodiment of the application.
  • FIG. 11 is a schematic structural diagram of another image stitching processing apparatus provided by an embodiment of the application.
  • FIG. 12 is a schematic structural diagram of another image stitching processing apparatus provided by an embodiment of the present application.
  • FIG. 13 is a schematic structural diagram of still another image stitching processing apparatus provided by an embodiment of the present application.
  • the embodiments of the present application provide a processing method, device, electronic system, electronic device, computer-readable storage medium and computer program for image stitching.
  • the target area in the overlapping images determines the seam, and then the images are spliced, which can make the determined seam more reasonable and effectively improve the image splicing effect.
  • an image stitching processing method disclosed in this embodiment of the present application will be described in detail below.
  • FIG. 1 a schematic structural diagram of an electronic system 100 according to an embodiment of the present application shown in FIG. 1 .
  • the electronic system can be used to implement the image stitching processing method and apparatus of the embodiments of the present application.
  • the electronic system 100 may include one or more processing devices 102, one or more storage devices 104, an input device 106, an output device 108, and one or more image acquisition devices 110, these components may be interconnected by a bus system 112 and/or other form of connection mechanism (not shown). It should be noted that the components and structures of the electronic system 100 shown in FIG. 1 are only exemplary and not limiting, and the electronic system may also have other components and structures as required.
  • the processing device 102 may be a server, an intelligent terminal, or a device including a central processing unit (CPU) or other forms of processing units with data processing capability and/or instruction execution capability.
  • CPU central processing unit
  • the data of the components are processed, and other components in the electronic system 100 can also be controlled to perform the processing function of image stitching.
  • Storage 104 may include one or more computer program products, which may include various forms of computer-readable storage media, such as volatile memory and/or non-volatile memory.
  • Volatile memory may include, for example, random access memory (RAM) and/or cache memory, among others.
  • Non-volatile memory may include, for example, read only memory (ROM), hard disk, flash memory, and the like.
  • One or more computer program instructions may be stored on the computer-readable storage medium, and the processing device 102 may execute the program instructions to implement the client functions (implemented by the processing device) in the following embodiments of the present application and/or other desired Function.
  • client functions implemented by the processing device
  • Various application programs and various data such as various data used and/or generated by the application program, etc., may also be stored in the computer-readable storage medium.
  • Input device 106 may be a device used by a user to input instructions, and may include one or more of a keyboard, mouse, microphone, touch screen, and the like.
  • the output device 108 may output various information (eg, images or sounds) to the outside (eg, a user), and may include one or more of a display, a speaker, and the like.
  • the image acquisition device 110 may acquire the images to be stitched and store the acquired images in the storage device 104 for use by other components.
  • the components in the processing method, device, and electronic system for implementing image stitching according to the embodiments of the present application may be set in an integrated manner, or may be set in a decentralized manner, such as the processing device 102, the storage device 104, the input device 106 and the The output device 108 is integrated into one body, and the image capturing device 110 is arranged at a designated position where images can be captured.
  • the electronic system can be implemented as a smart terminal such as a camera, a smart phone, a tablet computer, a computer, a vehicle-mounted terminal, and the like.
  • Another embodiment of the present application provides a processing method for image stitching.
  • the method can be applied to an electronic device.
  • the electronic device can be configured with the above electronic system.
  • the method can mainly include the following steps:
  • Step S202 acquiring a first overlapping image and a second overlapping image respectively corresponding to overlapping regions of the first image and the second image to be spliced.
  • the above image may be an image extracted from a video received by an image acquisition device, or may be an image obtained from a network or a third party.
  • the first image and the second image may be captured by cameras at different positions, and the types of cameras (telephoto cameras, wide-angle cameras, etc.) used to capture the two images may be the same or different.
  • the overlapping area of the first image and the second image and a part of the non-overlapping area may be formed into a first overlapping image and a second overlapping image with the same size respectively, so as to facilitate the subsequent image processing process.
  • Step S204 determining the fusion energy map corresponding to the first overlapping image and the second overlapping image; wherein, the pixel energy value of the pixel corresponding to the pixel at the target position in the first overlapping image and the second overlapping image in the fusion energy map is higher than other positions
  • the pixel energy value of the pixel at the target location includes the edge location and the location corresponding to the salient region.
  • the sensitive areas in the two images can be determined first.
  • edge detection may be performed on the first overlapped image and the second overlapped image, respectively.
  • suitable operators can be used for calculation, such as Sobel operator, Isotropic Sobel operator, Roberts operator, Prewitt operator, Laplacian operator, Canny operator Subs, etc., can be selected according to actual needs.
  • the edge detection results of the first overlapping image and the second overlapping image are presented in the form of an edge energy map, and edge energy maps corresponding to the first overlapping image and the second overlapping image can be obtained through edge detection processing.
  • the pixel energy value is used to represent the difference between a pixel in the overlapping image and other pixels around it. The greater the difference, the higher the pixel energy value corresponding to the pixel on the edge energy map. It can be represented by the pixel value at the corresponding position of the pixel.
  • salient object detection may be performed on the first overlapping image and the second overlapping image respectively.
  • a suitable neural network model can be used for calculation, such as: Deeply Supervised Saliency Detection Framework (DSS, Deeply Supervised Salient), agile aggregation multi-layer feature detection Framework (Agile Amulet), Progressive Attention Driven Regularization Framework (PAGR, Progressive Attention Graph Regularization), Pixel-wise Contextual Attention Network (PiCANet, Pixel-wise Contextual Attention network), Boundary-Aware Network (BASNet, Boundary-Aware Salient Object Detection network), etc., which can be selected according to actual needs.
  • the size of the saliency energy map is the same as the size of the input images (i.e. the first and second overlapping images), and the saliency energy map is a binary image, i.e.: in the saliency energy map, the saliency target of the overlapping image
  • the pixel value of the area corresponding to the area is a value greater than 0 (for example, 255), and the pixel value of the remaining area is 0.
  • the fusion method used in this embodiment may be weighted fusion and/or mean fusion, etc.
  • the specific fusion The method can be selected according to needs.
  • the edge energy map and the saliency energy map corresponding to the first overlapping image can also be fused to obtain the first fusion energy map of the first overlapping image; in the same way, the edge energy corresponding to the second overlapping image
  • the image and the saliency energy map are fused to obtain a second fused energy map of the second overlapping image.
  • the first fusion energy map and the second fusion energy map are fused to obtain fusion energy maps corresponding to the first overlapping image and the second overlapping image.
  • the positions corresponding to the edge regions and the salient regions of different overlapping images can be defined as target positions, and then the pixel values of the regions corresponding to the target positions in the energy maps can be adjusted. method or by adjusting the weight of the pixel values of different energy maps in the fusion process, so that the pixel value of the area corresponding to the target position in the fusion energy map is larger than the pixel value of the remaining area.
  • Step S206 determining the minimum energy path corresponding to the first graph edge to the second graph edge of the fusion energy graph; wherein the first graph edge and the second graph edge are graph edges having a relative relationship.
  • the fusion energy map After the fusion energy map is obtained, the fusion energy map has the upper edge, the lower edge, the left edge and the right edge in the four directions, respectively. edge, or the left edge and the right edge) are defined as the first edge and the second edge respectively, and the minimum energy path is searched along the direction from the first edge to the second edge on the fusion energy graph. Taking the upper and lower edges of the fusion energy map as the first edge and the second edge respectively, on the fusion energy map, we can take the pixels in each row in the order from the upper edge to the lower edge. The pixels whose value is less than the set pixel threshold (or the minimum value) are determined, and the minimum energy path corresponding to the first edge to the second edge of the fusion energy map is determined based on the determined pixels.
  • Step S208 determining the first mask corresponding to the first overlapping image and the second mask corresponding to the second overlapping image based on the minimum energy path; wherein, the sum of the pixel values of the pixels corresponding to the first mask and the second mask can be is 1.
  • pixels corresponding to the minimum energy path in the fusion energy map are found on the first overlapping image and the second overlapping image respectively, and these pixels on the first overlapping image constitute the patchwork of the first overlapping image. These pixels on the two overlapping images form the patchwork of the second overlapping image.
  • the image stitching method as left-right stitching
  • the first overlapping image on the left, and the second overlapping image on the right construct a first mask with the same size as the first overlapping image and a mask with the same size as the second overlapping image.
  • second mask construct a first mask with the same size as the first overlapping image and a mask with the same size as the second overlapping image.
  • the pixel value of the area corresponding to the left area of the seam in the first overlapping image is set to 1, and the pixel value of the remaining area is set to 0;
  • the pixel value of the area corresponding to the area on the right side of the seam in the second overlapping image in the second mask is set to 1, and the pixel value of the remaining area is set to 0.
  • Step S210 stitching the first image and the second image based on the first mask and the second mask to obtain a stitched image.
  • the first mask is used to perform mask processing on the region corresponding to the first overlapping image in the first image, that is, the pixel value of the first mask is multiplied by the pixel value of the region.
  • the above-mentioned image splicing processing method determines the target in the first overlapping image and the second overlapping image based on the pixel energy values of the fusion energy maps corresponding to the first overlapping image and the second overlapping image of the images to be spliced.
  • the region corresponding to the position (edge position and salient region), based on which the minimum energy path is determined between the relative edges of the fusion energy map, and then the masks corresponding to the first overlapping image and the second overlapping image are obtained.
  • the area corresponding to the target position can be determined according to the fusion energy map corresponding to the first overlapped image and the second overlapped image, and the patchwork searched by the minimum energy path based on the area corresponding to the target position can minimize as much as possible.
  • the seam is more conducive to image splicing, and the effect of image splicing is effectively improved.
  • step S204 that is, the step of determining the fusion energy map corresponding to the first overlapping image and the second overlapping image.
  • step S204 that is, the step of determining the fusion energy map corresponding to the first overlapping image and the second overlapping image.
  • Step S302 acquiring a first overlapping image and a second overlapping image corresponding to the overlapping regions of the first image and the second image to be spliced, respectively.
  • Step S304 determining a first edge energy map corresponding to the first overlapping image and a second edge energy map corresponding to the second overlapping image.
  • the color image can be grayscaled before edge detection, and the grayscale of the color image in a single channel can be obtained respectively.
  • image Take the first overlapping image and the second overlapping image both in RGB format as an example, that is, performing grayscale processing on the first overlapping image and the second overlapping image respectively to obtain the first overlapping image and the second overlapping image corresponding to R (red). ), G (green), and B (blue) three-channel grayscale images, and then use the specified image edge detection operator to perform edge detection on the grayscale image.
  • the result of the edge detection will be presented in the form of an edge energy map.
  • the edge energy maps of the R, G, and B channels of the first overlapping image and the second overlapping image will be obtained, wherein the pixel value of the edge energy map can reflect the energy value corresponding to the pixel.
  • the edge energy maps of the R, G, and B channels of the first overlapping image can be denoted as R1, G1, and B1, respectively, and the edge energy maps of the R, G, and B channels of the second overlapping image They are denoted as R2, G2, and B2, respectively.
  • the pixel values of R1, G1, and B1 can be added to obtain the first edge energy map corresponding to the first overlapping image, and the pixel values of R2, G2, and B2 can be added to obtain the second overlapping image.
  • the second edge energy map corresponding to the image can be added to obtain the first edge energy map corresponding to the first overlapping image.
  • Step S306 determining the first saliency energy map corresponding to the first overlapping image and the second saliency energy map corresponding to the second overlapping image.
  • Step S308 combining the first significant energy map and the second significant energy map to obtain a combined significant energy map.
  • a new pixel value can be obtained by direct summation of pixel values or weighted summation of pixel values, and the obtained new pixel value is the saliency energy
  • Step S310 determine the fusion energy map corresponding to the first overlapping image and the second overlapping image according to the significant energy combination map, the first edge energy map and the second edge energy map; wherein, the fusion energy map corresponds to the first overlapping image and the second overlapping image.
  • the pixel energy value of the pixel at the target position in the two overlapping images may be higher than the pixel energy value of the pixel at other positions; the target position may include edge positions and positions corresponding to salient regions.
  • Commonly used fusion methods may mainly include weighted fusion, mean fusion, etc.
  • the specific fusion method can be selected according to needs. Considering that the image stitching process is usually left-right stitching or top-bottom stitching, taking left-right stitching as an example, most of the images in the left area of the image to be stitched on the left remain unchanged, and the stitching is usually located in the right area of the image. It is determined that the seam is as far as possible in the right area of the image, and the above step S310 may include the following steps (1) to (3):
  • a part of the region near the left side of the first edge energy map can be determined as the enhanced energy region corresponding to the first edge energy map according to actual needs.
  • each pixel in the enhanced energy region can be multiplied by a first set value greater than 1, so that the pixel energy value of the enhanced energy region is increased, so that the subsequently determined seam can be bypassed in the first image.
  • the area corresponding to the boosted energy area is the area corresponding to the boosted energy area.
  • each corresponding pixel in the saliency energy combination map, the first edge enhancement energy map, and the second edge energy map may be weighted and summed to obtain the fusion energy maps corresponding to the first overlapped image and the second overlapped image.
  • Step S312 Determine the minimum energy path corresponding to the first graph edge to the second graph edge of the fusion energy graph, wherein the first graph edge and the second graph edge are graph edges having a relative relationship.
  • Step S314 determining the first mask corresponding to the first overlapping image and the second mask corresponding to the second overlapping image based on the minimum energy path; wherein the sum of the pixel values of the pixels corresponding to the first mask and the second mask is both 1.
  • Step S316 stitching the first image and the second image based on the first mask and the second mask to obtain a stitched image.
  • the enhanced energy region corresponding to the edge energy map is determined according to the splicing method of the images, and the pixel energy value of the enhanced energy region is increased to further ensure the subsequent
  • the seam determined according to the minimum energy path can not only reduce the number of passing through the sensitive area, but also bypass the enhanced energy area as much as possible, thereby effectively improving the effect of image stitching.
  • the original image Before image stitching, the original image is usually stretched, rotated, translated, affine transformation and other operations, but when these operations are performed on the original image, a black border area will appear near the edge of the image.
  • the scope of the target position is expanded, and the above target position may also include The position corresponding to the black border area of the first overlapping image and/or the position corresponding to the black border area of the second overlapping image. Based on this, the above step S310 may further include the following steps:
  • the fusion method of the energy map can be optimized, and then the search method of the seam can be optimized according to the fusion energy map corresponding to the overlapping area image, so that the determined seam can completely bypass the edge position in the image , salient areas, and black border areas that may exist during image processing, further improving the effect of image stitching.
  • the following takes the splicing of two images (tar and src) as an example, referring to another processing method of image splicing shown in Figure 4 and Figure 6, the method may specifically include the following steps:
  • Step S402 extracting the overlapping area between tar and src to obtain an image pair of the overlapping area.
  • Two images to be spliced, tar and src are known, both of which are in RGB format, with tar on the left and src on the right.
  • the overlapping area of the two images is calculated separately, and the overlapping area is marked on tar and src, respectively.
  • the pixels of the overlapping area in tar and the pixels of a part of the non-overlapping area are composed of the image tar_overlap, and the pixels of the overlapping area in src and the pixels of a part of the non-overlapping area are composed of the image src_overlap, so that the sizes of tar_overlap and src_overlap are exactly the same.
  • tar_overlap and src_overlap into a ⁇ tar_overlap, src_overlap> pair the image pair of the overlapping area is obtained.
  • Step S404 performing edge detection on the image pairs in the overlapping area to obtain an initial edge energy map.
  • the grayscale images of the R, G, and B channels of tar_overlap and src_overlap are extracted respectively, and the sobel operator is used to perform edge detection on each grayscale image respectively to obtain the edge energy map of the single-channel grayscale image.
  • the edge energy maps of the three single-channel grayscale images corresponding to tar_overlap are summed to obtain the initial edge energy map tar_overlap_sobel of tar_overlap.
  • the edge energy maps of the three single-channel grayscale images corresponding to src_overlap are summed to obtain the initial edge energy map src_overlap_sobel of src_overlap.
  • Step S406 performing saliency target detection on the image pairs in the overlapping area to obtain an initial saliency energy map.
  • the saliency regions in tar_overlap and src_overlap are calculated respectively, and the images corresponding to the saliency regions are obtained.
  • the initial saliency energy maps corresponding to tar_overlap and src_overlap are calculated based on the images corresponding to the above saliency areas, which are tar_overlap_salient and src_overlap_salient respectively, where the saliency target area is 1, the non-salient target area is 0, and the two initial saliency energies are
  • the graphs are all 0-1 binary distributions, and both have the same size as tar_overlap and src_overlap.
  • Step S408 performing operations such as stretching on src to determine the black edge region of src, and obtaining an initial black edge energy map src_black of src based on the black edge region.
  • the black area on the left is the black border area.
  • src_black the pixel value of the area corresponding to the black border area of src is 0 (black), and the pixel value of the remaining area is 255 (white).
  • Step S410 modify the pixel values of the initial edge energy map to obtain an optimized edge energy map.
  • a part of the tar_overlap_sobel near the left edge of the image is determined as the area where the pixel value needs to be modified.
  • the area percentage of this area in the tar_overlap_sobel can be determined according to experience (such as 50%, 30%, etc.);
  • the pixel values are multiplied by coefficients greater than 1 (such as 1.2, 1.5, etc.) to obtain the optimized edge energy map tar_overlap_sobel' corresponding to tar_overlap.
  • Step S412 modifying the pixel values of the initial saliency energy map to obtain an optimized saliency energy map.
  • the optimized saliency energy map of this embodiment is to combine the above tar_overlap_salient and src_overlap_salient. Specifically, the following formula can be used:
  • pitar_overlap_salient' is the pixel value corresponding to the pixel in the optimized saliency map tar_overlap_salient' of tar_overlap
  • pisrc_overlap_salient' is the pixel value corresponding to the pixel in the optimized saliency map src_overlap_salient' of src_overlap.
  • Step S414 modifying the pixel values of the initial black-edge energy map to obtain an optimized black-edge energy map.
  • pisrc_black_forbidden pisrc_black*max_value*A;
  • pisrc_black_forbidden is the pixel value corresponding to the pixel in the optimized black edge energy map src_black_forbidden corresponding to src
  • pisrc_black is the pixel value corresponding to the pixel in src_black
  • A is a preset fixed value greater than 1, and the value of A can be determined according to Experience to choose (such as 1.5, 2.5, etc.).
  • step S416 a fusion energy map is obtained based on the optimized edge energy map, the optimized saliency energy map, and the optimized black edge energy map.
  • Pienergy_map w1*Pitar_overlap_sobel′+w2*Pisrc_overlap_sobel+w3*Pitar_overlap_salient′+w4*Pisrc_overlap_salient′+w5*Pisrc_black_forbidden
  • Pienergy_map is the pixel value corresponding to the pixel in the fusion energy map energy_map
  • Pitar_overlap_sobel' is the pixel value corresponding to the pixel in tar_overlap_sobel'
  • the values of w1, w2, w3, w4, and w5 can be set according to experience (such as setting all is 1).
  • Step S418 Determine the minimum energy path corresponding to the upper edge to the lower edge of the fusion energy graph, and determine the masks corresponding to tar_overlap and src_overlap based on the minimum energy path.
  • the pixel with the smallest value in each row of pixels is determined on the fusion energy map in the order from the upper edge to the lower edge, and these determined pixels form the upper edge of the fusion energy map. to the minimum energy path corresponding to the edge in the figure below. Then find the pixels corresponding to the minimum energy path in the fusion energy map on tar_overlap and src_overlap respectively. These pixels on tar_overlap form the seam of tar_overlap, and these pixels on src_overlap form the seam of src_overlap, see figure The top-to-bottom curve shown in 5 is the seam corresponding to the minimum energy path.
  • tar_mask the pixel value of the area in the tar_mask corresponding to the area on the left side of the seam in tar_overlap can be set to 1, and the pixel value of the remaining area can be set to 0;
  • the pixel value of the area corresponding to the area on the right side of the seam is set to 1, and the pixel value of the remaining area is set to 0.
  • Step S420 splicing images based on tar_mask and src_mask to obtain a spliced image.
  • mask processing For the pixels in the area corresponding to tar_overlap in tar and the area corresponding to src_overlap in src, mask processing is performed. Specifically, the following formula can be used to calculate:
  • pioverlap_blend pitar_overlap*pitar_mask+pisrc_overlap*pisrc_mask; wherein, pioverlap_blend is the pixel value corresponding to the pixel in the fused overlap area image overlap_blend, pitar_overlap is the pixel value corresponding to the pixel in the area corresponding to tar_overlap in the tar, and pitar_mask is the pixel value in the tar_mask.
  • pisrc_overlap is the pixel value corresponding to the pixel in the area corresponding to src_overlap in src
  • pisrc_mask is the pixel value corresponding to the pixel in src_mask.
  • the values of the pixels in the area around the seams of tar and src are weighted and averaged to obtain the spliced image.
  • the mask corresponding to the overlapping image can be determined through a neural network.
  • the method may mainly include the following steps:
  • Step S702 acquiring a first overlapping image and a second overlapping image corresponding to the overlapping regions of the first image and the second image to be spliced, respectively.
  • Step S704 inputting the first overlapping image and the second overlapping image into the image segmentation network after training, to obtain the image segmentation result output by the image segmentation network; wherein, the image sample pairs in the training process of the image segmentation network may correspond to a mask truth value Yes, the mask truth value pair can be determined based on the minimum energy path of the fusion energy map corresponding to the image sample pair, wherein the pixel energy value of the pixel at the target position corresponding to the image sample pair in the fusion energy map is higher than Pixel energy values of pixels at other locations; the target location includes edge locations and locations corresponding to salient regions.
  • the above image segmentation result is a binary image of 0-1, and the size is the same as that of the first overlapping image and the second overlapping image.
  • the binary image of 0-1 can be used as the mask reference image of the first overlapping image and the second overlapping image. .
  • Step S706 determining the first mask corresponding to the first overlapping image and the second mask corresponding to the second overlapping image based on the above-mentioned image segmentation result; wherein, the sum of the pixel values of the pixels corresponding to the first mask and the second mask is equal to Can be 1.
  • the image segmentation result is the first mask corresponding to the first overlapping image
  • the image segmentation result is the second mask corresponding to the second overlapping image
  • Step S708 stitching the first image and the second image based on the first mask and the second mask to obtain a stitched image.
  • the above method can directly perform image segmentation processing on the first overlapping image and the second overlapping image through the image segmentation network, and obtain an image segmentation result. Based on the result, the first mask and the second overlapping image corresponding to the first overlapping image can be easily determined.
  • the image segmentation network in the training process of the sample labeling mask truth value pair is determined based on the minimum energy path of the corresponding fusion energy map of the image sample pair, And the pixel energy value of the pixel at the target position corresponding to the image sample pair in the fusion energy map is higher than the pixel energy value of the pixels at other positions, the target position includes the edge position and the position corresponding to the salient area, so the image obtained by training
  • the segmentation network can output reasonable image segmentation results, thereby ensuring the rationality of the mask, and improving the effect and speed of image stitching.
  • the network structure of the image segmentation network can be mainly divided into three parts: a downsampling network, an upsampling network and a skip connection network.
  • the skip connection network can connect the network layer of the downsampling network with the network layer corresponding to the upsampling;
  • the network layer of the downsampling network includes a convolution layer and a downsampling layer;
  • the network layer corresponding to the upsampling includes a deconvolution layer and upsampling layers.
  • Subsampled can reduce an image, and its main purpose is to make the image fit the size of the display area on the display device and to generate a thumbnail corresponding to the image.
  • Upsampling or image interpolation (interpolating) is the ability to upscale an image, with the main purpose of enabling the image to be displayed on a higher resolution display device.
  • the dotted line represents the skip connection network
  • both the down-sampling network and the up-sampling network include four network layers as an example.
  • the specific processing process of the image segmentation network can be divided into a down-sampling process and an up-sampling process. Among them, the downsampling process can be described as follows:
  • Step 1 after receiving the first overlapping image and the second overlapping image, the first downsampling network layer performs subsequent processing on the two overlapping images with two parallel branches respectively.
  • the first overlapping image is processed.
  • the first layer of convolution processing is performed on the first overlapping image to obtain the first initial feature map, and then the first initial feature map is down-sampled to obtain the first feature map, and the first feature map to the first feature map is output.
  • the first feature maps corresponding to the first overlapping image and the second overlapping image will be obtained, and the two first feature maps output to the skip connection network may be merged feature maps in form. .
  • Step 2 the second down-sampling network layer performs a second-layer convolution process on the first feature map to obtain a second initial feature map, and then performs down-sampling on the second initial feature map to obtain a second feature map, and outputs The second feature map to the third downsampling network layer and skip connection network.
  • step 3 the processing of the third and fourth downsampling network layers is similar to the above, and the third feature map and the fourth feature map are obtained.
  • the feature maps of the two branches can be merged, so that the two branches are merged into one branch, and then the feature maps corresponding to the branch are output to the next network layer.
  • the upsampling process may include the following steps 4 to 7:
  • Step 4 After the fourth upsampling network layer receives the fourth feature map, it first performs deconvolution processing to obtain the fifth initial feature map, and then performs upsampling processing to output the fifth feature map;
  • Step 5 the third upsampling network layer receives the fifth feature map and the third feature map, first splices and fuses the fifth feature map and the third feature map, and then performs deconvolution processing to obtain the sixth initial feature map, and then The sixth initial feature map is subjected to up-sampling processing to obtain a sixth feature map;
  • the splicing and fusion can be splicing two feature maps, or summing the pixels corresponding to the two feature maps,
  • Step 6 the second upsampling network layer performs processing similar to the above step 5 on the sixth feature map and the second feature map, and outputs the seventh feature map;
  • Step 7 the first upsampling network layer splices and fuses the first feature map and the seventh feature map, and then deconvolutes and upsamples to obtain the eighth feature map; in addition, the first upsampling network layer also includes A convolution layer is used to segment and classify the pixels in the eighth feature map through convolution, and output the Mask; the Mask is the mask of the first overlapping image, and the mask of the second overlapping image is obtained by subtracting Mask from 1 .
  • image segmentation network is only a feasible application mode, and is not intended to limit the present application. In practical applications, other image segmentation network structures may also be selected to implement the above technology.
  • an embodiment of the present application further provides a method for training an image segmentation network.
  • the method may mainly include the following steps:
  • Step S902 acquiring an image sample pair and a mask truth value pair corresponding to the image sample pair.
  • the two images that make up the image sample pair can be recorded as image 1 and image 2 respectively, the ground truth mask corresponding to image 1 is marked as GT1, and the ground truth mask corresponding to image 2 is marked as GT2, GT1 and GT2 form a mask-truth pair.
  • the acquisition methods of the image 1, the image 2, the GT1 and the GT2 are similar to the corresponding contents in the foregoing steps S202 to S208, and are not repeated here.
  • step S904 the image sample pair is divided into an initial network for the input image to obtain a mask prediction pair corresponding to the image sample pair.
  • the image composed of image 1 and image 2 can be divided into an initial network for the input image, and the image 1 and image 2 can be subjected to down-sampling and up-sampling processing in turn through the image segmentation initial network to obtain the prediction mask corresponding to image 1. , and then determine the prediction mask corresponding to image 2 based on the prediction mask, denoted as DT1 and DT2 respectively, and DT1 and DT2 form the mask prediction pair corresponding to the image sample pair.
  • Step S906 Calculate the training loss value according to the corresponding mask truth value pair and mask prediction pair corresponding to the image sample pair.
  • the first loss value can be calculated by the following formula:
  • gt is the pixel value in GT1
  • dt is the pixel value at the position corresponding to GT1 in DT1
  • mean is the mean function
  • abs is the absolute value function.
  • the second loss value is calculated using the following formula:
  • the total loss value between GT1 and DT1 can be calculated using the following formula:
  • the calculation method of the total loss value between GT2 and DT2 is similar to the above calculation method, and will not be repeated here.
  • the training loss values are calculated according to the above calculation methods for the mask ground-truth pairs and mask prediction pairs corresponding to all image sample pairs.
  • Step S908 determine whether the condition for stopping training is satisfied; wherein, the condition for stopping training may be that the above-mentioned training loss value converges to a preset convergence threshold or the number of training times is equal to the preset number of times threshold; if not (that is: the above-mentioned training If the loss value has not converged to the preset convergence threshold or the number of training times is less than the preset number of times threshold), perform the following step S910; if so (that is, the above-mentioned training loss value has converged to the preset convergence threshold or the number of training times is equal to the preset number of times threshold), The following step S912 is performed.
  • Step S910 adjust the initial network parameters for image segmentation according to the training loss value, and continue training.
  • Step S912 stop training, and use the current image segmentation initial network as the image segmentation network.
  • this embodiment optimizes the specific operation mode for obtaining image sample pairs, and the specific operation mode may be as follows:
  • the preset mode may include at least one of the following: rotation, translation, affine transformation, and the like.
  • the above-mentioned initial image samples may come from images in an existing image database, and the images in the image database may be independent images without a splicing relationship. In order to train the model, images with splicing basis can be generated based on these images.
  • the black border area generated in the above preprocessing process is defined as black border 1.
  • a batch of black border templates can be generated by projecting the initial sample of the image, and these black border templates are defined as black borders.
  • side 2 For the union of the above black border 1 and black border 2, black border 3 can be obtained, and the preset black border area may include the area corresponding to one or more of the above black border 1, black border 2 or black border 3. . Add a preset black border area to the original image sample and/or the image transformation sample to obtain the image sample pair.
  • the specific operation mode of obtaining the corresponding mask truth value pair of the image sample pair is optimized, and the specific operation mode can be as follows:
  • the mask truth pair of the sample pair is determined based on the minimum energy path of the fusion energy map corresponding to the image sample pair, and the fusion energy map corresponds to the pixel at the target position of the image sample pair.
  • the energy value of the pixel is higher than the pixel energy value of the pixels at other positions.
  • the target position includes the edge position and the position corresponding to the salient area. Therefore, the image segmentation network obtained by training can output a more reasonable image segmentation result, thereby ensuring the mask's Reasonable, improve the effect of image stitching and stitching speed.
  • another embodiment of the present application further provides an image stitching processing apparatus, which can be applied to the above-mentioned electronic equipment, and the apparatus can include the following modules:
  • the overlapped image acquisition module 81 may be configured to acquire the first overlapped image and the second overlapped image respectively corresponding to the overlapped regions of the first image and the second image to be spliced.
  • the fusion image determination module 82 can be configured to determine the fusion energy map corresponding to the first overlapping image and the second overlapping image; wherein, the fusion energy map corresponds to the first overlapping image.
  • Pixel energy values of pixels at target positions in the overlapping image and the second overlapping image may be higher than pixel energy values at other positions; the target positions may include edge positions and positions corresponding to salient regions.
  • the energy path determination module 83 can be configured to determine the minimum energy path corresponding to the first edge to the second edge of the fusion energy graph; wherein, the first edge and the first edge Two graph edges can be graph edges with relative relationship.
  • a mask acquisition module 84 which may be configured to determine a first mask corresponding to the first overlapping image and a second mask corresponding to the second overlapping image based on the minimum energy path; wherein , the sum of the pixel values of the pixels corresponding to the first mask and the second mask may both be 1.
  • the first stitching module 85 can be configured to stitch the first image and the second image based on the first mask and the second mask to obtain a stitched image.
  • the above-mentioned fused image determination module 82 may also be configured to: determine a first edge energy map corresponding to the first overlapping image and a second edge energy map corresponding to the second overlapping image; determine that the first overlapping image corresponds to The first saliency energy map and the second saliency energy map corresponding to the second overlapping image; combine the first saliency energy map and the second saliency energy map to obtain a saliency energy combination map; merge according to the saliency energy map, the first edge energy map and the second edge energy map to determine a fusion energy map corresponding to the first overlapping image and the second overlapping image.
  • the above-mentioned fused image determination module 82 may also be configured to: determine the enhanced energy region corresponding to the first edge energy map according to the splicing method of the first image and the second image; wherein, the splicing method may be Including left and right splicing or up and down splicing; multiplying each pixel in the enhanced energy area by a first set value to obtain a first edge enhanced energy map; wherein, the first set value may be greater than 1; according to the The merged saliency map, the first edge-enhanced energy map, and the second edge energy map determine fused energy maps corresponding to the first overlay image and the second overlay image.
  • the above-mentioned fused image determination module 82 may also be configured to: perform weighted summation on each corresponding pixel in the saliency energy combination map, the first edge-enhanced energy map, and the second edge energy map, to obtain the result. fusion energy maps corresponding to the first overlapping image and the second overlapping image.
  • the target position may further include a position corresponding to the black border area of the first overlapping image and/or a position corresponding to the black border area of the second overlapping image;
  • the fusion image determination module 82 may further It can be configured to: if the first overlapping image contains a black border region, obtain the black border region energy map corresponding to the black border region of the first overlapping image; if the second overlapping image contains a black border region, obtain a black-edge region energy map corresponding to the black-edge region of the second overlapping image; multiply each pixel in the obtained black-edge region energy map by a second set value to obtain a black-edge region energy enhancement map, Wherein, the second set value may be greater than 1; for each of the significant energy merge map, the first edge-enhanced energy map, the second edge energy map, and the black-bordered region energy-enhanced map The corresponding pixels are weighted and summed to obtain a fusion energy map corresponding to the first overlapping image and the second overlapping image;
  • still another embodiment of the present application also provides an image stitching processing apparatus, which can be applied to the above-mentioned electronic equipment, and the apparatus can include the following modules:
  • the image acquisition module 91 can be configured to acquire the first overlapping image and the second overlapping image corresponding to the overlapping area of the first image and the second image to be spliced respectively;
  • the image segmentation module 92 can be configured to input the first overlapping image and the second overlapping image into a trained image segmentation network to obtain an image segmentation result output by the image segmentation network;
  • the image sample pairs in the training process of the image segmentation network may correspond to a mask truth value pair, and the mask truth value pair may be determined based on the minimum energy path corresponding to the fusion energy map corresponding to the image sample pair, wherein , the pixel energy value of the pixel at the target position corresponding to the image sample pair in the fusion energy map may be higher than the pixel energy value of the pixel at other positions;
  • the target position may include edge positions and positions corresponding to salient regions;
  • a mask determination module 93 which may be configured to determine a first mask corresponding to the first overlapping image and a second mask corresponding to the second overlapping image based on the image segmentation result; wherein , the sum of the pixel values of the pixels corresponding to the first mask and the second mask may both be 1;
  • the second stitching module 94, the second stitching module 94 may be configured to stitch the first image and the second image based on the first mask and the second mask to obtain a stitched image.
  • the embodiment of the present application provides another image stitching processing device.
  • the device may also include a training module 95, and the training module 95 may be configured to be used for : obtain an image sample pair and a mask truth value pair corresponding to the image sample pair; input the image sample pair into the image segmentation initial network to obtain a mask prediction pair corresponding to the image sample pair; The training loss value is calculated for the corresponding mask truth value pair and mask prediction pair of the sample pair; according to the training loss value, it is judged whether the preset stop training condition is met; if not, the image segmentation is adjusted according to the training loss value Initial network parameters, continue training; if yes, stop training and use the current image segmentation initial network as the image segmentation network.
  • the above training module 95 may also be configured to: preprocess the initial image samples to obtain image transformation samples corresponding to the initial image samples; wherein, the preprocessing may include at least one of the following: rotation, translation, affine Transform; add a black border area to the initial image sample and/or the image transformed sample to obtain the image sample pair.
  • the above-mentioned training module 95 can also be configured to: obtain the first edge energy map corresponding to the initial image sample in the image sample pair and the second edge energy map corresponding to the image transformation sample by an edge detection method;
  • the target detection model obtains the first saliency energy map corresponding to the initial image sample and the second saliency energy map corresponding to the image transformation sample; merging the first saliency energy map and the second saliency energy map to obtain saliency energy Merge map; determine the fusion energy map corresponding to the image sample pair according to the significant energy combination map, the first edge energy map and the second edge energy map, and determine the fusion energy map based on the fusion energy map and the minimum energy path
  • the image sample pair corresponds to the mask ground-truth pair.
  • the above-mentioned mask determination module 93 may also be configured to: if the image segmentation result is the first mask corresponding to the first overlapped image, apply a mask that has the same size as the first mask and has a pixel value of 1. A difference between the image and the first mask is performed to obtain a second mask corresponding to the second overlapping image.
  • the processing apparatus 200 may include: a processor 40, a memory 41, a bus 42, and a communication interface 43.
  • the processor 40, the communication interface 43 and the The memory 41 may be connected by a bus 42 and the processor 40 may be configured to execute executable modules, such as computer programs, stored in the memory 41 .
  • the memory 41 may include a high-speed random access memory (RAM, Random Access Memory), and may also include a non-volatile memory (non-volatile memory), such as at least one disk memory.
  • RAM Random Access Memory
  • non-volatile memory such as at least one disk memory.
  • the communication connection between the network element of the system and at least one other network element is realized through at least one communication interface 43 (which may be wired or wireless), which may use the Internet, a wide area network, a local network, a metropolitan area network, and the like.
  • the bus 42 may be an ISA bus, a PCI bus, an EISA bus, or the like.
  • the bus can be divided into address bus, data bus, control bus and so on. For ease of representation, only one bidirectional arrow is shown in FIG. 13, but it does not mean that there is only one bus or one type of bus.
  • the memory 41 may be configured to store a program, and the processor 40 executes the program after receiving the execution instruction, and the method executed by the apparatus defined by the flow process disclosed in any of the foregoing embodiments of the present application may be applied to in the processor 40 or implemented by the processor 40 .
  • the processor 40 may be an integrated circuit chip with signal processing capability. In the implementation process, each step of the above-mentioned method can be completed by a hardware integrated logic circuit in the processor 40 or an instruction in the form of software.
  • the above-mentioned processor 40 can be a general-purpose processor, including a central processing unit (Central Processing Unit, referred to as CPU), a network processor (Network Processor, referred to as NP), etc.; can also be a digital signal processor (Digital Signal Processing, referred to as DSP) ), Application Specific Integrated Circuit (ASIC), Field-Programmable Gate Array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, and discrete hardware components.
  • CPU Central Processing Unit
  • NP Network Processor
  • DSP Digital Signal Processing
  • ASIC Application Specific Integrated Circuit
  • FPGA Field-Programmable Gate Array
  • a general purpose processor may be a microprocessor or the processor may be any conventional processor or the like.
  • the steps of the method disclosed in conjunction with the embodiments of the present application may be directly embodied as executed by a hardware decoding processor, or executed by a combination of hardware and software modules in the decoding processor.
  • the software modules may be located in random access memory, flash memory, read-only memory, programmable read-only memory or electrically erasable programmable memory, registers and other storage media mature in the art.
  • the storage medium can be located in the memory 41, and the processor 40 can read the information in the memory 41, and complete the steps of the above method in combination with its hardware.
  • Embodiments of the present application may further provide a computer program, where the computer program may include computer-readable codes, and when the computer-readable codes are executed on a computing and processing device, the computing and processing device may execute the above method.
  • the terms “installed”, “connected” and “connected” should be understood in a broad sense, for example, it may be a fixed connection or a detachable connection , or integrally connected; it can be a mechanical connection or an electrical connection; it can be a direct connection, or an indirect connection through an intermediate medium, or the internal communication between the two components.
  • installed should be understood in a broad sense, for example, it may be a fixed connection or a detachable connection , or integrally connected; it can be a mechanical connection or an electrical connection; it can be a direct connection, or an indirect connection through an intermediate medium, or the internal communication between the two components.
  • the computer program product for image stitching processing includes a computer-readable storage medium storing non-volatile program code executable by the processor, and the program code includes instructions that can be used to execute the foregoing method.
  • the program code includes instructions that can be used to execute the foregoing method.
  • the disclosed system, apparatus and method may be implemented in other manners.
  • the apparatus embodiments described above are only illustrative.
  • the division of the units is only a logical function division.
  • multiple units or components may be combined or Can be integrated into another system, or some features can be ignored, or not implemented.
  • the shown or discussed mutual coupling or direct coupling or communication connection may be through some communication interfaces, indirect coupling or communication connection of devices or units, which may be in electrical, mechanical or other forms.
  • the units described as separate components may or may not be physically separated, and components displayed as units may or may not be physical units, that is, may be located in one place, or may be distributed to multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution in this embodiment.
  • each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically alone, or two or more units may be integrated into one unit.
  • the functions, if implemented in the form of software functional units and sold or used as stand-alone products, may be stored in a processor-executable non-volatile computer-readable storage medium.
  • the technical solution of the present application can be embodied in the form of a software product in essence, or the part that contributes to the related technology or the part of the technical solution.
  • the computer software product is stored in a storage medium, including several
  • the instructions are used to cause a computer device (which may be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present application.
  • the aforementioned storage medium includes: U disk, mobile hard disk, Read-Only Memory (ROM, Read-Only Memory), Random Access Memory (RAM, Random Access Memory), magnetic disk or optical disk and other media that can store program codes .
  • the present application discloses a processing method, device, electronic system, electronic device, computer-readable storage medium, and computer program for image stitching.
  • the method includes: acquiring first images corresponding to overlapping regions of the first image and the second image to be stitched, respectively. Overlapping the image and the second overlapping image; determining the fusion energy map corresponding to the first overlapping image and the second overlapping image; determining the minimum energy path corresponding to the first edge to the second edge of the fusion energy map; determining the first edge based on the minimum energy path.
  • a first mask corresponding to an overlapping image and a second mask corresponding to the second overlapping image stitching the first image and the second image based on the first mask and the second mask.
  • the present application optimizes the area where the seam passes to a certain extent, and improves the image stitching effect.
  • processing method, apparatus, electronic system, electronic device, computer-readable storage medium and computer program of the present application are reproducible and can be applied in various industrial applications.
  • the image stitching processing method of the present application can be applied to the field of image processing.

Landscapes

  • Engineering & Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Theoretical Computer Science (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Image Processing (AREA)
  • Image Analysis (AREA)

Abstract

一种图像拼接的处理方法、装置、电子系统和设备、可读介质，该方法包括：获取待拼接的第一图像和第二图像的重叠区域分别对应的第一重叠图像和第二重叠图像（S202）；确定第一重叠图像和第二重叠图像对应的融合能量图（S204）；确定融合能量图的第一图边至第二图边对应的最小能量路径（S206）；基于最小能量路径确定第一重叠图像对应的第一掩模和第二重叠图像对应的第二掩模（S208）；基于第一掩模和第二掩模拼接第一图像和第二图像（S210）。上述方法在一定程度上优化了拼缝经过的区域，提升了图像的拼接效果。

Description

图像拼接的处理方法、装置、电子系统和设备、可读介质
相关申请的交叉引用
本申请要求于2021年2月9日提交中国国家知识产权局的申请号为202110186621.3、名称为“图像拼接的处理方法、装置和电子系统”的中国专利申请的优先权，其全部内容通过引用结合在本申请中。
技术领域
本申请涉及图像处理技术领域，尤其是涉及图像拼接的处理方法、装置、电子系统、电子设备、计算机可读存储介质以及计算机程序产品。
背景技术
图像拼接任务是将具有重叠视场的多幅图像组合在一起以获得大视场角的高分辨率图片，然而由于视差大、场景景深差异等影响，重叠区域部分的图像内容难以完全对齐，从而导致拼接结果出现鬼影、物体截断等问题。通过拼缝搜索可以在一定程度上缓解重叠区域部分的图像内容对不齐对拼接造成的影响。
当前主流的拼缝搜索方式是基于graph cut(图割)算法进行的，该算法通过图像像素值差异等定义的能量函数，搜索图像局部区域的最小能量路径，然而局部区域的能量最小化，对于差异不明显的重要区域(有显著性目标所在区域等)，也可能属于能量最小化的位置，进而被确定为拼缝，影响了后续图像拼接的效果。
发明内容
鉴于此，本申请提供了图像拼接的处理方法、装置、电子系统、电子设备、计算机可读存储介质以及计算机程序产品，以提升图像的拼接效果。
本申请的一些实施例提供了一种图像拼接的处理方法，所述方法可以应用于电子设备，所述方法可以包括：获取待拼接的第一图像和第二图像的重叠区域分别对应的第一重叠图像和第二重叠图像；确定所述第一重叠图像和所述第二重叠图像对应的融合能量图；其中，所述融合能量图中对应于所述第一重叠图像和所述第二重叠图像中的目标位置处像素的像素能量值可以高于其它位置的像素能量值；所述目标位置可以包括边缘位置和显著区域对应的位置；确定所述融合能量图的第一图边至第二图边对应的最小能量路径；其中，所述第一图边和所述第二图边可以为具有相对关系的图边；基于所述最小能量路径确定所述第一重叠图像对应的第一掩模和所述第二重叠图像对应的第二掩模；其中，所述第一掩模与所述第二掩模对应像素的像素值之和均可以为1；基于所述第一掩模和所述第二掩模拼接所述第一图像和所述第二图像，得到拼接图像。
结合本申请的一些实施例，本申请实施例提供了本申请一些实施例中的一种可能的实施方式，其中，确定所述第一重叠图像和所述第二重叠图像对应的融合能量图的步骤包括：确定所述第一重叠图像对应的第一边缘能量图和所述第二重叠图像对应的第二边缘能量图；确定所述第一重叠图像对应的第一显著能量图和所述第二重叠图像对应的第二显著能量图；合并所述第一显著能量图和所述第二显著能量图，得到显著能量合并图；根据所述显著能量合并图、所述第一边缘能量图和所述第二边缘能量图确定所述第一重叠图像和所述第二重叠图像对应的融合能量图。
结合本申请的一些实施例，本申请实施例提供了本申请一些实施例中的一种可能的实施方式，其中，根据所述显著能量合并图、所述第一边缘能量图和所述第二边缘能量图确定所述第一重叠图像和所述第二重叠图像对应的融合能量图的步骤包括：根据所述第一图像和所述第二图像的拼接方式，确定所述第一边缘能量图对应的增强能量区域；其中，所述拼接方式可以包括左右拼接或上下拼接；对所述增强能量区域中的每个像素乘以第一设定值，得到第一边缘增强能量图；其中，所述第一设定值可以大于1；根据所述显著能量合并图、所述第一边缘增强能量图和所述第二边缘能量图确定所述第一重叠图像和所述第二重叠图像对应的融合能量图。
结合本申请的一些实施例，本申请实施例提供了本申请一些实施例中的一种可能的实施方式，其中，当所述拼接方式为左右拼接、所述第一图像位于左侧时，将所述第一边缘能量图中靠近左侧的一部分区域确定为所述第一边缘能量图对应的增强能量区域。
结合本申请的一些实施例，本申请实施例提供了本申请一些实施例的一种可能的实施方式，其中，根据所述显著能量合并图、所述第一边缘增强能量图和所述第二边缘能量图确定所述第一重叠图像和所述第二重叠图像对应的融合能量图的步骤包括：对所述显著能量合并图、所述第一边缘增强能量图和所述第二边缘能量图中的每个对应像素进行加权求和，得到所述第一重叠图像和所述第二重叠图像对应的融合能量图。
结合本申请的一些实施例，本申请实施例提供了本申请一些实施例中的一种可能的实施方式，其中，根据所述显著 能量合并图、所述第一边缘增强能量图和所述第二边缘能量图确定所述第一重叠图像和所述第二重叠图像对应的融合能量图的步骤包括：如果所述第一重叠图像包含黑边区域，获取所述第一重叠图像的黑边区域对应的黑边区域能量图；如果所述第二重叠图像包含黑边区域，获取所述第二重叠图像的黑边区域对应的黑边区域能量图；将获取到的所述黑边区域能量图中的每个像素乘以第二设定值，得到黑边区域能量增强图，其中，所述第二设定值可以大于1；对所述显著能量合并图、所述第一边缘增强能量图、所述第二边缘能量图和所述黑边区域能量增强图中的每个对应像素进行加权求和，得到所述第一重叠图像和所述第二重叠图像对应的融合能量图。
结合本申请的一些实施例，本申请实施例提供了本申请一些实施例中的一种可能的实施方式，其中，合并所述第一显著能量图和所述第二显著能量图而得到显著能量合并图的步骤包括：对于所述第一显著能量图和所述第二显著能量图中的每个对应像素，采用像素值直接求和或像素值加权求和的方式得到新的像素值，得到的新的像素值即为所述显著能量合并图的像素值。
结合本申请的一些实施例，本申请实施例提供了本申请一些实施例中的一种可能的实施方式，其中，合并所述第一显著能量图和所述第二显著能量图而得到显著能量合并图的步骤包括：对于显著能量合并图中的每个像素，该像素对应的像素值max_value＝max(tar_overlap_salient，src_overlap_salient)；其中，tar_overlap_salient为所述第一显著能量图中的该像素对应的第一像素值；src_overlap_salient为所述第二显著能量图中的该像素对应的第二像素值；max函数为取所述第一像素值和所述第二像素值中的最大值。
本申请的另一些实施例提供了一种图像拼接的处理方法，所述方法可以应用于电子设备，所述方法可以包括：获取待拼接的第一图像和第二图像的重叠区域分别对应的第一重叠图像和第二重叠图像；将所述第一重叠图像和所述第二重叠图像输入训练后的图像分割网络，得到所述图像分割网络输出的图像分割结果；其中，所述图像分割网络训练过程中的图像样本对可以对应有掩模真值对，所述掩模真值对可以是基于所述图像样本对对应的融合能量图对应的最小能量路径确定的，其中，所述融合能量图中对应于所述图像样本对的目标位置处像素的像素能量值可以高于其它位置处像素的像素能量值；所述目标位置可以包括边缘位置和显著区域对应的位置；基于所述图像分割结果确定所述第一重叠图像对应的第一掩模和所述第二重叠图像对应的第二掩模；其中，所述第一掩模与所述第二掩模对应像素的像素值之和均可以为1；基于所述第一掩模和所述第二掩模拼接所述第一图像和所述第二图像，得到拼接图像。
结合本申请的另一些实施例，本申请实施例提供了本申请另一些实施例中的一种可能的实施方式，其中，所述图像分割网络的训练过程包括：获取图像样本对和所述图像样本对对应的掩模真值对；将所述图像样本对输入所述图像分割初始网络，得到所述图像样本对对应的掩模预测对；根据所述图像样本对对应的掩模真值对和掩模预测对计算训练损失值；根据所述训练损失值，判断是否满足停止训练条件；如果否，根据所述训练损失值调整所述图像分割初始网络参数，继续训练；如果是，停止训练，将当前的图像分割初始网络作为所述图像分割网络。
结合本申请的另一些实施例，本申请实施例提供了本申请另一些实施例中的一种可能的实施方式，其中，获取所述图像样本对的步骤包括：对图像初始样本进行目标处理，得到所述图像初始样本对应的图像变换样本；其中，所述目标处理可以包括以下至少之一：旋转、平移、仿射变换；为所述图像初始样本和/或所述图像变换样本添加黑边区域，得到所述图像样本对。
结合本申请的另一些实施例，本申请实施例提供了本申请另一些实施例中的一种可能的实施方式，其中，获取所述图像样本对对应的掩模真值对的步骤包括：通过边缘检测方法获取所述图像样本对中的图像初始样本对应的第一边缘能量图和所述图像变换样本对应的第二边缘能量图；通过显著性目标检测模型获取所述图像初始样本对应的第一显著能量图和所述图像变换样本对应的第二显著能量图；合并所述第一显著能量图和所述第二显著能量图，得到显著能量合并图；根据所述显著能量合并图、所述第一边缘能量图和所述第二边缘能量图确定所述图像样本对对应的融合能量图，并基于所述融合能量图和最小能量路径确定所述图像样本对对应的掩模真值对。
结合本申请的另一些实施例，本申请实施例提供了本申请另一些实施例中的一种可能的实施方式，其中，所述基于所述图像分割结果确定所述第一重叠图像对应的第一掩模和所述第二重叠图像对应的第二掩模的步骤包括：如果所述图像分割结果为所述第一重叠图像对应的第一掩模，应用与所述第一掩模尺寸相同且像素值均为1的图像与所述第一掩模做差，得到所述第二重叠图像对应的第二掩模。
结合本申请的另一些实施例，本申请实施例提供了本申请另一些实施例中的一种可能的实施方式，其中，所述基于所述图像分割结果确定所述第一重叠图像对应的第一掩模和所述第二重叠图像对应的第二掩模的步骤包括：如果所述图像分割结果为所述第二重叠图像对应的第二掩模，应用与所述第二掩模尺寸相同且像素值均为1的图像与所述第二掩模 做差，得到所述第一重叠图像对应的第一掩模。
本申请的又一些实施例提供了一种图像拼接的处理装置，所述装置可以应用于电子设备，所述装置可以包括：重叠图像获取模块，所述重叠图像获取模块可以配置成用于获取待拼接的第一图像和第二图像的重叠区域分别对应的第一重叠图像和第二重叠图像；融合图像确定模块，所述融合图像确定模块可以配置成用于确定所述第一重叠图像和所述第二重叠图像对应的融合能量图；其中，所述融合能量图中对应于所述第一重叠图像和所述第二重叠图像中的目标位置处像素的像素能量值可以高于所述其它位置处像素的像素能量值，所述目标位置可以包括边缘位置和显著区域对应的位置；能量路径确定模块，所述能量路径确定模块可以配置成用于确定所述融合能量图的第一图边至第二图边对应的最小能量路径；其中，所述第一图边和所述第二图边可以为具有相对关系的图边；掩模获取模块，所述掩模获取模块可以配置成用于基于所述最小能量路径确定所述第一重叠图像对应的第一掩模和所述第二重叠图像对应的第二掩模；其中，所述第一掩模与所述第二掩模对应像素的像素值之和均可以为1；第一拼接模块，所述第一拼接模块可以配置成用于基于所述第一掩模和所述第二掩模拼接所述第一图像和所述第二图像，得到拼接图像。
本申请的再一些实施例提供了一种图像拼接的处理装置，所述装置可以应用于电子设备，所述装置可以包括：图像获取模块，所述图像获取模块可以配置成用于获取待拼接的第一图像和第二图像的重叠区域分别对应的第一重叠图像和第二重叠图像；图像分割模块，所述图像分割模块可以配置成用于将所述第一重叠图像和所述第二重叠图像输入训练后的图像分割网络，得到所述图像分割网络输出的图像分割结果；其中，所述图像分割网络训练过程中的图像样本对可以对应有掩模真值对，所述掩模真值对可以是基于所述图像样本对对应的融合能量图对应的最小能量路径确定的，其中，所述融合能量图中对应于所述图像样本对的目标位置处像素的像素能量值可以高于其它位置处像素的像素能量值，所述目标位置包括边缘位置和显著区域对应的位置；掩模确定模块，所述掩模确定模块可以配置成用于基于所述图像分割结果确定所述第一重叠图像对应的第一掩模和所述第二重叠图像对应的第二掩模；其中，所述第一掩模与所述第二掩模对应像素的像素值之和均可以为1；第二拼接模块，所述第二拼接模块可以配置成用于基于所述第一掩模和所述第二掩模拼接所述第一图像和所述第二图像，得到拼接图像。
本申请的再另一些实施例提供了一种电子系统，所述电子系统可以包括：图像采集设备、处理设备和存储装置；所述图像采集设备可以配置成用于获取图像；所述存储装置上可以存储有计算机程序，所述计算机程序在被所述处理设备运行时执行如前述实施例中任一项所述的图像拼接的处理方法。
本申请的再又一些实施例提供了一种电子设备，所述电子设备可以包括处理器和存储器，所述处理器可以配置成在运行时执行如前述实施例中任一项所述的图像拼接的处理方法。
本申请的另外的一些实施例提供了一种计算机可读存储介质，所述计算机可读存储介质上可以存储有计算机程序，所述计算机程序被处理设备运行时可以执行如前述实施例中任一项所述的图像拼接的处理方法的步骤。
本申请的再另外的一些实施例提供了一种计算机程序产品，所述计算机程序产品可以包括计算机程序，当所述计算机程序被处理器执行时实现前述实施例中任一项所述的图像拼接的处理方法的步骤。
本申请实施例带来了至少以下有益效果：
本申请实施例提供的上述图像拼接的处理方法、装置、电子系统、电子设备、计算机可读存储介质以及计算机程序产品，基于待拼接图像的第一重叠图像和第二重叠图像对应的融合能量图的像素能量值，确定第一重叠图像和第二重叠图像中的目标位置(边缘位置和显著区域)对应的区域，基于该区域在融合能量图的相对图边之间确定最小能量路径，进而得到第一重叠图像和第二重叠图像对应的掩模，基于确定出的掩模拼接图像。通过上述操作方式，可根据第一重叠图像和第二重叠图像对应的融合能量图确定出目标位置对应的区域，基于目标位置对应的区域以最小能量路径搜索出的拼缝，能够尽可能少地经过目标位置对应的区域，使得拼缝更利于图像拼接，有效提高了图像拼接的效果。
本公开的其他特征和优点将在随后的说明书中阐述，或者，部分特征和优点可以从说明书推知或毫无疑义地确定，或者通过实施本公开的上述技术即可得知。
为使本公开的上述目的、特征和优点能更明显易懂，下文特举较佳实施例，并配合所附附图，作详细说明如下。
附图说明
为了更清楚地说明本申请具体实施方式或相关技术中的技术方案，下面将对具体实施方式或相关技术描述中所需要使用的附图作简单地介绍，显而易见地，下面描述中的附图是本申请的一些实施方式，对于本领域普通技术人员来讲，在不付出创造性劳动的前提下，还可以根据这些附图获得其他的附图。
图1为本申请实施例提供的一种电子系统的结构示意图；
图2为本申请实施例提供的一种图像拼接的处理方法的流程图；
图3为本申请实施例提供的另一种图像拼接的处理方法的流程图；
图4为本申请实施例提供的又一种图像拼接的处理方法的流程图；
图5为本申请实施例提供的一种黑边区域示意图；
图6为本申请实施例提供的另一种图像拼接的处理方法的示意图；
图7为本申请实施例提供的再一种图像拼接的处理方法的流程图；
图8为本申请实施例提供的一种图像分割网络的结构示意图；
图9为本申请实施例提供的一种图像分割网络的训练方法的流程示意图；
图10为本申请实施例提供的一种图像拼接的处理装置的结构示意图；
图11为本申请实施例提供的另一种图像拼接的处理装置的结构示意图；
图12为本申请实施例提供的又一种图像拼接的处理装置的结构示意图；
图13为本申请实施例提供的再一种图像拼接的处理装置的结构示意图。
具体实施方式
为使本申请实施例的目的、技术方案和优点更加清楚，下面将结合附图对本申请的技术方案进行清楚、完整地描述，显然，所描述的实施例是本申请一部分实施例，而不是全部的实施例。基于本申请中的实施例，本领域普通技术人员在没有做出创造性劳动前提下所获得的所有其他实施例，都属于本申请保护的范围。
考虑到基于现有的拼缝拼接的图像效果不佳的问题，本申请实施例提供了图像拼接的处理方法、装置、电子系统、电子设备、计算机可读存储介质以及计算机程序，该技术主要基于重叠图像中的目标区域确定拼缝，进而拼接图像，能够使确定的拼缝更合理，有效改善图像的拼接效果。为便于对本实施例提供的技术方案的理解，下面对本申请实施例所公开的一种图像拼接的处理方法进行详细介绍。
参照图1所示的本申请一实施例的电子系统100的结构示意图。该电子系统可以用于实现本申请实施例的图像拼接的处理方法和装置。
如图1所示的一种电子系统的结构示意图，电子系统100可以包括一个或多个处理设备102、一个或多个存储装置104、输入装置106、输出装置108以及一个或多个图像采集设备110，这些组件可以通过总线系统112和/或其它形式的连接机构(未示出)互连。应当注意，图1所示的电子系统100的组件和结构只是示例性的，而非限制性的，根据需要，电子系统也可以具有其他组件和结构。
处理设备102可以为服务器、智能终端，或者是包含中央处理单元(CPU)或者具有数据处理能力和/或指令执行能力的其它形式的处理单元的设备，处理设备102可以对电子系统100中的其它组件的数据进行处理，还可以控制电子系统100中的其它组件以执行图像拼接的处理功能。
存储装置104可以包括一个或多个计算机程序产品，计算机程序产品可以包括各种形式的计算机可读存储介质，例如易失性存储器和/或非易失性存储器。易失性存储器例如可以包括随机存取存储器(RAM)和/或高速缓冲存储器(cache)等。非易失性存储器例如可以包括只读存储器(ROM)、硬盘、闪存等。在计算机可读存储介质上可以存储一个或多个计算机程序指令，处理设备102可以运行程序指令，以实现下文的本申请实施例中(由处理设备实现)的客户端功能以及/或者其它期望的功能。在计算机可读存储介质中还可以存储各种应用程序和各种数据，例如应用程序使用和/或产生的各种数据等。
输入装置106可以是用户用来输入指令的装置，并且可以包括键盘、鼠标、麦克风和触摸屏等中的一个或多个。
输出装置108可以向外部(例如，用户)输出各种信息(例如，图像或声音)，并且可以包括显示器、扬声器等中的一个或多个。
图像采集设备110可以获取待拼接的图像，并且将采集到的图像存储在存储装置104中以供其它组件使用。
示例性地，用于实现根据本申请实施例的图像拼接的处理方法、装置和电子系统中的各器件可以集成设置，也可以分散设置，诸如将处理设备102、存储装置104、输入装置106和输出装置108集成设置于一体，而将图像采集设备110设置于可以采集到图像的指定位置。当上述电子系统中的各器件集成设置时，该电子系统可以被实现为诸如相机、智能手机、平板电脑、计算机、车载终端等智能终端。
本申请另一实施例提供一种图像拼接的处理方法，该方法可以应用于电子设备，该电子设备可以配置有上述电子系统，参见图2，该方法可以主要包括以下步骤：
步骤S202，获取待拼接的第一图像和第二图像的重叠区域分别对应的第一重叠图像和第二重叠图像。
上述图像可以是从图像采集设备接收到的视频中提取的图像，也可以是从网络或第三方得到的图像。例如，第一图像与第二图像可通过不同位置的摄像头拍摄得到，采集这两幅图像所用摄像头的类型(长焦摄像头、广角摄像头等)可以相同或不同。由于第一图像和第二图像之间具有重叠区域，而这部分重叠区域的信息会直接影响图像拼接效果，因此，在进行图像拼接之前，需要分别将这两幅不同图像的重叠区域的信息保存下来，为了便于操作，可将第一图像和第二图像的重叠区域以及一部分非重叠区域分别组成尺寸相同的第一重叠图像和第二重叠图像，以便后续图像处理过程。
步骤S204，确定第一重叠图像和第二重叠图像对应的融合能量图；其中，融合能量图中对应于第一重叠图像和第二重叠图像中的目标位置处像素的像素能量值高于其它位置处像素的像素能量值，该目标位置包括边缘位置和显著区域对应的位置。
具体地，在得到第一重叠图像和第二重叠图像后，首先可以确定这两幅图像中的敏感区域(如图像中显著性目标所在区域、图像的边缘区域等)。
例如，为了确定第一重叠图像和第二重叠图像的边缘区域(即图像中像素值有突变的区域)，可分别对第一重叠图像和第二重叠图像进行边缘检测。在对第一重叠图像和第二重叠图像进行图像边缘检测时，可以使用合适的算子进行计算，例如Sobel算子、Isotropic Sobel算子、Roberts算子、Prewitt算子、Laplacian算子、Canny算子等，具体可根据实际需要自行选择。
在本申请实施例中，第一重叠图像和第二重叠图像的边缘检测的结果会以边缘能量图的形式呈现，通过边缘检测处理可以得到第一重叠图像和第二重叠图像对应的边缘能量图，在边缘能量图上用像素能量值来表示重叠图像中某一像素与其周围其它像素之间的差异，差异越大，该像素对应在边缘能量图上的像素能量值越高，该像素能量值可以用像素对应位置处的像素值表示。
再例如，为了确定第一重叠图像和第二重叠图像中的显著性目标所在区域，可分别对第一重叠图像和第二重叠图像进行显著性目标检测。在对第一重叠图像和第二重叠图像进行显著性目标检测时，可以使用合适的神经网络模型进行计算，例如：深度监督显著性检测框架(DSS，Deeply Supervised Salient)、敏捷聚合多层特征检测框架(Agile Amulet)、渐进的注意驱动正则化框架(PAGR，Progressive Attention Graph Regularization)、像素级上下文注意网络(PiCANet，Pixel-wise Contextual Attention network)、边界感知网络(BASNet，Boundary-Aware Salient Object Detection network)等，具体可根据实际需要自行选择。
将第一重叠图像和第二重叠图像输入预先训练好的显著性目标检测模型，该模型的输出结果为第一重叠图像和第二重叠图像中的显著性目标区域分别对应的显著性能量图，显著性能量图的尺寸与输入图像(即第一重叠图像和第二重叠图像)的尺寸相同，显著性能量图为二值图像，即：在显著性能量图中，与重叠图像的显著性目标区域对应的区域的像素值为大于0的值(例如255)，剩余区域的像素值为0。
对上述边缘能量图和显著性能量图进行融合操作，得到第一重叠图像和第二重叠图像对应的融合能量图，本实施例使用的融合方式可以为加权融合和/或均值融合等，具体融合方式可根据需要自行选择。具体融合过程，也可以将上述第一重叠图像对应的边缘能量图和显著性能量图进行融合，得到第一重叠图像的第一融合能量图；同理，将上述第二重叠图像对应的边缘能量图和显著性能量图进行融合，得到第二重叠图像的第二融合能量图。然后再对第一融合能量图和第二融合能量图进行融合，得到第一重叠图像和第二重叠图像对应的融合能量图。
在对不同能量图进行融合的过程中，可将不同重叠图像的边缘区域对应的位置和显著区域对应的位置均定义为目标位置，进而可采用调整能量图中与目标位置对应的区域的像素值的方式或者采用调整不同能量图的像素值在融合过程中所占权重的方式，使融合能量图中与目标位置对应的区域的像素值大于剩余区域的像素值。
步骤S206，确定融合能量图的第一图边至第二图边对应的最小能量路径；其中，第一图边和第二图边为具有相对关系的图边。
在得到融合能量图后，融合能量图在上下左右四个方向上分别具有上图边、下图边、左图边和右图边，将相对关系的两个图边(上图边与下图边，或者，左图边与右图边)分别定义为第一图边和第二图边，在融合能量图上沿第一图边至第二图边的方向查找最小能量路径。以融合能量图的上图边和下图边分别为第一图边和第二图边为例，具体可在融合能量图上按照从上图边到下图边的顺序将每一行像素中取值小于设定像素阈值(或取值最小)的像素确定出来，基于确定出来的这些像素确定融合能量图的第一图边至第二图边对应的最小能量路径。
步骤S208，基于最小能量路径确定第一重叠图像对应的第一掩模和第二重叠图像对应的第二掩模；其中，第一掩模 与第二掩模对应像素的像素值之和均可以为1。
具体地，分别在第一重叠图像和第二重叠图像上找到与融合能量图中最小能量路径对应的像素，在第一重叠图像上的这些像素便组成了第一重叠图像的拼缝，在第二重叠图像上的这些像素便组成了第二重叠图像的拼缝。以图像拼接方式为左右拼接、第一重叠图像位于左侧、第二重叠图像位于右侧为例，构建与第一重叠图像的尺寸相同的第一掩模以及与第二重叠图像的尺寸相同的第二掩模。对于第一掩模来说，将第一掩模中与第一重叠图像中拼缝左侧区域对应的区域的像素值取1，将剩余区域的像素值取0；对于第二掩模来说，将第二掩模中与第二重叠图像中拼缝右侧区域对应的区域的像素值取1，将剩余区域的像素值取0。
步骤S210，基于第一掩模和第二掩模拼接第一图像和第二图像，得到拼接图像。
用第一掩模对第一图像中与第一重叠图像对应的区域进行掩模处理，即：将第一掩模的像素值与该区域的像素值相乘。用第二掩模对第二图像中与第二重叠图像对应的区域进行掩模处理，即将第二掩模的像素值与该区域的像素值相乘；之后将掩模处理后的第一图像与掩模处理后的第二图像在确定好的拼缝处进行拼接、对齐，得到拼接后的图像。
本申请实施例提供的上述图像拼接的处理方法，基于待拼接图像的第一重叠图像和第二重叠图像对应的融合能量图的像素能量值，确定第一重叠图像和第二重叠图像中的目标位置(边缘位置和显著区域)对应的区域，基于该区域在融合能量图的相对图边之间确定最小能量路径，进而得到第一重叠图像和第二重叠图像对应的掩模，基于确定出的掩模拼接图像。通过上述操作方式，可根据第一重叠图像和第二重叠图像对应的融合能量图确定出目标位置对应的区域，基于目标位置对应的区域以最小能量路径搜索出的拼缝，能够尽可能少地经过目标位置对应的区域，使得拼缝更利于图像拼接，有效提高了图像拼接的效果。
在上述实施例的基础上，为了提高图像拼接的效率，本申请又一实施例对上述步骤S204(即确定第一重叠图像和第二重叠图像对应的融合能量图的步骤)进行了优化，参见图3所示的另一种图像拼接的处理方法的流程图，该方法可以主要包括以下步骤：
步骤S302，获取待拼接的第一图像和第二图像的重叠区域分别对应的第一重叠图像和第二重叠图像。
步骤S304，确定第一重叠图像对应的第一边缘能量图和第二重叠图像对应的第二边缘能量图。
对于第一重叠图像和第二重叠图像均为彩色图像(如RGB格式等)的应用场景，在进行边缘检测前可以对彩色图像进行灰度化处理，分别得到彩色图像在单通道下的灰度图像。以第一重叠图像和第二重叠图像均为RGB格式为例，即分别对第一重叠图像和第二重叠图像进行灰度化处理，得到第一重叠图像和第二重叠图像对应在R(红色)、G(绿色)、B(蓝色)三个通道的灰度图像，之后利用指定的图像边缘检测算子对灰度图像进行边缘检测，边缘检测的结果会以边缘能量图的形式呈现，上述处理后将得到第一重叠图像和第二重叠图像的R、G、B三个通道的边缘能量图，其中，边缘能量图的像素值大小能够反映出该像素对应的能量值的高低。
为便于描述，可以将第一重叠图像的R、G、B三个通道的边缘能量图分别记为R1、G1、B1，将第二重叠图像的R、G、B三个通道的边缘能量图分别记为R2、G2、B2。为了最大程度保留彩色图像的边缘信息，可以将R1、G1、B1的像素值相加得到第一重叠图像对应的第一边缘能量图，将R2、G2、B2的像素值相加得到第二重叠图像对应的第二边缘能量图。
步骤S306，确定第一重叠图像对应的第一显著能量图和第二重叠图像对应的第二显著能量图。
步骤S308，合并第一显著能量图和第二显著能量图，得到显著能量合并图。
对于第一显著能量图和第二显著能量图中的每个对应像素，可采用像素值直接求和或像素值加权求和的方式得到新的像素值，得到的新的像素值即为显著能量合并图的像素值。例如，为了便于操作，具体操作方式可以为：对于显著能量合并图中的每个像素，该像素对应的像素值max_value＝max(tar_overlap_salient，src_overlap_salient)；其中，tar_overlap_salient为所述第一显著能量图中的该像素对应的第一像素值；src_overlap_salient为所述第二显著能量图中的该像素对应的第二像素值；max函数为取所述第一像素值和所述第二像素值中的最大值。
步骤S310，根据显著能量合并图、第一边缘能量图和第二边缘能量图确定第一重叠图像和第二重叠图像对应的融合能量图；其中，融合能量图中对应于第一重叠图像和第二重叠图像中的目标位置处像素的像素能量值可以高于其它位置处像素的像素能量值；该目标位置可以包括边缘位置和显著区域对应的位置。
常用的融合方式可以主要包括加权融合、均值融合等，具体融合方式可根据需要自行选择。考虑到图像拼接过程通常为左右拼接或上下拼接，以左右拼接为例，左侧待拼接的图像中大部分左侧区域的图像保留不变，拼缝通常位于该图像的右侧区域，为了保证确定出拼缝尽可能在该图像的右侧区域，上述步骤S310可以包括以下步骤(1)至(3)：
(1)根据第一图像和第二图像的拼接方式，确定第一边缘能量图对应的增强能量区域；其中，所述拼接方式包括左右拼接或上下拼接。
以拼接方式为左右拼接、第一图像位于左侧为例，可根据实际需要将第一边缘能量图中靠近左侧的一部分区域确定为第一边缘能量图对应的增强能量区域。
(2)对增强能量区域中的每个像素乘以第一设定值，得到第一边缘增强能量图；其中，所述第一设定值大于1。
本实施例可以对增强能量区域中的每一像素均乘以大于1的第一设定值，这样便提高了增强能量区域的像素能量值，以便后续确定的拼缝能够绕开第一图像中与增强能量区域对应的区域。
(3)根据显著能量合并图、第一边缘增强能量图和第二边缘能量图确定第一重叠图像和第二重叠图像对应的融合能量图。例如，可对显著能量合并图、第一边缘增强能量图和第二边缘能量图中的每个对应像素进行加权求和，得到第一重叠图像和第二重叠图像对应的融合能量图。
步骤S312，确定融合能量图的第一图边至第二图边对应的最小能量路径；其中，第一图边和第二图边为具有相对关系的图边。
步骤S314，基于最小能量路径确定第一重叠图像对应的第一掩模和第二重叠图像对应的第二掩模；其中，第一掩模与第二掩模对应像素的像素值之和均为1。
步骤S316，基于第一掩模和第二掩模拼接第一图像和第二图像，得到拼接图像。
本申请实施例的上述图像拼接的处理方法，在能量图的融合过程中，根据图像的拼接方式确定了边缘能量图对应的增强能量区域，并提高了增强能量区域的像素能量值，进一步保证后续根据能量最小路径确定出来的拼缝既可以减少经过敏感区域，又可以尽可能地绕开该增强能量区域，进而有效提高了图像拼接的效果。
在进行图像拼接前，通常对原始图像进行拉伸、旋转、平移、仿射变换等操作，但在对原始图像进行这些操作时会在靠近图像边缘的位置出现黑边区域。为了进一步提高图像拼接过程中的操作灵活性，在上述步骤S310中的目标位置已经包括边缘位置和显著区域对应的位置的基础上，对该目标位置的范围进行了拓展，上述目标位置还可以包括第一重叠图像的黑边区域对应的位置和/或第二重叠图像的黑边区域对应的位置。基于此，上述步骤S310还可以包括如下步骤：
(1)如果第一重叠图像包含黑边区域，获取第一重叠图像的黑边区域对应的黑边区域能量图；
(2)如果第二重叠图像包含黑边区域，获取第二重叠图像的黑边区域对应的黑边区域能量图；
(3)将获取到的黑边区域能量图中的每个像素乘以第二设定值，得到黑边区域能量增强图，其中，所述第二设定值大于1；
(4)对显著能量合并图、第一边缘增强能量图、第二边缘能量图和黑边区域能量增强图中的每个对应像素进行加权求和，得到第一重叠图像和第二重叠图像对应的融合能量图。
通过该操作方式，可对能量图的融合方式进行优化，进而可根据重叠区域图像对应的融合能量图对拼缝的搜索方式进行优化，使确定出来的拼缝能够完全绕开图像中的边缘位置、显著区域以及图像处理过程中可能存在的黑边区域，进一步提高了图像拼接的效果。
下面以两幅图像(tar与src)左右拼接为例，参见图4和图6所示的另一种图像拼接的处理方法，该方法可以具体包括如下步骤：
步骤S402，提取tar与src之间的重叠区域，得到重叠区域的图像对。
已知两幅待拼接的图像tar和src，这两幅图像均为RGB格式，tar位于左侧，src位于右侧。分别计算出两幅图像的重叠区域，并分别在tar和src上标识出重叠区域。将tar中重叠区域的像素以及一部分非重叠区域的像素组成图像tar_overlap，将src中重叠区域的像素以及一部分非重叠区域的像素组成图像src_overlap，使得tar_overlap和src_overlap的尺寸完全相同。将tar_overlap和src_overlap组成<tar_overlap,src_overlap>对，即得到了重叠区域的图像对。
步骤S404，对重叠区域的图像对进行边缘检测，得到初始边缘能量图。
具体地，分别提取tar_overlap和src_overlap的R、G、B三个通道的灰度图像，使用sobel算子分别对每一灰度图像进行边缘检测，得到单通道灰度图像的边缘能量图。将tar_overlap对应的三个单通道灰度图像的边缘能量图进行求和，得到tar_overlap的初始边缘能量图tar_overlap_sobel。将src_overlap对应的三个单通道灰度图像的边缘能量图进行求和，得到src_overlap的初始边缘能量图src_overlap_sobel。
步骤S406，对重叠区域的图像对进行显著性目标检测，得到初始显著性能量图。
基于DSS算法分别计算tar_overlap和src_overlap中的显著性区域，得到显著性区域对应的图像。基于上述显著性区 域对应的图像计算得到tar_overlap和src_overlap对应的初始显著性能量图，分别为tar_overlap_salient和src_overlap_salient，其中，显著性目标区域为1，非显著性目标区域为0，两个初始显著性能量图均为0-1二值分布，且均与tar_overlap和src_overlap具有相同的尺寸。
步骤S408，对src进行拉伸等操作，确定src的黑边区域，基于该黑边区域得到src的初始黑边能量图src_black。
具体地，以图像src的形状为方形为例，参见图5所示的黑边区域示意图，左侧的黑色区域即为黑边区域。在src_black中，与src的黑边区域对应的区域的像素值为0(黑色)，剩余区域的像素值为255(白色)。
步骤S410，修改初始边缘能量图的像素值，得到优化边缘能量图。
将tar_overlap_sobel中靠近左图边的一部分区域确定为需要修改像素值的区域，该区域在tar_overlap_sobel中所占的面积百分比可根据经验自行确定(如50％、30％等)；将该区域中像素的像素值均乘以大于1的系数(如1.2、1.5等)，得到tar_overlap对应的优化边缘能量图tar_overlap_sobel′。
步骤S412，修改初始显著能量图的像素值，得到优化显著性能量图。
对于tar_overlap_salient和src_overlap_salient中的每个像素，设置该像素对应于显著能量合并图中的像素的像素值max_value＝max(tar_overlap_salient，src_overlap_salient)；其中，tar_overlap_salient为所述第一显著能量图中的该像素对应的第一像素值；src_overlap_salient为所述第二显著能量图中的该像素对应的第二像素值；max函数为取所述第一像素值和所述第二像素值中的最大值；且该max_value的取值为0或者1。
计算优化显著能量图的像素值，本实施例的优化显著能量图即合并上述tar_overlap_salient和src_overlap_salient，具体可以采用以下公式：
pitar_overlap_salient′＝pisrc_overlap_salient′＝max_value；
pitar_overlap_salient′为tar_overlap的优化显著能量图tar_overlap_salient′中的像素对应的像素值，pisrc_overlap_salient′为src_overlap的优化显著能量图src_overlap_salient′中的像素对应的像素值。
步骤S414，修改初始黑边能量图的像素值，得到优化黑边能量图。
对于src_black中的每一像素，具体可以采用以下公式进行计算：
pisrc_black_forbidden＝pisrc_black*max_value*A；
其中，pisrc_black_forbidden为src对应的优化黑边能量图src_black_forbidden中的像素对应的像素值，pisrc_black为src_black中的像素对应的像素值，A为预先设置好的大于1的固定值，A的取值可根据经验自行选择(如1.5、2.5等)。
步骤S416，基于优化边缘能量图、优化显著性能量图、优化黑边能量图得到融合能量图。
对于得到的能量图中的每一个像素，具体可以采用以下公式进行计算：
Pienergy_map＝w1*Pitar_overlap_sobel′+w2*Pisrc_overlap_sobel+w3*Pitar_overlap_salient′+w4*Pisrc_overlap_salient′+w5*Pisrc_black_forbidden
其中，Pienergy_map为融合能量图energy_map中的像素对应的像素值，Pitar_overlap_sobel′为tar_overlap_sobel′中的像素对应的像素值，w1、w2、w3、w4、w5的取值可根据经验进行设置(如均设置为1)。
步骤S418，确定融合能量图的上图边至第下图边对应的最小能量路径，基于最小能量路径确定tar_overlap和src_overlap对应的掩模。
在得到融合能量图后，在融合能量图上按照从上图边到下图边的顺序将每一行像素中取值最小的像素确定出来，确定出来的这些像素组成了融合能量图的上图边至下图边对应的最小能量路径。再分别在tar_overlap和src_overlap上找到与融合能量图中最小能量路径对应的像素，在tar_overlap上的这些像素便组成了tar_overlap的拼缝，在src_overlap上的这些像素便组成了src_overlap的拼缝，参见图5所示的由上至下的曲线即为最小能量路径对应的拼缝。
构建与tar_overlap的尺寸相同的掩模tar_mask以及与src_overlap的尺寸相同的掩模src_mask。对于tar_mask来说，可以将tar_mask中与tar_overlap中拼缝左侧区域对应的区域的像素值取1，将剩余区域的像素值取0；对于src_mask来说，可以将第二掩模中与src_overlap中拼缝右侧区域对应的区域的像素值取1，将剩余区域的像素值取0。
步骤S420，基于tar_mask和src_mask对图像进行拼接，得到拼接图像。
对于tar中与tar_overlap对应的区域以及src中与src_overlap对应的区域中的像素，进行掩模处理，具体可以采用以下公式进行计算：
pioverlap_blend＝pitar_overlap*pitar_mask+pisrc_overlap*pisrc_mask；其中，pioverlap_blend为融合的重叠区域图像overlap_blend中的像素对应的像素值，pitar_overlap为tar中与tar_overlap对应的区域中的像素对应的像素值，pitar_mask 为tar_mask中的像素对应的像素值，pisrc_overlap为src中与src_overlap对应的区域中的像素对应的像素值，pisrc_mask为src_mask中的像素对应的像素值。
之后分别对tar和src的拼缝周围区域的像素的值进行加权求平均，得到拼接后的图像。
在上述实施例提供的图像拼接的处理方法的基础上，为了进一步提高图像拼接效率，可通过神经网络确定重叠图像对应的掩模。参见图7所示的本申请再一实施例的图像拼接的处理方法流程图，该方法可以主要包括以下步骤：
步骤S702，获取待拼接的第一图像和第二图像的重叠区域分别对应的第一重叠图像和第二重叠图像。
步骤S704，将第一重叠图像和第二重叠图像输入训练后的图像分割网络，得到图像分割网络输出的图像分割结果；其中，图像分割网络训练过程中的图像样本对可以对应有掩模真值对，该掩模真值对可以是基于图像样本对对应的融合能量图的最小能量路径确定的，其中，该融合能量图中对应于该图像样本对的目标位置处像素的像素能量值高于其它位置处像素的像素能量值；该目标位置包括边缘位置和显著区域对应的位置。
上述图像分割结果为0-1的二值图像，尺寸与第一重叠图像和第二重叠图像相同，该0-1的二值图像可以作为第一重叠图像和第二重叠图像的掩模参考图像。
步骤S706，基于上述图像分割结果确定第一重叠图像对应的第一掩模和第二重叠图像对应的第二掩模；其中，第一掩模与第二掩模对应像素的像素值之和均可以为1。
如果图像分割结果为第一重叠图像对应的第一掩模，应用与该第一掩模尺寸相同且像素值均为1的图像与该第一掩模做差，得到第二重叠图像对应的第二掩模。
如果图像分割结果为第二重叠图像对应的第二掩模，应用与该第二掩模尺寸相同且像素值均为1的图像与该第二掩模做差，得到第一重叠图像对应的第一掩模。
步骤S708，基于第一掩模和第二掩模拼接第一图像和第二图像，得到拼接图像。
上述方法通过图像分割网络可以直接对第一重叠图像和第二重叠图像进行图像分割处理，得到图像分割结果，基于该结果可以较容易地确定出第一重叠图像对应的第一掩模和第二重叠图像对应的第二掩模，进而完成后续的图像拼接过程，而该图像分割网络在训练过程中样本标注掩模真值对是基于图像样本对对应的融合能量图的最小能量路径确定的，且该融合能量图中对应于该图像样本对的目标位置处像素的像素能量值高于其它位置处像素的像素能量值，该目标位置包括边缘位置和显著区域对应的位置，因此训练得到的图像分割网络能够输出比较合理的图像分割结果，进而保障了掩模的合理性，提升了图像拼接的效果和拼接速度。
参见图8所示的图像分割网络的结构示意图，图像分割网络的网络结构可以主要分为三部分：下采样网络、上采样网络以及跳跃连接网络。其中，跳跃连接网络可以将下采样网络的网络层与上采样对应的网络层连接；下采样网络的网络层均包括卷积层和下采样层；上采样对应的网络层均包括反卷积层和上采样层。
下采样(subsampled)，或称为降采样(downsampled)，能够缩小图像，其主要目的在于使得图像符合显示设备上的显示区域的大小以及生成该图像对应的缩略图。
上采样(upsampling)，或称为图像插值(interpolating)，能够放大图像，其主要目的在于使得图像可以显示在更高分辨率的显示设备上。
在图8中，虚线表示跳跃连接网络，以下采样网络和上采样网络均包括四个网络层为例。图像分割网络的具体处理过程可以分为下采样过程和上采样过程。其中，下采样过程可以描述如下：
步骤1，第一下采样网络层接收到第一重叠图像和第二重叠图像后，分别以两个并行分支对这两个重叠图像进行后续处理，为了描述方便，以对第一重叠图像进行处理为例，先对第一重叠图像进行第一层卷积处理，得到第一初始特征图，再对第一初始特征图进行下采样处理，得到第一特征图，并输出第一特征图至第二下采样网络层和跳跃连接网络。
经上述第一下采样网络层处理后，会得到第一重叠图像和第二重叠图像分别对应的第一特征图，输出到跳跃连接网络的两个第一特征图形式上可以是合并的特征图。
步骤2，第二下采样网络层对上述第一特征图进行第二层卷积处理，得到第二初始特征图，再对第二初始特征图进行下采样处理，得到第二特征图，并输出第二特征图至第三下采样网络层和跳跃连接网络。
步骤3，第三和第四下采样网络层的处理同上类似，得到第三特征图和第四特征图。
需要说明的是，上述任一下采样处理过程中，可以在对两个分支的特征图进行合并处理，使两个分支合并为一个分支，然后输出该分支对应的特征图至下一个网络层。
在上述步骤1至步骤3(也即下采样处理过程)之后，进行后面的上采样处理过程。上采样处理过程可以包括如下步 骤4至步骤7：
步骤4，第四上采样网络层接收到第四特征图后，先进行反卷积处理，得到第五初始特征图，再进行上采样处理，输出第五特征图；
步骤5，第三上采样网络层接收第五特征图和第三特征图，先将第五特征图和第三特征图拼接融合，然后进行反卷积处理，得到第六初始特征图，再对第六初始特征图进行上采样处理，得到第六特征图；
该拼接融合可以是将两特征图拼接起来，也可以是两个特征图对应的像素求和，
步骤6，第二上采样网络层对第六特征图和第二特征图进行类似上述步骤5的处理，输出第七特征图；
步骤7，第一上采样网络层对第一特征图和第七特征图进行拼接融合，然后反卷积处理和上采样处理，得到第八特征图；另外，第一上采样网络层还包括了一个卷积层，用于通过卷积对第八特征图中的像素进行分割分类，输出Mask；该Mask即为第一重叠图像的掩模，第二重叠图像的掩模用1减去Mask得到。
上述图像分割网络仅是一种可行的应用方式，并不用于限定本申请，在实际应用中，也可以选择其它的图像分割网络结构实现上述技术。
参见图9，本申请实施例还提供一种图像分割网络的训练方法，该方法可以主要包括以下步骤：
步骤S902，获取图像样本对和该图像样本对对应的掩模真值对。
为了便于描述，可以将组成图像样本对的两幅图像分别记作图像1和图像2，将图像1对应的真值掩模记作GT1，将图像2对应的真值掩模记作GT2，GT1和GT2组成了一个掩模真值对。图像1、图像2、GT1和GT2的获取方式与上述步骤S202至步骤S208中对应内容类似，在此不再赘述。
步骤S904，将该图像样本对输入图像分割初始网络，得到该图像样本对对应的掩模预测对。
具体地，可将图像1和图像2组成的图像对输入图像分割初始网络，通过图像分割初始网络分别对图像1和图像2依次进行下采样处理和上采样处理，得到图像1对应的预测掩模，再基于该预测掩模确定图像2对应的预测掩模，分别记作DT1和DT2，DT1和DT2组成了该图像样本对对应的掩模预测对。
步骤S906，根据图像样本对对应的掩模真值对和掩模预测对计算训练损失值。
以计算GT1和DT1之间的损失值为例，对于GT1和DT1中对应位置的每一像素，可以采用以下公式计算得到第一损失值：
Loss1＝mean(abs(gt–dt))；
其中，gt为GT1中的像素值，dt为DT1中与GT1对应位置处的像素值，mean为均值函数，abs为绝对值函数。
对于DT1，将DT1中对应于GT1中像素值为255的区域(白色区域)的像素集合为a，将DT1中对应于GT1中像素值为0的区域(黑色区域)的像素集合为b，可以采用以下公式计算得到第二损失值：
Loss2＝std(a)+std(b)。
可以采用下述公式计算GT1和DT1之间的总损失值：
Loss＝Loss1+Loss2。
GT2和DT2之间的总损失值的计算方式与上述计算方式类似，在此不再赘述。对于所有图像样本对对应的掩模真值对和掩模预测对，均按上述计算方式计算训练损失值。
步骤S908，根据上述训练损失值，判断是否满足停止训练条件；其中，该停止训练条件可以为上述训练损失值收敛至预设收敛阈值或者训练次数等于预设次数阈值；如果否(即：上述训练损失值未收敛至预设收敛阈值或者训练次数小于预设次数阈值)，执行下述步骤S910；如果是(即：上述训练损失值收敛至预设收敛阈值或者训练次数等于预设次数阈值)，执行下述步骤S912。
步骤S910，根据训练损失值调整图像分割初始网络参数，继续训练。
步骤S912，停止训练，将当前的图像分割初始网络作为图像分割网络。
在上述训练方法的基础上，为了获取比较丰富的图像样本，本实施例对获取图像样本对的具体操作方式进行了优化，具体操作方式可以如下：
(1)对图像初始样本进行预处理，得到图像初始样本对应的图像变换样本；其中，预设方式可以包括以下至少之一：旋转、平移、仿射变换等。
上述图像初始样本可以来自现有图像数据库中的图像，而该图像数据库中的图像可能都是独立没有拼接关系的图像，为了训练模型，可以基于这些图像生成具有拼接基础的图像。
(2)为图像初始样本和/或图像变换样本添加预设的黑边区域，得到图像样本对。
对于上述预处理过程中产生的黑边区域定义为黑边1，除了这种黑边之外，还可通过对图像初始样本进行投影操作生成一批黑边模板，将这些黑边模板定义为黑边2。对于上述黑边1与黑边2取并集，可以得到黑边3，预设的黑边区域可以包括上述黑边1、黑边2或者黑边3中的一个或多个黑边对应的区域。为图像初始样本和/或图像变换样本添加预设的黑边区域，得到图像样本对。在添加预设的黑边区域时，可以仅为图像初始样本添加黑边区域，也可以仅为图像变换样本添加黑边区域，当然也可以为图像初始样本和图像变换样本均添加黑边区域。通过这种黑边添加方式，可以基于图像初始样本和图像变换样本得到多个图像样本对。
在上述训练方法的基础上，为了提高训练效率，对获取图像样本对对应的掩模真值对的具体操作方式进行了优化，具体操作方式可以如下：
(1)通过边缘检测方法获取图像样本对中的图像初始样本对应的第一边缘能量图和图像变换样本对应的第二边缘能量图。
(2)通过显著性目标检测模型获取图像初始样本对应的第一显著能量图和图像变换样本对应的第二显著能量图。
(3)合并第一显著能量图和第二显著能量图，得到显著能量合并图。
(4)根据显著能量合并图、第一边缘能量图和第二边缘能量图确定图像样本对对应的融合能量图，并基于融合能量图和最小能量路径确定所述图像样本对对应的掩模真值对。
上述掩模真值对的获取过程可以参考上述实施例中的方法，这里不再赘述。
上述图像分割网络在训练过程中，基于图像样本对对应的融合能量图的最小能量路径确定的样本对的掩模真值对，而该融合能量图中对应于该图像样本对的目标位置处像素的像素能量值高于其它位置处像素的像素能量值，该目标位置包括边缘位置和显著区域对应的位置，因此训练得到的图像分割网络能够输出比较合理的图像分割结果，进而保障了掩模的合理性，提升了图像拼接的效果和拼接速度。
在上述实施例的基础上，参见图10，本申请再另一实施例还提供一种图像拼接的处理装置，该装置可以应用于上述电子设备，该装置可以包括如下模块：
重叠图像获取模块81，重叠图像获取模块81可以配置成用于获取待拼接的第一图像和第二图像的重叠区域分别对应的第一重叠图像和第二重叠图像。
融合图像确定模块82，融合图像确定模块82可以配置成用于确定所述第一重叠图像和所述第二重叠图像对应的融合能量图；其中，所述融合能量图中对应于所述第一重叠图像和所述第二重叠图像中的目标位置处像素的像素能量值可以高于其它位置的像素能量值；所述目标位置可以包括边缘位置和显著区域对应的位置。
能量路径确定模块83，能量路径确定模块83可以配置成用于确定所述融合能量图的第一图边至第二图边对应的最小能量路径；其中，所述第一图边和所述第二图边可以为具有相对关系的图边。
掩模获取模块84，掩模获取模块84可以配置成用于基于所述最小能量路径确定所述第一重叠图像对应的第一掩模和所述第二重叠图像对应的第二掩模；其中，所述第一掩模与所述第二掩模对应像素的像素值之和均可以为1。
第一拼接模块85，第一拼接模块85可以配置成用于基于所述第一掩模和所述第二掩模拼接所述第一图像和所述第二图像，得到拼接图像。
上述融合图像确定模块82还可以配置成用于：确定所述第一重叠图像对应的第一边缘能量图和所述第二重叠图像对应的第二边缘能量图；确定所述第一重叠图像对应的第一显著能量图和所述第二重叠图像对应的第二显著能量图；合并所述第一显著能量图和所述第二显著能量图，得到显著能量合并图；根据所述显著能量合并图、所述第一边缘能量图和所述第二边缘能量图确定所述第一重叠图像和所述第二重叠图像对应的融合能量图。
上述融合图像确定模块82还可以配置成用于：根据所述第一图像和所述第二图像的拼接方式，确定所述第一边缘能量图对应的增强能量区域；其中，所述拼接方式可以包括左右拼接或上下拼接；对所述增强能量区域中的每个像素乘以第一设定值，得到第一边缘增强能量图；其中，所述第一设定值可以大于1；根据所述显著能量合并图、所述第一边缘增强能量图和所述第二边缘能量图确定所述第一重叠图像和所述第二重叠图像对应的融合能量图。
上述融合图像确定模块82还可以配置成用于：对所述显著能量合并图、所述第一边缘增强能量图和所述第二边缘能量图中的每个对应像素进行加权求和，得到所述第一重叠图像和所述第二重叠图像对应的融合能量图。
在上述装置的基础上，上述目标位置还可以包括所述第一重叠图像的黑边区域对应的位置和/或所述第二重叠图像的黑边区域对应的位置；上述融合图像确定模块82还可以配置成用于：如果所述第一重叠图像包含黑边区域，获取所述第 一重叠图像的黑边区域对应的黑边区域能量图；如果所述第二重叠图像包含黑边区域，获取所述第二重叠图像的黑边区域对应的黑边区域能量图；将获取到的所述黑边区域能量图中的每个像素乘以第二设定值，得到黑边区域能量增强图，其中，所述第二设定值可以大于1；对所述显著能量合并图、所述第一边缘增强能量图、所述第二边缘能量图和所述黑边区域能量增强图中的每个对应像素进行加权求和，得到所述第一重叠图像和所述第二重叠图像对应的融合能量图。
上述融合图像确定模块82还可以配置成用于：对于显著能量合并图中的每个像素，该像素对应的像素值max_value＝max(tar_overlap_salient，src_overlap_salient)；其中，tar_overlap_salient为所述第一显著能量图中的该像素对应的第一像素值；src_overlap_salient为所述第二显著能量图中的该像素对应的第二像素值；max函数为取所述第一像素值和所述第二像素值中的最大值。
在上述实施例的基础上，参见图11，本申请再又一实施例还提供一种图像拼接的处理装置，该装置可以应用于上述电子设备，该装置可以包括如下模块：
图像获取模块91，图像获取模块91可以配置成用于获取待拼接的第一图像和第二图像的重叠区域分别对应的第一重叠图像和第二重叠图像；
图像分割模块92，图像分割模块92可以配置成用于将所述第一重叠图像和所述第二重叠图像输入训练后的图像分割网络，得到所述图像分割网络输出的图像分割结果；其中，所述图像分割网络训练过程中的图像样本对可以对应有掩模真值对，所述掩模真值对可以是基于所述图像样本对对应的融合能量图对应的最小能量路径确定的，其中，所述融合能量图中对应于所述图像样本对的目标位置处像素的像素能量值可以高于其它位置处像素的像素能量值；所述目标位置可以包括边缘位置和显著区域对应的位置；
掩模确定模块93，掩模确定模块93可以配置成用于基于所述图像分割结果确定所述第一重叠图像对应的第一掩模和所述第二重叠图像对应的第二掩模；其中，所述第一掩模与所述第二掩模对应像素的像素值之和均可以为1；
第二拼接模块94，第二拼接模块94可以配置成用于基于所述第一掩模和所述第二掩模拼接所述第一图像和所述第二图像，得到拼接图像。
在上述装置的基础上，参见图12，本申请实施例提供了又一种图像拼接的处理装置，该装置除了包括上述模块之外，还可以包括训练模块95，训练模块95可以配置成用于：获取图像样本对和所述图像样本对对应的掩模真值对；将所述图像样本对输入所述图像分割初始网络，得到所述图像样本对对应的掩模预测对；根据所述图像样本对对应的掩模真值对和掩模预测对计算训练损失值；根据所述训练损失值，判断是否满足预设的停止训练条件；如果否，根据所述训练损失值调整所述图像分割初始网络参数，继续训练；如果是，停止训练，将当前的图像分割初始网络作为图像分割网络。
上述训练模块95还可以配置成用于：对图像初始样本进行预处理，得到所述图像初始样本对应的图像变换样本；其中，所述预处理可以包括以下至少之一：旋转、平移、仿射变换；为所述图像初始样本和/或所述图像变换样本添加黑边区域，得到所述图像样本对。
上述训练模块95还可以配置成用于：通过边缘检测方法获取所述图像样本对中的图像初始样本对应的第一边缘能量图和所述图像变换样本对应的第二边缘能量图；通过显著性目标检测模型获取所述图像初始样本对应的第一显著能量图和所述图像变换样本对应的第二显著能量图；合并所述第一显著能量图和所述第二显著能量图，得到显著能量合并图；根据所述显著能量合并图、所述第一边缘能量图和所述第二边缘能量图确定所述图像样本对对应的融合能量图，并基于融合能量图和最小能量路径确定所述图像样本对对应的掩模真值对。
上述掩模确定模块93还可以配置成用于：如果所述图像分割结果为所述第一重叠图像对应的第一掩模，应用与所述第一掩模尺寸相同且像素值均为1的图像与所述第一掩模做差，得到所述第二重叠图像对应的第二掩模。
参见图13，本申请另外的实施例还提供再一种图像拼接的处理装置200，处理装置200可以包括：处理器40、存储器41、总线42和通信接口43，处理器40、通信接口43和存储器41可以通过总线42连接，处理器40可以配置成用于执行存储器41中存储的可执行模块，例如计算机程序。
其中，存储器41可能包含高速随机存取存储器(RAM，Random Access Memory)，也可能还包括非易失存储器(non-volatile memory)，例如至少一个磁盘存储器。通过至少一个通信接口43(可以是有线或者无线)实现该系统网元与至少一个其他网元之间的通信连接，可以使用互联网、广域网、本地网、城域网等。
总线42可以是ISA总线、PCI总线或EISA总线等。总线可以分为地址总线、数据总线、控制总线等。为便于表示，图13中仅用一个双向箭头表示，但并不表示仅有一根总线或一种类型的总线。
其中，存储器41可以配置成用于存储程序，处理器40在接收到执行指令后，执行程序，前述本申请实施例中的任 一实施例揭示的流过程定义的装置所执行的方法可以应用于处理器40中，或者由处理器40实现。
处理器40可能是一种集成电路芯片，具有信号的处理能力。在实现过程中，上述方法的各步骤可以通过处理器40中的硬件的集成逻辑电路或者软件形式的指令完成。上述的处理器40可以是通用处理器，包括中央处理器(Central Processing Unit，简称CPU)、网络处理器(Network Processor，简称NP)等；还可以是数字信号处理器(Digital Signal Processing，简称DSP)、专用集成电路(Application Specific Integrated Circuit，简称ASIC)、现成可编程门阵列(Field-Programmable Gate Array，简称FPGA)或者其他可编程逻辑器件、分立门或者晶体管逻辑器件、分立硬件组件。可以实现或者执行本申请实施例中的公开的各方法、步骤及逻辑框图。通用处理器可以是微处理器或者该处理器也可以是任何常规的处理器等。结合本申请实施例所公开的方法的步骤可以直接体现为硬件译码处理器执行完成，或者用译码处理器中的硬件及软件模块组合执行完成。软件模块可以位于随机存储器、闪存、只读存储器、可编程只读存储器或者电可擦写可编程存储器、寄存器等本领域成熟的存储介质中。该存储介质可以位于存储器41中，处理器40可以读取存储器41中的信息，结合其硬件完成上述方法的步骤。
本申请实施例还可以提供一种计算机程序，该计算机程序可以包括计算机可读代码，当该计算机可读代码在计算处理设备上运行时，该计算处理设备可以执行上述方法。
本申请实施例所提供的装置，其实现原理及产生的技术效果和前述方法实施例相同，为简要描述，装置实施例部分未提及之处，可参考前述方法实施例中相应内容。
另外，在本申请实施例的描述中，除非另有明确的规定和限定，术语“安装”、“相连”、“连接”应做广义理解，例如，可以是固定连接，也可以是可拆卸连接，或一体地连接；可以是机械连接，也可以是电连接；可以是直接相连，也可以通过中间媒介间接相连，可以是两个元件内部的连通。对于本领域的普通技术人员而言，可以具体情况理解上述术语在本申请中的具体含义。
在本申请的描述中，需要说明的是，术语“中心”、“上”、“下”、“左”、“右”、“竖直”、“水平”、“内”、“外”等指示的方位或位置关系为基于附图所示的方位或位置关系，仅是为了便于描述本申请和简化描述，而不是指示或暗示所指的装置或元件必须具有特定的方位、以特定的方位构造和操作，因此不能理解为对本申请的限制。此外，术语“第一”、“第二”、“第三”仅用于描述目的，而不能理解为指示或暗示相对重要性。
本申请实施例所提供的进行图像拼接的处理的计算机程序产品，包括存储了处理器可执行的非易失的程序代码的计算机可读存储介质，所述程序代码包括的指令可用于执行前面方法实施例中所述的方法，具体实现可参见方法实施例，在此不再赘述。
所属领域的技术人员可以清楚地了解到，为描述的方便和简洁，上述描述的系统、装置和单元的具体工作过程，可以参考前述方法实施例中的对应过程，在此不再赘述。
在本申请所提供的几个实施例中，应该理解到，所揭露的系统、装置和方法，可以通过其它的方式实现。以上所描述的装置实施例仅仅是示意性的，例如，所述单元的划分，仅仅为一种逻辑功能划分，实际实现时可以有另外的划分方式，又例如，多个单元或组件可以结合或者可以集成到另一个系统，或一些特征可以忽略，或不执行。另一点，所显示或讨论的相互之间的耦合或直接耦合或通信连接可以是通过一些通信接口，装置或单元的间接耦合或通信连接，可以是电性，机械或其它的形式。
所述作为分离部件说明的单元可以是或者也可以不是物理上分开的，作为单元显示的部件可以是或者也可以不是物理单元，即可以位于一个地方，或者也可以分布到多个网络单元上。可以根据实际的需要选择其中的部分或者全部单元来实现本实施例方案的目的。
另外，在本申请各个实施例中的各功能单元可以集成在一个处理单元中，也可以是各个单元单独物理存在，也可以两个或两个以上单元集成在一个单元中。
所述功能如果以软件功能单元的形式实现并作为独立的产品销售或使用时，可以存储在一个处理器可执行的非易失的计算机可读取存储介质中。基于这样的理解，本申请的技术方案本质上或者说对相关技术做出贡献的部分或者该技术方案的部分可以以软件产品的形式体现出来，该计算机软件产品存储在一个存储介质中，包括若干指令用以使得一台计算机设备(可以是个人计算机、服务器、或者网络设备等)执行本申请各个实施例所述方法的全部或部分步骤。而前述的存储介质包括：U盘、移动硬盘、只读存储器(ROM，Read-Only Memory)、随机存取存储器(RAM，Random Access Memory)、磁碟或者光盘等各种可以存储程序代码的介质。
最后应说明的是：以上所述实施例，仅为本申请的具体实施方式，用以说明本申请的技术方案，而非对其限制，本 申请的保护范围并不局限于此，尽管参照前述实施例对本申请进行了详细的说明，本领域的普通技术人员应当理解：任何熟悉本技术领域的技术人员在本申请揭露的技术范围内，其依然可以对前述实施例所记载的技术方案进行修改或可轻易想到变化，或者对其中部分技术特征进行等同替换；而这些修改、变化或者替换，并不使相应技术方案的本质脱离本申请实施例技术方案的精神和范围，都应涵盖在本申请的保护范围之内。因此，本申请的保护范围应以权利要求的保护范围为准。
工业实用性
本申请公开了图像拼接的处理方法、装置、电子系统、电子设备、计算机可读存储介质以及计算机程序，该方法包括：获取待拼接的第一图像和第二图像的重叠区域分别对应的第一重叠图像和第二重叠图像；确定第一重叠图像和第二重叠图像对应的融合能量图；确定融合能量图的第一图边至第二图边对应的最小能量路径；基于最小能量路径确定第一重叠图像对应的第一掩模和第二重叠图像对应的第二掩模；基于第一掩模和第二掩模拼接第一图像和第二图像。本申请在一定程度上优化了拼缝经过的区域，提升了图像的拼接效果。
此外，可以理解的是，本申请的图像拼接的处理方法、装置、电子系统、电子设备、计算机可读存储介质以及计算机程序是可以重现的，并且可以应用在多种工业应用中。例如，本申请的图像拼接的处理方法可以应用于进行图像处理的领域。

Claims (20)

  1. 一种图像拼接的处理方法，其特征在于，所述方法应用于电子设备，所述方法包括：
    获取待拼接的第一图像和第二图像的重叠区域分别对应的第一重叠图像和第二重叠图像；
    确定所述第一重叠图像和所述第二重叠图像对应的融合能量图；其中，所述融合能量图中对应于所述第一重叠图像和所述第二重叠图像中的目标位置处像素的像素能量值高于其它位置处像素的像素能量值；所述目标位置包括边缘位置和显著区域对应的位置；
    确定所述融合能量图的第一图边至第二图边对应的最小能量路径；其中，所述第一图边和所述第二图边为具有相对关系的图边；
    基于所述最小能量路径确定所述第一重叠图像对应的第一掩模和所述第二重叠图像对应的第二掩模；其中，所述第一掩模与所述第二掩模对应像素的像素值之和均为1；
    基于所述第一掩模和所述第二掩模拼接所述第一图像和所述第二图像，得到拼接图像。
  2. 根据权利要求1所述的方法，其特征在于，确定所述第一重叠图像和所述第二重叠图像对应的融合能量图的步骤包括：
    确定所述第一重叠图像对应的第一边缘能量图和所述第二重叠图像对应的第二边缘能量图；
    确定所述第一重叠图像对应的第一显著能量图和所述第二重叠图像对应的第二显著能量图；
    合并所述第一显著能量图和所述第二显著能量图，得到显著能量合并图；
    根据所述显著能量合并图、所述第一边缘能量图和所述第二边缘能量图确定所述第一重叠图像和所述第二重叠图像对应的融合能量图。
  3. 根据权利要求2所述的方法，其特征在于，根据所述显著能量合并图、所述第一边缘能量图和所述第二边缘能量图确定所述第一重叠图像和所述第二重叠图像对应的融合能量图的步骤包括：
    根据所述第一图像和所述第二图像的拼接方式，确定所述第一边缘能量图对应的增强能量区域；其中，所述拼接方式包括左右拼接或上下拼接；
    对所述增强能量区域中的每个像素乘以第一设定值，得到第一边缘增强能量图；其中，所述第一设定值大于1；
    根据所述显著能量合并图、所述第一边缘增强能量图和所述第二边缘能量图确定所述第一重叠图像和所述第二重叠图像对应的融合能量图。
  4. 根据权利要求3所述的方法，其特征在于，当所述拼接方式为左右拼接、所述第一图像位于左侧时，将所述第一边缘能量图中靠近左侧的一部分区域确定为所述第一边缘能量图对应的增强能量区域。
  5. 根据权利要求3或4所述的方法，其特征在于，根据所述显著能量合并图、所述第一边缘增强能量图和所述第二边缘能量图确定所述第一重叠图像和所述第二重叠图像对应的融合能量图的步骤包括：
    对所述显著能量合并图、所述第一边缘增强能量图和所述第二边缘能量图中的每个对应像素进行加权求和，得到所述第一重叠图像和所述第二重叠图像对应的融合能量图。
  6. 根据权利要求3至5中的任一项所述的方法，其特征在于，根据所述显著能量合并图、所述第一边缘增强能量图和所述第二边缘能量图确定所述第一重叠图像和所述第二重叠图像对应的融合能量图的步骤包括：
    如果所述第一重叠图像包含黑边区域，获取所述第一重叠图像的黑边区域对应的黑边区域能量图；
    如果所述第二重叠图像包含黑边区域，获取所述第二重叠图像的黑边区域对应的黑边区域能量图；
    将获取到的所述黑边区域能量图中的每个像素乘以第二设定值，得到黑边区域能量增强图，其中，所述第二设定值大于1；
    对所述显著能量合并图、所述第一边缘增强能量图、所述第二边缘能量图和所述黑边区域能量增强图中的每个对应像素进行加权求和，得到所述第一重叠图像和所述第二重叠图像对应的融合能量图。
  7. 根据权利要求2至6中的任一项所述的方法，其特征在于，合并所述第一显著能量图和所述第二显著能量图而得到显著能量合并图的步骤包括：对于所述第一显著能量图和所述第二显著能量图中的每个对应像素，采用像素值直接求 和或像素值加权求和的方式得到新的像素值，得到的新的像素值即为所述显著能量合并图的像素值。
  8. 根据权利要求2至6中的任一项所述的方法，其特征在于，合并所述第一显著能量图和所述第二显著能量图而得到显著能量合并图的步骤包括：
    对于显著能量合并图中的每个像素，该像素对应的像素值max_value＝max(tar_overlap_salient，src_overlap_salient)；其中，tar_overlap_salient为所述第一显著能量图中的该像素对应的第一像素值；src_overlap_salient为所述第二显著能量图中的该像素对应的第二像素值；max函数为取所述第一像素值和所述第二像素值中的最大值。
  9. 一种图像拼接的处理方法，其特征在于，所述方法应用于电子设备，所述方法包括：
    获取待拼接的第一图像和第二图像的重叠区域分别对应的第一重叠图像和第二重叠图像；
    将所述第一重叠图像和所述第二重叠图像输入训练后的图像分割网络，得到所述图像分割网络输出的图像分割结果；其中，所述图像分割网络训练过程中的图像样本对对应有掩模真值对，所述掩模真值对是基于所述图像样本对对应的融合能量图对应的最小能量路径确定的，其中，所述融合能量图中对应于所述图像样本对的目标位置处像素的像素能量值高于其它位置处像素的像素能量值；所述目标位置包括边缘位置和显著区域对应的位置；
    基于所述图像分割结果确定所述第一重叠图像对应的第一掩模和所述第二重叠图像对应的第二掩模；其中，所述第一掩模与所述第二掩模对应像素的像素值之和均为1；
    基于所述第一掩模和所述第二掩模拼接所述第一图像和所述第二图像，得到拼接图像。
  10. 根据权利要求9所述的方法，其特征在于，所述图像分割网络的训练过程包括：
    获取图像样本对和所述图像样本对对应的掩模真值对；
    将所述图像样本对输入所述图像分割初始网络，得到所述图像样本对对应的掩模预测对；
    根据所述图像样本对对应的掩模真值对和掩模预测对计算训练损失值；
    根据所述训练损失值，判断是否满足停止训练条件；
    如果否，根据所述训练损失值调整所述图像分割初始网络参数，继续训练；
    如果是，停止训练，将当前的图像分割初始网络作为所述图像分割网络。
  11. 根据权利要求10所述的方法，其特征在于，获取所述图像样本对的步骤包括：
    对图像初始样本进行目标处理，得到所述图像初始样本对应的图像变换样本；其中，所述目标处理包括以下至少之一：旋转、平移、仿射变换；
    为所述图像初始样本和/或所述图像变换样本添加黑边区域，得到所述图像样本对。
  12. 根据权利要求11所述的方法，其特征在于，获取所述图像样本对对应的掩模真值对的步骤包括：
    通过边缘检测方法获取所述图像样本对中的图像初始样本对应的第一边缘能量图和所述图像变换样本对应的第二边缘能量图；
    通过显著性目标检测模型获取所述图像初始样本对应的第一显著能量图和所述图像变换样本对应的第二显著能量图；
    合并所述第一显著能量图和所述第二显著能量图，得到显著能量合并图；
    根据所述显著能量合并图、所述第一边缘能量图和所述第二边缘能量图确定所述图像样本对对应的融合能量图，并基于所述融合能量图和最小能量路径确定所述图像样本对对应的掩模真值对。
  13. 根据权利要求9至12中的任一项所述的方法，其特征在于，基于所述图像分割结果确定所述第一重叠图像对应的第一掩模和所述第二重叠图像对应的第二掩模的步骤包括：
    如果所述图像分割结果为所述第一重叠图像对应的第一掩模，应用与所述第一掩模尺寸相同且像素值均为1的图像与所述第一掩模做差，得到所述第二重叠图像对应的第二掩模。
  14. 根据权利要求9至12中的任一项所述的方法，其特征在于，基于所述图像分割结果确定所述第一重叠图像对应的第一掩模和所述第二重叠图像对应的第二掩模的步骤包括：
    如果所述图像分割结果为所述第二重叠图像对应的第二掩模，应用与所述第二掩模尺寸相同且像素值均为1的图像与所述第二掩模做差，得到所述第一重叠图像对应的第一掩模。
  15. 一种图像拼接的处理装置，其特征在于，所述装置应用于电子设备，所述装置包括：
    重叠图像获取模块，所述重叠图像获取模块配置成用于获取待拼接的第一图像和第二图像的重叠区域分别对应的第一重叠图像和第二重叠图像；
    融合图像确定模块，所述融合图像确定模块配置成用于确定所述第一重叠图像和所述第二重叠图像对应的融合能量图；其中，所述融合能量图中对应于所述第一重叠图像和所述第二重叠图像中的目标位置处像素的像素能量值高于其它位置处像素的像素能量值；所述目标位置包括边缘位置和显著区域对应的位置；
    能量路径确定模块，所述能量路径确定模块配置成用于确定所述融合能量图的第一图边至第二图边对应的最小能量路径；其中，所述第一图边和所述第二图边为具有相对关系的图边；
    掩模获取模块，所述掩模获取模块配置成用于基于所述最小能量路径确定所述第一重叠图像对应的第一掩模和所述第二重叠图像对应的第二掩模；其中，所述第一掩模与所述第二掩模对应像素的像素值之和均为1；
    第一拼接模块，所述第一拼接模块配置成用于基于所述第一掩模和所述第二掩模拼接所述第一图像和所述第二图像，得到拼接图像。
  16. 一种图像拼接的处理装置，其特征在于，所述装置应用于电子设备，所述装置包括：
    图像获取模块，所述图像获取模块配置成用于获取待拼接的第一图像和第二图像的重叠区域分别对应的第一重叠图像和第二重叠图像；
    图像分割模块，所述图像分割模块配置成用于将所述第一重叠图像和所述第二重叠图像输入训练后的图像分割网络，得到所述图像分割网络输出的图像分割结果；其中，所述图像分割网络训练过程中的图像样本对对应有掩模真值对，所述掩模真值对是基于所述图像样本对对应的融合能量图对应的最小能量路径确定的，其中，所述融合能量图中对应于所述图像样本对的目标位置处像素的像素能量值高于其它位置处像素的像素能量值；所述目标位置包括边缘位置和显著区域对应的位置；
    掩模确定模块，所述掩模确定模块配置成用于基于所述图像分割结果确定所述第一重叠图像对应的第一掩模和所述第二重叠图像对应的第二掩模；其中，所述第一掩模与所述第二掩模对应像素的像素值之和均为1；
    第二拼接模块，所述第二拼接模块配置成用于基于所述第一掩模和所述第二掩模拼接所述第一图像和所述第二图像，得到拼接图像。
  17. 一种电子系统，其特征在于，所述电子系统包括：图像采集设备、处理设备和存储装置；
    所述图像采集设备配置成用于获取图像；
    所述存储装置上存储有计算机程序，所述计算机程序在被所述处理设备运行时执行根据权利要求1至8中的任一项所述的方法，或者，所述计算机程序在被所述处理设备运行时执行根据权利要求9至14中的任一项所述的方法。
  18. 一种电子设备，其特征在于，所述电子设备包括处理器和存储器，所述处理器配置成在运行时执行根据权利要求1至8中的任一项所述的方法，或者，所述处理器在运行时执行根据权利要求9至14中的任一项所述的方法。
  19. 一种计算机可读存储介质，所述计算机可读存储介质上存储有计算机程序，其特征在于，所述计算机程序被处理器运行时执行根据权利要求1至8中的任一项所述的方法的步骤，或者，所述计算机程序被处理器运行时执行根据权利要求9至14中的任一项所述的方法的步骤。
  20. 一种计算机程序产品，其特征在于，所述计算机程序产品包括计算机程序，当所述计算机程序被处理器执行时实现根据权利要求1至8中的任一项所述的方法的步骤或者实现根据权利要求9至14中的任一项所述的方法的步骤。
PCT/CN2021/134126 2021-02-09 2021-11-29 图像拼接的处理方法、装置、电子系统和设备、可读介质 WO2022170824A1 (zh)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
CN202110186621.3 2021-02-09
CN202110186621.3A CN112862685B (zh) 2021-02-09 2021-02-09 图像拼接的处理方法、装置和电子系统

Publications (1)

Publication Number Publication Date
WO2022170824A1 true WO2022170824A1 (zh) 2022-08-18

Family

ID=75988140

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2021/134126 WO2022170824A1 (zh) 2021-02-09 2021-11-29 图像拼接的处理方法、装置、电子系统和设备、可读介质

Country Status (2)

Country Link
CN (1) CN112862685B (zh)
WO (1) WO2022170824A1 (zh)

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN116704376A (zh) * 2023-08-04 2023-09-05 航天宏图信息技术股份有限公司 基于单卫星图像的nDSM提取方法、装置及电子设备

Families Citing this family (9)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN112862685B (zh) * 2021-02-09 2024-02-23 北京迈格威科技有限公司 图像拼接的处理方法、装置和电子系统
CN113793382B (zh) * 2021-08-04 2024-10-18 北京旷视科技有限公司 视频图像的拼缝搜索方法、视频图像的拼接方法和装置
CN113902657A (zh) * 2021-08-26 2022-01-07 北京旷视科技有限公司 图像拼接方法、装置和电子设备
CN114143517A (zh) * 2021-10-26 2022-03-04 深圳华侨城卡乐技术有限公司 一种基于重叠区域的融合蒙板计算方法、系统及存储介质
CN114387229A (zh) * 2021-12-24 2022-04-22 佛山欧神诺云商科技有限公司 瓷砖的拼接检测方法、装置、计算机设备及可读存储介质
CN114723757A (zh) * 2022-06-09 2022-07-08 济南大学 一种基于深度学习算法的高精度晶圆缺陷检测方法及系统
CN115131549A (zh) * 2022-06-15 2022-09-30 浙江大学 一种基于自提升学习的显著性目标检测训练方法
CN115471403B (zh) * 2022-10-18 2024-02-13 如你所视(北京)科技有限公司 图像处理方法、装置及存储介质
CN115797210A (zh) * 2022-12-05 2023-03-14 深圳看到科技有限公司 用于水下拍摄图片的校正方法、装置及存储介质

Citations (7)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20160286137A1 (en) * 2013-09-16 2016-09-29 Duke University Method for combining multiple image fields
CN106204433A (zh) * 2015-05-27 2016-12-07 三星电子株式会社 用于显示医学图像的方法和设备
CN107610051A (zh) * 2017-09-22 2018-01-19 北京林业大学 无人机航拍图像拼接的正射图像生成方法及装置
US20180061006A1 (en) * 2016-08-26 2018-03-01 Multimedia Image Solution Limited Method for ensuring perfect stitching of a subject's images in a real-site image stitching operation
CN110782424A (zh) * 2019-11-08 2020-02-11 重庆紫光华山智安科技有限公司 图像融合方法、装置、电子设备及计算机可读存储介质
CN111915483A (zh) * 2020-06-24 2020-11-10 北京迈格威科技有限公司 图像拼接方法、装置、计算机设备和存储介质
CN112862685A (zh) * 2021-02-09 2021-05-28 北京迈格威科技有限公司 图像拼接的处理方法、装置和电子系统

Family Cites Families (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN104680501B (zh) * 2013-12-03 2018-12-07 华为技术有限公司 图像拼接的方法及装置
CN106940879B (zh) * 2017-03-27 2020-07-03 微鲸科技有限公司 一种图像拼接方法及装置
CN107146201A (zh) * 2017-05-08 2017-09-08 重庆邮电大学 一种基于改进图像融合的图像拼接方法
CN107464230B (zh) * 2017-08-23 2020-05-08 京东方科技集团股份有限公司 图像处理方法及装置

Patent Citations (7)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20160286137A1 (en) * 2013-09-16 2016-09-29 Duke University Method for combining multiple image fields
CN106204433A (zh) * 2015-05-27 2016-12-07 三星电子株式会社 用于显示医学图像的方法和设备
US20180061006A1 (en) * 2016-08-26 2018-03-01 Multimedia Image Solution Limited Method for ensuring perfect stitching of a subject's images in a real-site image stitching operation
CN107610051A (zh) * 2017-09-22 2018-01-19 北京林业大学 无人机航拍图像拼接的正射图像生成方法及装置
CN110782424A (zh) * 2019-11-08 2020-02-11 重庆紫光华山智安科技有限公司 图像融合方法、装置、电子设备及计算机可读存储介质
CN111915483A (zh) * 2020-06-24 2020-11-10 北京迈格威科技有限公司 图像拼接方法、装置、计算机设备和存储介质
CN112862685A (zh) * 2021-02-09 2021-05-28 北京迈格威科技有限公司 图像拼接的处理方法、装置和电子系统

Cited By (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN116704376A (zh) * 2023-08-04 2023-09-05 航天宏图信息技术股份有限公司 基于单卫星图像的nDSM提取方法、装置及电子设备
CN116704376B (zh) * 2023-08-04 2023-10-20 航天宏图信息技术股份有限公司 基于单卫星图像的nDSM提取方法、装置及电子设备

Also Published As

Publication number Publication date
CN112862685B (zh) 2024-02-23
CN112862685A (zh) 2021-05-28

Similar Documents

Publication Publication Date Title
WO2022170824A1 (zh) 图像拼接的处理方法、装置、电子系统和设备、可读介质
US10972672B2 (en) Device having cameras with different focal lengths and a method of implementing cameras with different focal lengths
US10389948B2 (en) Depth-based zoom function using multiple cameras
JP6316968B2 (ja) 対話型画像合成
US10909703B2 (en) Image processing method, electronic device and computer-readable storage medium
WO2020007320A1 (zh) 多视角图像的融合方法、装置、计算机设备和存储介质
JP6371553B2 (ja) 映像表示装置および映像表示システム
WO2021164234A1 (zh) 图像处理方法以及图像处理装置
CN111915483B (zh) 图像拼接方法、装置、计算机设备和存储介质
WO2017088533A1 (zh) 合并图像的方法和装置
KR20230084486A (ko) 이미지 효과를 위한 세그먼트화
JP2002208005A (ja) 画像処理装置、画像処理方法、画像処理プログラムおよび画像処理プログラムを記録したコンピュータ読み取り可能な記録媒体
CN111553841B (zh) 一种基于最佳缝合线更新的实时视频拼接方法
CN105023260A (zh) 一种全景图像融合方法及融合装置
CN114040179B (zh) 一种图像的处理方法及装置
US20190213749A1 (en) Method, image processing device, and system for generating disparity map
EP4200799A1 (en) Spatiotemporal recycling network
CN117173012A (zh) 无监督的多视角图像生成方法、装置、设备及存储介质
CN113610884B (zh) 图像处理方法、装置、电子设备及计算机可读存储介质
CN113436068B (zh) 一种图像拼接方法、装置、电子设备及存储介质
US12094079B2 (en) Reference-based super-resolution for image and video enhancement
CN115619636A (zh) 图像拼接方法、电子设备以及存储介质
WO2022182293A1 (en) Method and system for image stitching
CN114359162A (zh) 显著性检测方法、显著性检测设备和存储介质
Xiong Eliminating ghosting artifacts for panoramic images

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 21925471

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

32PN Ep: public notification in the ep bulletin as address of the adressee cannot be established

Free format text: NOTING OF LOSS OF RIGHTS PURSUANT TO RULE 112(1) EPC (EPO FORM 1205A DATED 27.11.2023)

122 Ep: pct application non-entry in european phase

Ref document number: 21925471

Country of ref document: EP

Kind code of ref document: A1