CN115797164B - Image stitching method, device and system in fixed view field - Google Patents

Abstract

The present disclosure relates to the field of image processing technologies, and in particular, to a method, an apparatus, a system, an electronic device, a storage medium, and a program for image stitching in a fixed field of view. An image stitching method comprises the steps of obtaining an image sequence, wherein the image sequence at least comprises a plurality of frame images of a moving target; determining the same position in the plurality of frame images as a splicing position; respectively determining t _n Time frame image and t _n+1 The foreground of the moving target in the time frame image is extracted, and the contextual characteristics of the image area at the foreground are extracted; determining a moving object at t according to the context characteristics _n Time frame image and t _n+1 A moving distance between time frame images; respectively corresponding to t according to the moving distance and the splicing position _n Time frame image and t _n+1 Cutting the time frame image to obtain a cut image; and splicing the clipping images. The method can realize the spliced display of the moving target in the fixed view field.

Description

Image splicing method, device and system in fixed field of view

Technical field

The present disclosure relates to the field of image processing technology, and more specifically, to an image splicing method, device, system, electronic device, storage medium and program product in a fixed field of view.

Background technique

Generalized image stitching is to stitch together several overlapping static images (which may be obtained at different times, different perspectives or different sensors) into a seamless panorama or high-resolution image. Existing image stitching technology generally includes two key technologies: image registration and image fusion. Image registration refers to using image processing methods to calculate the matching relationship between spliced images, obtain overlapping areas, and realize image splicing. Image fusion is to realize the smooth transition of the spliced images at the splicing point.

However, when the field of view of the camera or other shooting equipment is fixed and the splicing object is a moving target in the scene (for example, splicing a large moving vehicle, splicing objects conveyed on a conveyor belt, etc.), for the existing splicing methods, due to There is a background with the same field of view in the images to be stitched. When feature matching is used, it will lead to wrong image registration and thus wrong stitching results.

Contents of the invention

In view of this, embodiments of the present disclosure provide an image stitching method, device, system, electronic device, storage medium and program product.

One aspect of the present disclosure provides an image splicing method in a fixed field of view, including: acquiring an image sequence, the image sequence at least including a plurality of frame images containing a moving target; determining the same frame image in a plurality of the frame images. The position is the splicing position; respectively determine the foreground of the moving target in the frame image at time t _n and the frame image at time t _n+1 , and perform context feature extraction on the image area at the foreground; determine the The moving distance of the moving target between the frame image at time t _n and the frame image at time t _n+1 ; according to the moving distance and the splicing position, the frame image at time t _n and the frame image at t _n+ are respectively ₁ time frame image is trimmed to obtain a trimmed image; the trimmed images are spliced.

In some embodiments, before determining the foreground of the moving target in the frame image at time t _n and the frame image at time t _n+1 respectively, the method further includes: obtaining a background image in the frame image; and comparing the background image and the The frame images at time t _n are respectively subjected to mixed Gaussian modeling; based on the comparison results of the Gaussian mixture model of the background image and the Gaussian mixture model of the frame image at time t _n , it is determined whether the frame image at time t _n contains Moving target.

In some embodiments, when it is determined that the moving target is not included in the frame image at time _tn , the Gaussian mixture model of the frame image at time _tn is updated to the Gaussian mixture model of the background image.

In some embodiments, determining the same position in multiple frame images as the splicing position includes: setting a row or column of the frame image as the splicing position, and the splicing position satisfies: in the In the direction of movement of the moving target, the splicing position is located in front of the moving target in the frame image at time t _m , which is the first frame image containing the _moving target, m is a positive integer and m is less than or equal to n.

In some embodiments, determining the moving distance of the moving target between the frame image at time t _n and the frame image at time t _n+1 according to the contextual characteristics includes: comparing the frame image at time t _n The contextual features of the image and the contextual features of the frame image at time t _n+1 are feature matched, and the feature matching pair is obtained; according to the feature matching pair, the frame image at time tn and the frame image at time t _n+1 are obtained. The coordinate information in the frame image is used to calculate the moving distance of the moving target between the frame image at time t _n and the frame image at time t _n+1 .

In some embodiments, extracting contextual features from the image area at the foreground includes: extracting multiple contextual features within the image area at the foreground; extracting the frame image at time t _n Feature matching between the context features of the frame image at time t _n+1 and the context features of the frame image at time t _n +1 includes: combining multiple context features of the frame image at time t n and multiple contexts of the frame image at time t _n+1 Feature matching is performed using a combination of features.

In some embodiments, the frame image of the moving target at time t _n is calculated based on the feature matching pair of coordinate information in the frame image at time t n and the frame image at time t _{n +1} and the moving distance between the frame images at time t _n+1 includes: calculating the moving distance based on the average absolute difference or the intermediate value of the absolute differences of the coordinate information of the plurality of feature matching pairs.

In certain embodiments, at least one of said contextual features is related to an operating mode of a device that acquired said image sequence and/or an operating environment of said device.

In some embodiments, clipping the frame image at time t _n and the frame image at time t _n+1 according to the movement distance and the splicing position includes: determining a clipping area, the clipping area In order to use the splicing position as the starting point for clipping, a region whose length is the movement distance is defined along the opposite movement direction of the moving target with the clipping starting point.

In some embodiments, obtaining the image sequence includes: obtaining a video to be spliced, and decoding the video to obtain the image sequence.

In some embodiments, determining the foreground of the moving target in the frame image at time t _n and the frame image at time t _n+1 respectively includes: using a morphological method to determine the frame image at time t _n and the frame at time t _n+1 The foreground of the moving target described in the image.

In some embodiments, after the splicing of the cropped images, if it is determined that the moving target is not included in the frame image at time t _n+2 , the splicing of the cropped images will be terminated.

Another aspect of the present disclosure provides an image splicing device in a fixed field of view, including: an image sequence acquisition module, used to acquire an image sequence, the image sequence at least includes a plurality of frame images including a moving target; a splicing position The setting module is used to determine the same position in multiple frame images as the splicing position; the foreground determination module is used to determine the foreground of the moving target in the frame image at time t _n and the frame image at time t _n+1 respectively; A contextual feature extraction module, used to extract contextual features from the image area at the foreground; a moving distance calculation module, used to determine the frame image of the moving target at the time t n and the frame image of the moving target at _the t _n time according to the contextual features The movement distance between the frame images at time ₊₁ ; a clipping module, configured to clip the frame image at time t _n and the frame image at time t _n+1 respectively according to the movement distance and the splicing position to obtain a clipped image ; Splicing module, splicing the cropped images.

In some embodiments, it also includes: a background image acquisition module, used to acquire the background image in the frame image; and a mixed Gaussian modeling module, used to perform mixed Gaussian construction on the background image and the tn time frame image respectively. module; a moving target judgment module, configured to judge whether the t _n time frame image contains a moving target based on the comparison result of the mixed Gaussian model of the background image and the t _n time frame image.

In some embodiments, it also includes: a feature matching module, configured to perform feature matching on the contextual features of the frame image at time t _n and the contextual features of the frame image at time t _n+1 , and obtain the feature matching. right.

In some embodiments, it also includes: a video processing module, configured to obtain the video to be spliced and decode the video.

In some embodiments, the method further includes: a Gaussian mixture model update module, configured to update the Gaussian mixture model of the frame image at time t _n to the Gaussian mixture model when it is determined that the frame image at time t _n does not contain a moving target. Gaussian mixture model of the background image.

Another aspect of the present disclosure also provides an image splicing system in a fixed field of view, including: the image splicing device in a fixed field of view described in any one of the above, and an image acquisition device to collect and form images containing There are videos of moving targets as well as captured image backgrounds.

Another aspect of the present disclosure also provides an electronic device, including: one or more processors; a storage device for storing one or more programs, wherein when the one or more programs are processed by the one or more When executed by multiple processors, the one or more processors are caused to execute any of the methods described above.

Another aspect of the present disclosure also provides a computer-readable storage medium having executable instructions stored thereon, which when executed by a processor causes the processor to perform any one of the methods described above.

Another aspect of the present disclosure also provides a computer program product, including a computer program that implements any of the above methods when executed by a processor.

The image splicing method according to the embodiment of the present disclosure includes obtaining an image sequence, which at least includes multiple frame images containing moving targets; determining the same position in the multiple frame images as the splicing position; determining the frame image at time t _n and t respectively. The foreground of the moving target in the frame image at time _n+1 , and context feature extraction is performed on the image area in the foreground; determine the moving distance of the moving target between the frame image at time _tn and the frame image at time tn ₊₁ based on the context features; based on the movement The distance and splicing position are used to clip the frame image at time t _n and the frame image at time t _n+1 respectively to obtain a clipped image; the clipped images are spliced. In the method disclosed in the present disclosure, the moving distance of the moving object between adjacent frame images is determined by comparing the contextual features. According to the moving distance and the set splicing position, the frame image is trimmed to obtain the moving target part. Each trimming is By acquiring partial images of the moving target, and finally splicing the cropped areas of all frame images, a panoramic display of the moving target can be achieved.

Description of drawings

The above and other objects, features and advantages of the present disclosure will become more apparent from the following description of embodiments of the present disclosure with reference to the accompanying drawings, in which:

Figure 1 schematically shows a flow chart of a splicing method for moving targets in a fixed field of view according to an embodiment of the present disclosure;

Figure 2 schematically shows a flow chart of another embodiment of a splicing method for moving targets in a fixed field of view according to an embodiment of the present disclosure;

Figure 3 schematically shows a flow chart of another embodiment of a splicing method for moving targets in a fixed field of view according to an embodiment of the present disclosure;

Figure 4 schematically shows a flow chart of another embodiment of a splicing method for moving targets in a fixed field of view according to an embodiment of the present disclosure;

Figure 5 schematically shows a schematic diagram of a frame image of a moving target at time t _n according to an embodiment of the present disclosure;

Figure 6 schematically shows a schematic diagram of a frame image of a moving target at time t _n+1 according to an embodiment of the present disclosure;

Figure 7 schematically shows a schematic diagram of a frame image of a moving target at time t _n+2 according to an embodiment of the present disclosure;

Figure 8 schematically shows a structural block diagram of a splicing device for moving targets in a fixed field of view according to an embodiment of the present disclosure;

FIG. 9 schematically shows a block diagram of an electronic device suitable for implementing a splicing method of moving targets in a fixed field of view according to an embodiment of the present disclosure.

Detailed ways

Hereinafter, embodiments of the present disclosure will be described with reference to the accompanying drawings. It should be understood, however, that these descriptions are exemplary only and are not intended to limit the scope of the present disclosure. In the following detailed description, for convenience of explanation, numerous specific details are set forth to provide a comprehensive understanding of the disclosed embodiments. It will be apparent, however, that one or more embodiments may be practiced without these specific details. Furthermore, in the following description, descriptions of well-known structures and techniques are omitted to avoid unnecessarily confusing the concepts of the present disclosure.

The terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the disclosure. The terms "comprising," "comprising," and the like, as used herein, indicate the presence of stated features, steps, operations, and/or components but do not exclude the presence or addition of one or more other features, steps, operations, or components.

Where an expression similar to "at least one of A, B or C, etc." is used, it should generally be interpreted in accordance with the meaning that a person skilled in the art generally understands the expression to mean (for example, "having A, B or C "A system with at least one of" shall include, but is not limited to, systems with A alone, B alone, C alone, A and B, A and C, B and C, and/or systems with A, B, C, etc. ). The terms “first” and “second” are used for descriptive purposes only and shall not be understood as indicating or implying relative importance or implicitly indicating the quantity of indicated technical features. Thus, features defined as “first” and “second” may explicitly or implicitly include one or more of the described features.

The detailed background technology may include other technical problems in addition to the technical problems that are exclusively solved.

An embodiment of the present disclosure provides an image splicing method in a fixed field of view, including acquiring an image sequence, which at least includes multiple frame images containing a moving target; determining the same position in the multiple frame images as the splicing position; Determine the foreground of the moving target in the frame image at time t _n and the frame image at time t _n+1 respectively, and perform context feature extraction on the image area at the foreground; determine the location of the moving target in the frame image at time t _n and time t _n+1 based on the context features The movement distance between frame images; trim the frame image at time t _n and the frame image at time t _n+1 respectively according to the movement distance and splicing position to obtain a trimmed image; splice the trimmed images.

The image splicing method in the embodiment of the present disclosure is suitable for splicing panoramic images of moving objects in a fixed field of view in a fixed field of view environment. It is especially suitable for situations where the moving objects are large in size and the imaging range of the fixed field of view cannot A scene in which the entirety of a moving object is directly imaged.

It should be noted that the moving target in the embodiment of the present disclosure may be a moving object that needs to be photographed in a fixed field of view.

Figure 1 schematically shows an application scene diagram of an image stitching method in a fixed field of view according to an embodiment of the present disclosure.

As shown in Figure 1, the application scenario 100 according to this embodiment may include storage devices 101, 102, and 103. Network 104 is the medium used to provide communication links between storage devices 101, 102, 103 and server 105. Network 104 may include various connection types, such as wired, wireless communication links, or fiber optic cables, among others.

The user can interact with the server 105 through the network 104 using the storage devices 101, 102, 103 to upload image sequences to the server 105 for processing by the server 105.

The storage devices 101, 102, and 103 may be various electronic devices with display screens, including but not limited to smartphones, tablet computers, laptop computers, desktop computers, and the like.

The server 105 may be a server that performs contextual feature recognition, feature matching, image cropping, and image splicing and fusion for image sequences.

It should be noted that the splicing method of moving targets in a fixed field of view provided by the embodiments of the present disclosure can generally be executed by the server 105 . Correspondingly, the splicing device for moving targets in a fixed field of view provided by the embodiments of the present disclosure may generally be installed in the server 105 . The splicing method of moving targets in a fixed field of view provided by the embodiments of the present disclosure can also be executed by a server or server cluster that is different from the server 105 and can communicate with the storage devices 101, 102, 103 and/or the server 105. Correspondingly, the splicing system for moving targets in a fixed field of view provided by the embodiments of the present disclosure can also be set up in a server or server cluster that is different from the server 105 and can communicate with the storage devices 101, 102, 103 and/or the server 105.

It should be understood that the number of storage devices, networks and servers in Figure 1 is only illustrative. Depending on implementation needs, there can be any number of end devices, networks, and servers.

The following will describe in detail the splicing method of moving targets in a fixed field of view according to the embodiment of the present disclosure through FIGS. 2 to 7 based on the scene described in FIG. 1 .

FIG. 2 schematically shows a flow chart of a splicing method for moving targets in a fixed field of view according to an embodiment of the present disclosure.

As shown in FIG. 2 , the splicing method of moving targets in a fixed field of view in this embodiment includes operations S210 to S260.

In operation 210, an image sequence is obtained, where the image sequence at least includes a plurality of frame images containing the moving target at a moving time t.

The image sequence in the embodiment of the present disclosure may include a series of images obtained sequentially and continuously from a moving target at different times, and may also include multiple frame images obtained by processing videos, animations, etc. containing moving targets. In the application scenario of the embodiment of the present disclosure, the moving target moves in a fixed field of view. Correspondingly, the above-mentioned image sequence at least includes the video generated by the entire movement process of the moving target starting from entering the fixed field of view and completely leaving the fixed field of view. After analyzing and processing the above video, multiple frame images corresponding to the motion time t as the axis are obtained.

Combined with the application scenarios of the embodiments of the present disclosure, such as security inspection scenarios for large containers at ports, the shooting device in the fixed field of view will remain in working condition during working hours, but there will not be moving targets in the fixed field of view at all times. In motion, therefore, the image sequence may also include frame images that do not contain moving targets.

The method of obtaining an image sequence in an embodiment of the present disclosure may include: obtaining a video containing a moving target to be spliced, and decoding the video to obtain multiple frame images.

The video may include offline video files captured by cameras in a fixed field of view, and may include online video files uploaded to the Internet by users. The format of the video files may include but is not limited to general video formats such as MPEG, AVI, MOV, and WMV. The format of the decoded frame image may include, but is not limited to, common image formats such as PNG, BMP, or JPG.

The video decoding process in the embodiments of the present disclosure can be implemented by using the frame image processing function in relevant third-party software, such as photoshop, Adobe Premiere and other software.

It can be understood that when decoding a video, the number of frame images can be controlled to ensure that there are repeated parts between adjacent frames with moving targets, so that there is a basis for image splicing in the method provided by the embodiments of the present disclosure.

It should be noted that the frame images obtained by the video decoding process are named after the movement time t of the moving target, in order to facilitate understanding and explanation of the technical concepts of the embodiments of the present disclosure. For example, from the frame image at time t ₀ to the frame image at time t _n , in the following, in order to represent some frame images with special meanings or functions in the frame images, they are named frame images at time t _m to better distinguish them. Sub-explanation.

In operation S220, the same position in the plurality of frame images is determined as a splicing position.

The frame image obtained in the embodiment of the present disclosure is an image under a fixed field of view. Correspondingly, the size of the background image generated corresponding to the fixed field of view is the same. As shown in Figures 5 to 7, the background image is a pixel area including columns a1 to a18 and rows b1 to b14. In this embodiment, column a9 is used as the above-mentioned splicing position, and the splicing position is determined to facilitate subsequent image cropping and splicing.

It can be understood that different fields of view have different sizes of corresponding pixel areas of the background image.

It can be understood that the splicing position can also be a row of pixels in the pixel area, or a polygonal splicing line containing multiple rows or columns of pixels.

In order to further optimize the integrity of the spliced moving target, when setting the splicing position, the following conditions need to be met:

In the direction of motion of the moving target, the splicing position is located in front of the moving _target in the frame image at time t _m , which is the first frame image containing the moving target, and m is Positive integer and m is less than or equal to n.

Referring to Figure 5, the horizontal arrow in Figure 5 identifies the movement direction of the moving target. Assume that the current t _n time is the frame image at t _m time mentioned above, and it is assumed that the moving target is composed of c1, c2, c3 and other parts. In order to make the illustration clearer and avoid more overlaps, only the c1 and c2 parts of the moving target are shown in the drawings in this embodiment. At time t _m , the moving target enters the fixed field of view for the first time. At this time, it is necessary to ensure that the set splicing position is to the left of c1. The purpose of this setting is that in the frame image at time t _m , the splicing position If there is an intersection with the moving target, the moving target located to the left of the splicing position will not be clipped during subsequent image trimming, resulting in the inability to display the panoramic image of the moving target during image splicing.

In operation S230, the foreground of the moving target in the frame image at time t _n and the frame image at time t _n+1 is determined respectively, and context feature extraction is performed on the image area at the foreground.

In the embodiment of the present disclosure, a morphological method is used to determine the foreground of the moving target in the frame image at time t _n and the frame image at time t _n+1 . The background image and the foreground of the moving target are opened using morphological operations, that is, the binary image formed by the background image and the foreground is first corroded and then expanded to perform noise reduction processing on the frame image, thereby better distinguishing the background image. prospect.

Contextual features in embodiments of the present disclosure may include SIFT features, HOG features, etc. In the embodiment of the present disclosure, the SIFT feature extraction method may include the following operations:

Build DOG space;

Extract interest points. After finding all feature points, use corner points for identification, and use RANSAC for curve fitting of discrete points to obtain accurate location and scale information of key points;

Direction assignment, assigning values to feature points based on the local image structure of the detected key points, can use the gradient direction histogram method. When calculating the histogram, each sampling point added to the histogram is weighted using a circular Gaussian function. .

It can be understood that when extracting context features, a variety of context features can be extracted to compare multiple combinations of context features through subsequent operations to achieve a more precise clipping area to facilitate splicing. For example, the above-mentioned SIFT features and HOG features are extracted at the same time to further reduce the impact of external environments such as lighting on image splicing.

It can be understood that when extracting contextual features, it is necessary to ensure that at least one of the contextual features is related to the working mode of the device that collects the image sequence and/or the working environment of the device, so as to reduce the working mode and working environment of the device (for example, day detection and night detection, etc.) on image stitching.

It should be noted that the following operations need to be performed before using the morphological method to determine the foreground of the moving target in the frame image at time t _n and the frame image at time t _n+1 .

FIG. 3 schematically shows a flowchart of yet another embodiment of a splicing method for moving targets in a fixed field of view according to an embodiment of the present disclosure, including operations S201 to S203.

In operation S201, a background image in the frame image is obtained.

The background image is an image that does not contain moving objects. For example, the background image can be obtained by taking pictures using a camera device when the fixed field of view does not include a moving target.

In operation S202, Gaussian mixture modeling is performed on the background image and the frame image at time t _n respectively.

In operation S203, based on the comparison result of the Gaussian mixture model of the background image and the Gaussian mixture model of the frame image at time _tn , it is determined whether the frame image at time _tn contains a moving target. K Gaussian models are used in the mixed Gaussian model to characterize the characteristics of each pixel in the frame image. K usually takes a value of 3 to 5. When making the judgment, each pixel in the frame image at time t _n is matched with the Gaussian mixture model of the background image. If the match is successful, it means that the pixel is the background. If the match is unsuccessful, it means that the pixel is the background. prospect. After it is determined that the frame image at time t _n contains a moving target, the above-mentioned morphology-based method is used to further reduce noise and optimize the foreground in the frame image.

The collection of background images is easily affected by the external environment, such as lighting, shooting angle, distance, etc., which will cause changes in the Gaussian mixture model of the background image. It can be understood that the purpose of using Gaussian mixture modeling is to better distinguish it from the background image. Moving targets, so see Figure 4. In operation S2031, when it is determined that the frame image at time t _n does not contain a moving target, the Gaussian mixture model of the frame image at time t _n is updated to a mixture of the background image. Gaussian model to ensure that the background image is as close as possible to the background image in the frame image at time t _n .

Referring back to FIG. 2 , in operation S240 , the moving distance of the moving target between the frame image at time t _n and the frame image at time t _n+1 is determined according to the context feature.

In the embodiment of the present disclosure, feature matching is performed on the context features of the frame image at time t _n and the context features of the frame image at time t _n+1 , and the feature matching pair is obtained; according to the feature matching pair, The coordinate information in the frame image at time t _n and the frame image at time t _n+1 is used to calculate the moving distance of the moving target between the frame image at time t _n and the frame image at time t _n+1 . Taking the SIFT feature in the context feature as an example, the SIFT features on the moving target in the t _n time frame image and the t _{n +1} time frame image are respectively extracted, and the same part of the moving target is determined through the feature description in the t n time frame image and t _n time frame image. The SIFT features corresponding to the frame images at time t _n+1 are respectively used. Therefore, the SIFT features describing the same part of the moving object in different frame images are constructed into feature matching pairs, and the feature matching pairs are obtained in the pixel area of the frame image respectively. Coordinate position, for example, in the frame image at time t _n , the coordinates of one side of the feature matching pair are column a13 in the pixel area, and in the frame image at time t _n+1, the coordinates of the other side of the feature matching pair are column a9 in the pixel area. , then it can be calculated that the moving distance between the frame image at time t _n and the frame image at time t _n+1 is the length of four columns of pixel units.

It can be understood that when extracting the same context feature, features can be extracted from multiple locations on the moving target to obtain multiple feature matching pairs, and the average absolute difference or absolute difference of the coordinate information matched by multiple features can be obtained. The moving distance is calculated as an intermediate value to improve the accuracy of the moving distance calculation.

It can be understood that when extracting multiple contextual features, different features can be extracted from multiple locations on the moving target, thereby obtaining multiple different feature matching pairs, and the average absolute coordinate information of the matching coordinate information can be based on multiple different features. The moving distance is calculated using the difference or the intermediate value of the absolute difference to improve the accuracy of the moving distance calculation.

In operation S250, the frame image at time t _n and the frame image at time t _n+1 are respectively trimmed according to the movement distance and the splicing position to obtain a trimmed image.

For example, in this operation S250, a trimming area may be determined. The trimming area is based on the splicing position as the trimming starting point and the length of the trimming starting point in the direction opposite to the movement direction of the moving target. moving distance area. Referring to Figures 5 to 7, the movement direction of the moving target is from right to left in the drawing. The position of column a9 in the figure represents the splicing position explained in operation S220. At time t _n in Figure 5, the front end of part c1 of the moving target moves to column a13 of the pixel area. At time t _n+1 in Figure 6, the front end of part c1 of the moving target moves to column a9 of the pixel area. Using the method of calculating the moving distance using the above context features, the moving distance can be calculated as the length of four columns of pixel units, and the area indicated by the horizontal two-way arrow in Figure 6 can be obtained by delimiting the area using the method of determining the clipping area. That is the clipping area. This clipping area is applicable to at least the t _n time frame image and the t _{n + 1} time frame image. What is obtained by clipping the frame image at time t _n is a blank area that does not contain the moving target, and what is obtained by clipping the frame image at time t _n+1 is the shadow part of the moving target shown in Figure 6 . In this way, the frame image at time t _n+2 can be similarly cropped to obtain the shadow part of the moving target as shown in Figure 7.

In operation S260, the cropped images are spliced.

Taking the above-mentioned Figure 5 and Figure 7 as an example, by splicing the moving target in the clipping area, the complete c1 and c2 parts of the moving target can be obtained, that is, the blank area clipped in Figure 5, the blank area in Figure 6 and Figure 7 The shadow part of the moving target is spliced. When splicing the clipping areas, take the position a9 in Figure 5 as the starting point, and connect the position a13 in Figure 5 and the position a9 in Figure 6 to realize the splicing of the two clipping areas. The subsequent splicing methods are analogous and will not be described again.

It can be understood that the splicing of the clipping areas can be performed according to the above rules after all the clipping areas are obtained, or a splicing operation can be performed every time a clipping area is obtained.

After operation S260, when determining whether the frame image at time t _n+2 contains a moving target, if the frame image at time t _n+2 does not contain a moving target, the entire method of image splicing can be ended, indicating that the image The splicing operation has been completed, and this method is executed in a loop until a new moving target is detected again.

Based on the above splicing method of moving targets in a fixed field of view, embodiments of the present disclosure also provide a splicing device of moving targets in a fixed field of view. The device will be described in detail below with reference to FIG. 8 .

Figure 8 schematically shows a structural block diagram of a splicing device for moving targets in a fixed field of view according to an embodiment of the present disclosure.

As shown in Figure 8, the splicing device 300 includes an image sequence acquisition module 301, a splicing position setting module 302, a foreground determination module 303, a context feature extraction module 304, a movement distance calculation module 305, a cropping module 306, and a splicing module 307.

The image sequence acquisition module 301 is used to acquire an image sequence, which at least includes multiple frame images of the moving target at the moving time t.

The splicing position setting module 302 is used to determine the same position in multiple frame images as the splicing position.

The foreground determination module 303 is used to determine the foreground of the moving target in the frame image at time t _n and the frame image at time t _n+1 respectively.

The context feature extraction module 304 is used to extract context features from the image area in the foreground.

The moving distance calculation module 305 is configured to determine the moving distance of the moving target between the frame image at time t _n and the frame image at time t _n+1 according to the context characteristics.

The trimming module 306 is configured to trim the frame image at time t _n and the frame image at time t _n+1 respectively according to the movement distance and the splicing position to obtain a trimmed image.

The splicing module 307 is used to splice the cropped images.

For example, the splicing device for moving targets in a fixed field of view according to an embodiment of the present disclosure also includes a background image acquisition module for acquiring the background image in the frame image; and a Gaussian hybrid modeling module for calculating the background image and the t The frame images at time _n are respectively subjected to mixed Gaussian modeling; the moving target judgment module is used to judge the time t _n based on the comparison results of the mixed Gaussian model of the background image and the mixed Gaussian model of the frame image at time t _n Whether the frame image contains moving targets.

For example, the splicing device for moving targets in a fixed field of view according to an embodiment of the present disclosure further includes a feature matching module for characterizing the contextual features of the frame image at time t _n and the contextual features of the frame image at time t _n+1 match, and obtain matching pairs of the features.

For example, the device for splicing moving targets in a fixed field of view according to an embodiment of the present disclosure further includes: a video processing module, configured to obtain a video to be spliced and perform decoding processing on the video.

For example, the splicing device for moving targets in a fixed field of view according to an embodiment of the present disclosure further includes a hybrid Gaussian model update module, configured to update the frame image at time _tn when it is determined that the frame image at time _tn does not contain a moving target. The Gaussian mixture model is updated to be the Gaussian mixture model of the background image.

In an embodiment of the present disclosure, an image sequence is obtained, which at least includes multiple frame images containing a moving target; the same position in the multiple frame images is determined as the splicing position; and the frame image at time t _n and t _n+ are determined respectively. The foreground of the moving target in the frame image at time ₁ is extracted, and context features are extracted from the image area in the foreground; the moving distance of the moving target between the frame image at time t _n and the frame image at time t _n+1 is determined based on the context features; according to the moving distance and At the splicing position, the frame image at time t _n and the frame image at time t _n+1 are respectively trimmed to obtain a trimmed image; the trimmed images are spliced. By comparing the context features, the moving distance of the moving object between adjacent frame images is determined. According to the moving distance and the set splicing position, the frame image is trimmed to obtain the part of the moving target. The part of the moving target is obtained each time it is clipped. Images, and finally by splicing the cropped areas in all frame images, a panoramic display of the moving target can be achieved.

According to an embodiment of the present disclosure, any multiple modules in the image sequence acquisition module 301, the splicing position setting module 302, the foreground determination module 303, the contextual feature extraction module 304, the movement distance calculation module 305, the cropping module 306, and the splicing module 307 Can be combined into one module, or any module can be split into multiple modules. Alternatively, at least part of the functionality of one or more of these modules may be combined with at least part of the functionality of other modules and implemented in one module. According to an embodiment of the present disclosure, at least one of the image sequence acquisition module 301, the splicing position setting module 302, the foreground determination module 303, the contextual feature extraction module 304, the movement distance calculation module 305, the cropping module 306, and the splicing module 307 may at least is implemented partially as hardware circuitry, such as a field programmable gate array (FPGA), a programmable logic array (PLA), a system on a chip, a system on a substrate, a system on a package, an application specific integrated circuit (ASIC), or may be implemented via an application specific integrated circuit (ASIC). Any other reasonable way of integrating or encapsulating circuits, such as hardware or firmware, or any one of the three implementation methods of software, hardware and firmware or an appropriate combination of any of them. Alternatively, at least one of the image sequence acquisition module 301, the splicing position setting module 302, the foreground determination module 303, the context feature extraction module 304, the moving distance calculation module 305, the cropping module 306, and the splicing module 307 can be at least partially implemented as Computer program modules, when the computer program modules are run, can perform corresponding functions.

Embodiments of the present disclosure also provide a splicing system for moving targets in a fixed field of view. The image splicing device in the fixed field of view and the image acquisition device in the above embodiments are used to collect and form videos containing moving targets. , and collect image background.

FIG. 9 schematically shows a block diagram of an electronic device suitable for implementing a splicing method of moving targets in a fixed field of view according to an embodiment of the present disclosure.

As shown in FIG. 9 , an electronic device 400 according to an embodiment of the present disclosure includes a processor 401 that can be loaded into a random access memory (RAM) 403 according to a program stored in a read-only memory (ROM) 402 or from a storage part 408 program to perform various appropriate actions and processes. Processor 401 may include, for example, a general-purpose microprocessor (eg, a CPU), an instruction set processor and/or an associated chipset, and/or a special-purpose microprocessor (eg, an application specific integrated circuit (ASIC)), or the like. Processor 401 may also include onboard memory for caching purposes. The processor 401 may include a single processing unit or multiple processing units for performing different actions of the method flow according to embodiments of the present disclosure.

In the RAM 403, various programs and data required for the operation of the electronic device 400 are stored. The processor 401, ROM 402 and RAM 403 are connected to each other through a bus 404. The processor 401 performs various operations according to the method flow of the embodiment of the present disclosure by executing programs in the ROM 402 and/or RAM 403. It should be noted that the program may also be stored in one or more memories other than ROM 402 and RAM 403. The processor 401 may also perform various operations according to the method flow of embodiments of the present disclosure by executing programs stored in the one or more memories.

According to embodiments of the present disclosure, the electronic device 400 may further include an input/output (I/O) interface 405 that is also connected to the bus 404 . Electronic device 400 may also include one or more of the following components connected to I/O interface 405: an input portion 406 including a keyboard, mouse, etc.; including a cathode ray tube (CRT), liquid crystal display (LCD), etc., and an output section 407 such as a speaker; a storage section 408 including a hard disk and the like; and a communication section 409 including a network interface card such as a LAN card, a modem and the like. The communication section 409 performs communication processing via a network such as the Internet. Driver 410 is also connected to I/O interface 405 as needed. Removable media 411, such as magnetic disks, optical disks, magneto-optical disks, semiconductor memories, etc., are installed on the drive 410 as needed, so that a computer program read therefrom is installed into the storage portion 408 as needed.

The present disclosure also provides a computer-readable storage medium. The computer-readable storage medium may be included in the device/device/system described in the above embodiments; it may also exist independently without being assembled into the device/system. in the device/system. The above computer-readable storage medium carries one or more programs. When the above one or more programs are executed, the method according to the embodiment of the present disclosure is implemented.

According to embodiments of the present disclosure, the computer-readable storage medium may be a non-volatile computer-readable storage medium, which may include, but is not limited to, portable computer disks, hard disks, random access memory (RAM), and read-only memory (ROM). , erasable programmable read-only memory (EPROM or flash memory), portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination of the above. In embodiments of the present disclosure, a computer-readable storage medium may be any tangible medium that contains or stores a program for use by or in connection with an instruction execution system, apparatus, or device. For example, according to embodiments of the present disclosure, the computer-readable storage medium may include one or more memories other than ROM 402 and/or RAM 403 and/or ROM 402 and RAM 403 described above.

Embodiments of the present disclosure also include a computer program product including a computer program containing program code for performing the method illustrated in the flowchart. When the computer program product is run in the computer system, the program code is used to cause the computer system to implement the item recommendation method provided by the embodiments of the present disclosure.

When the computer program is executed by the processor 401, the above-described functions defined in the system/device of the embodiment of the present disclosure are performed. According to embodiments of the present disclosure, the systems, devices, modules, units, etc. described above may be implemented by computer program modules.

In one embodiment, the computer program may rely on tangible storage media such as optical storage devices and magnetic storage devices. In another embodiment, the computer program can also be transmitted and distributed in the form of a signal on a network medium, and downloaded and installed through the communication part 409, and/or installed from the removable medium 411. The program code contained in the computer program can be transmitted using any appropriate network medium, including but not limited to: wireless, wired, etc., or any suitable combination of the above.

In such embodiments, the computer program may be downloaded and installed from the network via communication portion 409 and/or installed from removable media 411 . When the computer program is executed by the processor 401, the above-described functions defined in the system of the embodiment of the present disclosure are performed. According to embodiments of the present disclosure, the systems, devices, devices, modules, units, etc. described above may be implemented by computer program modules.

According to the embodiments of the present disclosure, the program code for executing the computer program provided by the embodiments of the present disclosure may be written in any combination of one or more programming languages. Specifically, high-level procedural and/or object-oriented programming may be utilized. programming language, and/or assembly/machine language to implement these computational procedures. Programming languages include, but are not limited to, programming languages such as Java, C++, python, "C" language or similar programming languages. The program code may execute entirely on the user's computing device, partly on the user's device, partly on a remote computing device, or entirely on the remote computing device or server. In situations involving remote computing devices, the remote computing device may be connected to the user computing device through any kind of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computing device, such as provided by an Internet service. (business comes via Internet connection).

The flowcharts and block diagrams in the figures illustrate the architecture, functionality, and operations of possible implementations of systems, methods, and computer program products according to various embodiments of the present disclosure. In this regard, each block in the flowchart or block diagrams may represent a module, segment, or portion of code that contains one or more logic functions that implement the specified executable instructions. It should also be noted that, in some alternative implementations, the functions noted in the block may occur out of the order noted in the figures. For example, two blocks shown one after another may actually execute substantially in parallel, or they may sometimes execute in the reverse order, depending on the functionality involved. It will also be noted that each block in the block diagram or flowchart illustration, and combinations of blocks in the block diagram or flowchart illustration, can be implemented by special purpose hardware-based systems that perform the specified functions or operations, or may be implemented by special purpose hardware-based systems that perform the specified functions or operations. Achieved by a combination of specialized hardware and computer instructions.

Those skilled in the art will understand that the features described in the various embodiments and/or claims of the present disclosure may be combined and/or combined in various ways, even if such combinations or combinations are not explicitly described in the embodiments of the present disclosure. In particular, various combinations and/or combinations of features recited in the various embodiments and/or claims of the disclosure may be made without departing from the spirit and teachings of the disclosure. All such combinations and/or combinations fall within the scope of this disclosure.

The embodiments of the present disclosure have been described above. However, these examples are for illustrative purposes only and are not intended to limit the scope of the present disclosure. Although each embodiment is described separately above, this does not mean that the measures in the various embodiments cannot be used in combination to advantage. The scope of the disclosure is defined by the appended claims and their equivalents. Without departing from the scope of the present disclosure, those skilled in the art can make various substitutions and modifications, and these substitutions and modifications should all fall within the scope of the present disclosure.

Claims (17)

1. A method of image stitching in a fixed field of view, comprising:

acquiring an image sequence, wherein the image sequence at least comprises a plurality of frame images containing a moving target;

Determining the same position in a plurality of frame images as a splicing position;

respectively determining t _n Time frame image and t _n+1 Extracting a plurality of context features in an image area at the foreground, wherein n is a positive integer;

determining the moving target at t according to the context characteristics _n Time frame image and t _n+1 A moving distance between time frame images, wherein the moving distance comprises the following distance between time frame images _n Combination of multiple contextual features of a temporal frame image and t _n+1 Performing feature matching on the combination of multiple contextual features of the time frame image to obtain feature matching pairs; matching the pair at t according to the characteristics _n Time frame image and t _n+1 Calculating the coordinate information of the moving object in the time frame image at the t _n Time frame image and t _n+1 A moving distance between time frame images;

respectively aiming at the t according to the moving distance and the splicing position _n Time frame image and t _n+1 Cutting the time frame image to obtain a cut image; and

and splicing the clipping images.

2. The method of image stitching in a fixed field of view according to claim 1, wherein, in determining t separately _n Time frame image and t _n+1 Before the foreground of the moving object in the time frame image, the method further comprises:

acquiring a background image in a frame image;

for the background image and t _n Respectively carrying out Gaussian mixture modeling on the time frame images; and

based on the Gaussian mixture model of the background image and the t _n Comparing the results of the Gaussian mixture model of the time frame image, and judging the t _n Whether the moving object is contained in the time frame image.

3. The method of image stitching in a fixed field of view according to claim 2, wherein in determining the t _n When the time frame image does not contain the moving target, the t is calculated _n And updating the Gaussian mixture model of the moment frame image into the Gaussian mixture model of the background image.

4. The method of image stitching in a fixed field of view according to claim 2, wherein determining the same position in a plurality of the frame images as a stitching position comprises:

setting the row or column of the frame image as the splicing position, wherein the splicing position satisfies:

in the moving direction of the moving object, the splicing position is positioned at t _m In front of the moving object in the time frame image, wherein t is _m The time frame image is a frame image containing the moving target for the first time, m is a positive integer, and m is less than or equal to n.

5. The method of image stitching in a fixed field of view according to claim 1, wherein the matching pair is at the t according to the characteristics _n Time frame imageAnd t is as described _n+1 Calculating the coordinate information of the moving object in the time frame image at the t _n Time frame image and t _n+1 The moving distance between the time frame images includes:

and calculating the moving distance according to the average absolute difference or the intermediate value of the absolute differences of the coordinate information of the plurality of feature matching pairs.

6. The method of image stitching in a fixed field of view according to claim 5, wherein at least one of the contextual characteristics relates to an operating mode of a device acquiring the sequence of images and/or an operating environment of the device.

7. The method of image stitching in a fixed field of view according to claim 1, wherein the moving distance and the stitching position are respectively for the t _n Time frame image and t _n+1 The clipping of the time frame image comprises the following steps:

and determining a clipping region, wherein the clipping region is a region with the clipping starting point at the splicing position and the length of the clipping starting point is defined as the moving distance along the direction opposite to the moving direction of the moving target.

8. The method of image stitching in a fixed field of view according to any one of claims 1-7, wherein the acquiring the sequence of images includes:

and acquiring videos to be spliced, and decoding the videos to obtain the image sequence.

9. The method of image stitching in a fixed field of view according to any one of claims 1-7, wherein the respective determination of t _n Time frame image and t _n+1 The foreground of the moving object in the time frame image comprises:

determination of t using morphological methods _n Time frame image and t _n+1 And the foreground of the moving target in the moment frame image.

10. The method of claim 1, wherein after said stitching said cropped images, if said t is determined to be _n+2 And if the time frame image does not contain the moving target, ending the splicing of the clipping image.

11. An image stitching device in a fixed field of view, comprising:

an image sequence acquisition module, configured to acquire an image sequence, where the image sequence includes at least a plurality of frame images including a moving object;

the splicing position setting module is used for determining the same position in the plurality of frame images as a splicing position;

A foreground determining module for determining t respectively _n Time frame image and t _n+1 The foreground of the moving target in the time frame image;

the context feature extraction module is used for extracting context features of the image area at the foreground, wherein the context feature extraction module comprises the steps of extracting various context features in the image area at the foreground;

a moving distance calculation module for determining the moving object at the t according to the context characteristics _n Time frame image and t _n+1 A moving distance between time frame images, wherein the moving distance comprises the following distance between time frame images _n Combination of multiple contextual features of a temporal frame image and t _n+1 Performing feature matching on the combination of multiple contextual features of the time frame image to obtain feature matching pairs; matching the pair at t according to the characteristics _n Time frame image and t _n+1 Calculating the coordinate information of the moving object in the time frame image at the t _n Time frame image and t _n+1 A moving distance between time frame images;

the clipping module is used for respectively aiming at the t according to the moving distance and the splicing position _n Time frame image and t _n+1 Cutting the time frame image to obtain a cut image;

and the splicing module splices the cut images.

12. The image stitching device in a fixed field of view of claim 11 further comprising:

the background image acquisition module is used for acquiring a background image in the frame image;

a Gaussian mixture modeling module for modeling the background image and the t _n Respectively carrying out Gaussian mixture modeling on the time frame images;

a moving target judging module for judging the moving target according to the Gaussian mixture model of the background image and the t _n Comparing the results of the Gaussian mixture model of the time frame image, and judging the t _n Whether the moving object is contained in the time frame image.

13. The image stitching device in a fixed field of view of claim 11 further comprising:

the video processing module is used for acquiring videos to be spliced and decoding the videos.

14. The image stitching device in a fixed field of view of claim 11 further comprising:

a mixed Gaussian model updating module for judging the t _n When the time frame image does not contain the moving target, the t is calculated _n And updating the Gaussian mixture model of the time frame image into the Gaussian mixture model of the background image.

15. An image stitching system in a fixed field of view, comprising:

The image stitching device in a fixed field of view of any one of claims 11-14;

an image acquisition device for acquiring and forming a video including a moving object, and

an image background is acquired.

16. An electronic device, comprising:

one or more processors;

storage means for storing one or more programs,

wherein the one or more programs, when executed by the one or more processors, cause the one or more processors to perform the method of any of claims 1-10.

17. A computer readable storage medium having stored thereon executable instructions which, when executed by a processor, cause the processor to perform the method according to any of claims 1 to 10.