CN115170451A

CN115170451A - Sky background replacing method and device, computer equipment and storage medium

Info

Publication number: CN115170451A
Application number: CN202210784865.6A
Authority: CN
Inventors: 郑金云; 齐镗泉
Original assignee: Shenzhen Wondershare Software Co Ltd
Current assignee: Shenzhen Wondershare Software Co Ltd
Priority date: 2022-06-29
Filing date: 2022-06-29
Publication date: 2022-10-11

Abstract

The application relates to a method, a device, computer equipment and a storage medium for replacing a sky background, wherein the method comprises the steps of obtaining a target background and a to-be-processed sample selected by a user, and if the type of the to-be-processed sample is a video type, performing framing processing on the to-be-processed sample to obtain a to-be-processed video frame set; acquiring a current frame and an adjacent frame; the method comprises the steps of carrying out segmentation processing on a sky background in a current frame to obtain a sky mask image; constructing a transformation matrix based on the sky mask image and adjacent frames, and carrying out affine transformation processing on a target background based on the transformation matrix to obtain an initial sky background image; performing color harmony processing on the initial sky background image based on the target background to obtain a target sky background image of the current frame; when all video frame processing is completed, a target sky background video is generated. According to the invention, the sky background of the current frame is replaced by the information of the adjacent frames, which is beneficial to improving the accuracy of replacing the sky background.

Description

Sky background replacing method and device, computer equipment and storage medium

Technical Field

The present application relates to the field of image processing technologies, and in particular, to a method and an apparatus for replacing a sky background, a computer device, and a storage medium.

Background

The sky is an important component in an image or a video, and the expected sky is difficult to obtain when people shoot the sky due to uncontrollable weather. Therefore, the algorithm-based method is convenient for users to replace unsatisfied sky in videos or images into any desired appearance in a more convenient mode, and various weather special effects, such as sky in a million, moon in a million, or lightning and thunder, can be generated through one key of the algorithm.

The existing sky background replacing method is that semantic segmentation is carried out on an image or a video frame to obtain a mask image, confidence that each pixel point is the sky is marked, and then the original sky is replaced by a new sky background image according to the mask image to complete the function of changing the sky. Because the boundary between the new sky background and the sky background to be replaced is difficult to accurately identify, and when the scene of the front frame image and the scene of the rear frame image are changed, the change of the sky background cannot be accurately captured by the existing method, the sky background replacement accuracy of the existing method is poor. There is a need for a method capable of improving accuracy of sky background replacement.

Disclosure of Invention

An embodiment of the present application provides a method, an apparatus, a computer device and a storage medium for replacing a sky background, so as to improve accuracy of replacing the sky background.

In order to solve the above technical problem, an embodiment of the present application provides a method for replacing a sky background, including:

acquiring a target background and a sample to be processed selected by a user, and judging the type of the sample to be processed;

if the type of the sample to be processed is a video type, performing framing processing on the sample to be processed to obtain a video frame set to be processed;

acquiring a current frame and an adjacent frame corresponding to the current frame from the video frame set to be processed;

carrying out segmentation processing on the sky background in the current frame to obtain a sky mask image;

constructing a transformation matrix based on the sky mask image and the adjacent frames, and performing affine transformation processing on the target background based on the transformation matrix to obtain an initial sky background image;

performing color harmony processing on the initial sky background image based on the target background to obtain a target sky background image of the current frame;

judging whether the current frame is the last frame of the video frame set to be processed or not, if not, returning to execute the step of acquiring the current frame and the adjacent frame corresponding to the current frame from the video frame set to be processed until all the video frames are processed, obtaining a plurality of target sky background images of the current frame, and generating a target sky background video based on the plurality of target sky background images of the current frame.

In order to solve the above technical problem, an embodiment of the present application provides a sky background replacement device, including:

the device comprises a to-be-processed sample acquisition module, a to-be-processed sample acquisition module and a to-be-processed sample acquisition module, wherein the to-be-processed sample acquisition module is used for acquiring a target background and a to-be-processed sample selected by a user and judging the type of the to-be-processed sample;

a video frame set generating module, configured to perform framing processing on the sample to be processed if the type of the sample to be processed is a video type, to obtain a video frame set to be processed;

the to-be-processed image acquisition module is used for acquiring a current frame and an adjacent frame corresponding to the current frame from the to-be-processed video frame set;

a sky mask map generation module, configured to perform segmentation processing on the sky background in the current frame to obtain a sky mask map;

the initial sky background image generation module is used for constructing a transformation matrix based on the sky mask image and the adjacent frames, and carrying out affine transformation processing on the target background based on the transformation matrix to obtain an initial sky background image;

the color harmony processing module is used for performing color harmony processing on the initial sky background image based on the target background to obtain a target sky background image of the current frame;

and the target sky background video generation module is used for judging whether the current frame is the last frame of video frame in the set of video frames to be processed, if not, returning to execute the step of acquiring the current frame and the adjacent frame corresponding to the current frame from the set of video frames to be processed until all the video frames are processed, obtaining a plurality of target sky background images of the current frame, and generating a target sky background video based on the plurality of target sky background images of the current frame.

In order to solve the technical problems, the invention adopts a technical scheme that: a computer device is provided that includes, one or more processors; a memory for storing one or more programs for causing the one or more processors to implement the method for replacing a sky background as described in any one of the above.

In order to solve the technical problems, the invention adopts a technical scheme that: a computer-readable storage medium having stored thereon a computer program which, when executed by a processor, implements a method of replacing a background of the sky as recited in any one of the above.

The embodiment of the invention provides a sky background replacing method and device, computer equipment and a storage medium. The method comprises the following steps: acquiring a target background and a sample to be processed selected by a user, and judging the type of the sample to be processed; if the type of the sample to be processed is a video type, performing framing processing on the sample to be processed to obtain a video frame set to be processed; acquiring a current frame and an adjacent frame corresponding to the current frame from a video frame set to be processed; the method comprises the steps of carrying out segmentation processing on a sky background in a current frame to obtain a sky mask image; constructing a transformation matrix based on the sky mask image and adjacent frames, and carrying out affine transformation processing on a target background based on the transformation matrix to obtain an initial sky background image; performing color harmony processing on the initial sky background image based on the target background to obtain a target sky background image of the current frame; and judging whether the current frame is the last frame of the video frame set to be processed, if not, returning to the step of acquiring the current frame and the adjacent frame corresponding to the current frame from the video frame set to be processed until all the video frames are processed, obtaining a plurality of target sky background images of the current frame, and generating a target sky background video based on the target sky background images of the current frames. According to the embodiment of the invention, the frame division is carried out on the sample to be processed, the current frame and the adjacent frame are obtained, and the sky segmentation processing is carried out on the current frame, so that the segmentation of the foreground and the sky background in the image is realized, and the subsequent replacement of the sky background is facilitated; then, based on adjacent frames, a transformation matrix is constructed, and affine transformation processing is carried out on the target background based on the transformation matrix, so that the current frame sky background is replaced; after all video frames in the video frame set are processed, a target sky background video is generated, the sky background in the video is replaced, meanwhile, the sky background of the current frame is replaced by combining information of adjacent frames, and the accuracy of sky background replacement is improved.

Drawings

In order to more clearly illustrate the solution of the present application, the drawings needed for describing the embodiments of the present application will be briefly described below, and it is obvious that the drawings in the following description are some embodiments of the present application, and that other drawings can be obtained by those skilled in the art without inventive effort.

FIG. 1 is a flow chart of an implementation of a method flow for replacing a sky background according to an embodiment of the present disclosure;

FIG. 2 is a flow chart of another implementation of a sub-process of a method for replacing a sky background according to an embodiment of the present application;

FIG. 3 is a flow chart of another implementation of a sub-process of a method for replacing a sky background according to an embodiment of the present application;

FIG. 4 is a flow chart of another implementation of a sub-process of a method for replacing a sky background according to an embodiment of the present application;

FIG. 5 is a flow chart of another implementation of a sub-process of a method for replacing a sky background according to an embodiment of the present application;

fig. 6 is a schematic diagram illustrating an alternative comparison of a sky background of a first frame video frame according to an embodiment of the present disclosure;

fig. 7 is a schematic diagram illustrating a sky background replacement contrast of an inter-frame video frame according to an embodiment of the present disclosure;

FIG. 8 is a flow chart of another implementation of a sub-process of a method for replacing a sky background according to an embodiment of the present application;

fig. 9 is a schematic diagram illustrating a sky background replacement contrast of a first frame image according to an embodiment of the present disclosure;

fig. 10 is a schematic diagram illustrating a sky background replacement contrast of an inter-frame image according to an embodiment of the present disclosure;

FIG. 11 is a schematic view of an alternative apparatus for providing a sky background according to an embodiment of the present disclosure;

FIG. 12 is a schematic diagram of a computer device provided by an embodiment of the present application.

Detailed Description

Unless defined otherwise, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs; the terminology used in the description of the application herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the application; the terms "including" and "having," and any variations thereof, in the description and claims of this application and the description of the above figures are intended to cover non-exclusive inclusions. The terms "first," "second," and the like in the description and claims of this application or in the foregoing drawings are used for distinguishing between different objects and not for describing a particular sequential order.

Reference herein to "an embodiment" means that a particular feature, structure, or characteristic described in connection with the embodiment can be included in at least one embodiment of the application. The appearances of the phrase in various places in the specification are not necessarily all referring to the same embodiment, nor are separate or alternative embodiments mutually exclusive of other embodiments. It is explicitly and implicitly understood by one skilled in the art that the embodiments described herein can be combined with other embodiments.

In order to make the technical solutions better understood by those skilled in the art, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings.

The present invention will be described in detail below with reference to the accompanying drawings and embodiments.

It should be noted that, the replacing method of the sky background provided by the embodiment of the present application is generally executed by the server, and accordingly, the replacing device of the sky background is generally configured in the server.

Referring to fig. 1, fig. 1 shows an embodiment of a method for replacing a sky background.

It should be noted that, if there is substantially the same result, the method of the present invention is not limited to the flow sequence shown in fig. 1, and the method includes the following steps:

s1: and acquiring a target background and a sample to be processed selected by a user, and judging the type of the sample to be processed.

Specifically, when a user needs to replace a sky background in a certain image or video, the user selects a final sky background to be replaced, that is, a target background. After the server receives the target background and the sample to be processed, which are selected by the user, the type of the sample to be processed includes an image type and a video type, and the server adopts different replacement methods corresponding to different types, so that the type of the sample to be processed needs to be judged.

S2: and if the type of the sample to be processed is a video type, performing framing processing on the sample to be processed to obtain a video frame set to be processed.

Specifically, if the type of the sample to be processed is a video type, the sample to be processed is subjected to framing processing to obtain each frame of video frame, and then each frame of video frame is stored in the database to obtain a processed video frame set. Furthermore, the marking is carried out according to the sequence of the video frames, so that the sky background can be conveniently replaced from the first frame of the video frame, the subsequent judgment of the number of the obtained current frame belongs to the video frames is facilitated, the sky background can be conveniently replaced frame by frame of the video frames sequentially, and the subsequent successfully replaced video frames can be favorably superposed into the video.

S3: and acquiring a current frame and an adjacent frame corresponding to the current frame from the video frame set to be processed.

Specifically, the adjacent frame refers to a frame before the current frame, and if the current frame is the first frame in the set of video frames to be processed, the current frame at this time is taken as the adjacent frame.

S4: and carrying out segmentation processing on the sky background in the current frame to obtain a sky mask image.

Specifically, the sky background and the foreground in the current frame are identified, and the sky background is segmented to obtain a corresponding mask image, namely a sky mask image. Further, before step S4, it is determined whether the current frame is the last frame in the set of video frames to be processed, and if the current frame is the last frame, after the video frame is processed, all the video frames are processed.

Further, step S4 includes:

compressing the current frame to a preset size to obtain a current frame target image;

inputting the current frame target image into a pre-trained segmentation model, identifying the sky background in the current frame target image through the pre-trained segmentation model, and segmenting the sky background to obtain a sky mask image.

Specifically, the current frame is compressed to a preset size to obtain a current frame target image, where the preset size is set according to an actual situation, and is not limited here. In one embodiment, the current frame is compressed to a size of 256 × 256 × 3. And then, inputting the current frame target image into a pre-trained segmentation model for coding, namely, identifying the sky background in the current frame target image through convolution operation by the pre-trained segmentation model, and segmenting the sky background. If the current frame target image is 256 × 256 × 3 in size, a feature map of 16 × 16 × 112 in size is obtained after encoding processing; and performing decoding operation on the feature map coded by the pre-trained segmentation model, performing convolution and up-sampling processing on the feature map, and performing cascade operation on the feature map and the previous feature map to obtain a sky mask map. If the current frame target image has a size of 256 × 256 × 3, a sky mask image having a size of 256 × 256 × 1 is obtained after the decoding process. Further, the segmentation network used by the pre-trained segmentation model is Efficientnet-b0.

Further, before step S4, the segmentation network Efficientnet-b0 is trained, so as to obtain a pre-trained segmentation model. The training process is to input the marked sky background image and the target replacement background image, segment the marked sky background image through a segmentation network Efficientnet-b0, and compare the obtained result with the target replacement background image to obtain a loss value. And adjusting parameters of the segmentation network Efficientnet-b0 based on the loss value, and then segmenting the marked sky background map until the loss value reaches a preset threshold value, so as to obtain a pre-trained segmentation model. The preset threshold is set according to actual conditions, and is not limited herein.

S5: and constructing a transformation matrix based on the sky mask image and the adjacent frames, and carrying out affine transformation processing on the target background based on the transformation matrix to obtain an initial sky background image.

Specifically, a transformation matrix of the current frame is constructed according to information of comparing the sky mask image with adjacent frames, and then affine transformation processing is carried out on the target background based on the transformation matrix, so that the sky background of the sky mask image is replaced by the sky background in the target background, and the initial sky background image is obtained.

Referring to fig. 2, fig. 2 shows an embodiment of step S5, which is described in detail as follows:

s51: and carrying out corner detection processing on the sky mask image to obtain a corner feature image.

Referring to fig. 3, fig. 3 shows an embodiment of step S51, which is described in detail as follows:

s511: and carrying out corner detection processing on the sky mask image by adopting ShiTomasi algorithm to obtain an initial corner.

S512: and taking two adjacent initial angular points as angular point combinations, and calculating the distance between the angular points in the angular point combinations to obtain angular point distances.

S513: and acquiring a corner combination with the corner distance larger than the preset distance as a target corner combination, and generating a corner feature map based on the target corner combination.

Specifically, the ShiTomasi algorithm is an improvement of the Harris algorithm. The Harris algorithm is most originally defined by subtracting the determinant value of the matrix M from the trace of M and comparing the difference with a predetermined threshold. Later, shi and Tomasi proposed an improved method (i.e., shiTomasi algorithm) that resulted in strong corners if the smaller of the two eigenvalues was greater than a minimum threshold. In the embodiment of the application, a ShiTomasi algorithm is adopted to perform corner detection processing on a sky mask image to obtain initial corners, two adjacent initial corners are used as corner combinations, the distance between corners in the corner combinations is calculated to obtain corner distances, then the corner combinations with the corner distances larger than a preset distance are obtained and used as target corner combinations, the target corner combinations are reserved, and other corner combinations are deleted, so that a corner feature image is generated. The preset distance is set according to actual conditions, and is not limited herein. In one embodiment, the predetermined distance is 30 inter-pixel distances.

S52: and tracking the corner points of the adjacent frames based on the corner point feature map to obtain the corner points of the adjacent frames, which are the same as the corner point feature map, and using the corner points as the basic corner points.

Specifically, tracking the corner points in the corner point feature map based on the Lucas-Kanade algorithm. The Lucas-Kanade algorithm inputs adjacent frames and a current frame, and angular points detected in the previous step, and returns the coordinates of the tracked angular points in the previous frame and the coordinates of the current frame, namely the basic angular points. The base corner point includes position information (coordinates) at the current frame and position information (coordinates) at the neighboring frame. Among them, the Lucas-Kanade algorithm is a two-frame differential optical flow estimation algorithm, which was proposed by bruced. Since algorithms are easily applied to a set of points in the input image, they have become an important method for finding sparse optical flow.

S53: and acquiring a preset number of target angular points from the basic angular points, and constructing a transformation matrix based on the target angular points.

Referring to fig. 4, fig. 4 shows an embodiment of step S53, which is described in detail as follows:

s531: and acquiring a preset number of target angular points from the basic angular points by adopting a RANSAC random sampling mode.

S532: and respectively acquiring the positions of the target corner points in the current frame and the adjacent frame to obtain current position information and adjacent position information.

S533: and acquiring preset parameters, and calculating corresponding values of the preset parameters through a preset formula based on the current position information and the adjacent position information to obtain target parameter values.

Specifically, a random sample access method is adopted to acquire a preset number of target angular points from the basic angular points, and then the positions of the target angular points in the current frame and the adjacent frame are respectively acquired to obtain current position information and adjacent position information. And then acquiring preset parameters, and calculating corresponding values of the preset parameters through a preset formula based on the current position information and the adjacent position information to obtain target parameter values. For example, the preset parameters are a1, a2, a3, a4, b1, b2, where a1= s cos θ, a2= s (-sin θ), a3= s sin θ, and a4= s cos θ, where s represents a scaling ratio and θ represents a rotation angle, and then the following preset formula is used for calculation;

where x, y represent neighboring location information and x ', y' represent current location information.

The preset number is set according to actual conditions, and is not limited herein. In a specific embodiment, 3 target corner points are acquired. The RANSAC random sampling can be used for estimating parameters of a mathematical model from a group of observation data sets containing 'local outliers' in an iterative mode.

S534: based on the target parameter values, a transformation matrix is constructed.

Specifically, the target parameter values, that is, the values corresponding to a1, a2, a3, a4, b1, and b2, are obtained in step S533, and then the transformation matrix is constructed according to the target parameter values. Its transformation matrix is denoted T:

s54: and performing affine transformation processing on the target background based on the transformation matrix to obtain an initial sky background image.

Specifically, affine transformation processing is performed on the target background through the transformation matrix, so that the sky background of the target background is replaced with the sky background of the current frame, and an initial sky background image is obtained.

S6: and performing color harmony processing on the initial sky background image based on the target background to obtain a target sky background image of the current frame.

Referring to fig. 5, fig. 5 shows an embodiment of step S6, which is described in detail as follows:

s61: and respectively calculating the color sampling mean values of the initial sky background image and the target background to obtain an initial color sampling mean value and a target color sampling mean value.

S62: and calculating the difference value of the initial color sampling mean value and the target color sampling mean value.

S63: and performing color harmony processing on the initial sky background image based on the difference value to obtain a target sky background image of the current frame.

Specifically, the sky background of the target background is replaced with the sky background of the current frame to obtain an initial sky background image, and the target background and the current frame may have a large color difference, so that color harmony processing needs to be performed on the initial sky background image. The specific process is as follows: respectively calculating color sampling mean values of the initial sky background image and the target background to obtain an initial color sampling mean value and a target color sampling mean value, and then calculating a difference value between the initial color sampling mean value and the target color sampling mean value; and based on the difference value, carrying out color harmony processing on the initial sky background image by adopting a preset weight coefficient to obtain a target sky background image of the current frame. The preset weight is set according to actual conditions, and is not limited herein.

S7: judging whether the current frame is the last frame of the video frame set to be processed or not, if not, returning to execute the step of acquiring the current frame and the adjacent frame corresponding to the current frame from the video frame set to be processed until all the video frames are processed, obtaining a plurality of target sky background images of the current frame, and generating a target sky background video based on the target sky background images of the current frames.

Specifically, whether the current frame is the last frame of the video frame in the video frame set to be processed is judged, if yes, it is indicated that all the video frames in the video frame set to be processed are completely replaced, and a target sky background video can be generated; if not, returning to the step of obtaining the current frame and the adjacent frame corresponding to the current frame from the video frame set to be processed, namely returning to the step S3 to replace the sky background until all the video frames are processed, thereby obtaining the target sky background images of a plurality of current frames. And then, according to the sequence of the sky mask image and the video frames, overlapping a plurality of target sky background images of the current frame, thereby generating a target sky background video. Referring to fig. 6 and 7, fig. 6 is a schematic diagram illustrating a sky background replacement contrast of a first frame of a video frame according to an embodiment of the present disclosure; fig. 7 is a schematic diagram illustrating a sky background replacement contrast of an inter-frame video frame according to an embodiment of the present disclosure. In fig. 6 and 7, the left side of the image is the image before replacing the current frame sky background, and the right side of the image is the image after replacing the current frame sky background.

In the embodiment, a target background and a sample to be processed selected by a user are obtained, and the type of the sample to be processed is judged; if the type of the sample to be processed is a video type, performing framing processing on the sample to be processed to obtain a video frame set to be processed; acquiring a current frame and an adjacent frame corresponding to the current frame from a video frame set to be processed; the method comprises the steps of carrying out segmentation processing on a sky background in a current frame to obtain a sky mask image; constructing a transformation matrix based on the sky mask image and adjacent frames, and carrying out affine transformation processing on a target background based on the transformation matrix to obtain an initial sky background image; performing color harmony processing on the initial sky background image based on the target background to obtain a target sky background image of the current frame; and judging whether the current frame is the last frame of the video frame set to be processed, if not, returning to the step of acquiring the current frame and the adjacent frame corresponding to the current frame from the video frame set to be processed until all the video frames are processed, obtaining a plurality of target sky background images of the current frame, and generating a target sky background video based on the target sky background images of the current frames. According to the embodiment of the invention, the segmentation of the foreground and the sky background in the image is realized by framing the sample to be processed, acquiring the current frame and the adjacent frame and performing the sky segmentation processing on the current frame, so that the sky background can be replaced conveniently; then, based on adjacent frames, a transformation matrix is constructed, and affine transformation processing is carried out on the target background based on the transformation matrix, so that the current frame sky background is replaced; after all the video frames in the video frame set are processed, a target sky background video is generated, the sky background in the video is replaced, meanwhile, the sky background of the current frame is replaced by combining the information of the adjacent frames, and the accuracy of sky background replacement is improved.

Referring to fig. 8 to 10, fig. 8 is a schematic diagram illustrating an embodiment of a method for replacing a sky background, and fig. 9 is a schematic diagram illustrating a comparison between a sky background and a first frame image according to an embodiment of the present disclosure; fig. 10 is a schematic diagram illustrating a sky background replacement contrast of an intermediate frame image according to an embodiment of the present disclosure.

S1A: and if the type of the sample to be processed is the image type, performing sky background segmentation processing on the image to be processed in the sample to be processed to obtain an image sky background mask image.

S1B: and superposing the foreground of the sky background mask image and the target background to obtain an initial image sky background image.

S1C: and performing color harmony processing on the sky background image of the initial image based on the target background to obtain a sky background image of the target image.

S1D: and judging whether the to-be-processed image still exists in the to-be-processed sample, if so, returning to execute the step of performing sky background segmentation processing on the to-be-processed image in the to-be-processed sample to obtain an image sky background mask if the type of the to-be-processed sample is the image type, and obtaining a plurality of target image sky background pictures until all the to-be-processed images in the to-be-processed sample are processed.

S1E: and splicing the sky background images of the multiple target images to obtain a sky background video.

Specifically, if the type of the sample to be processed is an image type, the sky background segmentation processing is performed on the image to be processed in the sample to be processed to obtain an image sky background mask, the segmentation processing process is the same as that in step S4, and in order to avoid repetition, the process is not repeated here. And then superposing the foreground of the sky background mask image and the target background to obtain an initial image sky background image, and performing color harmony processing on the initial image sky background image based on the target background to obtain a target image sky background image. The color harmony processing is the same as step S6, and is not described herein again to avoid repetition. And judging whether the to-be-processed image still exists in the to-be-processed sample, if so, returning to execute the S1A until all the to-be-processed images in the to-be-processed sample are processed, and obtaining a plurality of target image sky background images. And finally, sequentially splicing the sky background images of the plurality of target images to obtain a sky background video.

S1E: splicing a plurality of target image sky background images to obtain a sky background video

Referring to fig. 11, as an implementation of the method shown in fig. 1, the present application provides an embodiment of a device for replacing a sky background, where the embodiment of the device corresponds to the embodiment of the method shown in fig. 1, and the device may be applied to various electronic devices.

As shown in fig. 11, the sky background replacement device of the present embodiment includes: a to-be-processed sample acquisition module 81, a video frame set generation module 82, a to-be-processed image acquisition module 83, a sky mask image generation module 84, an initial sky background image generation module 85, a color harmony processing module 86, and a target sky background video generation module 87, where:

a to-be-processed sample obtaining module 81, configured to obtain a target background and a to-be-processed sample selected by a user, and determine a type of the to-be-processed sample;

the video frame set generating module 82 is configured to, if the type of the sample to be processed is a video type, perform framing processing on the sample to be processed to obtain a video frame set to be processed;

a to-be-processed image obtaining module 83, configured to obtain a current frame and an adjacent frame corresponding to the current frame from a to-be-processed video frame set;

a sky mask map generating module 84, configured to segment the sky background in the current frame to obtain a sky mask map;

the initial sky background image generation module 85 is configured to construct a transformation matrix based on the sky mask image and the adjacent frames, and perform affine transformation processing on the target background based on the transformation matrix to obtain an initial sky background image;

a color harmony processing module 86, configured to perform color harmony processing on the initial sky background image based on the target background to obtain a target sky background image of the current frame;

the target sky background video generation module 87 is configured to determine whether the current frame is a last frame video frame in the set of video frames to be processed, if not, return to the step of acquiring the current frame and an adjacent frame corresponding to the current frame from the set of video frames to be processed until all video frames are processed, obtain target sky background images of a plurality of current frames, and generate a target sky background video based on the target sky background images of the plurality of current frames.

Further, the initial sky background map generation module 85 includes:

the corner detection processing unit is used for carrying out corner detection processing on the sky mask image to obtain a corner feature image;

the corner tracking processing unit is used for tracking and processing the corners of the adjacent frames based on the corner feature map to obtain the corners of the adjacent frames, which are the same as the corner feature map, and the corners are used as basic corners;

the target corner acquisition unit is used for acquiring a preset number of target corners from the basic corners and constructing a transformation matrix based on the target corners;

and the affine transformation unit is used for carrying out affine transformation processing on the target background based on the transformation matrix to obtain an initial sky background image.

Further, the corner detection processing unit includes:

an initial corner acquisition subunit, configured to perform corner detection processing on the sky mask image by using a ShiTomasi algorithm to obtain an initial corner;

the angular point distance calculation subunit is used for taking two adjacent initial angular points as an angular point combination, and calculating the distance between the angular points in the angular point combination to obtain an angular point distance;

and the corner feature map generation subunit is used for acquiring a corner combination with a corner distance greater than a preset distance as a target corner combination and generating a corner feature map based on the target corner combination.

Further, the target corner point acquiring unit includes:

the angular point sampling sub-unit is used for acquiring a preset number of target angular points from the basic angular points in a RANSAC random sampling mode;

the position information acquisition subunit is used for respectively acquiring the positions of the target corner points in the current frame and the adjacent frame to obtain current position information and adjacent position information;

the target parameter value calculation operator unit is used for acquiring a preset parameter, and calculating a corresponding value of the preset parameter through a preset formula based on the current position information and the adjacent position information to obtain a target parameter value;

and the transformation matrix constructing subunit is used for constructing a transformation matrix based on the target parameter values.

Further, the color harmony processing module 86 includes:

the color sampling average value calculating unit is used for calculating the color sampling average values of the initial sky background image and the target background respectively to obtain an initial color sampling average value and a target color sampling average value;

the difference value calculating unit is used for calculating the difference value between the initial color sampling mean value and the target color sampling mean value;

and the target sky background image generation unit is used for performing color harmony processing on the initial sky background image based on the difference value to obtain a target sky background image of the current frame.

The to-be-processed sample obtaining module 81 further comprises:

the image sky background segmentation module is used for performing sky background segmentation processing on the to-be-processed image in the to-be-processed sample to obtain an image sky background mask image if the type of the to-be-processed sample is an image type;

the sky background image generation module of the initial image is used for superposing the foreground of the sky background mask image and the target background to obtain a sky background image of the initial image;

the target image sky background image generation module is used for carrying out color harmony processing on the original image sky background image based on the target background to obtain a target image sky background image;

the target image sky background image generation module is used for judging whether an image to be processed still exists in the sample to be processed or not, if yes, returning to execute the step of performing sky background segmentation processing on the image to be processed in the sample to be processed if the type of the sample to be processed is the image type to obtain an image sky background mask until all the images to be processed in the sample to be processed are processed to obtain a plurality of target image sky background images;

and the sky background video generation module is used for splicing the sky background images of the multiple target images to obtain a sky background video.

Further, the sky mask map generating module 84 includes:

the current frame target image generating unit is used for compressing the current frame to a preset size to obtain a current frame target image;

and the model segmentation unit is used for inputting the current frame target image into a pre-trained segmentation model, identifying the sky background in the current frame target image through the pre-trained segmentation model, and segmenting the sky background to obtain a sky mask image.

In order to solve the technical problem, an embodiment of the present application further provides a computer device. Referring to fig. 12, fig. 12 is a block diagram of a basic structure of a computer device according to the present embodiment.

The computer device 9 includes a memory 91, a processor 92, and a network interface 93 communicatively connected to each other via a system bus. It is noted that only the computer device 9 having three components memory 91, processor 92, network interface 93 is shown, but it is understood that not all of the shown components are required to be implemented, and more or fewer components may be implemented instead. As will be understood by those skilled in the art, the computer device is a device capable of automatically performing numerical calculation and/or information processing according to a preset or stored instruction, and the hardware includes, but is not limited to, a microprocessor, an Application Specific Integrated Circuit (ASIC), a Programmable Gate Array (FPGA), a Digital Signal Processor (DSP), an embedded device, and the like.

The computer device may be a desktop computer, a notebook, a palmtop computer, a cloud server, or other computing device. The computer equipment can carry out man-machine interaction with a user through a keyboard, a mouse, a remote controller, a touch panel or voice control equipment and the like.

The memory 91 includes at least one type of readable storage medium including a flash memory, a hard disk, a multimedia card, a card type memory (e.g., SD or DX memory, etc.), a Random Access Memory (RAM), a Static Random Access Memory (SRAM), a Read Only Memory (ROM), an Electrically Erasable Programmable Read Only Memory (EEPROM), a Programmable Read Only Memory (PROM), a magnetic memory, a magnetic disk, an optical disk, etc. In some embodiments, the storage 91 may be an internal storage unit of the computer device 9, such as a hard disk or a memory of the computer device 9. In other embodiments, the memory 91 may also be an external storage device of the computer device 9, such as a plug-in hard disk, a Smart Media Card (SMC), a Secure Digital (SD) Card, a Flash memory Card (Flash Card), and the like provided on the computer device 9. Of course, the memory 91 may also include both an internal storage unit of the computer device 9 and an external storage device thereof. In this embodiment, the memory 91 is generally used for storing the program code of the operating system and various application software installed in the computer device 9, such as the method for replacing the sky background. Further, the memory 91 can also be used to temporarily store various types of data that have been output or are to be output.

Processor 92 may be a Central Processing Unit (CPU), controller, microcontroller, microprocessor, or other data Processing chip in some embodiments. The processor 92 is typically used to control the overall operation of the computer device 9. In this embodiment, the processor 92 is configured to execute the program code stored in the memory 91 or process data, for example, execute the program code of the replacing method for the sky background, so as to implement various embodiments of the replacing method for the sky background.

The network interface 93 may include a wireless network interface or a wired network interface, and the network interface 93 is generally used to establish a communication connection between the computer device 9 and other electronic devices.

The present application further provides another embodiment, which is a computer-readable storage medium storing a computer program, which is executable by at least one processor to cause the at least one processor to perform the steps of a method for replacing a sky background as described above.

Through the above description of the embodiments, those skilled in the art will clearly understand that the method of the above embodiments can be implemented by software plus a necessary general hardware platform, and certainly can also be implemented by hardware, but in many cases, the former is a better implementation manner. Based on such understanding, the technical solutions of the present application may be embodied in the form of a software product, which is stored in a storage medium (such as ROM/RAM, magnetic disk, optical disk) and includes instructions for enabling a terminal device (such as a mobile phone, a computer, a server, an air conditioner, or a network device) to execute the method of the embodiments of the present application.

It is to be understood that the above-described embodiments are merely illustrative of some, but not restrictive, of the broad invention, and that the appended drawings illustrate preferred embodiments of the invention and do not limit the scope of the invention. This application is capable of embodiments in many different forms and is provided for the purpose of enabling a thorough understanding of the disclosure of the application. Although the present application has been described in detail with reference to the foregoing embodiments, it will be apparent to one skilled in the art that the present application may be practiced without modification or with equivalents of some of the features described in the foregoing embodiments. All equivalent structures made by using the contents of the specification and the drawings of the present application are directly or indirectly applied to other related technical fields and are within the protection scope of the present application.

Claims

1. A method of replacing a background sky, comprising:

constructing a transformation matrix based on the sky mask image and the adjacent frames, and carrying out affine transformation processing on the target background based on the transformation matrix to obtain an initial sky background image;

and judging whether the current frame is the last frame of the video frame set to be processed, if not, returning to execute the step of acquiring the current frame and the adjacent frame corresponding to the current frame from the video frame set to be processed until all the video frames are processed, obtaining a plurality of target sky background images of the current frame, and generating a target sky background video based on the plurality of target sky background images of the current frame.

2. The sky background replacing method of claim 1, wherein said constructing a transformation matrix based on said sky mask map and said adjacent frames, and performing affine transformation on said target background based on said transformation matrix to obtain an initial sky background map comprises:

carrying out corner detection processing on the sky mask image to obtain a corner feature image;

based on the corner feature map, performing tracking processing on the corners of the adjacent frames to obtain the same corners of the adjacent frames as the corner feature map, and using the same corners as the corner feature map as basic corners;

acquiring a preset number of target corner points from the basic corner points, and constructing the transformation matrix based on the target corner points;

and carrying out affine transformation processing on the target background based on the transformation matrix to obtain the initial sky background image.

3. The method of replacing a sky background as defined in claim 2, wherein said processing said sky mask map for corner detection to obtain a corner feature map comprises:

performing corner detection processing on the sky mask image by adopting ShiTomasi algorithm to obtain an initial corner;

taking two adjacent initial angular points as angular point combinations, and calculating the distance between the angular points in the angular point combinations to obtain angular point distances;

and acquiring the corner combination with the corner distance larger than the preset distance as a target corner combination, and generating the corner feature map based on the target corner combination.

4. A method of replacing a sky background as claimed in claim 2, wherein said obtaining a predetermined number of target corner points from said base corner points and constructing said transformation matrix based on said target corner points comprises:

acquiring a preset number of target angular points from the basic angular points by adopting a RANSAC random sampling mode;

respectively acquiring the positions of the target corner points in the current frame and the adjacent frame to obtain current position information and adjacent position information;

acquiring preset parameters, and calculating corresponding values of the preset parameters through a preset formula based on the current position information and the adjacent position information to obtain target parameter values;

and constructing the transformation matrix based on the target parameter values.

5. The method of replacing sky background according to claim 1, wherein said performing a color harmony processing on said initial sky background map based on said target background to obtain a target sky background map of said current frame comprises:

respectively calculating the color sampling average values of the initial sky background image and the target background to obtain an initial color sampling average value and a target color sampling average value;

calculating a difference value between the initial color sampling mean value and the target color sampling mean value;

and performing color harmony processing on the initial sky background image based on the difference value to obtain a target sky background image of the current frame.

6. The sky background replacing method according to claim 1, wherein after obtaining a target background and a sample to be processed selected by a user and determining a type of the sample to be processed, the method further comprises:

if the type of the sample to be processed is an image type, performing sky background segmentation processing on the image to be processed in the sample to be processed to obtain an image sky background mask image;

superposing the foreground of the sky background mask image and the target background to obtain an initial image sky background image;

performing color harmony processing on the initial image sky background image based on the target background to obtain a target image sky background image;

judging whether the to-be-processed image still exists in the to-be-processed sample, if so, returning to execute the step of performing sky background segmentation processing on the to-be-processed image in the to-be-processed sample to obtain an image sky background mask if the type of the to-be-processed sample is the image type, and obtaining a plurality of target image sky background pictures until all the to-be-processed images in the to-be-processed sample are processed;

and splicing the sky background images of the target images to obtain a sky background video.

7. The method of replacing a sky background according to any one of claims 1 to 6, wherein said segmenting the sky background in the current frame to obtain a sky mask map comprises:

inputting the current frame target image into a pre-trained segmentation model, identifying the sky background in the current frame target image through the pre-trained segmentation model, and segmenting the sky background to obtain the sky mask image.

8. A sky background replacement device, comprising:

the video frame set generating module is used for performing framing processing on the sample to be processed to obtain a video frame set to be processed if the type of the sample to be processed is a video type;

a sky mask map generation module, configured to segment the sky background in the current frame to obtain a sky mask map;

the initial sky background map generation module is used for constructing a transformation matrix based on the sky mask map and the adjacent frames, and performing affine transformation processing on the target background based on the transformation matrix to obtain an initial sky background map;

and the target sky background video generation module is used for judging whether the current frame is the last frame of the video frame set to be processed, if not, returning to execute the step of acquiring the current frame and the adjacent frame corresponding to the current frame from the video frame set to be processed until all the video frames are processed, obtaining a plurality of target sky background images of the current frame, and generating a target sky background video based on the plurality of target sky background images of the current frame.

9. A computer device, characterized in that it comprises a memory in which a computer program is stored and a processor which, when executing the computer program, implements a method of replacing a sky background as claimed in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that the computer-readable storage medium has stored thereon a computer program which, when being executed by a processor, carries out a method of replacing a sky background as claimed in any one of the claims 1 to 7.