CN113947523A - Method and device for replacing background image - Google Patents
Method and device for replacing background image Download PDFInfo
- Publication number
- CN113947523A CN113947523A CN202111210425.1A CN202111210425A CN113947523A CN 113947523 A CN113947523 A CN 113947523A CN 202111210425 A CN202111210425 A CN 202111210425A CN 113947523 A CN113947523 A CN 113947523A
- Authority
- CN
- China
- Prior art keywords
- image
- background
- illumination
- replaced
- alternative
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
- 238000000034 method Methods 0.000 title claims abstract description 45
- 238000005286 illumination Methods 0.000 claims abstract description 177
- 238000012545 processing Methods 0.000 claims abstract description 59
- 238000013528 artificial neural network Methods 0.000 claims description 23
- 238000004364 calculation method Methods 0.000 claims description 17
- 238000001914 filtration Methods 0.000 claims description 12
- 238000012549 training Methods 0.000 claims description 11
- 230000008569 process Effects 0.000 claims description 3
- 238000012360 testing method Methods 0.000 claims description 3
- 230000002194 synthesizing effect Effects 0.000 claims description 2
- 230000000694 effects Effects 0.000 abstract description 5
- 101000755816 Homo sapiens Inactive rhomboid protein 1 Proteins 0.000 description 7
- 102100022420 Inactive rhomboid protein 1 Human genes 0.000 description 7
- 230000008859 change Effects 0.000 description 3
- 238000010586 diagram Methods 0.000 description 2
- 238000005516 engineering process Methods 0.000 description 2
- 238000013459 approach Methods 0.000 description 1
- 230000015572 biosynthetic process Effects 0.000 description 1
- 239000002131 composite material Substances 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 230000002093 peripheral effect Effects 0.000 description 1
- 238000003672 processing method Methods 0.000 description 1
- 238000012163 sequencing technique Methods 0.000 description 1
- 230000003068 static effect Effects 0.000 description 1
- 239000000126 substance Substances 0.000 description 1
- 238000003786 synthesis reaction Methods 0.000 description 1
- 230000000007 visual effect Effects 0.000 description 1
Images
Classifications
-
- G06T3/04—
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/08—Learning methods
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T3/00—Geometric image transformation in the plane of the image
- G06T3/40—Scaling the whole image or part thereof
Abstract
The application discloses a method for replacing a background image, which comprises the following steps: acquiring a background image to be replaced from an original image; to-be-replaced background images and each of all alternative background images: determining an illumination image and carrying out scaling processing according to a plurality of scales; processing the zoomed image by using a texture characteristic operator under each scale to obtain the texture characteristics of the image, taking the texture characteristics as illumination coding characteristics under the corresponding scale, and combining the illumination coding characteristics under each scale adopted by each background image to be replaced or alternative background image to form the illumination coding characteristics of the corresponding image; calculating the distance between the background image to be replaced and the illumination coding features of each alternative background image as a first distance; and for all the alternative background images, ranking the alternative background images based on the corresponding first distances and recommending the alternative background images to the user. By the method and the device, the whole image after background replacement can be more natural, the background replacement effect is improved, and the method and the device are particularly suitable for replacing the background image of the real-time video.
Description
Technical Field
The present application relates to image processing technologies, and in particular, to a method and an apparatus for replacing a background image.
Background
In various image processing applications, techniques for replacing background images are often involved, such as various types of retouching software. Especially in recent years, live video, online conferences, online teaching and the like have been rapidly developed, and background replacement technology has been widely applied in order to improve video effects and facilitate the use of people. Specifically, in a real-time video application, a real-time picture shot by a camera can be divided into a foreground and a background, the foreground is generally a real-time image of a person participating in a video conference, live broadcast and the like, the background is generally a real-time image of the current environment of the person, and for the reasons of privacy protection and the like, the system can provide various static images for a user and is used for replacing a background part in the real-time video according to the selection of the user. Of course, there is a similar process for background replacement of still images.
However, most of the existing video background replacement techniques focus on how to accurately segment the foreground and the background so as to replace the background with the existing background image while maintaining the intact original foreground part as much as possible.
However, in real-time scenes such as live video and online conferences, the background image after replacement sometimes has a large difference in scene atmosphere compared with the foreground image, so that people feel that the foreground and the background have a split feeling, and feel that the reality of the current background image is very poor.
Disclosure of Invention
The application provides a method and a device for replacing a background image, which can improve the overall sense and the sense of reality of the whole image after background replacement and improve the background replacement effect, and are particularly suitable for replacing the background image of a real-time video.
In order to achieve the purpose, the following technical scheme is adopted in the application:
a method of replacing a background image, comprising:
acquiring a background image to be replaced from an original image;
for each of the background image to be replaced and all alternative background images: determining an illumination image and carrying out scaling processing according to a plurality of scales; under each scale, processing the zoomed image by using a texture feature operator to obtain the texture feature of the image, wherein the texture feature is used as an illumination coding feature under the corresponding scale; combining the illumination coding features of each to-be-replaced background image or each alternative background image under each scale to form the illumination coding features of the corresponding to-be-replaced background image or each alternative background image;
calculating the distance between the background image to be replaced and the illumination coding features of each alternative background image as a first distance; and for all the alternative background images, ranking the alternative background images based on the corresponding first distances, and recommending the alternative background images to the user for replacing the background images.
Preferably, the processing the scaled image by using the texture feature operator to obtain the texture feature of the image includes:
and coding each pixel participating in statistics of the scaled image by using the texture feature operator, performing region division on the coded image, calculating the occurrence frequency of various coding values in each region participating in statistics, and combining the occurrence frequencies of set regions to form the texture feature.
Preferably, after performing the scaling process on each of the background image to be replaced and all the alternative background images, the method further includes: for the zoomed image with a preset scale, combining all pixel values together to serve as an additional illumination characteristic of a corresponding background image to be replaced or an alternative background; calculating the distance between the background image to be replaced and the additional illumination characteristic of each alternative background image as a second distance;
the recommending the all alternative background images to the user after being sorted according to the respective corresponding first distances comprises: and calculating the weighted sum of the first distance and the second distance of all the alternative background images and the background image to be replaced, and arranging all the alternative background images based on the weighted sum.
Preferably, when all the alternative background images are sorted, the sorting is performed according to the order of the distances between the alternative background images and the illumination coding features of the background image to be replaced from small to large.
Preferably, the background image to be replaced is an original image; the pixels participating in statistics in the background image to be replaced and the alternative background image are as follows: all pixels of the corresponding image, or pixels in the same position as the background part in the corresponding image; the areas participating in statistics in the background image to be replaced and the alternative background image are as follows: all regions of the corresponding image, or regions in the corresponding image that are in the same position as the background portion;
alternatively, the first and second electrodes may be,
the background image to be replaced is as follows: the background part image or the image formed by combining the background part with the foreground part filled with preset pixels; pixels participating in statistics in the background image to be replaced and the alternative background image are pixels with the same positions as the background part in the corresponding image; the area participating in statistics in the background image to be replaced and the alternative background image is an area with the same position as the background part in the corresponding image;
the background part and the foreground part are two parts divided from the original image.
Preferably, the calculating the occurrence frequency of various code values in each statistic participating region comprises: counting a histogram of the coding values in the region, wherein the abscissa of the histogram is the coding value, and the ordinate is the frequency of occurrence of the coding value; the ordinate in the histogram is normalized as the occurrence probability of each code value.
Preferably, the determining the illumination image comprises:
determining a gray image, and performing low-pass filtering processing on the gray image to obtain the illumination image; alternatively, the first and second electrodes may be,
inputting a pre-trained illumination neural network, and taking the output as the illumination image; alternatively, the first and second electrodes may be,
determining a gray level image, inputting the gray level image into a pre-trained illumination neural network, and outputting the gray level image as the illumination image.
Preferably, the low-pass filtering is gaussian filtering.
Preferably, the manner of generating the training sample of the illumination neural network includes:
generating a pure illumination image by using an illumination model, synthesizing the pure illumination image and a pure texture image, taking the synthesized image or a gray level image thereof as an input training image of the illumination neural network, and taking the pure illumination image as an output target image of the illumination neural network; alternatively, the first and second electrodes may be,
and adding standard illumination in a test scene without illumination, shooting, taking a shot image or a gray image thereof as an input training image of the illumination neural network, and taking the standard illumination image as an output target image of the illumination neural network.
Preferably, the texture feature operator is an LBP operator or an LTP operator.
Preferably, the calculating the distance between the background image to be replaced and the illumination coding feature of each alternative background image is as follows: and calculating the Euclidean distance or included angle distance between the background image to be replaced and the illumination coding features of each alternative background image.
Preferably, the calculating the distance between the background image to be replaced and the additional illumination feature of each alternative background image is as follows: and calculating the Euclidean distance or included angle distance between the background image to be replaced and the additional illumination characteristic of each alternative background image.
Preferably, said ranking all alternative background images based on said weighted sum comprises: and for all the alternative background images, sorting the alternative background images according to the sequence from small to large of the weighted sum of the alternative background images and the background image to be replaced.
Preferably, the plurality of scales include a scale corresponding to an original size of the grayscale image.
An apparatus for replacing a background image, the apparatus comprising: the device comprises an image acquisition unit, an illumination characteristic calculation unit, a distance calculation unit and an image sorting unit;
the image acquisition unit is used for acquiring a background image to be replaced from an original image;
the illumination feature calculation unit is configured to, for each of the background image to be replaced and all candidate background images: determining an illumination image and carrying out scaling processing according to a plurality of scales; under each scale, processing the scaled image by using a texture feature operator to obtain texture features of the image as illumination coding features under the corresponding scale; combining illumination coding features of each background image to be replaced or the alternative background image under each scale to form corresponding illumination coding features of the background image to be replaced or the alternative background image;
the distance calculation unit is used for calculating the distance between the background image to be replaced and the illumination coding feature of each alternative background image as a first distance;
the image sorting unit is used for sorting all the candidate background images based on the corresponding first distances and recommending the sorted background images to the user for replacing the background images.
According to the technical scheme, the background image to be replaced is obtained from the original image; then, to each of the replacement background image and all alternative background images: determining an illumination image and carrying out scaling processing according to a plurality of scales; under each scale, processing the scaled image by using a texture feature operator to obtain texture features of the image as illumination coding features under the corresponding scale; combining the illumination coding features of each background image to be replaced or alternative background image under each scale to form the illumination coding features of the corresponding background image to be replaced or alternative background image; and finally, calculating the distance between the background image to be replaced and the illumination coding features of each alternative background image as a first distance, and recommending all the alternative background images to a user after sorting according to the respective corresponding first distances for replacing the background images. Through the processing, the distance between each alternative background image and the background image to be replaced on the basis of the illumination characteristic represented by the texture characteristic can be determined, so that the alternative background images are recommended to the user on the basis of the distance, the illumination characteristics of the replaced image and the original image tend to be consistent, the overall sense and the sense of reality of the whole image after background replacement are improved, the background replacement effect is improved, and the method is particularly suitable for replacing the background image of the real-time video.
Drawings
Fig. 1 is a schematic flowchart of a background image replacement method in an embodiment of the present application;
fig. 2 is a schematic diagram of a basic structure of a background image replacing apparatus according to the present application.
Detailed Description
For the purpose of making the objects, technical means and advantages of the present application more apparent, the present application will be described in further detail with reference to the accompanying drawings.
The basic idea of the application is that: and analyzing the illumination characteristics of the background image to be replaced and the alternative background images by using the texture characteristic operator, sequencing the alternative background images based on the distance from the illumination characteristics of the background image to be replaced, and recommending the alternative background images to the user.
Fig. 1 is a schematic flowchart of a background image replacement method in an embodiment of the present application. As shown in fig. 1, the method includes:
The original image herein refers to an image before background replacement is performed. The processing of the present application can be applied to processing of still pictures as well as to processing of moving videos. For a dynamic video (for example, various real-time video applications), the method for specifically obtaining an original image may be performed in various existing manners, for example, a frame of image is randomly extracted from the dynamic video or a frame of image is extracted according to a set policy, which is not limited in this application.
For both still pictures and moving videos, the original image can be divided into foreground and background portions, wherein the background portion is the portion to be replaced. Here, the background image to be replaced may be only a background portion of the original image, or may also be an image in which the background portion of the original image and a foreground portion filled with a set pixel value are combined, and for the latter two cases, the size of the background image to be replaced is still the same as that of the original image, so that feature comparison between the subsequent background image and the alternative background image may be facilitated, which will be described in detail later.
Next, the following steps 102 and 103 are performed for each of the background image to be replaced and all the alternative background images, and the step 104 is selectively performed:
The following processing of this step is performed on each of the image to be replaced and all alternative background images, where the specific processing of each image is the same, and the processing of one image a is taken as an example for explanation here.
The illumination image of image a is first determined. The illumination image is an image which visually reflects the illumination information of the image. Two exemplary methods of acquiring an illumination image are given below.
1) The first method first uses a gray image to obtain brightness information of the image, and then performs low-pass filtering on the gray image to obtain an illumination image. Specifically, it is first necessary to determine the grayscale image of the image a, and the specific determination is performed in the existing manner, for example, assuming that the image a is an RGB image, the manner of converting into the grayscale image I may be: the gray-scale value Igray of the gray-scale image I is 0.299 × R +0.587 × G +0.114 × B.
Meanwhile, the illumination information in the image generally changes slowly and is mainly represented in the low-frequency part of the image; the present application therefore obtains the illumination image L by low-pass filtering the grayscale image I. The specific low-pass filtering may be implemented by various existing low-pass filtering methods, such as two-dimensional gaussian filtering G (x, y) for gray scaleThe image I is subjected to low-pass filtering to obtain the pixel value L of the illumination image LIllumination of lightIgray G (x, y), wherein,
(x, y) represents the pixel coordinates, σ, of a grayscale image2Representing the width parameter of the gaussian filter function.
2) The second approach utilizes a pre-trained illumination neural network to determine the illumination image. Specifically, an illumination neural network is first trained using pairs of illuminated images (color or grayscale) with the illuminated image M as an input and the corresponding illuminated image N as an output target image. A proper illumination neural network can be trained by using a sufficient number of paired images M and corresponding illumination images N; and then inputting the trained illumination neural network to the background image to be replaced or the alternative background image, or inputting the gray level image of the background image to be replaced or the alternative background image, and outputting the gray level image as an illumination image. The grayscale image of the background image to be replaced or the grayscale image of the alternative background image may be determined according to the grayscale image determination method described in the first method. In addition, when the illumination neural network is trained, it is necessary to generate an image with illumination and a corresponding illumination image pair as a training sample in advance, and a specific generation manner may be as follows:
a. training samples can be generated in a technical synthesis mode; specifically, a pure-illumination image can be generated by using an illumination model, the pure-illumination image and a pure-texture image are synthesized, the synthesized image or a gray-scale image thereof is used as an input training image (i.e., an image M with illumination) of an illumination neural network, and a corresponding pure-illumination image is used as an output target image (i.e., an illumination image N) of the illumination neural network; the illumination model can adopt various existing illumination models;
b. training samples can be generated in an actual shooting mode; specifically, a test scene without illumination may be preset, standard illumination may be added and a shot may be taken, the shot image or a grayscale image thereof may be used as an input training image (i.e., an image M with illumination) of the illumination neural network, and a standard image corresponding to the added standard illumination may be used as an output target image (i.e., an illumination image N) of the illumination neural network.
The illumination image can be determined by the above-described processing. Next, the photo image is scaled according to the set scales. The scale here refers to the scaled size of the target image, such as 64 × 64, 128 × 128, etc. In the specific processing, if only one scale is set, the scaling processing may be performed once for the scale, or if a plurality of scales are set, the scaling processing may be performed once for each set scale. For example, if the scale can be set to include 64 × 64 and 128 × 128, the illumination image is scaled to 64 × 64 pixels, resulting in a first scaled image; then, the illumination image is zoomed into 128 × 128 pixels to obtain a second zoomed image; thus for two scales, one illumination image may result in two scaled images. Typically, the minimum scale of the image is 4 × 4, and the scales are sequentially increased to 8 × 8, 16 × 16, 32 × 32,64 × 64 …, and in this embodiment, the scaled image with the maximum scale is set to 128 × 128. In addition, there is a special case that the dimension may also be the original size of the illumination image, so that in fact the illumination image may be directly obtained as a corresponding scaled image without processing, and in combination with the number of the dimensions mentioned above, when the dimension is the original size of the illumination image, and only this dimension is set, then the illumination image is the only scaled image.
103, processing the zoomed image by using a texture feature operator under each scale to obtain the texture features of the image, wherein the texture features are used as illumination coding features under corresponding scales; and for each background image to be replaced or alternative background image, combining the adopted illumination coding features under various scales to form the illumination coding features of the corresponding background image to be replaced or alternative background image.
Similarly to step 102, all the processing of this step is performed on the image after the processing of step 102 is performed on each of the background image to be replaced and all the alternative background images, and the processing for each image is the same, and the following description will take an example of the image after the image a is processed in step 102. The image a is processed in step 102 to obtain images of several scales, that is, the image a may obtain one or more images of different scales after being processed in step 102. In this step, similar processing is required for images corresponding to each scale regardless of whether one or a plurality of images are obtained, and the following description will be given by taking an image B corresponding to a scale X as an example.
Firstly, processing an image B by using a texture feature operator to obtain the texture feature of the image B. The specific processing may include: and (3) coding each pixel participating in statistics of the image B by using a texture feature operator, carrying out region division on the coded image, calculating the occurrence frequency of various coding values in each region participating in statistics, and combining the occurrence frequencies of set regions to form the texture feature of the image B. The set area may be an area that participates in the statistics entirely or may be an area that participates in the statistics partially.
The texture feature operator may use an existing Local Binary Pattern (LBP) operator or Local Ternary Pattern (LTP) operator. The LBP operator is described as an example. LBP is an operator used to describe local texture features of an image, and has significant advantages of rotation invariance and gray scale invariance. When encoding is carried out by using LBP operator, each pixel (x) participating in statistics is traversedc,yc) For the pixel (x)c,yc) The following calculations were performed:
wherein the content of the first and second substances,
icis a pixel (x)c,yc) Gray value of ipIs a pixel (x)c,yc) The gray value of the surrounding P-th pixel, P being the pixel participating in the LBP calculation (x)c,yc) The number of peripheral pixels of (a) is generally 8 (i.e., in pixels (x)c,yc) As a center, within 3 x 3 regionDivide pixel (x)c,yc) The other 8 pixels are the above-mentioned pixels (x)c,yc) The surrounding pixels of (2).
Processing all pixels participating in statistics in the image B by using an LBP operator to obtain an LBP image; then, uniformly dividing the LBP image into a plurality of regions, and counting the occurrence probability of the LBP code value in each region participating in the counting, where the occurrence probability of the LBP code value can be counted in various conventional manners, for example, counting a histogram of the code value in each region, and then normalizing a vertical coordinate in the histogram to obtain the occurrence probability Hi ═ { h0, h1, h2, …, hN, …, hN }, where an abscissa of the histogram can be various possible values of the code value obtained after the LBP calculation, the vertical coordinate is the number of times of occurrence of each code value in the region (that is, the number of pixels of each code value obtained by calculation), i is a region index, N is a value index of the code value, N is all the values of the code value, and generally, N is 255; finally, the probability of occurrence of the set regions is combined to form the illumination code feature Hbk ═ { H1, H2, …, Hi, … HM }, where M is the total number of regions and k is the index of the scale.
In the above processing of obtaining image texture features by processing a zoomed image with a texture feature operator, pixels participating in statistics and regions participating in statistics are involved, mainly considering that in the present application, the replacement of a background image is performed, and a foreground part of an original image is kept unchanged, so that when a background image with illumination characteristics consistent with that of the background image to be replaced is searched, mainly observing illumination conditions of the background part in the original image, and illumination conditions of a part of the image in a candidate background image, which is coincident with the background part of the original image, so that when determining image texture features, only regions and pixels coincident with the background of the original image can be counted, or, for convenience of processing, and considering that the common background and the foreground illumination conditions are not greatly different, all blocks and pixels of the whole image can also be counted. However, in the meantime, considering the many possibilities of the background image to be replaced described in step 101, the regions and pixels participating in statistics are specifically classified into the following cases:
1) the background image to be replaced is an original image, and then the pixels participating in statistics in the background image to be replaced and the alternative background image may be all pixels of the corresponding image, or the pixels participating in statistics in the background image to be replaced and the alternative background image may also be pixels in the same position as the background portion of the original image in the corresponding image; the area participating in statistics in the background image to be replaced and the alternative background image may be all areas of the corresponding image, or the area participating in statistics in the background image to be replaced and the alternative background image may also be an area with the same position as the background part in the corresponding image;
2) the background image to be replaced is a background partial image, or an image in which a background portion and a foreground portion filled with preset pixels are combined, then pixels participating in statistics in the background image to be replaced and the alternative background image are pixels in the same position as the background portion in the corresponding image, and regions participating in statistics in the background image to be replaced and the alternative background image may be regions in the same position as the background portion in the corresponding image.
The above-mentioned method for determining the illumination coding feature is described by taking the image B corresponding to the dimension X as an example. As described above, the image B is an image corresponding to the image a (i.e., the background image to be replaced or the alternative background image) at the scale X, the illumination coding features at each scale adopted by the image a are obtained in a manner similar to the processing of the image B, and then the illumination coding features at each scale adopted by the image a are combined together to form the illumination coding features Fb of the image a, which is { Hb1, Hb2, …, Hbk, …, HbK }, where K is the total number of scales.
The illumination coding characteristics of the image obtained in the above manner can describe the change characteristics of the image texture, and are used for representing the illumination change in each region. In this embodiment, when describing the illumination characteristics, in addition to describing from the perspective of illumination change, it is preferable that the illumination characteristics are additionally described from the perspective of intuitive illumination information, and the following step 104 is performed on each of the background image to be replaced and the alternative background image:
and 104, combining all pixel values of the zoomed image as the additional illumination characteristic of the corresponding background image to be replaced or the alternative background image for the zoomed image with the preset scale.
Like step 103, the processing of this step is performed on each of the background image to be replaced and all candidate background images after the processing of step 102 is performed, and the processing for each image is the same, and the following description will be given taking an image of image a after the processing of step 102 as an example. The image a is processed in step 102 to obtain images of several scales, that is, the image a may obtain one or more images of different scales after being processed in step 102. In the images of the plurality of scales, for the zoomed image of the preset scale, the gray value of the pixel of the image can be directly used as the visual illumination information. In consideration of the amount of calculation and the realizability, the scale is usually set to be a smaller scale, and in particular, the set scale may be limited to a scale lower than the set value, and the gray values of the image pixels of the corresponding scales are combined to be the additional illumination feature of the image, for example, in this embodiment, the first two scales are set to be small scales, and the gray values of the corresponding image pixels are combined to obtain the additional illumination feature Fs of the image a (F1, F2), where F1 is (G) and (F2) are1,1,…,Gm,n,…,G4,4)，F2＝(G1,1,…,Gm,n,…,G8,8)。
The processing of step 104 is a preferred processing method, and this processing may not be included in the most basic background image replacement method, and based on this, step 104 is shown by a dashed line box in fig. 1.
As described above, the processing of step 102-104 is performed for each of the background image to be replaced and all the alternative background images. For any image, firstly, step 102 needs to be executed, and then steps 103 and 104 need to be executed; the processing of steps 103 and 104 does not have a fixed sequence, and may be performed simultaneously, or step 103 may be performed first and then step 104 may be performed, or step 104 may be performed first and then step 103 may be performed. For different images, assuming that the processing of step 102 and step 104 for any image is regarded as a processing group, the processing groups for different images may be performed in parallel or sequentially, or step 102 may be performed for all background images to be replaced and alternative background images (hereinafter, referred to as all images), and then steps 103 and 104 are performed for the processing results of all images. In summary, for all the to-be-replaced background images and alternative background images regarding the processing of step 102 and step 104, those skilled in the art can determine the specific execution sequence according to the actual conditions and requirements, as long as it is ensured that for each image itself, step 102 is executed first, and then steps 103 and 104 are executed, which is not limited to the sequence given in the above embodiment.
Step 105, calculating the distance between the background image to be replaced and the illumination coding feature of each alternative background image as a first distance; if step 104 has been performed, the distance between the background image to be replaced and the additional illumination feature of each alternative background image is further calculated as the second distance.
The illumination coding features of the background image to be replaced and all the alternative background images are obtained through the processing in the previous step, preferably, additional illumination features of the background image to be replaced and all the alternative background images can be further obtained through the processing in the step 104.
Specifically, for each alternative background image, calculating a distance between the alternative background image and the illumination coding feature of the background image to be replaced, taking the distance as a first distance Dist1, and assuming that the alternative background image is image m and the background image to be replaced is image n, Dist1 is Dist (Fb _ m, Fb _ n); if step 104 is executed before, in this step, for each alternative background image, the distance between the alternative background image and the additional illumination feature of the background image to be replaced is calculated, and this distance is taken as the second distance Dist2, and given that the alternative background image is image m and the background image to be replaced is image n, Dist2 is Dist (Fs _ m, Fs _ n).
The distance may be calculated by various methods, such as euclidean distance or angular distance. The first distance Dist1 and the second distance Dist2 may be calculated in the same manner or may be calculated in different manners. For example, the first distance Dist1 uses the euclidean distance; the second distance Dist2 uses an angled distance.
And 106, for all the alternative background images, ranking the alternative background images based on the corresponding first distances, and recommending the alternative background images to the user for replacing the background images.
Through the foregoing step 105, the first distance Dist1 corresponding to each candidate background image and the background image to be replaced is obtained, and the distance reflects the difference between the candidate background image and the background image to be replaced, so that the degree of difference between the candidate background images and the background image to be replaced in terms of illumination can be determined based on the Dist1, and the smaller the first distance is, the smaller the difference between the illumination characteristics reflected when the images are processed by the texture feature operator is, and based on this, the candidate background images can be sorted by using the corresponding degree of difference, and sequentially recommended to the user for replacing the background images.
Specifically, if step 104 is not executed before this step, this step may perform sorting according to the first distance between each alternative background image and the background image to be replaced when sorting all the alternative background images, of course, the alternative background images may be sorted most simply by considering only the first distance Dist1, and specifically, the alternative background images may be sorted in the order of the first distance from small to large; alternatively, the user may sort all alternative background images in combination with the first distance and other reference features as desired. If step 104 is executed before this step, when this step sorts all the candidate background images, it may first calculate, for each candidate background image and the background image to be replaced, a weighted sum of the corresponding first distance and second distance as a final distance Dist (Dist 1 k + Dist2 (1-k), and then sort based on the final distance Dist, and similarly to the foregoing, most simply, it may only consider the final distance Dist to sort all the candidate background images, and specifically, it may sort all the candidate background images in order of the final distance from small to large; alternatively, the user may sort the alternative background images as desired in combination with the final distance and other reference features. Where k (0< k <1) is a weight of the first distance, and is determined by different importance of the illumination coding feature and the additional illumination feature, it is assumed in this embodiment that the importance of the illumination coding feature obtained by LBP coding is greater, so that k is greater than 0.5, for example, k is 0.7. Of course, the value of k can be determined by those skilled in the art according to actual needs.
So far, the background image replacement method in the present application ends. The application also provides a background image replacing device which can be used for implementing the replacing method. Fig. 2 is a schematic diagram of the basic structure of the device. As shown in fig. 2, the apparatus includes: the device comprises an image acquisition unit, an illumination characteristic calculation unit, a distance calculation unit and an image sorting unit.
The image acquisition unit is used for acquiring a background image to be replaced from an original image.
An illumination feature calculation unit, configured to: determining an illumination image and carrying out scaling processing according to a plurality of scales; and under each scale, processing the scaled image by using a texture feature operator to obtain texture features of the image as illumination coding features under the corresponding scale, and combining the illumination coding features under each scale adopted by each background image to be replaced or alternative background image to form the illumination coding features of the corresponding background image to be replaced or alternative background image.
And the distance calculation unit is used for calculating the distance between the background image to be replaced and the illumination coding feature of each alternative background image as a first distance. And the image sorting unit is used for sorting all the candidate background images based on the corresponding first distances and recommending the sorted background images to the user for replacing the background images.
In the background image replacement method and device, the illumination characteristic of the image is considered in the replacement of the background image, and a user can select the background image with the illumination characteristic closer to that of the original background image for replacement, so that the illumination characteristic of the replaced image and the illumination characteristic of the original image tend to be consistent, the overall sense and the sense of reality of the whole image after the background replacement are improved, the background replacement effect is improved, and the method and device are particularly suitable for the replacement of the background image of a real-time video, so that the obtained composite image is more natural in scenes such as a video conference and the like, and the application requirements are better met.
The above description is only for the purpose of illustrating the preferred embodiments of the present invention and is not to be construed as limiting the invention, and any modifications, equivalents, improvements and the like made within the spirit and principle of the present invention should be included in the scope of the present invention.
Claims (15)
1. A method for replacing a background image, comprising:
acquiring a background image to be replaced from an original image;
for each of the background image to be replaced and all alternative background images: determining an illumination image and carrying out scaling processing according to a plurality of scales; under each scale, processing the zoomed image by using a texture feature operator to obtain the texture feature of the image, wherein the texture feature is used as an illumination coding feature under the corresponding scale; combining the illumination coding features of each to-be-replaced background image or each alternative background image under each scale to form the illumination coding features of the corresponding to-be-replaced background image or each alternative background image;
calculating the distance between the background image to be replaced and the illumination coding features of each alternative background image as a first distance; and for all the alternative background images, ranking the alternative background images based on the corresponding first distances, and recommending the alternative background images to the user for replacing the background images.
2. The method of claim 1, wherein processing the scaled image with a texture operator to obtain texture features of the image comprises:
and coding each pixel participating in statistics of the scaled image by using the texture feature operator, performing region division on the coded image, calculating the occurrence frequency of various coding values in each region participating in statistics, and combining the occurrence frequencies of set regions to form the texture feature.
3. The method according to claim 1, wherein after performing the scaling process on each of the background image to be replaced and all alternative background images, further comprising: for the zoomed image with a preset scale, combining all pixel values together to serve as an additional illumination characteristic of a corresponding background image to be replaced or an alternative background; calculating the distance between the background image to be replaced and the additional illumination characteristic of each alternative background image as a second distance;
the recommending the all alternative background images to the user after being sorted according to the respective corresponding first distances comprises: and calculating the weighted sum of the first distance and the second distance of all the alternative background images and the background image to be replaced, and arranging all the alternative background images based on the weighted sum.
4. The method according to claim 1, wherein when all the alternative background images are sorted, the sorting is performed according to the order of the distances between the alternative background images and the illumination coding features of the background image to be replaced from small to large.
5. The method of claim 2,
the background image to be replaced is an original image; the pixels participating in statistics in the background image to be replaced and the alternative background image are as follows: all pixels of the corresponding image, or pixels in the same position as the background part in the corresponding image; the areas participating in statistics in the background image to be replaced and the alternative background image are as follows: all regions of the corresponding image, or regions in the corresponding image that are in the same position as the background portion;
alternatively, the first and second electrodes may be,
the background image to be replaced is as follows: the background part image or the image formed by combining the background part with the foreground part filled with preset pixels; pixels participating in statistics in the background image to be replaced and the alternative background image are pixels with the same positions as the background part in the corresponding image; the area participating in statistics in the background image to be replaced and the alternative background image is an area with the same position as the background part in the corresponding image;
the background part and the foreground part are two parts divided from the original image.
6. The method according to claim 2 or 5, wherein the calculating of the frequency of occurrence of the various code values in each statistical-participating region comprises: counting a histogram of the coding values in the region, wherein the abscissa of the histogram is the coding value, and the ordinate is the frequency of occurrence of the coding value; the ordinate in the histogram is normalized as the occurrence probability of each code value.
7. The method of claim 1, wherein determining the illumination image comprises:
determining a gray image, and performing low-pass filtering processing on the gray image to obtain the illumination image; alternatively, the first and second electrodes may be,
inputting a pre-trained illumination neural network, and taking the output as the illumination image; alternatively, the first and second electrodes may be,
determining a gray level image, inputting the gray level image into a pre-trained illumination neural network, and outputting the gray level image as the illumination image.
8. The method of claim 7, wherein the low-pass filtering is gaussian filtering.
9. The method of claim 7, wherein generating the training samples of the illuminated neural network comprises:
generating a pure illumination image by using an illumination model, synthesizing the pure illumination image and a pure texture image, taking the synthesized image or a gray level image thereof as an input training image of the illumination neural network, and taking the pure illumination image as an output target image of the illumination neural network; alternatively, the first and second electrodes may be,
and adding standard illumination in a test scene without illumination, shooting, taking a shot image or a gray image thereof as an input training image of the illumination neural network, and taking the standard illumination image as an output target image of the illumination neural network.
10. The method of claim 1 or 2, wherein the texture feature operator is an LBP operator or an LTP operator.
11. The method according to claim 1, wherein the calculating the distance between the background image to be replaced and the illumination coding feature of each alternative background image is: and calculating the Euclidean distance or included angle distance between the background image to be replaced and the illumination coding features of each alternative background image.
12. The method according to claim 3, wherein the calculating the distance between the background image to be replaced and the additional illumination feature of each alternative background image is: and calculating the Euclidean distance or included angle distance between the background image to be replaced and the additional illumination characteristic of each alternative background image.
13. The method of claim 3 or 12, wherein said ranking all alternative background images based on said weighted sum comprises: and for all the alternative background images, sorting the alternative background images according to the sequence from small to large of the weighted sum of the alternative background images and the background image to be replaced.
14. The method of claim 1, wherein the plurality of scales comprises a scale corresponding to an original size of the grayscale image.
15. An apparatus for replacing a background image, the apparatus comprising: the device comprises an image acquisition unit, an illumination characteristic calculation unit, a distance calculation unit and an image sorting unit;
the image acquisition unit is used for acquiring a background image to be replaced from an original image;
the illumination feature calculation unit is configured to, for each of the background image to be replaced and all candidate background images: determining an illumination image and carrying out scaling processing according to a plurality of scales; under each scale, processing the scaled image by using a texture feature operator to obtain texture features of the image as illumination coding features under the corresponding scale; combining illumination coding features of each background image to be replaced or the alternative background image under each scale to form corresponding illumination coding features of the background image to be replaced or the alternative background image;
the distance calculation unit is used for calculating the distance between the background image to be replaced and the illumination coding feature of each alternative background image as a first distance;
the image sorting unit is used for sorting all the candidate background images based on the corresponding first distances and recommending the sorted background images to the user for replacing the background images.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202111210425.1A CN113947523B (en) | 2021-10-18 | 2021-10-18 | Method and device for replacing background image |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202111210425.1A CN113947523B (en) | 2021-10-18 | 2021-10-18 | Method and device for replacing background image |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN113947523A true CN113947523A (en) | 2022-01-18 |
| CN113947523B CN113947523B (en) | 2022-06-14 |
Family
ID=79331049
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202111210425.1A Active CN113947523B (en) | 2021-10-18 | 2021-10-18 | Method and device for replacing background image |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN113947523B (en) |
Citations (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101436299A (en) * | 2008-11-19 | 2009-05-20 | 哈尔滨工业大学 | Method for detecting natural scene image words |
| CN101777180A (en) * | 2009-12-23 | 2010-07-14 | 中国科学院自动化研究所 | Complex background real-time alternating method based on background modeling and energy minimization |
| US20110243451A1 (en) * | 2010-03-30 | 2011-10-06 | Hideki Oyaizu | Image processing apparatus and method, and program |
| CN103793477A (en) * | 2014-01-10 | 2014-05-14 | 同观科技（深圳）有限公司 | System and method for video abstract generation |
| CN103810722A (en) * | 2014-02-27 | 2014-05-21 | 云南大学 | Moving target detection method combining improved LBP (Local Binary Pattern) texture and chrominance information |
| CN107241600A (en) * | 2017-04-20 | 2017-10-10 | 中国科学技术大学 | A kind of static background inner frame coding method and device |
| CN108961302A (en) * | 2018-07-16 | 2018-12-07 | Oppo广东移动通信有限公司 | Image processing method, device, mobile terminal and computer readable storage medium |
| CN110148155A (en) * | 2019-04-19 | 2019-08-20 | 浙江万里学院 | A kind of multiple features liquidate the method for tracking target combined with adaptive standard |
-
2021
- 2021-10-18 CN CN202111210425.1A patent/CN113947523B/en active Active
Patent Citations (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101436299A (en) * | 2008-11-19 | 2009-05-20 | 哈尔滨工业大学 | Method for detecting natural scene image words |
| CN101777180A (en) * | 2009-12-23 | 2010-07-14 | 中国科学院自动化研究所 | Complex background real-time alternating method based on background modeling and energy minimization |
| US20110243451A1 (en) * | 2010-03-30 | 2011-10-06 | Hideki Oyaizu | Image processing apparatus and method, and program |
| CN103793477A (en) * | 2014-01-10 | 2014-05-14 | 同观科技（深圳）有限公司 | System and method for video abstract generation |
| CN103810722A (en) * | 2014-02-27 | 2014-05-21 | 云南大学 | Moving target detection method combining improved LBP (Local Binary Pattern) texture and chrominance information |
| CN107241600A (en) * | 2017-04-20 | 2017-10-10 | 中国科学技术大学 | A kind of static background inner frame coding method and device |
| CN108961302A (en) * | 2018-07-16 | 2018-12-07 | Oppo广东移动通信有限公司 | Image processing method, device, mobile terminal and computer readable storage medium |
| CN110148155A (en) * | 2019-04-19 | 2019-08-20 | 浙江万里学院 | A kind of multiple features liquidate the method for tracking target combined with adaptive standard |
Also Published As
| Publication number | Publication date |
|---|---|
| CN113947523B (en) | 2022-06-14 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| Lim et al. | DSLR: Deep stacked Laplacian restorer for low-light image enhancement | |
| Matern et al. | Exploiting visual artifacts to expose deepfakes and face manipulations | |
| Ren et al. | Low-light image enhancement via a deep hybrid network | |
| Fang et al. | No reference quality assessment for screen content images with both local and global feature representation | |
| CN110598610B (en) | Target significance detection method based on neural selection attention | |
| Li et al. | Multi-scale single image dehazing using laplacian and gaussian pyramids | |
| Hosseini et al. | Encoding visual sensitivity by maxpol convolution filters for image sharpness assessment | |
| Po et al. | A novel patch variance biased convolutional neural network for no-reference image quality assessment | |
| CN111292264A (en) | Image high dynamic range reconstruction method based on deep learning | |
| CN107358141B (en) | Data identification method and device | |
| CN107958441A (en) | Image split-joint method, device, computer equipment and storage medium | |
| CN111062314A (en) | Image selection method and device, computer readable storage medium and electronic equipment | |
| CN108647696B (en) | Picture color value determining method and device, electronic equipment and storage medium | |
| CN113762107A (en) | Object state evaluation method and device, electronic equipment and readable storage medium | |
| CN112241745A (en) | Characteristic point extraction method based on illumination invariant color space | |
| CN113301408A (en) | Video data processing method and device, electronic equipment and readable storage medium | |
| Su et al. | Going the extra mile in face image quality assessment: A novel database and model | |
| Parekh et al. | A survey of image enhancement and object detection methods | |
| CN112712525A (en) | Multi-party image interaction system and method | |
| CN113947523B (en) | Method and device for replacing background image | |
| CN110489584B (en) | Image classification method and system based on dense connection MobileNet model | |
| CN114463379A (en) | Dynamic capturing method and device for video key points | |
| CN113901916A (en) | Visual optical flow feature-based facial fraud action identification method | |
| CN113706636A (en) | Method and device for identifying tampered image | |
| CN111950565B (en) | Abstract picture image direction identification method based on feature fusion and naive Bayes |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |