CN113301384B - Background replacing method and device, electronic equipment and readable storage medium - Google Patents
Background replacing method and device, electronic equipment and readable storage medium Download PDFInfo
- Publication number
- CN113301384B CN113301384B CN202110558649.5A CN202110558649A CN113301384B CN 113301384 B CN113301384 B CN 113301384B CN 202110558649 A CN202110558649 A CN 202110558649A CN 113301384 B CN113301384 B CN 113301384B
- Authority
- CN
- China
- Prior art keywords
- background
- image
- filtering
- replaced
- replacing
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 238000000034 method Methods 0.000 title claims abstract description 45
- 238000001914 filtration Methods 0.000 claims abstract description 167
- 238000012216 screening Methods 0.000 claims abstract description 13
- 230000004044 response Effects 0.000 claims abstract description 9
- 238000006243 chemical reaction Methods 0.000 claims description 32
- 239000011159 matrix material Substances 0.000 claims description 20
- 238000004590 computer program Methods 0.000 claims description 6
- 230000000694 effects Effects 0.000 abstract description 10
- 230000008569 process Effects 0.000 description 13
- 238000010586 diagram Methods 0.000 description 8
- 238000012545 processing Methods 0.000 description 5
- 230000008859 change Effects 0.000 description 4
- 238000004891 communication Methods 0.000 description 3
- 230000008878 coupling Effects 0.000 description 3
- 238000010168 coupling process Methods 0.000 description 3
- 238000005859 coupling reaction Methods 0.000 description 3
- 239000007787 solid Substances 0.000 description 3
- 238000005516 engineering process Methods 0.000 description 2
- 230000006870 function Effects 0.000 description 2
- 230000003993 interaction Effects 0.000 description 2
- 238000012986 modification Methods 0.000 description 2
- 230000004048 modification Effects 0.000 description 2
- 238000006467 substitution reaction Methods 0.000 description 2
- 230000003321 amplification Effects 0.000 description 1
- 238000004364 calculation method Methods 0.000 description 1
- 239000003086 colorant Substances 0.000 description 1
- 238000012937 correction Methods 0.000 description 1
- 238000011161 development Methods 0.000 description 1
- 230000007613 environmental effect Effects 0.000 description 1
- 238000005286 illumination Methods 0.000 description 1
- 238000003199 nucleic acid amplification method Methods 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
- 238000011160 research Methods 0.000 description 1
- 238000000926 separation method Methods 0.000 description 1
Images
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
- H04N21/20—Servers specifically adapted for the distribution of content, e.g. VOD servers; Operations thereof
- H04N21/23—Processing of content or additional data; Elementary server operations; Server middleware
- H04N21/234—Processing of video elementary streams, e.g. splicing of video streams or manipulating encoded video stream scene graphs
- H04N21/23418—Processing of video elementary streams, e.g. splicing of video streams or manipulating encoded video stream scene graphs involving operations for analysing video streams, e.g. detecting features or characteristics
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
- H04N21/20—Servers specifically adapted for the distribution of content, e.g. VOD servers; Operations thereof
- H04N21/23—Processing of content or additional data; Elementary server operations; Server middleware
- H04N21/234—Processing of video elementary streams, e.g. splicing of video streams or manipulating encoded video stream scene graphs
- H04N21/23424—Processing of video elementary streams, e.g. splicing of video streams or manipulating encoded video stream scene graphs involving splicing one content stream with another content stream, e.g. for inserting or substituting an advertisement
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
- H04N21/20—Servers specifically adapted for the distribution of content, e.g. VOD servers; Operations thereof
- H04N21/23—Processing of content or additional data; Elementary server operations; Server middleware
- H04N21/234—Processing of video elementary streams, e.g. splicing of video streams or manipulating encoded video stream scene graphs
- H04N21/2343—Processing of video elementary streams, e.g. splicing of video streams or manipulating encoded video stream scene graphs involving reformatting operations of video signals for distribution or compliance with end-user requests or end-user device requirements
- H04N21/234309—Processing of video elementary streams, e.g. splicing of video streams or manipulating encoded video stream scene graphs involving reformatting operations of video signals for distribution or compliance with end-user requests or end-user device requirements by transcoding between formats or standards, e.g. from MPEG-2 to MPEG-4 or from Quicktime to Realvideo
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
- H04N21/40—Client devices specifically adapted for the reception of or interaction with content, e.g. set-top-box [STB]; Operations thereof
- H04N21/43—Processing of content or additional data, e.g. demultiplexing additional data from a digital video stream; Elementary client operations, e.g. monitoring of home network or synchronising decoder's clock; Client middleware
- H04N21/44—Processing of video elementary streams, e.g. splicing a video clip retrieved from local storage with an incoming video stream or rendering scenes according to encoded video stream scene graphs
- H04N21/44008—Processing of video elementary streams, e.g. splicing a video clip retrieved from local storage with an incoming video stream or rendering scenes according to encoded video stream scene graphs involving operations for analysing video streams, e.g. detecting features or characteristics in the video stream
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
- H04N21/40—Client devices specifically adapted for the reception of or interaction with content, e.g. set-top-box [STB]; Operations thereof
- H04N21/43—Processing of content or additional data, e.g. demultiplexing additional data from a digital video stream; Elementary client operations, e.g. monitoring of home network or synchronising decoder's clock; Client middleware
- H04N21/44—Processing of video elementary streams, e.g. splicing a video clip retrieved from local storage with an incoming video stream or rendering scenes according to encoded video stream scene graphs
- H04N21/44016—Processing of video elementary streams, e.g. splicing a video clip retrieved from local storage with an incoming video stream or rendering scenes according to encoded video stream scene graphs involving splicing one content stream with another content stream, e.g. for substituting a video clip
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
- H04N21/40—Client devices specifically adapted for the reception of or interaction with content, e.g. set-top-box [STB]; Operations thereof
- H04N21/43—Processing of content or additional data, e.g. demultiplexing additional data from a digital video stream; Elementary client operations, e.g. monitoring of home network or synchronising decoder's clock; Client middleware
- H04N21/44—Processing of video elementary streams, e.g. splicing a video clip retrieved from local storage with an incoming video stream or rendering scenes according to encoded video stream scene graphs
- H04N21/4402—Processing of video elementary streams, e.g. splicing a video clip retrieved from local storage with an incoming video stream or rendering scenes according to encoded video stream scene graphs involving reformatting operations of video signals for household redistribution, storage or real-time display
- H04N21/440218—Processing of video elementary streams, e.g. splicing a video clip retrieved from local storage with an incoming video stream or rendering scenes according to encoded video stream scene graphs involving reformatting operations of video signals for household redistribution, storage or real-time display by transcoding between formats or standards, e.g. from MPEG-2 to MPEG-4
Landscapes
- Engineering & Computer Science (AREA)
- Multimedia (AREA)
- Signal Processing (AREA)
- Business, Economics & Management (AREA)
- Marketing (AREA)
- Image Processing (AREA)
Abstract
The application provides a background replacing method, a background replacing device, an electronic device and a readable storage medium, wherein an HSV (hue, saturation, value) format image of a background image to be replaced is determined in response to a background replacing request; aiming at the HSV-format image, screening the image according to a preset first filtering threshold set, and determining an initial background area in the HSV-format image; performing pixel filtering on the initial background area according to the determined second filtering threshold group to determine a target background area of the HSV-format image; the filtering range of the second set of filtering thresholds is smaller than the filtering range of the first set of filtering thresholds; and replacing the background image to be replaced with a preset replacement background image based on the target background area, and determining the replaced target image. Therefore, through image screening and filtering twice, the target background area in the background image to be replaced is accurately determined, background replacement is carried out on the target background area, and the accuracy of background switching and the switching display effect can be improved.
Description
Technical Field
The present application relates to the field of live video technologies, and in particular, to a method and an apparatus for replacing a background, an electronic device, and a readable storage medium.
Background
With the rapid development of network technology, the interaction modes among users become various, video conferences and online video teaching are representative interaction modes, and during the video conferences and online video teaching, live broadcast backgrounds are generally required to be switched to show live broadcast contents to other users watching videos more abundantly.
At the present stage, a speaker who carries out a video conference and gives lessons on an online video places a background curtain of a pure color to switch video backgrounds, but in the background switching process, due to the influence of external environment factors, there is a situation of misoperation in the matting process, which results in an unsatisfactory effect after background switching, how to more accurately determine the background area to switch backgrounds, and the problem to be solved urgently is solved.
Disclosure of Invention
In view of this, an object of the present application is to provide a method and an apparatus for replacing a background, an electronic device, and a readable storage medium, which accurately determine a target background area in a background image to be replaced by two times of image screening and filtering, perform background replacement on the target background area, and can improve accuracy of background switching and a switching display effect.
In a first aspect, an embodiment of the present application provides a method for replacing a background, where the method includes:
responding to the background replacing request, performing image format conversion on the background image to be replaced, and determining an HSV (hue, saturation and value) format image of the background image to be replaced;
aiming at the HSV-format image, screening the image according to a preset first filtering threshold group to determine an initial background area in the HSV-format image;
performing pixel filtering on the initial background area according to the determined second filtering threshold value group to determine a target background area of the HSV-format image; wherein a filtering range of the second set of filtering thresholds is less than a filtering range of the first set of filtering thresholds;
and replacing the background image to be replaced with a preset replacement background image based on the target background area, and determining the replaced target image.
In one possible implementation, the first set of filtering thresholds includes a hue filtering threshold, a saturation filtering threshold, and a first brightness filtering threshold.
In one possible embodiment, the second set of filter thresholds is determined by:
determining a luminance mean value based on pixel luminance of each pixel included in the initial background region, the luminance mean value being determined as a second luminance filtering threshold;
determining the second set of filtering thresholds based on the hue filtering threshold, the saturation filtering threshold, and the second brightness filtering threshold.
In a possible embodiment, the performing, in response to a background replacement request, image format conversion on a background image to be replaced to determine an HSV-format image of the background image to be replaced includes:
responding to a background replacing request, and converting the background image to be replaced into a BGR format image;
and determining the HSV format image based on the BGR format image.
In a possible implementation manner, the replacing the background image to be replaced with a preset replacement background image based on the target background area, and determining a replaced target image includes:
generating a mask matrix corresponding to the background image to be replaced based on the target background area; wherein each element in the mask matrix corresponds to each pixel included in the background image to be replaced, and each element value indicates whether the corresponding pixel belongs to the target background region;
determining a replacing background image based on the mask matrix, the BGR format image and a preset replacing background image;
replacing the background image to be replaced with the replacement background image, and determining the target image.
In a second aspect, an embodiment of the present application further provides a replacement apparatus for a background, where the replacement apparatus includes:
the image format conversion module is used for responding to the background replacement request, performing image format conversion on the background image to be replaced and determining an HSV (hue, saturation and value) format image of the background image to be replaced;
the initial background determining module is used for screening the HSV format image according to a preset first filtering threshold set so as to determine an initial background area in the HSV format image;
the target background determining module is used for performing pixel filtering on the initial background area according to the determined second filtering threshold group to determine a target background area of the HSV-format image; wherein the filtering range of the second set of filtering thresholds is less than the filtering range of the first set of filtering thresholds;
and the background image replacing module is used for replacing the background image to be replaced with a preset replacing background image based on the target background area and determining a replaced target image.
In one possible implementation, the first set of filtering thresholds includes a hue filtering threshold, a saturation filtering threshold, and a first brightness filtering threshold.
In a possible implementation, the replacement apparatus further includes a filtering threshold determination module, and the filtering threshold determination module is configured to:
determining a luminance mean value based on pixel luminance of each pixel included in the initial background region, the luminance mean value being determined as a second luminance filtering threshold;
determining the second set of filtering thresholds based on the hue filtering threshold, the saturation filtering threshold, and the second brightness filtering threshold.
In a possible embodiment, when the image format conversion module is configured to perform image format conversion on the background image to be replaced in response to the background replacement request, and determine an HSV format image of the background image to be replaced, the image format conversion module is configured to:
responding to a background replacing request, and converting the background image to be replaced into a BGR format image;
determining the HSV format image based on the BGR format image.
In a possible implementation manner, when the background image replacement module is configured to replace the background image to be replaced with a preset replacement background image based on the target background area, and determine a replaced target image, the background image replacement module is configured to:
generating a mask matrix corresponding to the background image to be replaced based on the target background area; wherein each element in the mask matrix corresponds to each pixel included in the background image to be replaced, and each element value indicates whether the corresponding pixel belongs to the target background region;
determining a replacing background image based on the mask matrix, the BGR format image and a preset replacing background image;
replacing the background image to be replaced with the replacement background image, and determining the target image.
In a third aspect, an embodiment of the present application further provides an electronic device, including: a processor, a storage medium and a bus, the storage medium storing machine-readable instructions executable by the processor, the processor and the storage medium communicating over the bus when the electronic device is operating, the processor executing the machine-readable instructions to perform the steps of the method of the background of any of the first aspects.
In a fourth aspect, the present application further provides a computer-readable storage medium, on which a computer program is stored, where the computer program is executed by a processor to perform the steps of the method for replacing the background of the first aspect.
The background replacing method, the background replacing device, the electronic equipment and the readable storage medium provided by the embodiment of the application respond to a background replacing request, perform image format conversion on a background image to be replaced, and determine an HSV (hue, saturation and value) format image of the background image to be replaced; for the HSV format image, screening the image according to a preset first filtering threshold set to determine an initial background area in the HSV format image; performing pixel filtering on the initial background area according to the determined second filtering threshold value group to determine a target background area of the HSV-format image; wherein the filtering range of the second set of filtering thresholds is less than the filtering range of the first set of filtering thresholds; and replacing the background image to be replaced with a preset replacement background image based on the target background area, and determining the replaced target image.
When a background replacement request is received, performing image format conversion on a background image to be replaced, determining an HSV (hue, saturation and value) format image corresponding to the background image to be replaced, performing image filtering on the determined HSV format image according to a first filtering threshold value, and determining an initial background area; and performing pixel filtering on the initial background area by using the determined second filtering threshold group to determine a target background area of the HSV-format image, replacing the background image to be replaced with a preset replacement background image according to the determined target background area, and determining the replaced target image. Therefore, the accuracy of background switching and the switching display effect can be improved.
In order to make the aforementioned objects, features and advantages of the present application comprehensible, preferred embodiments accompanied with figures are described in detail below.
Drawings
In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings that are required to be used in the embodiments will be briefly described below, it should be understood that the following drawings only illustrate some embodiments of the present application and therefore should not be considered as limiting the scope, and for those skilled in the art, other related drawings can be obtained from the drawings without inventive effort.
FIG. 1 is a flow chart of an alternative method of context provided by an embodiment of the present application;
FIG. 2 is a schematic diagram of a background switching process;
FIG. 3 is a schematic structural diagram of an alternative apparatus of the background according to an embodiment of the present disclosure;
fig. 4 is a second schematic structural diagram of an alternative apparatus of the background according to the embodiment of the present application;
fig. 5 is a schematic structural diagram of an electronic device according to an embodiment of the present application.
Detailed Description
In order to make the objects, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application, and it is obvious that the described embodiments are only a part of the embodiments of the present application, and not all the embodiments. The components of the embodiments of the present application, generally described and illustrated in the figures herein, can be arranged and designed in a wide variety of different configurations. Thus, the following detailed description of the embodiments of the present application, presented in the accompanying drawings, is not intended to limit the scope of the claimed application, but is merely representative of selected embodiments of the application. Every other embodiment that can be obtained by a person skilled in the art without making creative efforts based on the embodiments of the present application falls within the protection scope of the present application.
First, an application scenario to which the present application is applicable will be described. The application can be applied to the technical field of video live broadcast.
Research shows that, in the present stage, a speaker who carries out a video conference and online video teaching usually places a plain background curtain to switch video backgrounds, but in the background switching process, due to the influence of external environmental factors, there is a situation of misoperation in the matting process, which results in an unsatisfactory effect after background switching, how to more accurately determine the background area to switch backgrounds, and the problem to be solved urgently is solved.
Based on this, the embodiment of the application provides a method for replacing a background, so as to improve the accuracy of background switching and the switching display effect.
Referring to fig. 1, fig. 1 is a flowchart illustrating an alternative method of the background according to an embodiment of the present disclosure. As shown in fig. 1, an alternative method of the background provided by the embodiment of the present application includes:
s101, responding to the background replacing request, performing image format conversion on the background image to be replaced, and determining an HSV (hue, saturation and value) format image of the background image to be replaced.
In the step, in response to a background replacement request of a user, a current image to be replaced is determined, and the determined image to be replaced is subjected to image format replacement to determine the HSV image format of the image to be replaced.
Here, the application scene that this application embodiment was directed to can be scenes such as live video, video conference, video recording, and in live video course or video conference process, often need to live the camera shooting background of the director or live broadcast lecturer and replace to reach the purpose of abundant show, realize the function that the portrait is scratched and then further carry out the background replacement through the curtain that uses green curtain or other pure color backgrounds usually.
Here, for the application scene, the background image to be replaced may be a frame of image corresponding to a background replacement request initiated by a user in a live video broadcast process, where the background image to be replaced includes a background area and a non-background area, and in this embodiment of the present application, the corresponding background area is determined from the background image to be replaced, and background replacement is performed.
Here, the responding to the background replacement request may be that the user sends the background replacement request through a touch operation on the replacement control, or receives the background replacement request sent by the user through voice control, and is not limited in particular.
The touch operation of the user on the replacement control can be click operation, sliding operation, long-time pressing operation and the like.
Here, the background to be replaced for the embodiment of the present application is generally a curtain including a background of a solid color, such as a solid green, a solid blue, and the like.
Here, the original background image to be replaced is generally a YUV (brightness, chrominance) image, and when filtering each pixel in the image in different channels, the hue channel, the saturation channel, and the luminance channel of the image are targeted, so when determining a background area in the image by image filtering, the background image to be replaced needs to be converted from a YUV format to an HSV format image.
In a modern color television system, a three-tube color camera or a color CCD camera is generally used for image capture, then the obtained color image signals are subjected to color separation and respective amplification and correction to obtain RGB, a matrix conversion circuit is further used to obtain a luminance signal Y and two color difference signals R-Y (i.e., U) and B-Y (i.e., V), and finally, a transmitting end respectively encodes the luminance and two color difference signals, and transmits the encoded signals through the same channel. This color representation method is called YUV color space representation.
In this embodiment of the present application, the step of performing image format conversion on the background image to be replaced in response to the background replacement request, and determining an HSV format image of the background image to be replaced includes:
a1: and responding to a background replacement request, and converting the background image to be replaced into the BGR format image.
In the step, when a background replacement request of a user is received, the format of the image to be replaced is converted into a BGR format from a YUV format, and the image in the BGR format is obtained.
Here, at the time of image conversion, the format of the image to be replaced can be converted from YUV format to BGR format by the following formula:
R＝Y+1.403×(V-128)；
G＝Y-0.343×(U-128)-0.714×(V-128)；
B＝Y+1.770×(U-128)；
a2: and determining the HSV format image based on the BGR format image.
In the step, after obtaining the image in the BGR format from the background image to be replaced, image conversion is carried out according to the image in the BGR format to obtain the HSV image.
S102, aiming at the HSV-format image, image screening is carried out according to a preset first filtering threshold set, and an initial background area in the HSV-format image is determined.
In this step, for the HSV image determined in step S101, image screening is performed according to a preset first filtering threshold set, so as to determine an initial background area in the HSV image.
In the embodiment of the present application, the filtering on the background image to be replaced is filtering on a pixel level.
In the process of carrying out video live broadcast or video conference, in order to facilitate the switching of the background and the matting of non-background images in the images, generally non-background objects generally select colors different from the color of a background curtain so as to distinguish the non-background objects from the background, so that the filtering of the background images to be replaced by the first filtering threshold value group is the process of filtering parts obviously not belonging to the background in the background images to be replaced.
When filtering an image, each pixel included in a background image to be replaced needs to be filtered in a hue channel, a saturation channel and a brightness channel, so as to obtain an initial background area in the background image to be replaced.
S103, performing pixel filtering on the initial background area according to the determined second filtering threshold group, and determining a target background area of the HSV-format image; wherein the filtering range of the second set of filtering thresholds is smaller than the filtering range of the first set of filtering thresholds.
In the step, the initial background area is filtered again in a pixel level according to the determined second filtering threshold value group, and a target background area in the HSV-format background image to be replaced is determined.
Here, the filtering range of the second filtering threshold group is smaller than that of the first filtering threshold group, and the chroma threshold value and the saturation threshold value in the second filtering threshold group are the same as those in the first filtering threshold group, except that the luminance threshold value in the luminance channel is different.
Here, in an actual application scenario, with a change of an ambient lighting environment, characteristics such as brightness of the same green screen may also change correspondingly, which will greatly affect a matting effect, and is specifically represented as mistaking a non-background region in an image as a background region, so that a person in the matting is not fully displayed after the background is replaced, which affects a display effect.
For example, a person in front of the background generates a shadow due to illumination, and for a shadow region, its hue and saturation do not change much with respect to the original image, and mainly the luminance information changes much, so that it may be considered to perform pixel filtering again on the original background region according to the luminance channel.
In an embodiment of the present application, the second filtering threshold set for filtering the background image to be replaced may be determined by:
b1: determining a luminance mean value based on a pixel luminance of each pixel included in the initial background region, the luminance mean value being determined as a second luminance filtering threshold.
In this step, for the determined initial background region, the pixel luminance of each pixel included in the initial background region is determined, a luminance average value in the initial background region is determined based on the pixel luminance of each pixel, and the determined luminance average value is determined as a second luminance filtering threshold value in the second filtering threshold value group.
Taking the shadow part in the background image as an example, in the whole background green screen, the whole green screen is brighter, the brightness of the shadow part is lower, the dark green of the shadow part only occupies a small part of the original image, after the average value calculation is carried out on the brightness (V) channel of the whole green (HSV format), a brightness range which is up and down floated on the average value is used, so that the HSV threshold range used for image matting can be reduced, and a more accurate target background area is determined.
b2: determining the second set of filtering thresholds based on the hue filtering threshold, the saturation filtering threshold, and the second brightness filtering threshold.
In this step, a second filtering threshold set is determined according to the hue filtering threshold value, the saturation filtering threshold value and the determined second brightness filtering threshold value in the first filtering threshold set.
Here, the initial background area is secondarily filtered in a narrower range through the second filtering threshold group, so that the target background area included in the background image to be replaced is more accurately determined.
And S104, replacing the background image to be replaced with a preset replacement background image based on the target background area, and determining the replaced target image.
In the step, the determined target background area in the background image to be replaced is replaced by a preset replacement background image, and the replaced target image is determined.
Here, the determined target background area in the image to be replaced is replaced with a preset replacement background image, so that after the replaced target image is obtained, the target image is displayed, and the switching of the background image is completed in the process of live video or video conference.
In this embodiment of the present application, the step of replacing the background image to be replaced with a preset replacement background image based on the target background area, and determining the replaced target image includes:
c1: generating a mask matrix corresponding to the background image to be replaced based on the target background area; wherein each element in the mask matrix corresponds to each pixel included in the background image to be replaced, and each element value indicates whether the corresponding pixel belongs to the target background region.
In the step, according to the determined target background area in the background image to be replaced, a mask matrix which is corresponding to the background image to be replaced and can represent whether the area where each pixel in the background image to be replaced is a background area or a non-background area is generated.
Wherein each element in the mask matrix corresponds to each pixel included in the background image to be replaced, and each element value indicates whether the corresponding pixel belongs to the target background region.
Here, since the mask matrices have two values of only 0 and 1, it can be specified that 0 represents a non-background region and 1 represents a background region, and then a corresponding mask matrix is generated for filling a subsequent preset replacement background image.
c2: and determining a replacing background image based on the mask matrix, the BGR format image and a preset replacing background image.
In the step, a replacing background image is determined according to the determined mask matrix, the converted BGR image and a preset background image.
And then, determining a converted alternative background image according to the preset background image with the determined target background area and the converted BGR format image.
Here, since the format of the background image to be replaced is YUV format, the format of the replacement background image should also be YUV format, and the BGR format image after replacement needs to be converted into YUV format to obtain the converted replacement background image.
Here, at the time of image conversion, the format of the image to be replaced can be converted from the BGR format to the YUV format by the following formula:
Y＝0.299R+0.587G+0.114B；
V＝0.500R-0.419G-0.081B+128；
U＝-0.169R-0.331G+0.500B+128；
c3: replacing the background image to be replaced with the replacement background image, and determining the target image.
In the step, the determined background image to be replaced is replaced by a replacement background image, and the target image after the background is replaced is determined, so that the background of the live video or the video conference is switched.
The following provides a specific background switching embodiment, which can perform background image replacement in video live broadcast through a background replacement device, where the background replacement device includes: referring to fig. 2, fig. 2 is a schematic diagram of a background switching process, and as shown in fig. 2, specific background replacement steps are as follows:
(1) The camera video frame reading module sends a green screen background replacement request and an original image (the original image is usually in a YUV420P format) to the green screen matting module; the camera video frame reading module is connected with the green curtain cutout module through an external interface API of the green curtain cutout module;
(2) The green curtain matting module sends the received original image to the format conversion module;
(3) After the format conversion module receives the original image, the processing is divided into two steps: firstly, converting an original image into a BGR format; secondly, converting the BGR format image into an HSV format, and sending the HSV format image and a mask generation request to a mask generation module;
(4) The mask generation module acquires a current green curtain background area according to a wide HSV threshold interval based on a received mask generation request, and calculates the pixel mean value of a V channel in H/S/V three channels of the area; according to the V-channel pixel mean value calculated in the last step, the V-channel pixel mean value is used as a change factor coefficient, acts on the default HSV threshold value of the color interval, resets the filtering range and generates a mask; sending the mask and the original image in the BGR format to a background pixel processing module;
(5) The background pixel processing module carries out pixel operation through the received mask, the original image in the BGR format and a preset background image to replace the green curtain background into the preset background image; returning the BGR format image with the replaced background to a format conversion module;
(6) And the format conversion module converts the BGR format image back into an original YUV420P format image and returns the image after background replacement to an interface API for subsequent processing.
The method for replacing the background, provided by the embodiment of the application, is used for performing image format conversion on the background image to be replaced in response to a background replacement request, and determining an HSV (hue, saturation and value) format image of the background image to be replaced; for the HSV format image, screening the image according to a preset first filtering threshold set to determine an initial background area in the HSV format image; performing pixel filtering on the initial background area according to the determined second filtering threshold value group to determine a target background area of the HSV-format image; wherein the filtering range of the second set of filtering thresholds is less than the filtering range of the first set of filtering thresholds; and replacing the background image to be replaced with a preset replacement background image based on the target background area, and determining the replaced target image.
When a background replacement request is received, performing image format conversion on a background image to be replaced to determine an HSV (hue, saturation and value) format image corresponding to the background image to be replaced, performing image filtering on the determined HSV format image according to a first filtering threshold value, and determining an initial background area; and performing pixel filtering on the initial background area by using the determined second filtering threshold group to determine a target background area of the HSV-format image, replacing the background image to be replaced with a preset replacement background image according to the determined target background area, and determining the replaced target image. Therefore, the accuracy of background switching and the switching display effect can be improved.
Based on the same inventive concept, a background switching device corresponding to the background replacement method is also provided in the embodiments of the present application, and since the principle of solving the problem of the device in the embodiments of the present application is similar to the background replacement method in the embodiments of the present application, the implementation of the device may refer to the implementation of the method, and the repeated parts are not described again.
Referring to fig. 3 and 4, fig. 3 is a first schematic structural diagram of a background replacement device provided in an embodiment of the present application, and fig. 4 is a second schematic structural diagram of the background replacement device provided in the embodiment of the present application. As shown in fig. 3, the replacement apparatus 300 includes:
the image format conversion module 310 is configured to perform image format conversion on the background image to be replaced in response to the background replacement request, and determine an HSV format image of the background image to be replaced;
an initial background determining module 320, configured to perform image screening on the HSV format image according to a preset first filtering threshold set, and determine an initial background region in the HSV format image;
a target background determining module 330, configured to perform pixel filtering on the initial background region according to the determined second filtering threshold group, and determine a target background region of the HSV format image; wherein the filtering range of the second set of filtering thresholds is less than the filtering range of the first set of filtering thresholds;
and a background image replacing module 340, configured to replace the background image to be replaced with a preset replacement background image based on the target background area, and determine a replaced target image.
Further, as shown in fig. 4, the replacing apparatus 300 further includes a filtering threshold determining module 350, where the filtering threshold determining module 350 is configured to:
determining a luminance mean value based on pixel luminance of each pixel included in the initial background region, the luminance mean value being determined as a second luminance filtering threshold;
determining the second set of filtering thresholds based on the hue filtering threshold, the saturation filtering threshold, and the second brightness filtering threshold.
Further, the first set of filtering thresholds includes a hue filtering threshold, a saturation filtering threshold, and a first brightness filtering threshold.
Further, when the image format conversion module 310 is configured to perform image format conversion on the background image to be replaced in response to the background replacement request, and determine an HSV format image of the background image to be replaced, the image format conversion module 310 is configured to:
responding to a background replacing request, and converting the background image to be replaced into a BGR format image;
and determining the HSV format image based on the BGR format image.
Further, when the background image replacing module 340 is configured to replace the background image to be replaced with a preset replacement background image based on the target background area, and determine a replaced target image, the background image replacing module 340 is configured to:
generating a mask matrix corresponding to the background image to be replaced based on the target background area; wherein each element in the mask matrix corresponds to each pixel included in the background image to be replaced, and each element value indicates whether the corresponding pixel belongs to the target background region;
determining a replacing background image based on the mask matrix, the BGR format image and a preset replacing background image;
replacing the background image to be replaced with the replacement background image, and determining the target image.
The background replacing device provided by the embodiment of the application responds to a background replacing request, performs image format conversion on a background image to be replaced, and determines an HSV (hue, saturation and value) format image of the background image to be replaced; aiming at the HSV-format image, screening the image according to a preset first filtering threshold group to determine an initial background area in the HSV-format image; performing pixel filtering on the initial background area according to the determined second filtering threshold value group to determine a target background area of the HSV-format image; wherein the filtering range of the second set of filtering thresholds is less than the filtering range of the first set of filtering thresholds; and replacing the background image to be replaced with a preset replacement background image based on the target background area, and determining the replaced target image.
When a background replacement request is received, performing image format conversion on a background image to be replaced to determine an HSV (hue, saturation and value) format image corresponding to the background image to be replaced, performing image filtering on the determined HSV format image according to a first filtering threshold value, and determining an initial background area; and performing pixel filtering on the initial background area by using the determined second filtering threshold group to determine a target background area of the HSV-format image, replacing the background image to be replaced with a preset replacement background image according to the determined target background area, and determining the replaced target image. Therefore, the accuracy of background switching and the switching display effect can be improved.
Referring to fig. 5, fig. 5 is a schematic structural diagram of an electronic device according to an embodiment of the present disclosure. As shown in fig. 5, the electronic device 500 includes a processor 510, a memory 520, and a bus 530.
The memory 520 stores machine-readable instructions executable by the processor 510, when the electronic device 500 runs, the processor 510 communicates with the memory 520 through the bus 530, and when the machine-readable instructions are executed by the processor 510, the steps of the method for replacing the background in the method embodiment shown in fig. 1 may be performed.
An embodiment of the present application further provides a computer-readable storage medium, where a computer program is stored on the computer-readable storage medium, and when the computer program is executed by a processor, the step of the background replacement method in the method embodiment shown in fig. 1 may be executed.
It is clear to those skilled in the art that, for convenience and brevity of description, the specific working processes of the above-described systems, apparatuses and units may refer to the corresponding processes in the foregoing method embodiments, and are not described herein again.
In the several embodiments provided in the present application, it should be understood that the disclosed system, apparatus and method may be implemented in other ways. The above-described embodiments of the apparatus are merely illustrative, and for example, the division of the units is only one logical division, and there may be other divisions when actually implemented, and for example, a plurality of units or components may be combined or integrated into another system, or some features may be omitted, or not executed. In addition, the shown or discussed coupling or direct coupling or communication connection between each other may be through some communication interfaces, indirect coupling or communication connection between devices or units, and may be in an electrical, mechanical or other form.
The units described as separate parts may or may not be physically separate, and parts displayed as units may or may not be physical units, may be located in one place, or may be distributed on a plurality of network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the solution of the embodiment.
In addition, functional units in the embodiments of the present application may be integrated into one processing unit, or each unit may exist alone physically, or two or more units are integrated into one unit.
The functions, if implemented in the form of software functional units and sold or used as a stand-alone product, may be stored in a non-volatile computer-readable storage medium executable by a processor. Based on such understanding, the technical solution of the present application or portions thereof that substantially contribute to the prior art may be embodied in the form of a software product stored in a storage medium and including instructions for causing a computer device (which may be a personal computer, a server, or a network device) to execute all or part of the steps of the method according to the embodiments of the present application. And the aforementioned storage medium includes: various media capable of storing program codes, such as a usb disk, a removable hard disk, a Read-Only Memory (ROM), a Random Access Memory (RAM), a magnetic disk, or an optical disk.
Finally, it should be noted that: the above-mentioned embodiments are only specific embodiments of the present application, and are used for illustrating the technical solutions of the present application, but not limiting the same, and the scope of the present application is not limited thereto, and although the present application is described in detail with reference to the foregoing embodiments, those skilled in the art should understand that: those skilled in the art can still make modifications or changes to the embodiments described in the foregoing embodiments, or make equivalent substitutions for some features, within the technical scope of the present disclosure; such modifications, changes or substitutions do not depart from the spirit and scope of the exemplary embodiments of the present application, and are intended to be covered by the scope of the present application. Therefore, the protection scope of the present application shall be subject to the protection scope of the claims.
Claims (6)
1. A method of replacing a background, the method comprising:
responding to the background replacing request, performing image format conversion on the background image to be replaced, and determining an HSV (hue, saturation and value) format image of the background image to be replaced;
for the HSV format image, screening the image according to a preset first filtering threshold set to determine an initial background area in the HSV format image; the first filtering threshold set comprises a hue filtering threshold, a saturation filtering threshold and a first brightness filtering threshold;
performing pixel filtering on the initial background area according to the determined second filtering threshold value group to determine a target background area of the HSV-format image; wherein the filtering range of the second set of filtering thresholds is less than the filtering range of the first set of filtering thresholds;
replacing the background image to be replaced with a preset replacement background image based on the target background area, and determining a replaced target image; determining the second set of filtering thresholds by:
determining a luminance mean value based on pixel luminance of each pixel included in the initial background region, the luminance mean value being determined as a second luminance filtering threshold;
determining the second set of filtering thresholds based on the hue filtering threshold, the saturation filtering threshold, and the second brightness filtering threshold.
2. The replacing method according to claim 1, wherein the step of performing image format conversion on the background image to be replaced in response to the background replacing request to determine an HSV format image of the background image to be replaced comprises the steps of:
responding to a background replacing request, and converting the background image to be replaced into a BGR format image;
and determining the HSV format image based on the BGR format image.
3. The replacing method according to claim 2, wherein the replacing the background image to be replaced with a preset replacing background image based on the target background area, and determining the replaced target image comprises:
generating a mask matrix corresponding to the background image to be replaced based on the target background area; wherein each element in the mask matrix corresponds to each pixel included in the background image to be replaced, and each element value indicates whether the corresponding pixel belongs to the target background region;
determining a replacing background image based on the mask matrix, the BGR format image and a preset replacing background image;
replacing the background image to be replaced with the replacement background image, and determining the target image.
4. A background replacement apparatus, characterized in that the replacement apparatus comprises:
the image format conversion module is used for responding to the background replacement request, performing image format conversion on the background image to be replaced and determining an HSV (hue, saturation and value) format image of the background image to be replaced;
the initial background determining module is used for screening the HSV format image according to a preset first filtering threshold set so as to determine an initial background area in the HSV format image; the first set of filtering thresholds includes a hue filtering threshold, a saturation filtering threshold, and a first brightness filtering threshold;
the target background determining module is used for performing pixel filtering on the initial background area according to the determined second filtering threshold group to determine a target background area of the HSV-format image; wherein the filtering range of the second set of filtering thresholds is less than the filtering range of the first set of filtering thresholds;
the background image replacing module is used for replacing the background image to be replaced with a preset replacing background image based on the target background area and determining a replaced target image;
the replacement device further comprises a filtering threshold determination module configured to:
determining a luminance mean value based on pixel luminance of each pixel included in the initial background region, the luminance mean value being determined as a second luminance filtering threshold;
determining the second set of filtering thresholds based on the hue filtering threshold, the saturation filtering threshold, and the second brightness filtering threshold.
5. An electronic device, comprising: processor, memory and bus, the memory storing machine-readable instructions executable by the processor, the processor and the memory communicating over the bus when the electronic device is run, the machine-readable instructions when executed by the processor performing the steps of the alternative method of any of the background claims 1 to 3.
6. A computer-readable storage medium, characterized in that a computer program is stored on the computer-readable storage medium, which computer program, when being executed by a processor, performs the steps of the method of replacing the background of claims 1 to 3.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110558649.5A CN113301384B (en) | 2021-05-21 | 2021-05-21 | Background replacing method and device, electronic equipment and readable storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110558649.5A CN113301384B (en) | 2021-05-21 | 2021-05-21 | Background replacing method and device, electronic equipment and readable storage medium |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN113301384A CN113301384A (en) | 2021-08-24 |
| CN113301384B true CN113301384B (en) | 2023-03-24 |
Family
ID=77323724
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202110558649.5A Active CN113301384B (en) | 2021-05-21 | 2021-05-21 | Background replacing method and device, electronic equipment and readable storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN113301384B (en) |
Families Citing this family (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN114449355B (en) * | 2022-01-24 | 2023-06-20 | 腾讯科技（深圳）有限公司 | Live interaction method, device, equipment and storage medium |
| CN114567794B (en) * | 2022-03-11 | 2023-06-30 | 浙江理工大学 | Live video background replacement method |
Family Cites Families (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101371274B (en) * | 2005-12-30 | 2012-09-19 | 意大利电信股份公司 | Edge comparison in video sequence partition |
| CN101686338B (en) * | 2008-09-26 | 2013-12-25 | 索尼株式会社 | System and method for partitioning foreground and background in video |
| JP7207957B2 (en) * | 2018-11-07 | 2023-01-18 | キヤノン株式会社 | Image processing device, image processing method, and program |
| CN111292337B (en) * | 2020-01-21 | 2024-03-01 | 广州虎牙科技有限公司 | Image background replacement method, device, equipment and storage medium |
-
2021
- 2021-05-21 CN CN202110558649.5A patent/CN113301384B/en active Active
Also Published As
| Publication number | Publication date |
|---|---|
| CN113301384A (en) | 2021-08-24 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US10896634B2 (en) | Image signal processing apparatus and control method therefor | |
| JP6717858B2 (en) | Calibration of defective image sensor element | |
| CN113301384B (en) | Background replacing method and device, electronic equipment and readable storage medium | |
| RU2491760C2 (en) | Image processing device, image processing method and programme | |
| EP2426927B1 (en) | Image processing apparatus, image processing method and computer program | |
| CN110958401B (en) | Super night scene image color correction method and device and electronic equipment | |
| JPWO2006059573A1 (en) | Color adjustment apparatus and method | |
| JP2003224769A (en) | Digital camera | |
| JPWO2011118071A1 (en) | Image processing method and apparatus, image processing program, and medium storing this program | |
| JP6574878B2 (en) | Image processing apparatus, image processing method, imaging apparatus, program, and storage medium | |
| US11089211B2 (en) | Image processing apparatus, image processing method, and program for switching between two types of composite images | |
| US20090252412A1 (en) | Image processing apparatus and image processing method | |
| CN111327827B (en) | Shooting scene recognition control method and device and shooting equipment | |
| US11361408B2 (en) | Image processing apparatus, system, image processing method, and non-transitory computer-readable storage medium | |
| JP2005210495A (en) | Image processing apparatus, method, and program | |
| US10091415B2 (en) | Image processing apparatus, method for controlling image processing apparatus, image pickup apparatus, method for controlling image pickup apparatus, and recording medium | |
| KR20120024448A (en) | Imaging apparatus, signal processing method, and program | |
| CN115804080A (en) | Correction of color-rendered pixels captured under low light conditions | |
| KR100915598B1 (en) | Automatic white balance apparatus and method thereof | |
| WO2016086493A1 (en) | Immersive video presentation method for intelligent mobile terminal | |
| JP2004088408A (en) | Digital camera | |
| JP4069522B2 (en) | Video signal processing apparatus, color video camera, and video signal processing method | |
| KR100581526B1 (en) | A digital still camera operating white correction in response to variable light source | |
| US20240163567A1 (en) | Image processing apparatus, image processing method, and image capture apparatus | |
| JP5166859B2 (en) | White balance control device, imaging device using the same, and white balance control method |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |