CN112330692B

CN112330692B - Matting method, matting device, matting equipment and storage medium

Info

Publication number: CN112330692B
Application number: CN202011253435.9A
Authority: CN
Inventors: 朱玉荣; 黄建超
Original assignee: Anhui Wenxiang Technology Co ltd
Current assignee: Anhui Wenxiang Technology Co ltd
Priority date: 2020-11-11
Filing date: 2020-11-11
Publication date: 2022-06-28
Anticipated expiration: 2040-11-11
Also published as: CN112330692A

Abstract

The embodiment of the application discloses a matting method, a matting device, matting equipment and a storage medium, wherein the method comprises the following steps: acquiring a background image (not including a target object) and an image to be segmented aiming at a static scene; determining the transparency of a first pixel point according to the brightness difference between the first pixel point in the image to be segmented and a corresponding second pixel point in the background image and the color difference between the first pixel point and the second pixel point; if the transparency of the first pixel point is larger than the transparency threshold, determining the first pixel point as a target pixel point, and determining the target pixel point as a pixel point in a target object; and according to the target pixel points, segmenting the target object from the image to be segmented. Based on the scheme of this application, need not to build specific background and can realize the cutout to the complexity and the cutout cost of cutout have been reduced.

Description

Matting method, matting device, matting equipment and storage medium

Technical Field

The present application relates to the field of data processing, and in particular, to a matting method, apparatus, device, and storage medium.

Background

The virtual matting can be used to matte a target object from an image of one scene and merge the matte target object with images of other scenes to generate a new image. The virtual image matting is widely applied to various scenes, such as recording of teaching videos, shooting of movie and television plays and the like.

At present, the main way of virtual image matting is: and (3) building a specific background, such as a blue-green background, then recording a video of the target object under the blue-green background, and further matting the target object from the video image to be synthesized with other required scene images. However, the existing virtual matting scheme needs to construct a specific background to perform matting, which increases the complexity of matting and also increases the matting cost.

Disclosure of Invention

In order to solve the technical problem, the application provides a matting method, a device, equipment and a storage medium, which can realize the segmentation of a target object under a static scene without additionally building a background, thereby reducing the complexity of virtual matting and the matting cost.

The embodiment of the application discloses the following technical scheme:

in one aspect, an embodiment of the present application provides a matting method, where the method includes:

acquiring a background image and an image to be segmented aiming at a static scene; wherein the background image does not include a target object;

determining the transparency of a first pixel point according to the brightness difference between the first pixel point in the image to be segmented and a corresponding second pixel point in the background image and the color difference between the first pixel point and the second pixel point;

If the transparency of the first pixel point is larger than a transparency threshold value, determining that the first pixel point is a target pixel point, and the target pixel point is a pixel point in the target object;

and according to the target pixel points, the target object is segmented from the image to be segmented.

Optionally, the process of determining the transparency of the first pixel point includes:

if the brightness value of the first pixel point is smaller than or equal to the brightness threshold value, determining the transparency of the first pixel point by using the brightness difference between the first pixel point and the second pixel point;

and if the brightness value of the first pixel point is larger than the brightness threshold value, determining the transparency of the first pixel point by using the brightness difference between the first pixel point and the second pixel point and the color difference between the first pixel point and the second pixel point.

Optionally, the determining the transparency of the first pixel point by using the brightness difference between the first pixel point and the second pixel point includes: the transparency of the first pixel point is calculated by the following formula,

f(x,y)＝p×|I_x,y-I_d,x,y|+q

wherein alpha is_x,yThe transparency of the first pixel point is obtained; i is_x,yThe brightness of the first pixel point is obtained; i is_d,x,yThe brightness of the second pixel point is obtained; p and q are both preset parameters.

Optionally, the determining the transparency of the first pixel point by using the brightness difference between the first pixel point and the second pixel point and the color difference between the first pixel point and the second pixel point includes: the transparency of the first pixel point is calculated by the following formula,

g(x,y)＝a×((R_x,y-R_d,x,y)²+(G_x,y-G_d,x,y)²)+b×|I_x,y-I_d,x,y|+c

wherein alpha is_x,yThe transparency of the first pixel point is obtained; r_x,yThe value of the red channel of the first pixel point is obtained; r_d,x,yThe value of the red channel of the second pixel point is obtained; g_x,yThe value of the green channel of the first pixel point is obtained; g_d,x,yThe value of the green channel of the second pixel point is obtained; i is_x,yThe brightness of the first pixel point is obtained; i is_d,x,yThe brightness of the second pixel point is obtained; a, b and c are all preset parameters.

Optionally, before determining that the first pixel point is the target pixel point if the transparency of the first pixel point is greater than the transparency threshold, the method further includes:

calculating a transparency bias value corresponding to the first pixel point according to the brightness difference and the transparency difference of the first pixel point and the pixel points around the first pixel point;

summing the transparency of the first pixel point and the transparency offset value to obtain the optimized transparency of the first pixel point;

Correspondingly, if the transparency of the first pixel point is greater than the transparency threshold, determining that the first pixel point is a target pixel point includes:

and if the optimized transparency of the first pixel point is larger than a transparency threshold, determining the first pixel point as a target pixel point.

Optionally, the background image is obtained by processing at least part of images in the first N frames of images of the target video.

Optionally, the background image is determined by:

determining an nth frame image in the previous N frames of images as a first image, and determining an (N-1) th frame image in the previous N frames of images as a second image;

calculating the color distance between a third pixel point in the first image and a fourth pixel point in the first background image; the first background image is a background image corresponding to the second image, and pixel points in the first background image have corresponding color thresholds; the third pixel point is any one pixel point in the first image, and the four pixel points correspond to the third pixel point;

if the color distance corresponding to each pixel point in the first image is smaller than or equal to the color threshold corresponding to the corresponding pixel point in the first background image, taking the first background image as the background image; if not, then,

Updating the first background image based on the color distance to obtain a background image corresponding to the first image; and determining the (N + 1) th frame image in the previous N frames of images as a first image, determining the nth frame image as a second image, and returning to the step of calculating the color distance between the third pixel point in the first image and the fourth pixel point in the first background image.

Optionally, the updating the first background map based on the color distance includes:

if the color distance is larger than the color threshold corresponding to the fourth pixel point, updating the color threshold corresponding to the fourth pixel point to the color distance, and updating the color value of the fourth pixel point according to the color value of the third pixel point and the color value of the fourth pixel point;

and if the color distance is smaller than or equal to the color threshold corresponding to the fourth pixel point, keeping the color value of the fourth pixel point and the corresponding color threshold unchanged.

Optionally, the method further includes:

determining the target color value of the target pixel point according to the color value and the transparency of the target pixel point and the color value of the second pixel point;

And updating the color value of each target pixel point in the target object to be a corresponding target color value.

In another aspect, an embodiment of the present application provides a matting device, where the matting device includes:

the device comprises an acquisition unit, a segmentation unit and a processing unit, wherein the acquisition unit is used for acquiring a background image and an image to be segmented aiming at a static scene; wherein the background image does not include a target object;

the calculation unit is used for determining the transparency of the first pixel point according to the brightness difference between the first pixel point in the image to be segmented and the corresponding second pixel point in the background image and the color difference between the first pixel point and the second pixel point;

the determining unit is used for determining the first pixel point as a target pixel point if the transparency of the first pixel point is greater than a transparency threshold value, and the target pixel point is a pixel point in the target object;

and the segmentation unit is used for segmenting the target object from the image to be segmented according to the target pixel point.

Optionally, the computing unit is configured to:

Optionally, when the computing unit determines the transparency of the first pixel point by using the brightness difference between the first pixel point and the second pixel point, the computing unit is configured to:

the transparency of the first pixel point is calculated by the following formula,

f(x,y)＝p×|I_x,y-I_d,x,y|+q

wherein alpha is_x,yThe transparency of the first pixel point is obtained; I.C. A_x,yThe brightness of the first pixel point is obtained; i is_d,x,yThe brightness of the second pixel point is obtained; p and q are both preset parameters.

Optionally, the calculating unit is configured to determine the transparency of the first pixel point by using the brightness difference between the first pixel point and the second pixel point and the color difference between the first pixel point and the second pixel point: the transparency of the first pixel point is calculated by the following formula,

g(x,y)＝a×((R_x,y-R_d,x,y)²+(G_x,y-G_d,x,y)²)+b×|I_x,y-I_d,x,y|+c

wherein alpha is_x,yThe transparency of the first pixel point is obtained; r_x,yThe value of the red channel of the first pixel point is obtained; r_d,x,yThe value of the red channel of the second pixel point is obtained; g_x,yThe value of the green channel of the first pixel point is obtained; g _d,x,yThe value of the green channel of the second pixel point is obtained; I.C. A_x,yThe brightness of the first pixel point is obtained; I.C. A_d,x,yThe brightness of the second pixel point is obtained; a, b and c are all preset parameters.

Optionally, the method further includes:

the transparency optimization unit is used for calculating a transparency offset value corresponding to the first pixel point according to the brightness difference and the transparency difference of the first pixel point and the pixel points around the first pixel point; summing the transparency of the first pixel point and the transparency offset value to obtain the optimized transparency of the first pixel point;

correspondingly, the determining unit is specifically configured to: and if the optimized transparency of the first pixel point is larger than a transparency threshold, determining the first pixel point as a target pixel point.

Optionally, the method further includes:

and the background image acquisition unit is used for processing at least part of images in the first N frames of images of the target video to obtain the background image.

Optionally, the background image acquiring unit is specifically configured to:

determining an nth frame image in the previous N frame images as a first image, and determining an (N-1) th frame image in the previous N frame images as a second image;

Optionally, when the background image obtaining unit updates the first background image based on the color distance, the background image obtaining unit is configured to:

Optionally, the method further includes:

the color optimization unit is used for determining the target color value of the target pixel point according to the color value and the transparency of the target pixel point and the color value of the second pixel point; and updating the color value of each target pixel point in the target object to be the corresponding target color value.

In another aspect, an embodiment of the present application provides a matting device, where the matting device includes a processor and a memory:

the memory is used for storing a computer program and transmitting the computer program to the processor;

the processor is configured to execute the matting method according to any one of the above-mentioned items according to instructions in the computer program.

In another aspect, an embodiment of the present application provides a computer-readable storage medium for storing a computer program, where the computer program is configured to execute the matting method described in any one of the above.

According to the technical scheme, the matting method comprises the following steps: acquiring a background image (not including a target object) and an image to be segmented aiming at a static scene; determining the transparency of a first pixel point according to the brightness difference between the first pixel point in the image to be segmented and a corresponding second pixel point in the background image and the color difference between the first pixel point and the second pixel point; if the transparency of the first pixel point is larger than the transparency threshold, determining the first pixel point as a target pixel point, and determining the target pixel point as a pixel point in a target object; and segmenting the target object from the image to be segmented according to the target pixel points. Based on the scheme of this application, need not to build specific background and can realize the cutout to the complexity and the cutout cost of cutout have been reduced.

Drawings

In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings needed to be used in the description of the embodiments or the prior art will be briefly introduced below, and it is obvious that the drawings in the following description are only some embodiments of the present application, and it is obvious for those skilled in the art that other drawings can be obtained according to the drawings without inventive exercise.

Fig. 1 is a flowchart of an implementation of a matting method provided in an embodiment of the present application;

fig. 2 is an exemplary diagram of a background image and an image to be segmented according to an embodiment of the present application;

fig. 3 is a flowchart of an implementation of optimizing transparency of a first pixel point according to an embodiment of the present disclosure;

FIG. 4 is a flowchart of an implementation of determining a background image according to an embodiment of the present disclosure;

fig. 5 is a schematic structural diagram of a matting device provided in an embodiment of the present application;

fig. 6 is a schematic structural diagram of a server according to an embodiment of the present application;

fig. 7 is a schematic structural diagram of a terminal according to an embodiment of the present application.

Detailed Description

Embodiments of the present application are described below with reference to the accompanying drawings.

The matting method provided by the application can be executed by data processing equipment, such as terminal equipment, a server and the like. The terminal device may be, for example, a smart phone, a computer, a Personal digital assistant (PDA for short), a tablet computer, or the like. The servers may be stand-alone servers or servers in a cluster.

The matting method provided by the embodiment of the present application is described below, and with reference to fig. 1, this figure shows a flowchart of an implementation of the matting method provided by the embodiment of the present application, and may include:

step S101: a background image and an image to be segmented for a static scene are acquired.

The target object to be subjected to matting in the embodiment of the present application may be any object, and for example, may be a human body (such as a host), or may be a car, or may be an animal, and the like.

It should be noted that the target object in the present application may be in a static scene, and the static scene may be any static scene, such as a scene of an office, or a scene of a living room, etc. No specific background such as a blue-green background is built in the static scene. Of course, the general background of the solution of the present application is not limited to a scene with a blue-green background, but the solution of the present application has a wider application range, and is not limited to a specific background such as a blue-green background, but can be used in a background other than a specific background. That is, if the static scene is a specific background such as a blue-green background of construction, the scheme of the present application is also applicable.

In the embodiment of the application, when the shooting device shoots a video for a static scene, in the beginning stage of shooting, a target object does not enter the static scene, the target object enters the static scene after shooting for a certain time, and the target object is not static in the static scene but can move randomly. Thus, in a video (for convenience of description, referred to as a target video) captured by the capturing apparatus, only a still scene is present in the first several frames of images (for convenience of description, referred to as the first N frames of images), and there is no target object.

The background image in the embodiment of the application is obtained based on the image which does not contain the target object in the target video, and the image to be segmented is the image which contains the target object in the target video.

Step S102: and determining the transparency of the first pixel point according to the brightness difference between the first pixel point in the image to be segmented and the corresponding second pixel point in the background image and the color difference between the first pixel point and the second pixel point.

In this embodiment of the present application, one pixel point in the image to be segmented may correspond to one pixel point in the background image.

For example, referring to fig. 2, which shows an exemplary diagram of a background image and an image to be segmented provided by an embodiment of the present application, as shown in fig. 2, the background image shows a wall surface, and the image to be segmented shows that a person is in front of the wall surface, where a first pixel point 201 in the background image corresponds to a second pixel point 202 in the image to be segmented, and both real background scenes corresponding to the two pixel points are at the same position on the wall surface.

The color difference between the first pixel point and the second pixel point can be the difference value of each color channel between the first pixel point and the second pixel point, or the difference value of partial color channels between the first pixel point and the second pixel point. The color channels of the pixel points may be a red channel (denoted as R), a green channel (denoted as G), and a blue channel (denoted as B). If the red, green and blue channel values of the first pixel point are recorded as R in sequence_x,y，G_x,yAnd B_x,yThe red, green and blue channel values of the second pixel point are recorded as R in sequence_d,x,y，G_d,x,yAnd B_d,x,yThen, as an example, the color difference between the first pixel point and the second pixel point may be (R)_x,y-R_d,x,y)，(G_x,y-G_d,x,y) And (B)_x,y-B_d,x,y) Or, the color difference between the first pixel point and the second pixel point may be (R)_x,y-R_d,x,y)，(G_x,y-G_d,x,y)。

Step S103: if the transparency of the first pixel point is larger than the transparency threshold, the first pixel point is indicated as a foreground pixel point, and the first pixel point is determined as a target pixel point.

The target pixel point may refer to a pixel point in the target object.

If the first pixel point is smaller than or equal to the transparency threshold value, the first pixel point is indicated as a background pixel point, and the first pixel point is determined as a non-target pixel point. Optionally, the transparency threshold may be 0, if the transparency of the first pixel is greater than 0, the first pixel is determined to be a target pixel, and if the transparency of the first pixel is equal to 0, the first pixel is determined to be a non-target pixel.

Step S104: and segmenting the target object from the image to be segmented according to the target pixel points.

After each pixel point in the image to be segmented is determined to be a target pixel point through the method, the target can be extracted from the image to be segmented according to all target pixel points in the image to be segmented.

And the target object can be segmented from the image to be segmented according to the edge determined by the target pixel point. And all the pixel points in the target object are target pixel points.

According to the technical scheme, the matting method provided by the embodiment of the application acquires the background image (not including the target object) and the image to be segmented aiming at the static scene; determining the transparency of a first pixel point according to the brightness difference between the first pixel point in the image to be segmented and a corresponding second pixel point in the background image and the color difference between the first pixel point and the second pixel point; if the transparency of the first pixel point is larger than the transparency threshold, determining the first pixel point as a target pixel point, and determining the target pixel point as a pixel point in a target object; and according to the target pixel points, segmenting the target object from the image to be segmented. Due to the fact that the cutout can be achieved without building a specific background, the cutout method provided by the embodiment of the application reduces the complexity of cutout and the cutout cost.

In an optional embodiment, one implementation manner of determining the transparency of the first pixel point according to the brightness difference between the first pixel point in the image to be segmented and the corresponding second pixel point in the background image and the color difference between the first pixel point and the second pixel point may be:

and if the brightness value of the first pixel point is less than or equal to the brightness threshold, determining the transparency of the first pixel point by using the brightness difference between the first pixel point and the second pixel point.

That is to say, if the brightness of the first pixel point is smaller, the transparency of the first pixel point is determined only by using the brightness difference between the first pixel point and the second pixel point.

And if the brightness value of the first pixel point is greater than the brightness threshold value, determining the transparency of the first pixel point by using the brightness difference between the first pixel point and the second pixel point and the color difference between the first pixel point and the second pixel point.

That is to say, if the brightness of the first pixel point is larger, the transparency of the first pixel point can be determined by combining the brightness difference between the first pixel point and the second pixel point and the color difference between the first pixel point and the second pixel point.

In an optional embodiment, one implementation manner of determining the transparency of the first pixel point by using the luminance difference between the first pixel point and the second pixel point may be: the transparency of the first pixel point is calculated by the following formula,

f(x,y)＝p×|I_x,y-I_d,x,y|+q

Wherein alpha is_x,yThe transparency of the first pixel point is shown, and the coordinates of the first pixel point are (x, y); I.C. A_x,yThe brightness of the first pixel point; I.C. A_d,x,yThe brightness of the second pixel point; and p and q are both preset parameters. The coordinates of the second pixel point are also (x, y).

In an optional embodiment, one implementation manner for determining the transparency of the first pixel point by using the brightness difference between the first pixel point and the second pixel point and the color difference between the first pixel point and the second pixel point may be: the transparency of the first pixel point is calculated by the following formula,

g(x,y)＝a×((R_x,y-R_d,x,y)²+(G_x,y-G_d,x,y)²)+b×|I_x,y-I_d,x,y|+c

wherein alpha is_x,yThe transparency of the first pixel point is shown, and the coordinate of the first pixel point is (x, y); r_x,yThe value of the red channel of the first pixel point; r_d,x,yThe value of the red channel of the second pixel point; g_x,yIs green of the first pixel pointThe value of the channel; g_d,x,yThe value of the green channel of the second pixel point; i is_x,yBrightness of the first pixel point; i is_d,x,yThe brightness of the second pixel point; a, b and c are all preset parameters. The coordinates of the second pixel point are also (x, y).

The value of sign (u) is determined according to the value of u, if u is greater than or equal to 0, the value of sign (u) is 1, otherwise, the value of sign (u) is-1. u is f (x, y) or g (x, y).

In this embodiment, the color difference between the first pixel point and the second pixel point is a difference between partial color channels of the first pixel point and the second pixel point, that is, a difference between a red color channel and a green color channel.

In an alternative embodiment, in order to improve the matting accuracy, the transparency of the first pixel point determined in step S102 may be optimized. Optionally, the transparency of the first pixel point may be optimized by using the brightness difference and the transparency difference between the first pixel point and the pixel points around the first pixel point. Specifically, an implementation flowchart for optimizing the transparency of the first pixel point, provided in the embodiment of the present application, is shown in fig. 3, and may include:

step S301: and calculating a transparency bias value corresponding to the first pixel point according to the brightness difference and the transparency difference of the first pixel point and the pixel points around the first pixel point.

Alternatively, the transparency bias value corresponding to the first pixel point may be calculated according to the following formula

Wherein the content of the first and second substances,

step S302: and summing the transparency of the first pixel point and the corresponding transparency offset value to obtain the optimized transparency of the first pixel point. Can be expressed by the formula:

wherein, alpha'_x,yRepresenting the optimized transparency, alpha, of the first pixel point _x,yAnd representing the transparency of the first pixel point determined in the step S102.

Optionally, the optimization process shown in fig. 3 may be executed multiple times, and the specific execution times may be preset times. When the jth (j is an integer larger than 1) sub-optimization is carried out on the transparency of the first pixel point, alpha in the formula is_x,yTo the transparency of the first pixel pointLine j-1 suboptimal transparency.

k is a preset parameter, and the value of k may be 0.25 as an example.

In an optional embodiment, the background image is obtained by processing at least a part of images in the first N frames of images of the target video. As shown in fig. 4, a flowchart for determining a background image according to an embodiment of the present application may include:

step S401: and determining the nth frame image in the previous N frame images as a first image, and determining the (N-1) th frame image in the previous N frame images as a second image. n is an integer greater than 1.

Step S402: calculating the color distance between a third pixel point in the first image and a fourth pixel point in the first background image;

the first background image is a background image corresponding to the second image, and pixel points in the first background image have corresponding color thresholds; the third pixel point is any one pixel point in the first image, and the four pixel points correspond to the third pixel point.

When n is 2, the second image is the 1 st frame image, the background image corresponding to the 1 st frame image is the 1 st frame image itself, that is, the first background image is the 1 st frame image, and at this time, the color threshold corresponding to each pixel point in the first background image is an initial value, for example, 0.

When n >2, the background image corresponding to the second image and the color threshold corresponding to the pixel point in the background image are determined through the subsequent steps.

Taking the first image as the nth frame image as an example, the color distance D between the third pixel point and the fourth pixel point_x,yCan be calculated by the following formula:

wherein, the color values of the three channels of the third pixel point are divided into (R)_n,x,y，G_n,x,y，B_n,x,y) The color values of the three channels of the fourth pixel point are respectively (R)_d,x,y，G_d,x,y，B_d,x,y)。

Alternatively, the first and second electrodes may be,

color distance D between third pixel point and fourth pixel point_x,yCan be calculated by the following formula:

wherein, (R'_n,x,y，G’_n,x,y，B’_n,x,y) Is the normalized color value of the three channels of the third pixel, (R'_d,x,y，G’_d,x,y，B’_d,x,y) Is the normalized color value of the three channels of the fourth pixel.

Wherein the content of the first and second substances,

step S403: if the color distance corresponding to each pixel point in the first image is smaller than or equal to the color threshold corresponding to the corresponding pixel point in the first background image, taking the first background image as the background image, and ending the process; otherwise, the process proceeds to step S404.

Step S404: updating the first background image based on the color distance to obtain a background image corresponding to the first image; and determining the (N + 1) th frame image in the previous N frames of images as a first image, determining the nth frame image as a second image, and returning to execute the step S402 and the subsequent steps.

Optionally, one implementation manner of updating the first background image based on the color distance may be as follows:

and if the color distance is larger than the color threshold corresponding to the fourth pixel point, updating the color threshold corresponding to the fourth pixel point to be the color distance, and updating the color value of the fourth pixel point according to the color value of the third pixel point and the color value of the fourth pixel point.

Optionally, taking the first image as the nth frame image as an example, one implementation manner of updating the color value of the fourth pixel point according to the color value of the third pixel point and the color value of the fourth pixel point may be: updating the color value of the fourth pixel point according to the following formula:

wherein, equal-numbered left side (R'_d,x,y,G’_d,x,y,B’_d,x,y) Is the updated color value of the fourth pixel point (i.e. the color value of the fourth pixel point in the background image corresponding to the first image), and is right of equal sign (R'_d,x,y,G’_d,x,y,B’_d,x,y) (R ') for the pre-update color value of the fourth pixel (i.e., the color value of the fourth pixel in the first background image)' _n,x,y,G’_n,x,y,B’_n,x,y) The color value of the third pixel point in the first image. It should be noted that the color values involved in the formula are normalized color values, and in an alternative embodiment, the color values involved in the formula may also be non-normalized color values.

And after all the pixel points of the nth frame image are calculated, an nth background image corresponding to the nth frame image can be obtained.

And if the color distance is less than or equal to the color threshold corresponding to the fourth pixel point, keeping the color value of the fourth pixel point and the corresponding color threshold unchanged.

Optionally, if the nth frame image is the last frame image in the previous N frame images (that is, the nth frame image is the nth frame image in the target video), and the nth frame image has the first type of pixel points, and the color distance corresponding to the first type of pixel points is greater than the color threshold corresponding to the pixel points corresponding to the first type of pixel points in the nth-1 th background image, the nth background image corresponding to the nth frame image is used as the background image for detecting the target object.

As an example, the color distance between the third pixel point in the 2 nd frame image and the corresponding fourth pixel point in the 1 st background image may be calculated first. The third pixel point is any pixel point in the 2 nd frame image; and the fourth pixel point is a pixel point in the 1 st background image corresponding to the third pixel point in the 2 nd frame image.

If the color distance between the third pixel point in the 2 nd frame image and the corresponding fourth pixel point in the 1 st background image is greater than the color threshold corresponding to the fourth pixel point, the color threshold corresponding to the fourth pixel point is updated to the color distance between the third pixel point in the 2 nd frame image and the corresponding fourth pixel point in the 1 st background image, the color value of the fourth pixel point in the 1 st background image is updated according to the color value of the third pixel point in the 2 nd frame image and the color value of the fourth pixel point in the 1 st background image, and otherwise, the color value of the fourth pixel point in the 1 st background image and the corresponding color threshold are kept unchanged. And after all the pixel points of the 2 nd frame image are calculated, obtaining a 2 nd background image corresponding to the 2 nd frame image.

And then calculating a third color distance between a third pixel point in the 3 rd frame image and a corresponding fourth pixel point in the 2 nd background image. The third pixel point is any pixel point in the 3 rd frame image; and the fourth pixel point is a pixel point in the 2 nd background image corresponding to the third pixel point in the 3 rd frame image.

If the color distance between the third pixel point in the 3 rd frame image and the corresponding fourth pixel point in the 2 nd background image is greater than the color threshold corresponding to the fourth pixel point, the color threshold corresponding to the fourth pixel point is updated to be the color distance between the third pixel point in the 3 rd frame image and the corresponding fourth pixel point in the 2 nd background image, the color value of the fourth pixel point in the 2 nd background image is updated according to the color value of the third pixel point in the 3 rd frame image and the color value of the fourth pixel point in the 2 nd background image, and otherwise, the color value of the fourth pixel point in the 2 nd background image and the corresponding color threshold are kept unchanged. And after all the pixel points of the 3 rd frame image are calculated, a 3 rd background image corresponding to the 3 rd frame image can be obtained.

Then the color distance between the third pixel point in the 4 th frame image and the corresponding fourth pixel point in the 3 rd background image is calculated … …, and so on.

And if the color distance corresponding to each pixel point in the 2 nd frame image is smaller than or equal to the color threshold corresponding to the pixel point in the 1 st background image, taking the 1 st background image corresponding to the 1 st frame image as the background image for detecting the target object, and ending the process.

Similarly, if the color distance corresponding to each pixel point in the 3 rd frame image is less than or equal to the color threshold corresponding to the pixel point in the 2 nd background image, the 2 nd background image corresponding to the 2 nd frame image is used as the background image for detecting the target object, and the process is ended.

And so on.

After determining the background image for detecting the target object, the process of determining the background image and the color threshold is also ended.

In an optional embodiment, since the color value of the pixel point in the target object may not be the color value of the pixel point itself, and it may have a certain transparency, in order to obtain the real color of the target object, in this embodiment of the application, after the target object is segmented from the image to be segmented, the method may further include:

And determining the target color value of the target pixel point according to the color value and the transparency of the target pixel point and the color value of the second pixel point.

Optionally, the target color value of the target pixel point may be determined by the following formula:

wherein (R)_f,x,y，G_f,x,y，B_f,x,y) Is the target color value of the target pixel point, (R)_s,x,y，G_s,x,y，B_s,x,y) Is the color value of the target pixel point segmented from the image to be segmented, (R)_d,x,y，G_d,x,y，B_d,x,y) And alpha is the transparency of the target pixel point segmented from the image to be segmented.

And updating the color value of each target pixel point in the target object to be the corresponding target color value.

In an alternative embodiment, the preset parameters a, b, c, p and q may be set by the user according to the actual lighting condition and the parameters of the camera, and may be specifically determined as follows:

after determining the background image, displaying a parameter setting interface; the parameter setting interface may be automatically displayed after determining the background image, or may be displayed in response to the user performing a preset operation after determining the background image, for example, after the user clicks a preset button/key.

Receiving parameter values input for a, b, c, p and q; i.e. the user inputs parameter values for a, b, c, and p and q.

Calculating the brightness value of each pixel point in a reference image based on the input parameters, wherein the reference image is an image of any frame in a target video with a target object; the manner of calculating the brightness value may refer to the foregoing embodiments, and is not described herein again.

And displaying the reference image according to the calculated brightness value so that the user judges the difference between the brightness value of the foreground and the brightness value of the background. If the user determines that the difference between the brightness of the foreground and the brightness of the background is small, the values of a, b, c, p, and q may be modified until the user determines that the difference between the brightness of the foreground and the brightness of the background is large, for example, the brightness of the foreground is close to 1, and the brightness of the background is close to 0.

The values of a, b, c, and p and q are all greater than 0. As an example, one optional value for a may be 521; an optional value for b may be 1033, an optional value for c may be 0.5, an optional value for p may be 10, and an optional value for q may be 7.

Based on the matting method provided above, an embodiment of the present application further provides a matting device, which is shown in fig. 5 as a schematic structural diagram, and includes: an acquisition unit 501, a calculation unit 502, a determination unit 503, and a division unit 504; wherein, the first and the second end of the pipe are connected with each other,

The acquiring unit 501 is configured to acquire a background image and an image to be segmented for a still scene; wherein the background image does not include a target object;

the calculating unit 502 is configured to determine the transparency of a first pixel point according to a luminance difference between the first pixel point in the image to be segmented and a corresponding second pixel point in the background image, and a color difference between the first pixel point and the second pixel point;

the determining unit 503 is configured to determine that the first pixel is a target pixel if the transparency of the first pixel is greater than a transparency threshold, where the target pixel is a pixel in the target object;

the segmentation unit 504 is configured to segment the target object from the image to be segmented according to the target pixel point.

The matting device provided by the embodiment of the application acquires a background image (not including a target object) and an image to be segmented aiming at a static scene; determining the transparency of a first pixel point according to the brightness difference between the first pixel point in the image to be segmented and a corresponding second pixel point in the background image and the color difference between the first pixel point and the second pixel point; if the transparency of the first pixel point is larger than the transparency threshold, determining the first pixel point as a target pixel point, and determining the target pixel point as a pixel point in a target object; and according to the target pixel points, segmenting the target object from the image to be segmented. Because the cutout can be realized without building a specific background, the cutout device provided by the embodiment of the application reduces the complexity and the cutout cost of the cutout.

In one possible implementation, the computing unit is configured to:

In a possible implementation manner, when the computing unit determines the transparency of the first pixel point by using the luminance difference between the first pixel point and the second pixel point, the computing unit is configured to:

f(x,y)＝p×|I_x,y-I_d,x,y|+q

In a possible implementation manner, the calculating unit is configured to, when determining the transparency of the first pixel point by using the luminance difference between the first pixel point and the second pixel point and the color difference between the first pixel point and the second pixel point: the transparency of the first pixel point is calculated by the following formula,

g(x,y)＝a×((R_x,y-R_d,x,y)²+(G_x,y-G_d,x,y)²)+b×|I_x,y-I_d,x,y|+c

In one possible implementation manner, the method further includes:

In a possible implementation manner, the background image obtaining unit is specifically configured to:

In a possible implementation manner, when the background image obtaining unit updates the first background image based on the color distance, the background image obtaining unit is configured to:

In one possible implementation manner, the method further includes:

the color optimization unit is used for determining the target color value of the target pixel point according to the color value and the transparency of the target pixel point and the color value of the second pixel point; updating the color value of each target pixel point in the target object to a corresponding target color value

Corresponding to the method embodiment, an apparatus may specifically be a server, fig. 6 is a schematic structural diagram of the server provided in the embodiment of the present application, and the server 600 may generate a relatively large difference due to different configurations or performances, and may include one or more Central Processing Units (CPUs) 622 (e.g., one or more processors) and a memory 632, and one or more storage media 630 (e.g., one or more mass storage devices) storing an application program 642 or data 644. Memory 632 and storage medium 630 may be, among other things, transient or persistent storage. The program stored in the storage medium 630 may include one or more modules (not shown), each of which may include a series of instruction operations for the server. Still further, the central processor 622 may be configured to communicate with the storage medium 630 and execute a series of instruction operations in the storage medium 630 on the server 600.

The server 600 may also include one or more power supplies 626, one or more wired or wireless network interfaces 650, one or more input-output interfaces 658, and/or one or more operating systems 641, such as Windows Server, Mac OS XTM, UnixTM, LinuxTM, FreeBSDTM, and so forth.

The steps performed by the server in the above embodiments may be based on the server structure shown in fig. 6.

The CPU622 is configured to execute the following steps:

Optionally, the CPU622 is further configured to execute steps of any implementation manner of the matting method provided in the embodiment of the present application.

The embodiment of the present application further provides another device, where the device may specifically be a terminal, and the terminal may be a desktop, a notebook, or the like, as shown in fig. 7, for convenience of description, only a part related to the embodiment of the present application is shown, and specific technical details are not disclosed, please refer to a method part in the embodiment of the present application. Taking a terminal as a desktop as an example:

fig. 7 is a block diagram illustrating a structure of a desktop part related to a terminal provided in an embodiment of the present application. Referring to fig. 7, the desktop includes: radio Frequency (RF) circuit 710, memory 720, input unit 730, display unit 740, sensor 750, audio circuit 760, wireless fidelity (WiFi) module 770, processor 780, and power supply 790. Those skilled in the art will appreciate that the desktop architecture shown in fig. 7 does not constitute a limitation on desktop, and may include more or fewer components than shown, or some components may be combined, or a different arrangement of components.

The following specifically describes each constituent element of the desktop with reference to fig. 7:

the RF circuit 710 may be used for receiving and transmitting signals during a message or call. In general, RF circuit 710 includes, but is not limited to, an antenna, at least one Amplifier, a transceiver, a coupler, a Low Noise Amplifier (Low Noise Amplifier; LNA), a duplexer, and the like. In addition, the RF circuit 710 may also communicate with networks and other devices via wireless communication.

The memory 720 may be used to store software programs and modules, and the processor 780 performs various functional applications and data processing of the desktop by operating the software programs and modules stored in the memory 720. The memory 720 may mainly include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application program required by at least one function (such as a sound playing function, an image playing function, etc.), and the like; the storage data area may store data (such as audio data, etc.) created according to the use of desktop, etc. Further, the memory 720 may include high speed random access memory, and may also include non-volatile memory, such as at least one magnetic disk storage device, flash memory device, or other volatile solid state storage device.

The input unit 730 may be used to receive input numeric or character information and generate key signal inputs related to user setting and function control of the desktop. Specifically, the input unit 730 may include a touch panel 731 and other input devices 732. The touch panel 731, also referred to as a touch screen, can collect touch operations of a user (e.g. operations of the user on or near the touch panel 731 by using any suitable object or accessory such as a finger, a stylus, etc.) and drive the corresponding connection device according to a preset program. Alternatively, the touch panel 731 may include two portions of a touch detection device and a touch controller. The touch detection device detects the touch direction of a user, detects a signal brought by touch operation and transmits the signal to the touch controller; the touch controller receives touch information from the touch sensing device, converts it to touch point coordinates, and sends the touch point coordinates to the processor 780, and can receive and execute commands from the processor 780. In addition, the touch panel 731 may be implemented by various types, such as a resistive type, a capacitive type, an infrared ray, and a surface acoustic wave. The input unit 730 may include other input devices 732 in addition to the touch panel 731. In particular, other input devices 732 may include, but are not limited to, one or more of a physical keyboard, function keys (such as volume control keys, switch keys, etc.), a trackball, a mouse, a joystick, and the like.

The display unit 740 may be used to display information input by the user or information provided to the user and various menus of the desktop. The Display unit 740 may include a Display panel 741, and optionally, the Display panel 741 may be configured in the form of a Liquid Crystal Display (LCD), an Organic Light-Emitting Diode (OLED), or the like. Further, the touch panel 731 can cover the display panel 741, and when the touch panel 731 detects a touch operation on or near the touch panel 731, the touch operation is transmitted to the processor 780 to determine the type of the touch event, and then the processor 780 provides a corresponding visual output on the display panel 741 according to the type of the touch event. Although in fig. 7, the touch panel 731 and the display panel 741 are two independent components to implement the input and output functions of the desktop, in some embodiments, the touch panel 731 and the display panel 741 may be integrated to implement the input and output functions of the desktop.

The desktop may also include at least one sensor 750, such as light sensors, motion sensors, and other sensors. Specifically, the light sensor may include an ambient light sensor that may adjust the brightness of the display panel 741 according to the brightness of ambient light, and a proximity sensor that may turn off the display panel 741 and/or the backlight when the desktop moves to the ear. As one of the motion sensors, the accelerometer sensor can detect the magnitude of acceleration in each direction (generally, three axes), detect the magnitude and direction of gravity when stationary, and can be used for applications of recognizing a desktop gesture, related functions of vibration recognition (such as pedometer and tapping), and the like; as for the desktop, other sensors such as a gyroscope, a barometer, a hygrometer, a thermometer, and an infrared sensor may be further configured, which are not described herein.

Audio circuitry 760, speaker 761, and microphone 762 may provide an audio interface between the user and the desktop. The audio circuit 760 can transmit the electrical signal converted from the received audio data to the speaker 761, and the electrical signal is converted into a sound signal by the speaker 761 and output; on the other hand, the microphone 762 converts the collected sound signal into an electric signal, converts the electric signal into audio data after being received by the audio circuit 760, processes the audio data output processor 780, and then transmits the processed audio data to, for example, another desktop via the RF circuit 710, or outputs the audio data to the memory 720 for further processing.

WiFi belongs to short-distance wireless transmission technology, desktop can help users to receive and send e-mails, browse webpages and access streaming media and the like through a WiFi module 770, and wireless broadband Internet access is provided for the users. Although fig. 7 shows the WiFi module 770, it is understood that it does not belong to the essential constitution of the desktop, and can be omitted entirely as needed within the scope not changing the essence of the invention.

The processor 780 is a control center of the desktop, connects each part of the entire desktop by using various interfaces and lines, and performs various functions of the desktop and processes data by running or executing software programs and/or modules stored in the memory 720 and calling data stored in the memory 720, thereby performing overall monitoring of the desktop. Optionally, processor 780 may include one or more processing units; preferably, the processor 780 may integrate an application processor, which primarily handles operating systems, user interfaces, applications, etc., and a modem processor, which primarily handles wireless communications. It will be appreciated that the modem processor described above may not be integrated into processor 780.

The desktop also includes a power supply 790 (e.g., a battery) for supplying power to the various components, which may preferably be logically connected to the processor 780 via a power management system, so as to manage charging, discharging, and power consumption via the power management system.

Although not shown, the desktop may further include a camera, a bluetooth module, and the like, which are not described in detail herein.

In the embodiment of the present application, the processor 780 included in the terminal further has the following functions:

Optionally, the processor 780 is further configured to perform the steps of any one implementation of the matting method of the present application.

The embodiments of the present application further provide a computer-readable storage medium for storing a computer program, where the computer program is configured to execute any one implementation manner of the matting method described in the foregoing embodiments.

Embodiments of the present application further provide a computer program product including instructions, which when executed on a computer, cause the computer to perform any one of the embodiments of a matting method described in the foregoing embodiments.

Those of ordinary skill in the art will understand that: all or part of the steps of implementing the method embodiments may be implemented by hardware associated with program instructions, where the program may be stored in a computer-readable storage medium, and when executed, performs the steps including the method embodiments; and the aforementioned storage medium may be at least one of the following media: various media that can store program codes, such as read-only memory (ROM), RAM, magnetic disk, or optical disk.

It should be noted that, in the present specification, all the embodiments are described in a progressive manner, and the same and similar parts among the embodiments may be referred to each other, and each embodiment focuses on the differences from the other embodiments. In particular, the apparatus and system embodiments, because they are substantially similar to the method embodiments, are described in a relatively simple manner, and reference may be made to some of the descriptions of the method embodiments for related points. The above-described embodiments of the apparatus and system are merely illustrative, and the units described as separate parts may or may not be physically separate, and the parts displayed as units may or may not be physical units, may be located in one place, or may be distributed on a plurality of network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment. One of ordinary skill in the art can understand and implement it without inventive effort.

The above description is only one specific embodiment of the present application, but the scope of the present application is not limited thereto, and any changes or substitutions that can be easily conceived by those skilled in the art within the technical scope of the present application should be covered within the scope of the present application. Therefore, the protection scope of the present application shall be subject to the protection scope of the claims.

Claims

1. A matting method, characterized in that the method comprises:

determining the transparency of a first pixel point according to the brightness difference between the first pixel point in the image to be segmented and a corresponding second pixel point in the background image and the color difference between the first pixel point and the second pixel point; the method comprises the following steps:

if the brightness value of the first pixel point is smaller than or equal to the brightness threshold value, determining the transparency of the first pixel point by using the brightness difference between the first pixel point and the second pixel point; the transparency of the first pixel point is calculated by the following formula,

f(x,y)＝p×|I_x,y-I_d,x,y|+q

wherein alpha is_x,yThe transparency of the first pixel point is obtained; i is _x,yThe brightness of the first pixel point is obtained; i is_d,x,yThe brightness of the second pixel point is obtained; p and q are both preset parameters;

if the brightness value of the first pixel point is larger than the brightness threshold value, determining the transparency of the first pixel point by using the brightness difference between the first pixel point and the second pixel point and the color difference between the first pixel point and the second pixel point; the transparency of the first pixel point is calculated by the following formula,

g(x,y)＝a×((R_x,y-R_d,x,y)²+(G_x,y-G_d,x,y)²)+b×|I_x,y-I_d,x,y|+c

wherein alpha is_x,yThe transparency of the first pixel point is obtained; r_x,yThe value of the red channel of the first pixel point is obtained; r_d,x,yThe value of the red channel of the second pixel point is obtained; g_x,yThe value of the green channel of the first pixel point is obtained; g_d,x,yThe value of the green channel of the second pixel point is obtained; i is_x,yThe brightness of the first pixel point is obtained; i is_d,x,yThe brightness of the second pixel point is obtained; a, b and c are all preset parameters;

2. The method of claim 1, wherein before determining the first pixel point as a target pixel point if the transparency of the first pixel point is greater than the transparency threshold, further comprising:

Calculating a transparency offset value corresponding to the first pixel point according to the brightness difference and the transparency difference of the first pixel point and the pixel points around the first pixel point;

3. The method of claim 1, wherein the background image is obtained by processing at least a portion of an image in the first N frames of the target video.

4. The method of claim 3, wherein the background image is determined by:

If the color distance corresponding to each pixel point in the first image is smaller than or equal to the color threshold corresponding to the corresponding pixel point in the first background image, taking the first background image as the background image; if not, then the mobile terminal can be switched to the normal mode,

5. The method of claim 4, wherein the updating the first background map based on the color distance comprises:

if the color distance is larger than the color threshold corresponding to the fourth pixel point, updating the color threshold corresponding to the fourth pixel point to be the color distance, and updating the color value of the fourth pixel point according to the color value of the third pixel point and the color value of the fourth pixel point;

6. The method of claim 1, further comprising:

7. A matting device, characterized in that the device comprises:

the calculation unit is used for determining the transparency of the first pixel point according to the brightness difference between the first pixel point in the image to be segmented and the corresponding second pixel point in the background image and the color difference between the first pixel point and the second pixel point; the computing unit is to:

f(x,y)＝p×|I_x,y-I_d,x,y|+q

wherein alpha is _x,yThe transparency of the first pixel point is obtained; I.C. A_x,yThe brightness of the first pixel point is obtained; I.C. A_d,x,yThe brightness of the second pixel point is obtained; p and q are both preset parameters;

g(x,y)＝a×((R_x,y-R_d,x,y)²+(G_x,y-G_d,x,y)²)+b×|I_x,y-I_d,x,y|+c

8. The apparatus of claim 7, further comprising:

9. The apparatus of claim 7, further comprising:

10. The apparatus according to claim 9, wherein the background image obtaining unit is specifically configured to:

11. The apparatus of claim 10, wherein the background image obtaining unit, when updating the first background map based on the color distance, is configured to:

12. The apparatus of claim 7, further comprising:

the color optimization unit is used for determining the target color value of the target pixel point according to the color value and the transparency of the target pixel point and the color value of the second pixel point; and updating the color value of each target pixel point in the target object to be a corresponding target color value.

13. A matting device characterized in that the device comprises a processor and a memory:

the processor is used for executing the matting method of any one of claims 1 to 6 according to instructions in the computer program.

14. A computer-readable storage medium, characterized in that the computer-readable storage medium is used for storing a computer program for performing the matting method as claimed in any one of claims 1 to 6.