KR20180065441A - Object extraction method and apparatus - Google Patents

Abstract

An object extraction method and an apparatus thereof are disclosed. The object region extraction method includes a step of receiving a color image; a step of receiving joint information on an object; a step of generating skeleton information on the object using the joint information; and a step of extracting an object region using color information and the skeleton information on the object in the color image. It is possible to extract the object region with high accuracy.

Description

     Field of the Invention [0001] The present invention relates to an object extraction method and apparatus,

The present invention relates to a method and apparatus for separating and extracting object regions from a color image.

Human Body Segmentation technology, which separates background and person from images in order to make realistic effects such as human and background composition in broadcasting or film, is a very important factor.

Conventional techniques for dividing a human region in an image can be roughly classified into several types according to information to be used.

The first method is to use the color information in the image to search for a position in a human color range based on the corresponding color information or to extract feature vectors from the combination of feature points and search for features An example of how to find it.

In the second method, not only color information but also depth information is used. Because color information is prone to noise depending on the state of light and lighting, it is a method of increasing accuracy by using depth information that is not greatly affected by light and illumination.

In the case of the first method, as described above, not only is it greatly affected by light and lighting, but also when there is a complicated or similar background color around a person (for example, furniture similar to a person's color) A phenomenon appears. Broadcast and film studios use chroma key to solve this problem. Chroma key is often referred to as a green screen or a blue screen. It is a technique used for composing other images by removing a single color in the editing process after shooting in front of a monochromatic background.

However, the field of application of the chroma key is limited to a case of a pre-fabricated studio or movie set. That is, there is a problem that a person can not be used when he / she wants to recognize and divide a person in an image taken based on a real environment.

In the second method, additional depth information is used, so that even if there are complicated or similar backgrounds, this background can be filtered to some extent through the distance from the camera. However, due to the nature of the sensor that acquires the depth information, the spatial resolution is rather low and the obtained depth value is often inaccurate. Therefore, the boundary of the divided person area is inaccurate, or the "Flickering "There is a problem that the phenomenon occurs.

As described above, color information and depth information, which are basic information used in conventional human recognition and segmentation, are problematic in terms of reliability due to confusion with the background and noise caused by sensor characteristics.

The present invention provides a method and apparatus for separating and extracting object regions from a color image.

The present invention also provides an object region extracting method and apparatus for extracting an object region with high accuracy using color information and skeleton information of an object.

According to an aspect of the present invention, a method of extracting an object region from a color image is provided.

According to an embodiment of the present invention, there is provided an image processing method comprising: receiving a color image; Receiving joint information on an object; Generating skeleton information for the object using the joint information; And extracting an object region using the color information of the object and the skeleton information in the color image.

According to another embodiment of the present invention, there is provided an image processing method comprising the steps of: receiving a color image and depth information; Generating joint information using the depth information; Generating skeleton information for the object using the joint information; And extracting an object region using the color information of the object and the skeleton information in the color image.

The method may further include generating a skeleton weight map for each pixel of the color image using the skeleton information before extracting the object region.

The skeleton weight map may be generated such that as each pixel of the color image is moved away from the skeleton position included in the skeleton information, the skeleton weight of each pixel is attenuated.

The skeleton weight map may be generated by varying the degree of skeleton weight reduction of each pixel according to the reliability corresponding to each joint of the skeleton included in the skeleton information.

The skeleton weight map may be generated by varying the degree of skeleton weight reduction of each pixel according to the depth value of the skeleton included in the skeleton information.

The extracting of the object region may include generating a graph structure having each pixel of the color image as each node, and connecting each of the nodes to a foreground node and a background node, respectively; Assigning weights for edges between each of the nodes, edges between the nodes and the foreground nodes, or edges between each of the nodes and the background node in the graph structure; And extracting the object region by cutting an edge in the graph structure using a GraphCut algorithm.

Wherein the assigning of the weight of the edge assigns the weight of the edge connected to the foreground node to increase according to whether the color of each pixel is similar to the color model of the predetermined object region, The weight of the edge connected to the background node may be increased in accordance with the similarity with the color model of the background region.

The assigning of the weights may be such that as the skeleton weight included in the skeleton weight map corresponding to each pixel increases, the weights of the edges connecting each node and the foreground node increase.

According to another aspect of the present invention, there is provided an image processing apparatus for extracting an object region from a color image.

According to an embodiment of the present invention, an input unit for acquiring joint information on a color image and an object from a plurality of sensors, respectively; A skeleton generating unit for generating skeleton information for the object using the joint information; And an extracting unit for extracting an object region using color information of the object and the skeleton information in the color image.

According to another embodiment of the present invention, there is provided an image processing apparatus including an input unit for obtaining color image and depth information from at least one sensor; A joint information generating unit for generating joint information using the depth information; A skeleton generating unit for generating skeleton information on the object using the joint information; And an extracting unit for extracting an object region using color information of the object and the skeleton information in the color image.

And a weight map generator for generating a skeleton weight map for each pixel of the color image using the skeleton information,

The extractor may extract the object region using the skeleton weight map.

Wherein the extracting unit is configured to generate a graph structure having each pixel of the color image as each node, and generate a graph structure using the color model for the object and the skeleton weight map to the foreground node And the object region can be extracted by cutting an edge in the graph structure using a GraphCut algorithm.

The method and apparatus for extracting an object region according to an embodiment of the present invention have an advantage of accurately separating an object region with high accuracy by using color information and skeleton information in extracting an object region from a color image .

1 is a flowchart illustrating an object region extracting method according to an embodiment of the present invention;
2 is a diagram illustrating an operation process of a Kalman filter according to an embodiment of the present invention.
Figure 3 is a visualization of joint information for an object in accordance with an embodiment of the present invention.
4 is a visualization of a skeleton weight map according to an embodiment of the present invention.
5 is a diagram illustrating a graph cut according to an embodiment of the present invention;
6 illustrates a color image according to an embodiment of the present invention.
FIG. 7 is a diagram illustrating an image of a skeleton information extracted from a depth image according to an exemplary embodiment of the present invention, on a color image. FIG.
FIG. 8 is a view illustrating an image in which skeleton information is projected on a depth image according to an embodiment of the present invention; FIG.
9 is a view showing a result of extracting an object region using only conventional color information.
FIGS. 10 and 11 are diagrams illustrating a result of extracting object regions using only conventional depth information. FIG.
12 is a view illustrating a result of extracting an object region using color information and skeleton information in a color image according to an exemplary embodiment of the present invention.
13 is a block diagram schematically showing an internal configuration of an image processing apparatus according to an embodiment of the present invention;

As used herein, the singular forms "a", "an" and "the" include plural referents unless the context clearly dictates otherwise. In this specification, the terms "comprising ", or" comprising "and the like should not be construed as necessarily including the various elements or steps described in the specification, Or may be further comprised of additional components or steps. Also, the terms "part," "module, " and the like in the specification mean units for processing at least one function or operation, which may be implemented by hardware or software or by a combination of hardware and software.

Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings.

FIG. 1 is a flowchart illustrating an object region extracting method according to an embodiment of the present invention. FIG. 2 is a flowchart illustrating an operation of a Kalman filter according to an embodiment of the present invention. FIG. FIG. 4 is a diagram illustrating a visualization of a skeleton weighting map according to an embodiment of the present invention. FIG. 5 is a view illustrating a graph cut according to an embodiment of the present invention. FIG. 6 is a diagram illustrating a color image according to an embodiment of the present invention. FIG. 7 is a diagram illustrating a color image of a skeleton information extracted from a depth image according to an exemplary embodiment of the present invention, FIG. 8 is a diagram illustrating an image in which skeleton information is projected on a depth image according to an exemplary embodiment of the present invention. FIG. 9 is a view illustrating an example of extracting an object region using only conventional color information. FIGS. 10 and 11 are views showing the result of extracting an object region using only conventional depth information. FIG. 12 is a view showing a result of extracting color information and skeleton information from a color image according to an embodiment of the present invention. And extracting the object region using the extracted region.

In step 110, the image processing apparatus 100 receives the color image and the depth information, respectively.

The image processing apparatus 100 can receive the color image and the depth information through a sensor described later.

In this specification, it is assumed that a sensor generates color images and depth information at the same time in order to facilitate understanding and explanation, but it is natural that a color image and depth information can be inputted through a plurality of sensors, respectively .

For example, the image processing apparatus 100 may receive a plurality of color images continuously through an image sensor such as a camera, and may generate depth information through a plurality of color images.

In step 115, the image processing apparatus 100 generates joint information using the depth information.

The method of generating joint information using depth information itself is obvious to those skilled in the art, and a separate description thereof will be omitted.

As another example, the image processing apparatus 100 may be coupled to a key knot and may receive joint information through a key knot. In addition to the Kinect, most motion recognition devices provide joint information, and the image processing apparatus 100 may receive joint information about the object through these motion recognition devices.

In operation 120, the image processing apparatus 100 corrects joint information generated using the Kalman filter.

Joint information may not be properly measured due to objects being masked by other objects or backgrounds or due to noise in the depth image. Accordingly, the image processing apparatus 100 can correct the joint information using the Kalman filter.

For example, when using a Kinect or the like, there are provided 25 joint information for the object and reliability (for example, three steps) for each joint information. Therefore, a joint with low reliability using Kalman filter estimates the position of the current joint through the predicted value, and the joint with high reliability uses the current joint position provided through the keynote. This allows the joint information to be corrected. The operation of the Kalman filter is as shown in Fig. Since the Kalman filter itself is already known to those skilled in the art, a further description thereof will be omitted.

In this specification, it is assumed that a Kalman filter is used for joint information correction in order to facilitate understanding and explanation. However, it is a matter of course that other techniques may be applied in addition to the Kalman filter.

FIG. 3 is a diagram visualizing the names of the 25 joints obtainable through the Kinect and corresponding body parts when the object is a person. In the present specification, in order to facilitate understanding and explanation, it is assumed that an object is a person, but an object

In operation 125, the image processing apparatus 100 generates skeleton information using the joint information.

For example, the image processing apparatus 100 can generate 24 skeletons using 25 joint information obtained through a Kinect sensor or the like. That is, the image processing apparatus 100 can obtain a straight line passing through each joint, and designate all the pixels on the straight line as a skeleton. Thus, each joint can be connected to generate skeleton information for an object.

Also, when skeleton information is generated based on a depth image such as a keynote, the coordinate system between the skeleton information and the color image may be different from each other. Therefore, the coordinate system of the skeleton information should be converted to the coordinate system of the color image.

In addition, a preprocessing process for correcting the resolution and the viewpoint difference between the color image and the depth image may be additionally performed according to the implementation method.

In operation 130, the image processing apparatus 100 generates a skeleton weight map using the generated skeleton information.

The skeleton weight map can be generated to have a higher weight value as the likelihood of the object region in the color image is higher based on the skeleton information. FIG. 4 illustrates an example of visualizing and visualizing a skeleton weight map according to an embodiment of the present invention.

Since skeleton information is skeleton information for an object, a pixel closer to the skeleton position in the color image is more likely to be included in the object region, and a pixel far from the skeleton position is less likely to be included in the object region. Accordingly, the image processing apparatus 100 can project the skeleton information onto the color image in order to generate the skeleton weight map using the skeleton information. The image processing apparatus 100 can generate a skeleton weight map so that the weight value for each pixel decreases as each pixel of the color image moves away from the skeleton position included in the skeleton information.

That is, as shown in FIG. 4, the image processing apparatus 100 can generate a skeleton weight map so as to have a higher weight value as the joint position and the position of the skeleton become closer to each other.

In one embodiment of the present invention, a Gaussian function may be utilized in generating a skeleton weight map using skeleton information.

This can be expressed by the following equation (1).

는 가우시안 함수의 표준 편차를 나타낸다. Here, (x, y) represents the coordinates of the image, and (x _s , y _s ) represents the skeleton coordinates close to (x, y). Also,

Represents the standard deviation of the Gaussian function.

의 값은 스켈레톤 신뢰도, 스켈레톤 정보에 포함되는 각 조인트의 신뢰도, 스켈레톤의 깊이값 중 적어도 하나에 따라 각각 상이하게 결정될 수 있다.In generating a skeleton weighting map, the weighting factors for each pixel

May be determined differently depending on at least one of the skeleton reliability, the reliability of each joint included in the skeleton information, and the depth value of the skeleton.

값은 스켈레톤 신뢰도에 따라 상이하게 결정될 수 있다. 영상 처리 장치(100)는 스켈레톤 신뢰도가 높을스록

의 값을 낮은 값으로 설정하여 스켈레톤 정보가 객체 영역 분할에 많은 영향을 미치도록 할 수 있다.E.g,

The value may be determined differently depending on the skeleton reliability. The image processing apparatus 100 is a system in which the reliability of the skeleton is high

Can be set to a low value so that the skeleton information has a great influence on the object area division.

의 값을 높은 값으로 설정하여 스켈레톤 정보가 객체 영역 분할에 보조적인 역할만 수행하도록 할 수도 있다. On the other hand, when the reliability of the skeleton information is low,

Can be set to a high value so that the skeleton information plays only an auxiliary role in object region division.

의 값을 다르게 설정할 수도 있다. In another example, the reliability may be different for each joint in the skeleton information. Accordingly, the image processing apparatus 100 can perform the image processing

May be set differently.

값을 다르게 설정할 수도 있다. 즉 영상 처리 장치(100)는 스켈레톤의 깊이 값이 클수록

가 작은 값을 가지도록 설정하고, 스켈레톤의 깊이 값이 작을수록

가 큰 값을 가지도록 설정할 수도 있다.As another example, since the size of the object area becomes smaller as the distance from the camera increases, the image processing apparatus sets the width according to the depth value of the skeleton position

Values can be set differently. That is, in the image processing apparatus 100, as the depth value of the skeleton becomes larger

Is set to have a small value, and the smaller the depth value of the skeleton

Can be set to have a large value.

In operation 135, the image processing apparatus 100 separates (extracts) the object region from the color image using the color image and the skeleton weight map.

In more detail, the image processing apparatus 100 may generate a color model for an object region and a background region using a color image, respectively. In this case, a Gaussian mixture model may be used.

Of course, in using the color model of the object region, the image processing apparatus 100 may generate a color model of the object region using the pixel information of the object region extracted from the previous frame.

For example, the image processing apparatus 100 may generate a color model of the object region using pixels of the object region extracted in the previous frame and pixels of the skeleton generated in the current frame.

The color model of the background region can be generated using pixels belonging to all the remaining regions except the object region to be extracted.

In another example, the image processing apparatus 100 may generate a color model for a background region using pixels other than pixels belonging to an object region within a bounding box including an object region.

The image processing apparatus 100 may divide (extract) the object region using the color model of the object region and the background region and the skeleton weight map.

For example, the image processing apparatus 100 may separate (extract) an object region using a graph-based region segmentation technique.

In more detail, the image processing apparatus 100 can separate (extract) object regions using a GraphCut technique.

A method of separating (extracting) an object region in the image processing apparatus 100 using GraphCut technology will be described with reference to FIG.

Each pixel corresponds to a node in the graph, and neighboring pixels are connected to edges in the graph. At this time, the graph may be generated in a unit of a pixel rather than a pixel or a super pixel. The weight of the edge can be set using the difference value between connected nodes. A color value difference between nodes connected in more detail can be used as a weight setting method of an edge.

Each node may be separately connected to the foreground node s and the background node t (63 of FIG. 5). As each node is closer to the foreground, a higher weight is assigned to the edge connected to the foreground node. On the other hand, as each node is closer to the background, a higher weight is assigned to the edge connected to the background node.

As described above, according to an embodiment of the present invention, the image processing apparatus 100 has a color model for an object region and a background region, The higher the weight of the edge connected to the foreground node, the higher the color model. On the other hand, as the color value of each node is similar to the color model of the background area, the image processing apparatus 100 can assign a higher weight to the edge connected to the background node.

In addition, the image processing apparatus 100 can allocate a weight of an edge connected to the foreground node to a higher value as the skeleton weight of each node is higher, using the skeleton weight map.

The image processing apparatus 100 can separate (extract) the object region in accordance with the method specified by the GraphCut method (e.g., max-flow or min-cut) after completing the weight assignment for all the edges .

The GraphCut method operates by cutting the edges in the graph structure and operates so that the sum of the weights of the cut edges is minimized (64 in FIG. 5).

According to an embodiment of the present invention, since the image processing apparatus 100 sets the edge weight using the skeleton weight map, the object region can be set only in the vicinity of the skeleton.

FIG. 6 is a view illustrating a color image according to an embodiment of the present invention. FIG. 7 is a view illustrating an image of a skeleton information extracted from a depth image according to an exemplary embodiment of the present invention, Is a diagram illustrating an image in which skeleton information is projected on a depth image according to an embodiment of the present invention.

In the case of extracting an object using only color information in a complex color image as shown in FIG. 6, surrounding backgrounds similar to the color of an object region (for example, a human region) are recognized as the same object So that a problem of division occurs.

FIGS. 10 and 11 are diagrams showing regions obtained stepwise when object regions (i.e., human regions) are extracted using only depth information. FIG. 11 shows that although the object region is extracted roughly from a binary image in which pixels having a specific threshold value or more in a depth image are displayed in black, the boundary of the extracted object region is considerably inaccurate due to inaccurate information of the depth image.

FIG. 12 shows a result of extracting an object region using color information and skeleton information in a color image according to an exemplary embodiment of the present invention. By extracting an object region using skeleton information as well as color information, It can be seen that the object region can be accurately separated (extracted) even in an image having a similar or complicated background or a difference in depth between the foreground and the background.

FIG. 13 is a block diagram schematically illustrating an internal configuration of an image processing apparatus according to an embodiment of the present invention.

13, an image processing apparatus 100 according to an exemplary embodiment of the present invention includes an input unit 1310, a joint information generation unit 1315, a correction unit 1320, a skeleton generation unit 1325, A setting unit 1335, an extracting unit 1340, a memory 1345, and a processor 1350, as shown in FIG.

The input unit 1310 is means for acquiring color image and depth information from at least one sensor.

For example, the input unit 1310 may acquire joint information on a color image and an object from a plurality of sensors. As already described above, the image processing apparatus 100 may acquire joint information about an object through a sensor such as a Kinect. When the image processing apparatus 100 acquires the joint information, it is needless to say that the joint information generating unit 1315 described below may not be included in the configuration.

The joint information generation unit 1315 is means for generating joint information on the object using the obtained depth information.

The correction unit 1320 is means for correcting the joint information generated by the joint information generation unit 1315 or acquired through the input unit 1310. [ As described above, the corrector 1320 can correct the joint information using the Kalman filter. The detailed operation of the Kalman filter is obvious to those skilled in the art, so a detailed description thereof will be omitted.

The skeleton generation unit 1325 is means for generating skeleton information using joint information. You can create skeleton information by connecting each acquired joint to an object.

The weight map generating unit 1330 is a unit for generating a skeleton weight map in which a skeleton weight for each pixel of the color image is set using skeleton information.

The method of setting the skeleton weight for each pixel can be given a skeleton weight so that the weight of each pixel of the color image decreases as the distance from the skeleton position included in the skeleton information decreases.

For another example, a skeleton weight for each pixel may be given by varying the degree of weight reduction depending on the reliability corresponding to each joint of the skeleton included in the skeleton information.

As another example, the skeleton weight value may be given to each pixel by varying the weight reduction degree according to the depth value of the skeleton included in the skeleton information. Since this is the same as described above, redundant description will be omitted.

The setting unit 1335 is a means for setting a color model for the object region and a color model for the background region using the color information of each pixel of the color image.

Since this is the same as described above, redundant description will be omitted.

The extraction unit 1340 extracts the object region using the color information and the skeleton information about the object in the color image.

As described above, the extracting unit 1340 generates a graph structure in which each pixel of the color image is a node, and then generates an edge between each node, an edge between each node and a foreground node (an object node) After assigning the weights of the edges between nodes, we can extract the object region by cutting the edges using the graph cut algorithm.

This is the same as that described above with reference to FIG. 1, so duplicate descriptions will be omitted.

The memory 1345 stores various algorithms necessary for performing a method of extracting an object region using color information and skeleton information of a color image according to an exemplary embodiment of the present invention, various data derived from the algorithm, It is means.

The processor 1350 includes internal components (e.g., an input unit 1310, a joint information generation unit 1315, a correction unit 1320, and a skeleton generation unit) of the image processing apparatus according to an exemplary embodiment of the present invention. A weight map generating unit 1330, a setting unit 1335, an extracting unit 1340, a memory 1345, and the like).

On the other hand, the components of the above-described embodiment can be easily grasped from a process viewpoint. That is, each component can be identified as a respective process. Further, the process of the above-described embodiment can be easily grasped from the viewpoint of the components of the apparatus.

In addition, the above-described technical features may be implemented in the form of program instructions that can be executed through various computer means and recorded in a computer-readable medium. The computer-readable medium may include program instructions, data files, data structures, and the like, alone or in combination. The program instructions recorded on the medium may be those specially designed and constructed for the embodiments or may be available to those skilled in the art of computer software. Examples of computer-readable media include magnetic media such as hard disks, floppy disks and magnetic tape; optical media such as CD-ROMs and DVDs; magnetic media such as floppy disks; Magneto-optical media, and hardware devices specifically configured to store and execute program instructions such as ROM, RAM, flash memory, and the like. Examples of program instructions include machine language code such as those produced by a compiler, as well as high-level language code that can be executed by a computer using an interpreter or the like. The hardware device may be configured to operate as one or more software modules to perform the operations of the embodiments, and vice versa.

100: image processing device
1310:
1315: joint information generating unit
1320:
1325: Skeleton generation unit
1330: Weight map generating unit
1335: Setting section
1340:
1345: Memory
1350: Processor

Claims (14)

Receiving a color image;
Receiving joint information on an object;
Generating skeleton information for the object using the joint information; And
And extracting an object region using the color information of the object and the skeleton information in the color image.
Receiving color image and depth information;
Generating joint information using the depth information;
Generating skeleton information for the object using the joint information; And
And extracting an object region using the color information of the object and the skeleton information in the color image.
3. The method according to claim 1 or 2,
Before the step of extracting the object region,
And generating a skeleton weight map for each pixel of the color image using the skeleton information.
The method of claim 3,
Wherein the skeleton weight map is generated by reducing the weight as each pixel of the color image is away from the skeleton position included in the skeleton information.
The method of claim 3,
Wherein the skeleton weight map is generated by reducing the weights differently according to the reliability corresponding to each joint of the skeleton included in the skeleton information.
The method of claim 3,
Wherein the skeleton weight map is generated by reducing the weight according to the depth value of the skeleton included in the skeleton information.
The method of claim 3,
Wherein the extracting of the object region comprises:
Generating a graph structure having each pixel of the color image as each node, and connecting each node to a foreground node and a background node;
Assigning weights for edges between each of the nodes, edges between the nodes and the foreground nodes, or edges between each of the nodes and the background node in the graph structure; And
And extracting the object region by cutting an edge in the graph structure using a GraphCut algorithm.
8. The method of claim 7,
Wherein assigning the weights of the edges comprises:
And assigning a weight of an edge connected to the foreground node to increase according to whether the color of each pixel is similar to a color model of a predetermined object area, and if the color of each pixel is similar to a color model of a predetermined background area And allocating the weight of the edge connected to the background node to increase the weight of the edge connected to the background node.
8. The method of claim 7,
Wherein the assigning of the weights comprises:
Wherein a weight of an edge connecting each of the nodes and the foreground node increases as the skeleton weight included in the skeleton weight map corresponding to each pixel increases.
10. A computer-readable recording medium having recorded thereon a program code for performing the method according to any one of claims 1 to 9.
An input unit for acquiring joint information on a color image and an object from a plurality of sensors, respectively;
A skeleton generating unit for generating skeleton information for the object using the joint information; And
And an extracting unit for extracting an object region using color information of the object and the skeleton information in the color image.
An input unit for obtaining a color image and depth information from at least one sensor;
A joint information generating unit for generating joint information using the depth information;
A skeleton generating unit for generating skeleton information on the object using the joint information; And
And an extracting unit for extracting an object region using color information of the object and the skeleton information in the color image.
13. The method according to claim 11 or 12,
And a weight map generator for generating a skeleton weight map for each pixel of the color image using the skeleton information,
Wherein the extracting unit extracts the object region using the skeleton weight map.
14. The method of claim 13,
The extracting unit extracts,
A graph structure is generated using each pixel of the color image as each node, and a weight of an edge to a foreground node, which is a target of dividing each node and the object region, is used by using a color model for the object and the skeleton weight map Quot;
And extracts the object region by cutting an edge in the graph structure using a GraphCut algorithm.

Images