TWI494865B - Depth-based subject extraction method - Google Patents

Description

Method of intercepting a target from depth information

The invention relates to a method for intercepting a target object from depth information, in particular to a method for intercepting a target human body from depth information.

Human images taken with a camera need to be separated from the target body (foreground) and background. There have been many studies on foreground extraction methods, such as frame difference method, background modeling and optical flow method, which are briefly described below.

Frame difference method: subtracts the image from the image of the previous frame, and takes the portion whose change value is greater than a threshold as the foreground.

Background Modeling: Create a background model with less changes in the picture, and subtract the input image from the background to get the foreground.

Optical flow method: Calculate the motion vector of each point in the picture, and take the part with constant speed movement or large amount of movement as the foreground.

However, these methods are mostly based on the assumption that "moving objects are foreground", and are limited by color image input, which is easily interfered by the light in the picture, the shadow of the person and the background of the change, and there is a problem that the background of the same color cannot be discerned.

As a result of his position, the inventor, in view of the lack of prior art, thought of and improved the idea of invention, and finally invented the "method of intercepting objects from in-depth information" in this case.

The method proposed by the present invention is applicable to a depth matrix obtained by a depth sensing device, and can utilize additional distance dimension information to separate objects in a space compared to a method for grayscale or color two-dimensional images. Compared with the general use of the depth threshold (the front and rear depth of the target is set to the minimum/maximum depth, the portion other than the filtering), the method proposed by the present invention does not need to define an accurate threshold value, and allows a larger The range of human body movement can more effectively intercept the contours of the human body. In addition, compared with the method of using depth information to locate various parts and joint positions of the human body, the method proposed by the invention does not require training and comparison of previous data, and can reduce the calculation amount at the time of operation.

The main purpose of this case is to provide a method for intercepting objects from deep information. It can use additional distance dimension information to separate objects in space without defining an accurate threshold value and allowing a larger range of human body movement. The human body contour is intercepted more effectively, and the method proposed by the invention can reduce the calculation amount at the time of operation without training and comparison of the previous data.

Another main object of the present invention is to provide a method for intercepting an object from a depth information, comprising the steps of: providing a depth matrix for generating a plurality of three-dimensional spatial points; and resampling according to the plurality of three-dimensional spatial points And adding at least one extension point to generate a plurality of three-dimensional connected blocks; when any one of the plurality of three-dimensional connected blocks has a volume greater than a predetermined value, calculating one of the three-dimensional connected blocks a distance from a centroid of a heart to a known target block; calculating a minimum value of each of the distances, and setting the three-dimensional connected block specified as one of the minimum values as a primary block; When any one of the plurality of three-dimensional connected blocks has the volume less than or equal to the preset value and is located at a front view boundary of the main block, and has a depth value smaller than a depth value of the main block And setting the three-dimensional connected block whose volume is less than or equal to one of the preset values as a front end block; and associating the main block with a point of the front end block as the known item Blocks.

A primary object of the present invention is to provide a method for intercepting an object from a depth information, comprising the steps of: providing a depth matrix for generating a plurality of three-dimensional connected blocks; and when the plurality of three-dimensional connected blocks are When any one of the volumes has a volume greater than a predetermined value, calculating a distance between a centroid of the three-dimensional connected block and a centroid of a known target block; calculating a minimum value of each of the distances, And setting the three-dimensional connected block whose distance is one of the minimum values to a primary block; and when any one of the plurality of three-dimensional connected blocks has the volume being less than or equal to the preset value, and having When a depth value is less than a depth value of the primary block, the three-dimensional connected block whose volume is less than or equal to one of the preset values is set as a front end block; and the main block and the front end are One of the blocks is a collection of points as the known target block.

A further main object of the present invention is to provide a method for intercepting a target from a depth matrix, comprising the steps of: obtaining a plurality of blocks from the depth matrix; determining a primary block from the blocks; Determining a front end block in the blocks; and associating the main block with the front end block as the target.

The above described objects, features, and advantages of the present invention will become more apparent and understood.

The method proposed by the present invention is a target human body capturing method based on depth information, which uses a depth sensing device to obtain a continuous depth matrix of a scene, and then performs subsequent processing on the depth information to define objects in the space. To take out the target of interest. First, this method converts the single depth matrix into a spatial point and resamples the depth value and extends to connect the points of an object. Then, the object and its volume are judged from the processed space point, the small object is filtered out, the human body is positioned according to the position of the front sheet, and the problem that the joint is shielded is dealt with. In this way, the goal of capturing the target human body is achieved.

The method of the present invention extracts the position occupied by the target from the continuous depth matrix obtained by a sensing device. The depth matrix is a matrix that records a scene depth information, indicating the parallel distance of the object to the sensing device in the scene. If the sensing range of the sensing device is a long h width w, the depth matrix is a matrix of h rows and w columns. The depth of one of the elements of the matrix from the point to the surface of the sensing device is smaller and smaller, and the larger the distance, the larger the value. As shown in the first figure, there is a human body with a forward arm in the scene. The sensing range of the sensing device is 120 pixels long and 160, so that a matrix of 160x120 can be obtained. The element value at (75, 50) represents the value of the human arm distance sensing device in the original picture.

In this example, the distance value is measured in metric and ranges from about 500 to 6,000. The value of the trunk portion is greater than the palm portion, that is, the palm separation device is closer. If the device can return multiple depth matrices per second, all the depth matrices in a period of time are collected sequentially, which is called a continuous depth matrix. As shown in the second figure, it can be used to represent the depth information at each time point. . This depth matrix can be obtained in a variety of ways, such as TOF camera, Stereo Camera, and the like.

The following action flow refers to the third figure and is performed on one of the continuous depth matrices, and is set as the kth depth matrix:

1. Convert the depth matrix to a spatial point: the element originally located in the matrix (x, y) has a value of z, which is converted into a three-dimensional coordinate point (x, y, z). As shown in the fourth figure, it can be seen that the objects in the scene have a depth interval in the three-dimensional space. However, the points on an object are not completely adjacent, so the following two steps are required.

i. Depth resampling: Divide the z-axis values at all points by a sampling range s to make the distance of each point narrow. Representing this range, as long as there is a punctuation point, it is considered to have a value.

Ii. Complementary extension point: Make a supplementary point motion at a certain distance t backward on the z-axis of each point. After these two steps, the points belonging to the same object in the picture are connected as much as possible. The results of the above two steps after execution are as shown in the fifth (a) and fifth (b), respectively. That is, the fifth (a) and fifth (b) drawings respectively show a two-dimensional top view and a re-set of the unprocessed point set obtained according to the preferred embodiment of the present invention and the flowchart shown in the third figure. A two-dimensional top view after sampling and complementing.

2. Three-dimensional connected block mark: Connect the block set after the above-mentioned sampling and complementing points, and mark the connected points as the same area. For example, (2,1,2) and (2,1,3), (2,2,2) will be judged as connected blocks, but (9,8,7) will not belong to the above block (if there is no Other connected points). After the step of connecting the block marking process, the result is as shown in the sixth figure, which shows a two-dimensional connected block mark obtained according to the preferred embodiment of the present invention and the flow chart shown in the third figure. Schematic diagram. The figure is numbered as the mark number of the block (can be color-coded on the screen of the display screen), the circle indicates the centroid position of the block, and the circle size represents the number of block points.

3. Make the following judgments for each block:

i. Divide the block into large blocks and blocks by its volume

Ii. From the large block, according to the target position of the k-1th depth matrix, the block with the closest centroid distance is selected as the main block. (When initializing k=0, use manual circle selection or face recognition to determine the initial target position.)

Iii. From the cell block, according to the main block of the previous step, select the following:

a. In the front view, the edges are adjacent. And

b. The depth value is smaller than the main block

Add eligible cell blocks as front-end blocks

Finally, the main block and the front block are merged, which is the new target. The seventh picture is the final single target extraction result, and the eighth picture is the continuous depth matrix target extraction result.

Example:

1. A method of intercepting a target from a depth information, comprising the steps of:

Providing a depth matrix for generating a plurality of three-dimensional spatial points; re-sampling and supplementing at least one extension point according to the plurality of three-dimensional spatial points to generate a plurality of three-dimensional connected blocks; and when the plurality of three-dimensional connected blocks are When any one of the volumes has a volume greater than a predetermined value, calculating a distance between a centroid of the three-dimensional connected block and a centroid of a known target block; calculating a minimum value of each of the distances, And setting the three-dimensional connected block whose distance is one of the minimum values to a primary block; and when any one of the plurality of three-dimensional connected blocks has the volume less than or equal to the preset value and located at the a front view boundary of one of the main blocks, and having a depth value smaller than a depth value of the main block, setting the three-dimensional connected block whose volume is less than or equal to one of the preset values as a front end area a block; and associating the primary block with a point set of the front end block as the known target block.

2. The method of embodiment 1, wherein the step of providing a depth matrix further comprises the steps of:

Performing a depth resampling to divide a z-axis coordinate value of each of the plurality of three-dimensional spatial points by a sampling range, wherein when at least one punctuation exists in the sampling range, the value is regarded as having a value; On a z-axis of a plurality of three-dimensional space points, a complementary extension point action is performed for a fixed distance.

3. The method of embodiment 1 or 2, wherein the step of providing a depth matrix further comprises the steps of:

Each of the plurality of three-dimensional spatial points adjacent to each other is marked as the same area to generate the plurality of three-dimensional connected blocks.

4. The method according to any of the preceding embodiments, wherein the initial target position of one of the known target blocks is determined by a manual circle or by a face recognition mode, and the target is a human body.

5. A method of intercepting a target from a depth information, comprising the steps of:

Providing a depth matrix for generating a plurality of three-dimensional connected blocks; and when any one of the plurality of three-dimensional connected blocks has a volume greater than a predetermined value, calculating a centroid of the three-dimensional connected block a distance from a centroid of a known target block; calculating a minimum value of each of the distances, and setting the three-dimensional connected block specified as one of the minimum values as a primary block; If any one of the plurality of three-dimensional connected blocks has the volume less than or equal to the preset value, and has a depth value smaller than a depth value of the primary block, the volume is less than or equal to the preset value. A specific three-dimensional connected block is set as a front end block; and the main block is combined with a point of the front end block as the known target block.

6. The method of embodiment 5, wherein the front end block is located at a front view boundary of the main block, and the step of providing a depth matrix further comprises the steps of:

Performing a depth resampling to divide a z-axis coordinate value of each of the plurality of three-dimensional spatial points by a sampling range, wherein when at least one punctuation exists in the sampling range, the value is regarded as having a value; a z-axis of a three-dimensional space point is made a complementary extension point motion by a certain distance t; and each of the plurality of three-dimensional space points adjacent to each other is marked as the same area to generate the plurality of three-dimensional connected blocks.

7. A method of intercepting a target from a depth matrix, comprising the steps of:

Obtaining a plurality of blocks from the depth matrix; determining a primary block from the blocks; determining a front end block from the blocks; and associating the main block with the front end block as the Target.

8. The method of embodiment 7, wherein the step of obtaining a plurality of blocks from the depth matrix further comprises the steps of:

Using a depth value z of one of the elements (x, y) in the depth matrix to generate a three-dimensional coordinate point (x, y, z); when the depth value z of the three-dimensional coordinate point (x, y, z) When less than a predetermined value, the three-dimensional coordinate point (x, y, z) represents a three-dimensional spatial point, so that a plurality of three-dimensional spatial points are generated; and according to the plurality of three-dimensional spatial points, re-sampling and supplementing at least one extension Point, 俾 produces the plurality of blocks.

9. The method of embodiment 7 or 8, wherein the step of determining a primary block from the blocks further comprises the steps of:

When any one of the plurality of blocks has a volume greater than a predetermined value, calculating a distance between a centroid of the block and a centroid of a known target block; and calculating the distance One of the minimum values, and the distance is specified as one of the minimum values, the block is set as a main block, and the step of determining a front end block from the blocks further includes a step: when When any one of the plurality of three-dimensional connected blocks has a volume less than or equal to the preset value and is located at a front view boundary of the main block, and has a depth value smaller than a depth value of the main block, Then, the three-dimensional connected block whose volume is less than or equal to one of the preset values is set as a front end block.

10. The method according to any of the preceding embodiments, wherein each of the blocks is a three-dimensional connected block, and an initial target position of the known target block is manually circled or recognized by a face. To decide, and the target is a human body.

In summary, the present invention provides a method for intercepting a target object from depth information, which can utilize additional distance dimension information to separate objects in space without defining an accurate threshold value and allowing a larger range of human body movement. The human body contour can be intercepted more effectively, and the method proposed by the invention does not require the training and comparison of the prior data, and can reduce the calculation amount at the time of operation, so it is indeed progressive and novel.

Therefore, even though the present invention has been described in detail by the above-described embodiments, it can be modified by those skilled in the art, and is not intended to be protected as claimed.

First: a schematic diagram showing a depth matrix corresponding to the input of the depth value of each point in grayscale color;

The second figure: a schematic diagram showing a continuous depth matrix corresponding to the depth value of each point in gray scale color;

FIG. 3 is a flow chart showing a method for intercepting an object from a depth information according to a preferred embodiment of the present invention;

The fourth picture: a three-dimensional diagram showing the conversion of the depth matrix to the three-dimensional space point;

Fifth (a) and fifth (b): respectively showing a two-dimensional top view of an unprocessed set of points obtained according to the preferred embodiment of the present invention and the flow chart shown in FIG. Re-sampling and replenishing the 2D top view;

Figure 6 is a two-dimensional diagram showing a three-dimensional connected block mark obtained according to the preferred embodiment of the present invention and the flow chart shown in Figure 3;

Figure 7: A two-dimensional diagram showing the results of a single target capture;

Figure 8: A two-dimensional diagram showing the results of a continuous foreground capture.

Claims (10)

A method for intercepting an object from a depth information, comprising the steps of: providing a depth matrix for generating a plurality of three-dimensional spatial points; re-sampling and supplementing at least one extension point according to the plurality of three-dimensional spatial points;俾 generating a plurality of three-dimensional connected blocks; when any one of the plurality of three-dimensional connected blocks has a volume greater than a predetermined value, calculating a centroid of the three-dimensional connected block and a known target block a distance of one centroid; calculating a minimum value of each of the distances, and setting the three-dimensional connected block specified as one of the minimum values as a main block; when the plurality of three-dimensional connected blocks If any one of the volumes has a volume less than or equal to the preset value and is located at a front view boundary of the main block, and has a depth value smaller than a depth value of the main block, the volume is less than or equal to One of the preset values specifies that the three-dimensional connected block is set as a front end block; and the main block is combined with a point of the front end block as the known target block. The method of claim 1, wherein the step of providing a depth matrix further comprises the steps of: performing a depth resampling, and dividing a z-axis coordinate value of each of the plurality of three-dimensional spatial points by one sampling. a range, wherein when at least one punctuation exists in the sampling range, it is regarded as having a value; and on a z-axis of each of the plurality of three-dimensional space points, a complementary extension point action is performed on the latter fixed distance. The method of claim 2, wherein the providing a depth matrix step further comprises the steps of: marking each of the plurality of three-dimensional spatial points adjacent to each other as the same area to generate the plurality of three-dimensional Connected blocks. The method of claim 1, wherein the initial target position of the known target block is determined by a manual circle or by a face recognition method, and the target is a human body. A method for intercepting an object from a depth information, comprising the steps of: providing a depth matrix for generating a plurality of three-dimensional connected blocks; and when any one of the plurality of three-dimensional connected blocks has a volume greater than a predetermined value, calculating a distance between a centroid of the three-dimensional connected block and a centroid of a known target block; calculating a minimum value of each of the distances, and the distance is the minimum One of the three-dimensional connected blocks is set as a main block; when any one of the plurality of three-dimensional connected blocks has the small size When the value is equal to or greater than the preset value, and the depth value is less than the depth value of the primary block, the three-dimensional connected block whose volume is less than or equal to one of the preset values is set as a front end block. And associating the primary block with a point set of the front end block as the known target block. The method of claim 5, wherein the front end block is located at a front view boundary of the main block, and the step of providing a depth matrix further comprises the steps of: performing a depth resampling, each of which A z-axis coordinate value of a plurality of three-dimensional spatial points is equal to a sampling range, wherein when at least one punctuation exists in the sampling range, it is regarded as having a value; on a z-axis of each of the plurality of three-dimensional spatial points A complementary extension point action is performed for a certain distance t; and each of the plurality of three-dimensional space points adjacent to each other is marked as the same area to generate the plurality of three-dimensional connected blocks. A method for extracting a target from a depth matrix, comprising the steps of: obtaining a plurality of blocks from the depth matrix; and when any one of the plurality of blocks has a volume greater than a predetermined value , calculating one of the centroids of the block and one of the known target blocks a distance of the centroid; calculating a minimum value of each of the distances, and setting the block to be one of the minimum values to be a primary block; determining a front end region from the plurality of blocks a block; and a union of the main block and the front end block as the target. The method of claim 7, wherein the step of obtaining a plurality of blocks from the depth matrix further comprises the step of using a depth value z of one of the elements (x, y) in the depth matrix To generate a three-dimensional coordinate point (x, y, z); when the depth value z of the three-dimensional coordinate point (x, y, z) is less than a predetermined value, that is, the three-dimensional coordinate point (x, y, z) Representing a three-dimensional spatial point such that a plurality of three-dimensional spatial points are generated; and the plurality of blocks are generated by re-sampling and supplementing at least one extended point based on the plurality of three-dimensional spatial points. The method of claim 7, wherein the step of determining a front end block from the blocks further comprises the step of: when any one of the plurality of three-dimensional connected blocks has the volume is less than Or equal to the preset value and a front view boundary of the main block, and having a depth value smaller than a depth value of the main block, the volume is less than or equal to one of the preset values. The three-dimensional connected block is set as a front end block. The method of claim 7, wherein each of the blocks is a three-dimensional connected block, and an initial target position of the known target block is A manual circle selection method or a face recognition method is used, and the target object is a human body.