TW202230286A - Method and computer program product for filtering an object - Google Patents

Abstract

The invention provides a method for filtering an object, which has: detecting an area of an existing object from a video; establishing a sampling space based on a detecting result of the area; capturing a color feature from the sampling space; adding the color feature to a filtering condition; and filtering an interest object from the video based on the filtering condition.

Description

Method for screening objects and computer program product

The present invention relates to a method of image processing, in particular to a method of screening objects in a video.

In the existing image processing methods, objects moving around in a place may be our attention objects, for example, a specific target such as a vehicle on the road or a customer in a store. When the area of the venue reaches a certain level, the existing practice is to set up human stagnation points or set up cameras, and use manual methods to find objects of interest in the venue or in the videos of multiple cameras. However, this approach creates problems of labor-intensive and inefficient screening of objects of interest.

In view of the above-mentioned existing problems, the present invention provides a method for screening objects, which enables a user to more efficiently screen out objects of interest to be screened from a video.

The method for screening objects of the present invention includes: detecting a region where an object exists from a video; establishing a sampling space according to the detection result of the region; extracting a color feature from the sampling space; incorporating the color feature into a filtering conditions; and filtering an object of interest in the video according to the filtering conditions.

In an embodiment, the step of extracting the color feature from the sampling space further includes: extracting at least one feature point from the sampling space; generating a polygon according to the at least one feature point; and extracting the polygon from an area formed by the polygon. color characteristics.

In one embodiment, the object of interest is a person, and the feature point includes any one of human limbs and joints.

In one embodiment, the method for screening objects further includes: extracting an attribute feature from the sampling space; incorporating the attribute feature into the screening condition; and screening the object of interest in the video according to the screening condition.

In a certain embodiment, the object of interest is a character, and the attribute feature includes any one of human gender, age, and clothing.

In a certain embodiment, the method for screening objects further includes: performing a re-identification model transformation on the sampling space corresponding to the object to obtain a feature vector; performing the re-identification model transformation in a sampling space of the object of interest corresponding to the object of interest , to obtain a feature vector of the object of interest; calculate a distance between the feature vector and the feature vector of the object of interest in a multi-dimensional space; and when the feature vector is included in the screening condition, by comparing the distance with a critical value to obtain Determine whether the object of interest meets the filtering condition.

In one embodiment, the re-identification model is converted into a matrix operation.

In a certain embodiment, the method for screening objects further comprises: according to the screening conditions, screening out a second object of interest in a second video; and comparing a plurality of detected objects of interest and the second object of interest Any one of time, a plurality of locations, and a plurality of speeds to filter the second object of interest.

Regarding the computer program product of the present invention, the program is loaded through a computer, so that the processor of the computer executes any one of the aforementioned methods for screening objects.

In a certain embodiment, the processor further executes: according to the screening condition, filtering out the second object of interest in the second video; and comparing a plurality of times when the object of interest and the second object of interest were detected , a plurality of locations, and a plurality of speeds to filter the second object of interest.

According to the present invention, the user can more efficiently select the object of interest to be selected from the video.

The above and other objects and advantages of the present invention will become more apparent upon reference to the detailed description hereinafter described in conjunction with the accompanying drawings.

FIG. 1 is a flow chart illustrating a method for screening objects according to Embodiments 1 to 4 of the present invention.

[Example 1] First, Embodiment 1 of the present invention will be described. The steps corresponding to Embodiment 1 of the present invention are steps ST01, ST02, ST03, ST04, and ST09.

First, in step ST01 , the region where the object 21 exists is detected from the video 20 . Among them, the video 20 is shot by a camera in a certain environment. There may be various static or dynamic people, cars, etc. (hereinafter referred to as "objects 21") in the environment. The method of object detection can be implemented by using existing object detection technology. For example, it is possible to use a large amount of data to perform machine learning on a neural network in advance.

Next, in step ST02, when an object 21 is detected in a certain area of the video 20, a sampling space 22 is created according to the detection result of the area.

Please refer to Figure 2. FIG. 2 is a schematic diagram illustrating an example of creating a sampling space 22 according to the detection result of the area in step ST02 . As shown in the figure, the sampling space 22 may be a frame selection area surrounded by two coordinate points located approximately at the upper left and lower right of the object 21 (human in the figure). However, although the sampling space 22 in the drawings is a rectangle, it is not limited to this in other embodiments; that is, the sampling space 22 may also be other polygons.

Back to Figure 1. In step ST03 , after the sampling space 22 is established, color features are extracted from the sampling space 22 . The detailed operation flow of extracting color features from the sampling space 22 is described below.

FIG. 3 is a flowchart illustrating the detailed operation process of extracting color features from the sampling space 22 in step ST03 . As shown in the figure, step ST03 further includes steps ST31 to ST33. Steps ST31 to ST33 will be described below, respectively.

First, in step ST31 , feature points are extracted from the sampling space 22 . Specifically, if the detection result of the object 21 is a person, the extracted feature points may be human limbs and joints, such as left and right shoulders, shafts, palms, hips, knees, soles, necks, etc. of the body. As shown in FIG. 4A , 10 feature points A to J are extracted from the sampling space 22 . Feature points A and B correspond to the left and right shoulders, feature points C and D correspond to the elbow joint, feature points E and F correspond to the left and right palms, feature points G and H correspond to the left and right sides of the buttocks, and feature points I and J correspond to the left and right soles of the feet. If the detection result of the object 21 is a car, the extracted feature points may be a license plate, a wheel, a car roof, a car window, and the like.

Next, in step ST32, after the feature points are extracted, a polygon is generated according to the extracted feature points. As shown in FIG. 4B , assuming that the detection result of the object 21 is a person, the generated polygon can be the upper body of the person, that is, the oblique line area surrounded by the four feature points A, B, G, and H. In addition, polygons of various shapes can also be generated according to the number of extracted feature points.

Please refer to Figure 5A and Figure 5B. 5A and 5B are schematic diagrams illustrating an example of generating a polygon based on feature points in step ST32 in FIG. 3 . For example, if a feature point 40 is captured and its coordinates in the video 20 are (x, y), then the polygon generated by the feature point 40 can be (x+h, y+h) as shown in Fig. 5A , (x-h, y+h), (x-h, y-h), (x+h, y-h) are the squares enclosed by the above four coordinate points. In other words, the side length of the square is 2h, and the feature point 40 is located at the center of the square. It should be noted that the value of h should not be too large, and it is better to be less than 5 (pixels) in implementation.

On the other hand, if two feature points 41 and 42 are captured and their coordinates in the video 20 are (x ₁ , y ₁ ) and (x ₂ , y ₂ ), respectively, then the feature points 41 and 42 generate The polygons can be as shown in Fig. 5B as (x ₁ +hcosθ, y ₁ +hsinθ), (x ₁ -hcosθ, y ₁ -hsinθ), (x ₂ +hcosθ, y ₂ +hsinθ), (x ₂ -hcosθ, y ₂ -hsinθ) The rectangle enclosed by the above four coordinate points. Wherein, θ is the angle between the line segment formed by the feature points 41 and 42 and the y-axis direction, and the length of one side of the rectangle is 2h. As mentioned above, the value of h should not be too large, preferably less than 5 (pixels).

In addition, the above-mentioned rule for establishing a polygon based on at least one feature point is only one example, but the implementation is not limited thereto.

Returning to Fig. 3, in step ST33, color features are extracted from the polygonal region. Specifically, the polygon generated in step ST32 includes a plurality of pixels. Each pixel can define its color using color spaces such as RGB (red, green, blue), HSL (hue, saturation, lightness), HSV (hue, saturation, lightness). For the convenience of description, the following takes the HSL color space as an example to describe the extraction of a representative color from a plurality of pixels in a polygon as an example of the color feature of the sampling space 22 .

First, from a plurality of pixels in the polygon, calculate the median h' of all pixels in the H component. Next, from the plurality of pixels in the sampling space 22, take out the pixels whose H component is between [h'-20, h'+20], and calculate the median s' of these pixels in the S component. Finally, from the plurality of pixels taken out in the previous step, take out the pixels whose S component is between [s'-20, s'+20], and calculate the median l' of these pixels in the L component. As described above, a representative color (h', s', l') can be obtained as the color feature of the sampling space 22. After steps ST31 to ST33 are executed, the complete flow of step ST03 in FIG. 1 is completed.

Returning to FIG. 1, in step ST04, after the color features are extracted from the sampling space 22, the color features are included in the screening conditions. After the color feature of the sampling space 22 is included in the filtering conditions, in step ST09, other detected objects in the video 20 can be screened to filter out the object of interest. The so-called "objects of interest" here refers to preparing to filter out objects that are identical or similar to the object 21 in a certain characteristic.

Specifically, if the color feature of the sampling space 22 is included in the screening condition, step ST09 is to compare the representative color (h', s', l') of the sampling space 22 and the representative color corresponding to the object of interest to determine Whether other objects of interest in the video 20 are similar to the object 21 must be screened out. For example, it can be specified in advance that if the difference between the H components is less than or equal to 10, the difference between the S components is less than or equal to 10, and the difference between the L components is less than or equal to 20, the representative color of the object of interest and the representative color of the sampling space 22 are determined. (h', s', l') are similar and must be filtered out. Conversely, if either of the above conditions is not satisfied between the two representative colors, it is determined that the representative color of the object of interest is not similar to the representative color (h', s', l') of the sampling space 22, and no need to be screened come out.

In addition, although the above is based on the condition that the difference between the H components of the two representative colors is less than or equal to 10, the difference between the S components is less than or equal to 10, and the difference between the L components is less than or equal to 20, the above judgment conditions can also be adjusted according to the situation. So far, the flow of steps ST01 , ST02 , ST03 , ST04 , and ST09 in Embodiment 1 of the present invention has been described in detail.

[Example 2] Next, Example 2 of this invention is demonstrated. The steps corresponding to Embodiment 2 of the present invention are steps ST01, ST02, ST05, ST06, and ST09. Since steps ST01 and ST02 have been described in Embodiment 1, the description is omitted here.

First, in step ST05 , after the sampling space 22 is established, the attribute features are extracted from the sampling space 22 . The detailed operation flow of extracting attribute features from the sampling space 22 is described below.

For example, if the detection result of the object 21 is a person, the attribute characteristics such as gender, age, and clothing of the human can be extracted from the sampling space 22 , for example, the style of clothing, the object to be carried, the object to carry, the object to wear, etc. can be described human characteristics. The above attribute features such as gender, age, and clothing can also be implemented using existing object detection technology.

The detection results of the above attribute features are usually expressed as percentages. Specifically, if the attribute feature of the character A in the sampling space 22 is detected, the detection result, for example, the probability of the gender being "male" is 92%, and the probability of "female" is 8%; The probability of wearing a "dress" is 3%, the probability of "trousers" is 97%; the probability of wearing a hat is "yes" 6%, the probability of "no" is 94%, and so on. Therefore, the attribute features of person A can be extracted from the sampling space 22 as "gender male", "wearing trousers", and "not wearing a hat".

Then, in step ST06, after the attribute features are extracted from the sampling space 22, the attribute features are included in the filtering conditions. After the attributes of the sampling space are included in the filtering conditions, in step ST09, other detected objects in the video 20 can be screened to filter out the objects of interest. For example, if "gender male" and "wearing trousers" are selected as the filtering objects, all concerned objects that match the attribute characteristics of "gender male" and "wearing trousers" can be filtered out.

It should be noted that, although step ST09 in the second embodiment is to include the attribute feature in the filter condition after the attribute feature is extracted from the sampling space 22 in step ST06. However, in step ST09 of the second embodiment, both the color feature (step ST04 ) and the attribute feature (step ST06 ) can be used as filter conditions to filter the objects of interest in the video at the same time. In other words, Embodiment 1 and Embodiment 2 do not have a strict execution sequence. Steps ST03 to ST04 of Embodiment 1 and steps ST05 to ST06 of Embodiment 2 may also be executed separately or simultaneously.

[Example 3] Next, Example 3 of this invention is demonstrated. The steps corresponding to Embodiment 3 of the present invention are steps ST01, ST02, ST07, ST08, and ST09. Since steps ST01 and ST02 have been described in Embodiment 1, the description is omitted here.

First, in step ST07, after the sampling space 22 is established, the re-identification model conversion is performed on the sampling space to obtain the feature vector.

Specifically, in step ST07, the re-identification model conversion uses a matrix operation to convert the 22-dimensional image in the 2-dimensional sampling space into the feature vector in the 256-dimensional space (or other multi-dimensional space). That is to say, the feature vector can be regarded as the direction corresponding to the origin in the 256-dimensional space pointing to a certain coordinate point. Since the re-identification model conversion can be implemented by using an existing algorithm, the detailed description is omitted here.

In addition, in step ST07, the re-identification model conversion is also performed for other objects of interest in the video 20 to obtain the object feature vector of each other object of interest.

Then, in step ST08, after the feature vector is acquired, the feature vector is included in the filter condition. After the feature vector converted from the sampling space 22 is included in the filtering condition, in step ST09, other detected objects in the video 20 can be screened to filter out the object of interest.

Specifically, in step ST09, the method of judging whether the other object of interest is the same as or similar to object 21 is to calculate the distance between the feature vector and the feature vector of the object of interest in the 256-dimensional space (or other multi-dimensional space). As described above, since the feature vector can be regarded as the direction corresponding to the origin in the 256-dimensional space pointing to a certain coordinate point, calculating the distance between two vectors is equivalent to calculating the distance between two coordinate points. Then, the distance is compared with a critical value. If the distance is less than or equal to the threshold value, it is determined that the object of interest is similar to the object 21 and meets the screening conditions and must be screened out. Conversely, if the distance is greater than the critical value, it is determined that the object of interest is not similar to the object 21 and does not meet the screening conditions, and does not need to be screened out.

It should be noted that although step ST09 of the third embodiment is to include the feature vector in the screening condition after the feature vector is obtained in step ST08. However, in step ST09 of the third embodiment, the color feature (step ST04), the attribute feature (step ST06), and the feature vector (step ST08) can be used as filter conditions simultaneously, and the objects of interest in the movie can be screened together. In other words, Embodiment 1 to Embodiment 3 do not have a strict order of execution. Steps ST03 to ST04 of Embodiment 1, steps ST05 to ST06 of Embodiment 2, and steps ST07 to ST08 of Embodiment 3 can also be executed separately. or synchronously.

[Example 4] Next, Example 4 of the present invention will be described. Steps corresponding to Embodiment 4 of the present invention are steps ST09, ST10, ST11, and ST12.

In Embodiment 4 of the present invention, when the object of interest has been screened, the filtered object of interest is then filtered. Therefore, in Embodiment 4, it is assumed that any one of steps ST04, ST06, and ST08 in the first figure has been executed, and according to the color feature (step ST04), the attribute feature (step ST06) and the feature vector (step ST08) ) to filter out the object of interest from the video 20 (ie, step ST09 is executed). In addition, Embodiment 4 also uses any one of the aforementioned steps at the same time to filter out another "second object of interest" that is different from the object of interest from another "second video" that is different from the video 20. The video 20 and the second video, for example, are captured by different cameras. The step of filtering out the second object of interest is marked as step ST10 in FIG. 1 .

However, although the execution sequence of step ST10 is shown after step ST09 in FIG. 1, the execution sequence of step ST10 may also be executed before step ST09, including the possibility of two steps occurring simultaneously.

Then, in step ST11, the time, place, moving speed, etc. of the object of interest and the second object of interest are compared for the object of interest and the second object of interest that have been screened.

Then, in step ST12, the second object of interest is filtered according to the comparison result of the time, place, and moving speed of the object of interest and the second object of interest.

According to Embodiment 4 of the present invention, the time, location, and movement speed of the object of interest and the second object of interest detected by the two cameras can be taken into consideration, so as to eliminate the possibility that the second object of interest is mistakenly identified as the object of interest. For example, under the screening of the methods of Embodiments 1 to 3 of the present invention, the object of interest has been screened out from the video 20, and the second object of interest similar to the object of interest has been screened out from the second video. However, according to the implementation results of Embodiment 4 of the present invention, the time, location, and moving speed of the object of interest and the second object of interest are compared, and it is found that the object of interest and the second object of interest are detected at the same time, and the object of interest is detected at the same time. The two cameras that detect the object of interest and the second object of interest are in a stationary state in the video 20 and the second video respectively, but the two cameras that detect the object of interest and the second object of interest are 100 kilometers away from each other. Considering the relative distance between the two cameras, it is impossible for the same object to appear in both film 20 and film 2 in a short period of time. Therefore, in step ST12, the second object of interest may be additionally filtered to exclude the possibility that the second object of interest is the object of interest.

To sum up, according to Embodiments 1 to 3 of the present invention, using the color features, attribute features, and feature vectors of the sample space 22 of the video 20 to screen other objects of interest in the video 20 can allow users to more efficiently Screen out the objects of interest to be screened in the video.

In addition, after any one of Embodiments 1 to 3 is executed, according to Embodiment 4 of the present invention, filtering can be performed from the already screened objects of interest to further reduce the search scope. Based on the aspect of Embodiment 4, it can be applied to one-time screening of images (or objects) captured by multiple cameras. Therefore, the user can more efficiently select the object of interest to be selected from the video.

In addition, each method step described in Embodiments 1 to 4 of the present invention may be implemented by hardware, software, or a combination of hardware and software.

When Embodiments 1 to 4 of the present invention are implemented in hardware, steps ST01 to ST12 in FIG. 1, steps ST31 to ST33 in FIG. 3, and algorithms in FIGS. 5A and 5B may be performed by dedicated It can be realized by hardware, but it is not limited to this. Dedicated hardware, for example, corresponds to a single circuit, a composite circuit, a programmable processor, a parallel programmable processor, an application specific integrated circuit, a field programmable gate array, or a combination of these.

When the embodiments 1 to 4 of the present invention are implemented by software, the software is used as a program and stored in the memory of the computer. When the processor of the computer reads the program stored in the memory, the computer is caused to execute steps ST01 to ST12 in FIG. 1, steps ST31 to ST33 in FIG. 3, and algorithms in FIGS. 5A and 5B. For example, a computer is equivalent to a central processing unit, a central processing unit, a processing unit, an arithmetic unit, a microprocessor, a microcomputer, a processor, or a digital signal processor.

Embodiments 1 to 4 of the present invention have been described in detail above. It should be noted that the above-mentioned Embodiments 1 to 4 are only illustrative to illustrate the principles and effects of the present invention, and are not intended to limit the scope of the present invention. Those skilled in the art can modify and change the embodiments without departing from the technical principles and spirit of the present invention. Therefore, the protection scope of the right of the present invention should be based on the description of the following patent application scope.

20: Videos 21: Objects 22: Sampling space 40~42: Feature points A~J: Feature points ST01~ST12: Steps ST31~ST33: Steps

FIG. 1 is a flow chart illustrating a method for screening objects according to Embodiments 1 to 4 of the present invention. FIG. 2 is a schematic diagram illustrating an example of creating a sampling space 22 according to the detection result of the area in step ST02 in FIG. 1 . FIG. 3 is a flowchart illustrating the detailed operation flow of step ST03 in FIG. 1 . FIG. 4A is a schematic diagram illustrating an example of 10 feature points A to J extracted from the object 21 in the sampling space 22 . FIG. 4B is a schematic diagram illustrating an example of generating a polygon with four feature points. 5A and 5B are schematic diagrams illustrating an example of generating a polygon based on feature points in step ST32 in FIG. 3 .

ST01~ST12: Steps

Claims (10)

A method of filtering objects, including: Detecting an area where an object is present from a video; Create a sampling space according to the detection result of the area; extracting a color feature from the sampling space; include the color feature in a filter; and According to the filter condition, filter an object of interest in the video. Such as the method of filtering objects of claim 1, Wherein, the step of extracting the color feature from the sampling space further includes: extracting at least one feature point from the sampling space; generating a polygon according to the at least one feature point; and The color feature is extracted from the area formed by the polygon. As in the method of filtering objects of claim 2, Wherein, the object of interest is a person, and the feature point includes any one of human limbs, joints, and the like. For example, the method for filtering objects in request item 1, further includes: extracting an attribute feature from the sampling space; include the attribute characteristic in the filter; and According to the filter condition, filter the object of interest in the video. As in the method of filtering objects of claim 4, Wherein, the object of interest is a character, and the attribute feature includes any one of human gender, age, and clothing. For example, the method for filtering objects in request item 1, further includes: performing a double identification model transformation on the sampling space corresponding to the object to obtain a feature vector; Performing the re-identification model transformation in a sample space of the object of interest corresponding to the object of interest to obtain a feature vector of the object of interest; calculating a distance between the feature vector and the feature vector of the object of interest in a multi-dimensional space; and When the feature vector is included in the filtering condition, it is determined whether the object of interest meets the filtering condition by comparing the distance with a threshold. As in the method of filtering objects of claim 6, Wherein, the re-identification model is converted into a matrix operation. If the method for filtering objects in any one of the requirements 1 to 7, it also includes: According to the filter condition, filter out a second object of interest in a second video; and Compare any one of a plurality of times, a plurality of locations, and a plurality of speeds at which the object of interest and the second object of interest are detected to filter the second object of interest. A computer program product, the program is loaded through a computer, so that the processor of the computer executes the method of screening objects according to any one of claims 1 to 7. For the computer program product of claim 9, Among them, the processor more executes: According to the filter condition, filter out the second object of interest in the second video; and Compare any one of a plurality of times, a plurality of locations, and a plurality of speeds at which the object of interest and the second object of interest are detected to filter the second object of interest.

Images