TWI517055B - Image foreground object screening method - Google Patents

Description

Image foreground object screening method

The invention relates to an image processing method, in particular to an object feature generation method, an object feature comparison method and an object screening method.

When the policeman handles the case, it is often necessary to read the video data of the surveillance camera and compare them one by one to find suspicious people.

Take a bank robbery as an example. When the robbery bank is robbed and escapes, the surveillance camera in the bank or nearby roadway may record the image of the robbery, and the investigator can rob the image recorded by the surveillance camera. The image data of a period of time before and after are compared one by one to find suspicious clues, but in the long-term viewing, the concentration of the investigators is easily reduced due to fatigue, which in turn affects the judgment of the image data and causes the mistakes of the comparison. In addition, if the robbers change their appearance in order to escape, it is easier to affect the comparison of the investigators.

In addition, the above-mentioned reporters have a shortcoming in comparing the methods of image data. That is, after reading a piece of image data, if the investigator has a new clue to conduct the investigation, it is necessary to look again. Image data, which causes considerable waste of time, therefore, an improved way of identifying existing image data must be provided.

Accordingly, it is an object of the present invention to provide a method of generating object features that aid in aligning images.

Accordingly, it is another object of the present invention to provide an image processing method that facilitates comparison of images.

Accordingly, it is still another object of the present invention to provide an object screening method that facilitates comparison of images.

Thus, the object feature generating method of the present invention comprises:

(A) Display a photo or video material.

(B) receiving a plurality of coordinate values representing a plurality of locations input by the user.

(C) generating a set of coordinates including the coordinate values.

(D) According to the coordinate set, the range of the photo or image data surrounded by the positions represented by the coordinates is taken to form a target object data.

(E) processing the target object data to obtain a second eigenvalue in the form of an array.

Preferably, the coordinate set sequentially stores the coordinate values representing the positions of the respective clicks according to the order in which the user clicks on the photo or the image data, and the range is to sequentially connect the respective positions according to the sequence. And around the scope of formation.

Preferably, wherein the step (E) comprises at least one of the following substeps Step (E1): For each target object data, use the moving window scan to obtain RGB color information values for each window and record them in the RGB chromaticity histogram respectively. After the target object data is scanned, the RGB is obtained by three 1×. 256 color one-dimensional array; and (E2) for each target object data using local binary pattern (LBP) processing, find the LBP value of each 3X3 block and record it in an LBP histogram, the overall iteration An iterative result is a 1×256 texture one-dimensional array; the second characteristic value is a set of the one-dimensional array of the colors and/or the one-dimensional array of the texture.

Therefore, the object feature comparison method of the present invention comprises the steps of: calculating foreground data of a plurality of image data for a video material, and processing, for each of the foreground data, a first feature value in the form of an array.

The object feature generation method generates a target object data and processes the target object data to obtain a second feature value in the form of an array.

The second feature value is compared with each of the first feature values of each of the foreground materials, and a correlation score is calculated respectively.

Sort according to each of the relevance scores and output the sort result.

Thus, the method for screening an object of the present invention comprises the following steps:

(a) Calculate the foreground data of a plurality of image data for a video material.

(b) analyzing the objects in the foreground data in the imagery, A plurality of objects having the same moving object feature are assigned to the same object.

(c) receiving a selected condition defined by more than three locations entered by the user and determining whether each of the objects meets the selected condition during the movement to exclude or retain each of the objects.

Preferably, the connection of the positions of the input surrounds an area with the vertices of the positions, and the selected condition is: if the object of the foreground data in the image data falls into the area during the movement, The object is retained, otherwise the object is excluded.

Preferably, the connection of the positions of the input surrounds an area with the vertices of the positions, and the selected condition is: if the object of the foreground data in the image data falls into the area during the movement, The object is then excluded, otherwise the object is retained.

Preferably, the positions of the inputs are a first position, a second position, and a third position, respectively, wherein the first position and the second position define a line segment, and the third position defines a vertical line segment and faces the first a vector on one side of the three positions, the selected condition is: if the object of the foreground data in the image data passes through the line segment during the movement and passes in the direction of the vector when passing or the angle between the forward direction and the vector is less than A preset value is retained for the object, otherwise the object is excluded.

Preferably, the step (a) further obtains a first feature value in the form of an array for each of the foreground data processing, the method further comprising the step (d) performed after the step (a), and the step (c) And (d) steps (e) and (f) performed after the execution is completed:

(d) The object feature generation method generates a target object data and processes the target object data to obtain a second eigenvalue in the form of an array.

(e) aligning the second eigenvalue with the first eigenvalue of the portion of the object selected by the step (c) corresponding to the foreground data, and calculating a correlation score.

(f) Sorting according to each of the relevance scores and outputting the sort result.

The effect of the invention is that the second object value is generated according to the target object data formed by the user's selection, so that the subsequent comparison work can be more accurate.

S11~S16‧‧‧Steps

S171~S176‧‧‧Steps

S181~S186‧‧‧Steps

S19~S21‧‧‧Steps

S200~S206‧‧‧Steps

S211~S214‧‧‧Steps

S221~S226‧‧‧Steps

S231~S236‧‧‧Steps

S24~S25‧‧‧Steps

1‧‧‧Electronic device

11‧‧‧ memory unit

12‧‧‧Display unit

13‧‧‧Control unit

The other features and advantages of the present invention will be apparent from the following detailed description of the preferred embodiments of the invention, wherein: FIG. 1 is a flow diagram illustrating a preferred embodiment of the image processing method of the present invention; Is a block diagram illustrating the architecture of an electronic device embodying the present invention; FIG. 3 is a schematic diagram of a Gaussian mixture model; FIG. 4 is a flow diagram illustrating the preferred embodiment; FIG. 5 is a schematic diagram of an operation screen illustrating a "mask" Figure 6 is a schematic diagram of an operation screen illustrating "retention screening"; Figure 7 is a schematic diagram of an operation screen illustrating "directional filtering"; Figure 8 is a flow diagram illustrating a comparison of the feature comparison methods of the present invention. Good example; FIG. 9 is a schematic flow chart showing step S21; and FIG. 10 is a schematic diagram of an operation screen for selecting an object.

Referring to FIG. 1 and FIG. 2, a preferred embodiment of the image processing method of the present invention is implemented by an electronic device 1. The electronic device 1 includes a memory unit 11, a display unit 12, and a memory unit 11 and a display unit 12. Control unit 13. The memory unit 11 stores an application related to the image processing method and the object feature comparison method of the present invention, and at least one video material. The control unit 13 reads the application and performs the steps of the following image processing method.

Step S11 - reading the video material. The video material is, for example, a movie recorded by a surveillance camera at a certain time.

Step S12 - extracting a plurality of image data from the recorded data, for example, taking 3 images per second, assuming that a total of 5400 image data are extracted from the 30-minute video data.

Step S13 - Using the image data to establish a Gaussian Mixture Model as illustrated in FIG. The purpose of establishing a Gaussian mixture model is to establish the basic background of this video material. When the Gaussian mixture model is represented by the graph shown in FIG. 3, the horizontal axis is time and the vertical axis is the probability intensity of the pixel; therefore, the Gaussian mixture model includes a plurality of probability strengths respectively representing a certain pixel set changing with time. The Gaussian distribution can be expressed by the equations of Equations 1 and 2.

(Formula 2)

Where, at time t, P(x _t ) represents the probability intensity of the pixel with pixel value X, W _i,t represents the weight of the ith Gaussian distribution, η is the Gaussian probability density function, μ _i,t and Σ _{i, t} are the average and covariance of the i-th Gaussian distribution _, respectively.

After the Gaussian mixture model is established, the video data can continue to be transmitted and the image data is continuously increased, and the time axis is elongated and the parameters are continuously maintained and updated.

Step S14 - obtaining a background image data from the Gaussian mixture model.

In the Gaussian mixture model, simply speaking, a long-term and high-intensity set of pixels indicates that most of these image data appear, usually representing the background; objects appearing only in a few pieces of image data, mixed in Gaussian In the model, the probability is usually low and the distribution is short. Therefore, several Gaussian distributions that best represent the background can be found from the Gaussian mixture model, and the set of pixels corresponding to the found Gaussian distribution can establish background image data.

In the embodiment, the method for finding the Gaussian distribution of the background is to separately calculate the weight of the Gaussian distribution divided by the standard deviation, and arrange the Gaussian distributions of the previous names according to the ratio size to represent the background.

Step S15 - using pixel-wise classification, the R/G/B chromaticity of each pixel of each image data is respectively compared with the R/G/B chromaticity of the corresponding pixel of the background image data. A difference is obtained; because the difference is large, the pixel is represented in the background image data, and the chromaticity is much different in a certain image data, that is, the object is moved in. Therefore, in this step, pixels having a difference greater than a predetermined function are separated to form a preliminary foreground data of a pair of image data.

Step S16 - The result obtained by S15 is subjected to shadow filtering by region-based classification, and the portion of the brightness change and the shadow portion is further filtered out, which is the real foreground data.

In this step, the gain value is calculated for the pixel in the preliminary foreground data obtained in step S15, and the gain value is equal to the difference between the grayscale value of one pixel of each image data and the grayscale value of the corresponding pixel of the background image data, and The background image data corresponds to the ratio of the grayscale values of the pixels; and the pixels whose gain values are close to and adjacent to each other are merged into one region. When the average gain value of the region is less than a preset threshold, the optical intensity is less changed than the background. And if it belongs to the shadow, it is further deleted to get the foreground data.

The next steps are divided into color analysis (S171~S176) and texture analysis (S181~S186) to analyze the foreground data and obtain the feature values. In this embodiment, it is assumed that the foreground data obtained in the previous step is a person, so the foreground data is divided into n parts, n=3, that is, the head, the body and the foot, and the weighting calculation is performed according to the preset weight (S176). S186), or performing weighting calculation (S25) when calculating the relevance score in the identification process, helps to improve the recognition. However, the present invention is not limited to n=3, and if the whole body is directly compared, n=1.

In terms of color analysis, set i=1~n, j=1~n part The number of pixels, i = 1 at the beginning, j = 1.

Step S171- Scanning the jth pixel of the i-th portion (for example, the head) of the foreground data by using a sliding window to obtain the RGB color information value of the pixel, which is the R chrominance value and the G chromaticity in this embodiment. Value, and B chromaticity value. In this embodiment, the chromaticity value is a value from 0 to 255.

Step S172— Recording the R chrominance value, the G chrominance value, and the B chrominance value in the R chromaticity histogram, the G chromaticity histogram, and the B chromaticity histogram, respectively, and the horizontal axis of the histogram is 0 to 255. The vertical axis is the number of pixels.

Step S173 - Check whether j is equal to the number of pixels of the i-th portion of the foreground data, that is, whether the i-th portion has been scanned. If yes, proceed to step S174, and if no, let j=j+1 scan the next pixel.

Suppose that the i-th part has three pixels. In terms of R chromaticity, the R chrominance values of the three pixels are, for example, 2, 250, and 250 respectively. On the R chromaticity histogram, the corresponding value at the horizontal axis position 2 is For 1, the corresponding value of the horizontal axis position 250 is 2, and the others are all 0.

Step S174 - reading the values in the R chroma histogram, the G chroma histogram, and the B chroma histogram corresponding to the i-th portion, and converting them into an R-dimensional array, a G-dimensional array, and a B-dimensional array. . In the previous example, we get a 1×256 array. Only the 3rd and 251th elements have values of 1 and 2, and the others are 0.

Step S175 - Check if i is equal to n, that is, check whether all parts of the foreground data have been scanned. If yes, go to step S176. If no, let i=i+1 and process the next part.

Step S176 - multiplying the first to nth portions by a predetermined weight and adding them to obtain an R color one-dimensional array, a G color one-dimensional array, and a B color one-dimensional array. In the case of n=3, when the body features are important, the preset weights can be 0.2, 0.6, and 0.2.

It should be noted that the RGB color one-dimensional array of each part can also be directly stored without weighting calculation, and is calculated when the identification process calculates the score.

The flow of the method for performing texture analysis on foreground data by using a local binary pattern (LBP) in the present embodiment will be described below. In terms of texture analysis, the same number of pixels of i=1~n, k=1~nth part is set, and i=1 and k=1 at the beginning.

Step S181 - The LBP value is used to process the LBP value of each 3X3 block of the i-th part. In detail, for the first 3X3 block of the i-th part of the foreground data (for example, the pixel in the upper left corner is the center of the nine-square grid, and the portion of the pixel is not the same as the hypothetical pixel with the same gray-scale value), the LBP value is calculated substantially. It is the grayscale value of each grid minus the grayscale value of the central grid and then binarized to become 0 or 1, and then multiply each weight by the weight (2 to the power of 0 to the power of 7) to obtain the LBP value. In this embodiment, the LBP value is a value from 0 to 255.

Step S182 - Record the LBP value in an LBP histogram having a horizontal axis of 0 to 255 and a vertical axis of the number of pixels.

Step S183 - It is checked whether k is equal to the number of pixels of the i-th portion of the foreground data, that is, whether the i-th portion has been scanned. If If yes, go to step S184. If no, let k=k+1, so that the nine squares move one space.

Step S184 - Read out the values in the LBP histogram to obtain a 1 x 256 texture one-dimensional array.

Step S185 - Detect whether i is equal to n, that is, check whether all parts of the foreground data have been scanned. If yes, go to step S186, if no, let i=i+1 and process the next part.

Step S186 - multiplying the first to nth portions by a predetermined weight and adding them to obtain an overall texture one-dimensional array. Similarly, the texture one-dimensional array of each part can also be directly stored without weighting calculation, and is calculated when the identification process calculates the score.

Step S19 - After performing color analysis and texture analysis, storing each one-dimensional array corresponding to a certain image data in the memory unit 11 to obtain a first feature value, where the first feature value is a set of the one-dimensional arrays.

It should be noted that the present invention is not limited to color analysis and texture analysis parallel collocation processing, and may also use only the result of color analysis as the eigenvalue of each image data, or only use the result of texture analysis as the eigenvalue; When color and texture are processed together, there is no limit to the order in which they are processed. Referring now to Figure 4, the subsequent steps are continued.

Step S20- Perform an analysis of the object tracking, and the objects in the foreground data in the image data have the specific moving object features attributed to the same object. For example, if you capture 5 images of the car on the road, you can analyze the 5 images after the analysis of this step. The foreground data of the car part should be identified as the same object. The algorithm for calculating the characteristics of the moving object is, for example, whether the object is the same object by the center of gravity of the moving part in the foreground data, or the Kalman filter tracking method is used to calculate and track the moving object feature.

The purpose of this step is to provide a subsequent object screening step (step S200). For example, if the moving object is to be filtered through an area on the screen, if the car is located in the area in the two image data, Then all the five image data containing the car can be found out because they are analyzed to belong to the same moving object.

Step S200 - Perform object screening. In the present embodiment, this step is for adding further restriction conditions to narrow the final screening result, and step S20 and the present step may be omitted, and step S25 described later may be directly performed with the first feature value of step S19.

This step includes the following sub-steps S201-S206. In this embodiment, the sub-steps can be divided into three types; "mask screening", "retention filtering", and "directional filtering", one of which can be selected for each type. Or select several of them to perform, as explained below.

"Mask screening" refers to receiving an area surrounded by a line of three or more vertices selected by a user (see FIG. 5) (step S201), and determining that the object of the foreground data in the image data has If the area falls into the area during the movement (step S202), the object is excluded and is not displayed in the result, otherwise the object is retained.

The "retention filter" is also an area that is pre-received by a user-selected line with more than three vertices (see Figure 6). Step S203), however, the difference is that if the object of the foreground material in the image data has fallen into the area during the movement, the object is retained, otherwise the object is excluded.

"Directive screening" refers to pre-receiving a line segment defined by the user and a direction which is a direction perpendicular to the vector of the line segment. For example, the user clicks on the first position P1 and the first in the screen. Two positions P2, specify two end points of the line segment to define a line segment in the upper left-lower right direction, and then click on any third position P3 on the upper right side of the line segment, thus equal to the selection from the lower left to the upper right The direction (as shown in FIG. 7), and then, if the object of the foreground data in the image data passes through the line segment and advances in the selected direction when the passage is passed or the angle between the forward direction and the vector is less than a preset value, then Keep the object, otherwise exclude the object.

It should be noted that each screening may be performed by one, two, or three. For example, if the intersection of three screening results is taken, the object must not fall within the selected area of the "mask screening", Falls into the selected area of the Retention Filter and travels through the selected line segment in the Directional Filter and in the selected direction. In addition, the screening method is not limited to the above, and as long as the position or movement behavior of the object is restricted, it can be used as a screening condition.

After the object is screened, the first characteristic value of the portion corresponding to the foreground material in the foreground data is subjected to the comparison of step S25. For example, in the foreground data of the object representing the car, there may be objects representing the pedestrian at the same time. If the object representing the car passes the screening, the partial pixels representing the object of the car in the foreground data are The first characteristic value should be subjected to the comparison of step S25.

In the case where this step is omitted, the subsequent step is performed on the first feature value of each foreground material.

When all the collected video data, the image processing method shown in FIG. 1 is pre-processed, and each image data is stored into corresponding first feature values in several one-dimensional arrays, and the video data needs to be read in the future. When identifying, the efficiency can be greatly improved, which is quite helpful. At the beginning of the video recording, the image processing method shown in FIG. 4 is performed on all the video materials, and the object tracking is performed, and the required object screening is selected to further limit the number of results.

Referring to FIGS. 2 and 8, the control unit 13 of the electronic device 1 reads the application to perform the following steps of the object feature comparison method.

Step S21 - receiving a target object data. The target object information is, for example, a photo of the suspect obtained by the investigator, and is selected from the photo frame or selected from a certain image data in the already locked video material. Wherein, the foregoing frame selection is not limited to being selected in a checkered or irregular shape. Referring to FIG. 9, in this embodiment, the step further includes the following sub-steps S211 to S214:

Step S211 - Display the photo or image data.

Step S212 - receiving a plurality of coordinate values representing a plurality of locations clicked by the user on the photo or video material. For example, if the user wants to select a pedestrian in a picture (as shown in FIG. 10), click on a plurality of points of the pedestrian contour to select the pedestrian object as the target object data to be compared next.

Step S213 - generating a set of coordinates including the coordinate values. It should be noted that the coordinate set stores the order in which the coordinate values are clicked in step S212.

Step S214 - extracting a range of the photo or image data surrounded by the coordinates represented by the coordinates according to the coordinate set to form the target object data.

In addition, if the first feature value of the image data is obtained by the image processing method of the present invention, the first feature value corresponding to the frame selection portion may be defined as the second feature value, and step S25 and subsequent steps are directly performed. .

The steps to be performed next are the same as the color analysis steps S171 to S176 and the texture analysis steps S181 to S186 of the foreground data in the image processing method of FIG. 1, except that the analysis target is the target object data, and therefore only the following description will be briefly described.

In terms of color analysis, set the number of pixels of i=1~n, j=1~nth part, i=1, j=1 at the beginning.

Step S221 - Scanning the jth pixel of the i-th portion of the target object data to obtain the R chrominance value, the G chrominance value, and the B chromaticity value of the pixel.

Step S222 - Recording the R chrominance value, the G chrominance value, and the B chrominance value in the R chromaticity histogram, the G chromaticity histogram, and the B chromaticity histogram, respectively.

Step S223 - It is checked whether j is equal to the number of pixels of the i-th part of the target object data. If yes, proceed to step S224, if no, then order j=j+1, scan the next pixel.

Step S224 - reading the values in the R chromaticity histogram, the G chromaticity histogram, and the B chromaticity histogram corresponding to the i-th portion, and converting them into an R-dimensional array, a G-dimensional array, and a B-dimensional array. .

Step S225 - Check if i is equal to n. If yes, go to step S226, if no, let i=i+1 and process the next part.

Step S226 - Multiplying the first to nth portions by a predetermined weight and adding them to obtain an R color one-dimensional array, a G color one-dimensional array, and a B color one-dimensional array.

It should be noted that the one-dimensional array of RGB colors of each part can also be directly stored without weighting calculation, and is calculated when the correlation score is calculated in step S25.

On the other hand, the LBP is used to perform texture analysis on the target object data.

Step S231 - processing the LBP value of each 3X3 block of the i-th part of the target object data by using the LBP technique.

Step S232 - Record the LBP value in an LBP histogram.

Step S233 - It is checked whether k is equal to the number of pixels of the i-th part of the target object data. If yes, proceed to step S234. If no, let k=k+1 move the nine squares to one grid.

Step S234 - Read out the values in the LBP histogram to obtain a 1 x 256 texture one-dimensional array.

Step S235 - Detect whether i is equal to n, that is, check whether all parts of the target object data have been scanned. If yes, go to step S236. If no, let i=i+1 and process the next part.

Step S236 - Multiplying the first to nth portions by a predetermined weight and adding them to obtain an overall texture one-dimensional array.

After performing color analysis and texture analysis, step S24 is performed to store each one-dimensional array of target object data in the memory unit 11 to obtain a second feature value, which is a set of the one-dimensional arrays.

Step S25 - Comparing the second feature value with a first feature value, and calculating a relevance score, wherein the first feature value is obtained by step S19 or step S200. Since the first eigenvalue and the second eigenvalue are both a set of one-dimensional arrays, each of the one-dimensional arrays is equivalent to a feature vector, so the present embodiment calculates a distance between the two feature vectors using a cosine distance formula.

In detail, in the foregoing examples, the color one-dimensional array and the texture one-dimensional array of each part have been respectively weighted and combined into an overall color one-dimensional array and a texture one-dimensional array. Therefore, this step is to compare the color one-dimensional array of the target object data with the one-dimensional array of the foreground data, calculate a color correlation score, and make the texture one-dimensional array of the target object data and the texture of the foreground data one-dimensional. The array is compared, a texture correlation score is calculated, and then the two correlation scores are respectively multiplied by a predetermined weight, and then the correlation scores of the target object data and the foreground data are obtained.

On the other hand, the present invention can also not merge the one-dimensional array of each part in the image processing stage, that is, without steps S176, S186, S226 and S236, then in this step, each part of the target object data must be made. The one-dimensional array is compared with the one-dimensional array corresponding to the foreground data, and the correlation score is calculated, and then the correlation scores of the respective parts are multiplied by a predetermined weight and then added to obtain the correlation between the target object data and the foreground data. Degree score. After the color one-dimensional array and the texture one-dimensional array calculate the correlation scores in the same manner, the two correlation scores are respectively multiplied by a predetermined weight, and then the correlation scores of the target object data and the foreground data are obtained. .

Step S26 - sorting according to the relevance score and outputting the sort result through the display unit 12. For example, the display unit 12 displays the foreground data among the five image data having the highest relevance score.

In summary, the video processing method of the present invention converts video data into a plurality of image data and stores them as feature values in an array form. Therefore, correlation processing can be conveniently performed in the object feature comparison method, and high correlation is efficiently found. The object, the fast and well-identified automatic identification effect, can provide the investigator with considerable help in image comparison.

In addition, the target object data formed according to the user's selection further generates the second characteristic value for the target object data, and reduces the number of comparisons according to various screening methods, and then performs correlation calculation to make the final ratio The result can be more precise, so it can achieve the purpose.

The above is only the preferred embodiment of the present invention, and the scope of the present invention is not limited thereto, that is, the simple equivalent changes and modifications made by the patent application scope and patent specification content of the present invention, All remain within the scope of the invention patent.

S211~S214‧‧‧Steps

Claims (4)

An object screening method comprises the steps of: (a) calculating foreground data of a plurality of image data for a video material; and (b) analyzing objects in the foreground data of the image data, having the same moving object feature. The plurality of objects are assigned to the same item; and (c) receiving a selected condition defined by more than three locations input by the user, and determining whether each of the objects meets the selected condition during the movement to exclude or retain each of the items object. The object screening method of claim 1, wherein the connection of the positions of the input surrounds an area with the vertex as the vertex, and the selected condition is: if the object of the foreground data in the image data has a moving process When it falls into the area, the object is retained, otherwise the object is excluded. The object screening method of claim 1, wherein the connection of the positions of the input surrounds an area with the vertex as the vertex, and the selected condition is: if the object of the foreground data in the image data has a moving process When falling into the area, the item is excluded, otherwise the item is retained. The object screening method of claim 1, wherein the positions of the inputs are a first position, a second position, and a third position, respectively, wherein the first position and the second position define a line segment, and the third position defines a vertical position. The line segment and the vector facing one side of the third position, the selection condition is: if the object of the foreground data in the image data passes through the line segment during the movement and passes in the direction of the vector or the direction of advancement when passing With the vector If the angle is less than a preset value, the object is retained, otherwise the object is excluded.