JP2011154632A - Image processing apparatus, method and program - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide an image processing apparatus, method and program, capable of achieving highly precise mobile object detection, and for achieving the reduction of the computational complexity and the simplification of an algorithm in post-processing such as edge detection, labeling, tracking and Hough transformation. <P>SOLUTION: The image processing apparatus includes: a matrix generator 21 for, when m×m(m: integer of 2 or more) pixels in one frame in an image are defined as one block, generating a matrix configured of luminance data about each of a plurality of corresponding blocks in temporally sequential frames for the sequential frames; a generator 22 for generating a covariance matrix by using the corresponding matrix; an calculator 23 for calculating a characteristic value and a characteristic vector from the covariance matrix, and for calculating an Euclidean distance between the corresponding blocks of the sequential frames based on the characteristic value and the characteristic vector; and a processor 24 for performing image processing by using the Euclidean distance and a threshold. <P>COPYRIGHT: (C)2011,JPO&INPIT

Description

  The present invention relates to an image processing apparatus, an image processing method, and an image processing program suitable for detecting various moving objects such as a person, a human hand, or a vehicle.

  In the conventional image processing, background difference and inter-frame difference are performed in units of one pixel (pixel) for moving object detection, and there is a problem that it is difficult to set a threshold value for performing detection with high accuracy.

  Also, in moving object detection, post-processing such as edge detection, labeling, tracking, and Hough transform is often performed (see Patent Document 1), but when a difference is obtained in units of one pixel as described above, There has been a problem that the amount of calculation of the post-processing described above becomes enormous and the algorithm becomes complicated.

JP 2006-98119 A

  The present invention has been made as a solution to the problems in the conventional image processing as described above, and its purpose is to detect a moving object with high accuracy, and to detect edges, labeling, tracking, Hough transform, etc. To provide an image processing apparatus, an image processing method, and an image processing program capable of reducing the amount of calculation in post-processing and simplifying an algorithm.

  The image processing apparatus according to the present invention uses m × m (m: integer greater than or equal to 2) pixels in one frame in an image as one block, and uses a plurality of corresponding blocks in temporally preceding and following frames according to the luminance data. Matrix creating means for creating a matrix for the previous and subsequent frames, a generating means for generating a covariance matrix using the corresponding matrix, eigenvalues and eigenvectors are obtained from the covariance matrix, and the previous and next frames are calculated based on the eigenvalues and eigenvectors. Computation means for obtaining a Euclidean distance between corresponding blocks, and processing means for performing image processing using the Euclidean distance and a threshold value are provided.

  The image processing apparatus according to the present invention is characterized in that all blocks in one frame or thinned blocks are used for creating a matrix.

  The image processing apparatus according to the present invention is characterized in that the processing means performs at least one of edge detection, tracking, labeling, and Hough transform of a moving object.

  The image processing method according to the present invention uses m × m (m: an integer of 2 or more) pixels in one frame in an image as one block, and uses a plurality of corresponding blocks in temporally preceding and following frames according to respective luminance data. A matrix creation step for creating a matrix for the previous and next frames, a generation step for generating a covariance matrix using the corresponding matrix, an eigenvalue and an eigenvector are obtained from the covariance matrix, and the previous and next frames are calculated based on the eigenvalue and eigenvector. A calculation step for obtaining a Euclidean distance between corresponding blocks and a processing step for performing image processing using the Euclidean distance and a threshold value are executed.

  The image processing method according to the present invention is characterized in that all blocks or thinned out blocks in one frame are used for creating a matrix.

  In the image processing method according to the present invention, in the processing step, at least one of edge detection, tracking, labeling, and Hough transform of a moving object is performed.

  An image processing program according to the present invention uses a computer that performs image processing as m × m (m: integer of 2 or more) pixels in one frame of an image as one block, and a plurality of corresponding frames in temporally preceding and following frames. Matrix generating means for generating a matrix based on the respective luminance data for the previous and subsequent frames, a generating means for generating a covariance matrix using the corresponding matrix, eigenvalues and eigenvectors are obtained from the covariance matrix, and eigenvalues and eigenvectors are obtained. And calculating means for calculating the Euclidean distance between corresponding blocks of the preceding and following frames, and processing means for performing image processing using the Euclidean distance and a threshold value.

  The image processing program according to the present invention is characterized in that the computer functions so that all blocks in one frame or thinned blocks are used for creating a matrix.

  The image processing program according to the present invention is characterized in that a computer functions as a processing means to perform at least one of edge detection, tracking, labeling, and Hough transform of a moving object.

  According to the present invention, a matrix is created with m × m (m: integer greater than or equal to 2) pixels in one frame in an image as one block, the Euclidean distance between corresponding blocks in the preceding and succeeding frames is obtained, and Euclidean Since the image processing is performed using the distance and the threshold, it is possible to detect the moving object with high accuracy by appropriately setting the threshold, and to reduce the amount of calculation in post-processing such as edge detection, tracking, labeling, and Hough transform. It is possible to simplify the algorithm.

The block diagram which shows the Example of the image processing apparatus which concerns on this invention. The figure for demonstrating the block division by the Example of the image processing apparatus which concerns on this invention. The figure for demonstrating the matrix preparation by the Example of the image processing apparatus which concerns on this invention. The figure for demonstrating the operation | movement from the matrix preparation by the Example of the image processing apparatus which concerns on this invention to covariance matrix production | generation. The figure for demonstrating operation | movement until it calculates | requires Euclidean distance from the covariance matrix by the Example of the image processing apparatus which concerns on this invention. The figure which shows the state which arranged the Euclidean distance calculated | required by the Example of the image processing apparatus which concerns on invention in the matrix form. The figure for demonstrating the operation | movement of the edge detection by the Example of the image processing apparatus which concerns on invention. The figure for demonstrating the operation | movement of labeling by the Example of the image processing apparatus which concerns on invention. 5 is a flowchart for explaining an image processing operation according to the embodiment of the image processing apparatus according to the invention.

  Embodiments of an image processing apparatus, an image processing method, and an image processing program according to the present invention will be described below with reference to the accompanying drawings. In each figure, the same components are denoted by the same reference numerals, and redundant description is omitted. FIG. 1 shows a configuration diagram of an image processing apparatus according to the embodiment. The image processing apparatus includes a CPU 10 and a memory (main memory and external storage memory) 11, and is configured by a computer system in which a display unit 13 for displaying information is connected to the CPU 10.

  An image processing program is stored in the memory 11, and the CPU 10, which is a computer, functions as the matrix creation means 21, the generation means 22, the calculation means 23, and the processing means 24 by the image processing program.

  The matrix creation means 21 sets m × m (m: integer greater than or equal to 2) pixels in one frame in an image signal of a moving image as one block, and sets luminance data for a plurality of corresponding blocks in temporally preceding and following frames. This creates a matrix for the previous and next frames. The image signal may be sent from a camera or stored in a storage medium.

  For example, when one frame is composed of 320 × 240 pixels, if 5 × 5 pixels are taken as one block as shown in FIG. 2, it is divided into 64 × 48 blocks. Hereinafter, the pixel configuration of one block is referred to as division granularity. In the above description, the division granularity is 5 × 5 pixels, but the division granularity may be 10 × 10 pixels or the division granularity may be 2 × 2 pixels.

  All blocks or thinned blocks in one frame are used to create a matrix. When the division granularity is thinned out by 5 × 5 pixels, for example, as shown in FIG. 3, in the first frame Frame1 and the second frame Frame2, sampling is performed every other column from the leftmost column and every other row from the uppermost row. Block (32 × 24 in total). In the second frame Frame2 and the third frame Frame3, blocks sampled every other column from the next column of the leftmost column and every other row from the uppermost row (32 × 24 in total) are used.

  In the third frame Frame3 and the fourth frame Frame4, blocks sampled every other column from the next row of the leftmost column and every other row from the next row of the uppermost row (32 × 24 in total) are used. In the fourth frame Frame4 and the fifth frame Frame5, blocks (total 32 × 24) blocks sampled every other column from the leftmost column and every other row from the next row of the uppermost row are used. Thereafter, sampling is performed in the same manner. By sampling in the interlaced manner as described above, uniform sampling can be performed.

  When a matrix is created and processed using all blocks, processing with higher accuracy than the above can be performed, and accuracy can be improved by reducing the division granularity. When creating a matrix with a division granularity of 5 × 5 pixels, for example, one matrix is created for every four blocks.

  As shown in FIG. 4, the luminance information of the four blocks of pixels indicated by 1 to 4 in the first frame Frame1 is a 25 × 4 matrix. That is, the luminance information of 5 × 5 = 25 pixels in the block indicated by 1 is arranged in the first column, and the luminance information of 5 × 5 = 25 pixels in the block indicated by 2 is arranged in the second column. The luminance information of 5 × 5 = 25 pixels in the indicated block is arranged in the third column, and the luminance information of 5 × 5 = 25 pixels in the block indicated by 4 is arranged in the fourth column to form a total of four rows. The matrix is the matrix 1 in FIG.

  Similarly, the luminance information of the pixels of the four blocks indicated by 5 to 8 in the second frame Frame2 is a 25 × 4 matrix. That is, luminance information of 5 × 5 = 25 pixels in the block indicated by 5 is arranged in the first column, luminance information of 5 × 5 = 25 pixels in the block indicated by 6 is arranged in the second column, and The luminance information of 5 × 5 = 25 pixels in the indicated block is arranged in the third column, and the luminance information of 5 × 5 = 25 pixels in the block indicated by 8 is arranged in the fourth column to form a total of four rows. The matrix is the matrix 2 in FIG. In the same manner, matrices are created for all blocks.

The generation unit 22 generates a covariance matrix using a corresponding matrix in the matrix created by the matrix creation unit 21. As shown in S11 in FIG. 4, a matrix 3 of 25 rows × 8 columns in which the matrix 1 and the matrix 2 are arranged is generated, and a transposed matrix 3 ^T is generated (S12). The transposed matrix ^{3 T} is 8 rows × 25 columns. Further, generating unit 22 performs multiplication of the transposed matrix 3 ^T and the matrix 3, as shown to step S13 in FIG. 4, to generate the covariance matrix shown in S14. Similarly, the generation unit 22 generates a covariance matrix from a matrix created by corresponding blocks of preceding and following frames. When the screen is vertically long or square, generally, when the granularity is m × m (m: integer of 2 or more) and one frame is divided into N blocks, (2 × N) × (2 × N) A covariance matrix is generated.

The computing means 23 obtains eigenvalues and eigenvectors from the covariance matrix and obtains Euclidean distances between corresponding blocks in the preceding and succeeding frames based on the eigenvalues and eigenvectors. As shown in FIG. 5, eigenvalues and eigenvectors are first obtained from the covariance matrix by a known calculation (S15). The obtained eigenvalues are set to λ ₁ to λ ₈ , and the eigenvector V is set to the following (formula 1).

Next, the calculation means 23 calculates ((matrix 3) · V) ^T · (matrix 3) using the eigenvector V of (Equation 1) and the above (matrix 3 ^T ) and (matrix 3) to calculate the feature vector. Is calculated. The feature vector calculated here is 8 rows and 8 columns (generally k rows and k columns) as shown in (Expression 2) below, and the elements arranged in the row direction in (Expression 2) are collected. Feature vectors c1 to c8 are obtained. _{That, c1 = {c 11, ···} , c 81} T, c2 = {c 12, ···, c 82} T, ···, c8 = {c 18, ···, c 88} T Get. In (Expression 2), the matrix 3 is indicated by Q.

  Further, the computing means 23 calculates the distance between each element of the feature vectors c1 to c8. In this embodiment, when the distance between each element is dmn, the Euclidean distance is calculated by the following (Equation 3). It is obtained (S16 in FIG. 5). (Expression 3) is a general expression, and in this example where the covariance matrix is 8 rows × 8 columns, k = 8.

  The Euclidean distance obtained by the calculation means 23 is 8 rows × 8 columns as shown in the distance matrix D shown in FIG. 5, and 4 blocks indicated by 1 to 4 of the first frame Frame1 and 5 of the second frame Frame2. The distances for the corresponding blocks in the four blocks indicated by 8 are D (1,5), D (2,6), D (3,7), D (4,8) or D (4,8) or It exists in D (5,1), D (6,2), D (7,3), and D (8,4) arranged diagonally on the lower left side. In FIG. 5, in addition to the above, the position of the distance D (8, 1) between the block 1 of the first frame Frame1 and the block 8 of the second frame Frame2, the block 6 of the second frame Frame2, and the block of the second frame Frame2 The structure of the distance matrix D is described by also showing the position of the distance D (7, 6) to 7.

  The calculating means 23 calculates the Euclidean distance by the same calculation for the remaining corresponding blocks in the previous and subsequent frames. Here, since one frame is 320 × 240 pixels and the division granularity is 5 × 5 pixels, 64 × 48 = 3072 Euclidean distances are obtained as a whole.

  The processing means 24 in FIG. 1 performs image processing using the Euclidean distance and the threshold value. When the division granularity is 5 × 5 pixels, the Euclidean distance between corresponding blocks in one frame before and after is obtained 64 × 48 and can be arranged in the matrix of FIG. On the other hand, for example, a threshold created using the Euclidean distance obtained from an image in which the moving object actually moves and a block that exceeds the threshold by obtaining the Euclidean distance of the matrix are selected and determined as a moving object. The processing means 24 performs at least one of moving object edge detection, tracking, labeling, and Hough transform. Here, the processing means 24 first performs edge detection and labeling.

  For example, 1 is assigned to a block that exceeds the threshold, 0 is assigned to a block that is less than or equal to the threshold, and 1 frame is composed of 1 and 0, and the value changes from 0 to 1 or from 1 to 0 in the preceding and following frames. Edge detection is performed to detect blocks that are present. After this edge detection, the blocks shown as hatched in FIG. 7 are looped as shown in FIG. 7 (a) or not looped as shown in FIG. 7 (b). If a loop is detected, labeling that assigns a different number to the loop area is performed.

  For example, when the moving object A (number 1) and the moving object B (number 2) are detected in one frame by labeling as shown in FIG. 8, tracking is performed in a plurality of frames. In this case, since the Euclidean distance of all the blocks is obtained in each frame, the processing unit 24 tracks the moving object A and the moving object B using the Euclidean distance.

  As a tracking method, for example, the moving object A and the moving object are used in each frame by using the size of the area exceeding the above-described threshold or using the average or distribution of the Euclidean distance of the labeled area. B is identified. For example, a red rectangular frame is displayed in the tracked moving object A area, and a black rectangular frame is displayed in the tracked moving object B area, for example, and displayed on the display unit 13 of FIG. 1 shown in FIG. To do. Accordingly, the moving object A can be tracked by the red rectangle by visually observing the displayed video, and the moving object B can be tracked by the black rectangle, which is effective for monitoring.

  The processing means 24 may perform Hough conversion. For example, a moving object is detected in a plurality of frames by the tracking described above, and a line segment indicating the moving direction of the moving object is obtained by Hough transform. A moving object is detected as a single block or a block of multiple blocks by tracking, and it is only necessary to perform Hough transform using the position information (coordinates) of this moving object, and obtain and display the result of high-precision Hough transform. Can do.

  FIG. 9 is a flowchart corresponding to the image processing program. Since each means shown in FIG. 1 is realized by this flowchart, the operation of the CPU 10 will be described based on this flowchart. When the image processing apparatus is started and started, the i-th (initially i = 1) frame of the image signal is captured (S21), and the i + 1-th frame of the image signal is further captured (S22). Next, the CPU 10 takes in corresponding j blocks in the i-th frame and the i + 1-th frame and creates a matrix (S23). Accordingly, when the division granularity of 1 flock is 5 × 5 pixels, the matrix shown in FIG. 2 is created.

  Next, the CPU 10 obtains eigenvalues and eigenvectors using the above matrix, and generates a covariance matrix using the eigenvalues and eigenvectors as described above (S24). This covariance matrix is as shown in FIG.

  Subsequent to step S24, a Euclidean distance between corresponding blocks in the i-th frame and the i + 1-th frame is obtained using a covariance matrix (S25). The obtained Euclidean distance is a distance matrix D of 8 rows × 8 columns as shown in FIG.

  When the Euclidean distance is obtained in step S25, it is detected whether the processing of all target blocks has been completed (S26). If NO, the selected block is updated (S27), and the process returns to step S23 to continue the processing. If it is detected in step S26 that all the target blocks have been processed, the Euclidean distance of the corresponding block is compared with a threshold value prepared in advance, and a block exceeding the threshold value is extracted (S28).

  Further, following step S28, it is detected whether there is an extracted block (S29). If there is an extracted block, edge detection and labeling are performed as described above (S30) to perform tracking and Hough transform. Whether the predetermined number of frames has been processed is detected (S31). If the predetermined number of frames has been processed, the above-described tracking is performed (S32) to detect a preset line segment or the like. The Hough transform is performed (S33).

  After the end of step S33, or when NO in steps S29 and S31, the image display is performed using the processed image signal together with the processing result (S34). Following this display, the presence or absence of the next processing frame is detected (S35), and if there is a next processing frame, i is incremented by 1 (S36), and the process returns to step S22 to continue the processing. If there is no processing frame, the end.

  As described above, by dividing one frame into blocks and calculating the Euclidean distance of the corresponding block in the preceding and succeeding frames, it is possible to detect a moving object with high accuracy compared to a threshold, and edge detection, tracking, It is possible to reduce the amount of computation and simplify the algorithm in post-processing such as labeling and Hough transform. In the above-described embodiment, each unit is realized by software. However, each unit is also realized by a combination of hardware such as an arithmetic circuit and a logic circuit, thereby configuring the image processing apparatus. good.

10 CPU
DESCRIPTION OF SYMBOLS 11 Memory 13 Display part 21 Matrix preparation means 22 Generation means 23 Calculation means 24 Processing means

Claims (9)

A matrix for creating a matrix of luminance data for each of a plurality of corresponding blocks in temporally preceding and following frames by using m × m (m: integer greater than or equal to 2) pixels in one frame in an image as a block. Creating means;
Generating means for generating a covariance matrix using the corresponding matrix;
Calculating means for obtaining eigenvalues and eigenvectors from the covariance matrix, and obtaining Euclidean distances between corresponding blocks of the preceding and following frames based on the eigenvalues and eigenvectors;
An image processing apparatus comprising: processing means for performing image processing using a Euclidean distance and a threshold value.   The image processing apparatus according to claim 1, wherein all blocks or thinned blocks in one frame are used for creating a matrix.   The image processing apparatus according to claim 1, wherein the processing unit performs at least one of edge detection, tracking, labeling, and Hough transform of a moving object. A matrix for creating a matrix of luminance data for each of a plurality of corresponding blocks in temporally preceding and following frames by using m × m (m: integer greater than or equal to 2) pixels in one frame in an image as a block. Creation steps,
Generating a covariance matrix using the corresponding matrix;
An eigenvalue and an eigenvector are obtained from the covariance matrix, and an Euclidean distance between corresponding blocks of the preceding and succeeding frames based on the eigenvalue and eigenvector is calculated.
An image processing method comprising: performing a process step of performing image processing using a Euclidean distance and a threshold value.   5. The image processing method according to claim 4, wherein all blocks or thinned blocks in one frame are used for creating a matrix.   6. The image processing method according to claim 4, wherein in the processing step, at least one of edge detection, tracking, labeling, and Hough transform of a moving object is performed. A computer that performs image processing
A matrix for creating a matrix of luminance data for each of a plurality of corresponding blocks in temporally preceding and following frames by using m × m (m: integer greater than or equal to 2) pixels in one frame in an image as a block. Creation means,
Generating means for generating a covariance matrix using the corresponding matrix;
An operation means for obtaining eigenvalues and eigenvectors from the covariance matrix and obtaining Euclidean distances between corresponding blocks of the preceding and succeeding frames based on the eigenvalues and eigenvectors,
An image processing program that functions as a processing unit that performs image processing using a Euclidean distance and a threshold value.   8. The image processing program according to claim 7, which causes the computer to function so as to use all blocks or thinned blocks in one frame for creating a matrix.   The image processing program according to claim 7 or 8, wherein the computer functions as a processing unit to perform at least one of edge detection, tracking, labeling, and Hough transform of a moving object.