JP2007257026A - Mobile object monitoring device - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To perform precise matching while suppressing the elongation of time necessary for computation processing in a mobile object monitoring device. <P>SOLUTION: The device comprises a mobile object detection means 10 for detecting a plurality of blocks Bij including an approaching another vehicle 200 on the basis of images P(n), P(n+1) consecutive in time series composing a video photographed by a camera 80; a correlation method computing means 40 for obtaining an optical flow OF through a correlation method. The computing means 40 has a compensation interpolating means 41 for applying subpixelization to local sections in which the differences from averaged brightness of the blocks Bij are larger than a prescribed threshold among the blocks Bij as templates T and the whole of the next image P(n+1) as a search range, and template matching is implemented at pixel intervals obtained by the subpixelization. <P>COPYRIGHT: (C)2008,JPO&INPIT

Description

  The present invention relates to a moving object monitoring apparatus, and more particularly to an improvement in optical flow detection based on video obtained from a camera.

  In recent years, there has been proposed a moving body monitoring apparatus that obtains an optical flow of a moving body that approaches the camera based on an image captured by the camera.

  Here, the optical flow is one of moving body analysis methods, and means the velocity field of each point of the image generated by the relative movement of the moving body with respect to the camera. The motion of each part in the image is analyzed based on the signal), and this motion is expressed by the velocity vector to express the motion of the moving body. The correlation method (template matching (block matching) method) and the gradient method are is there.

  The correlation method is suitable for detection of an optical flow of a moving object such as the presence of a characteristic texture, and movement in an arbitrary image (Nth frame) among a plurality of time-sequential images constituting a video. By template matching using an image portion including a body (a part of an image of the Nth frame) as a template and an image (the (N + 1) th frame) following this arbitrary image (Nth frame) in time series as a search range , An image portion identical to or similar to the template (a part of the image of the (N + 1) th frame) is detected from the subsequent image (the (N + 1) th frame), and between both images (the image of the Nth frame and the (N + 1) th frame). Spatial positional relationship (the Nth frame) between the corresponding image parts in the frame) (a part of the image of the Nth frame and a part of the image of the (N + 1) th frame) Definitive position (Xn, Yn) → position in the (N + 1) frame based on (X (n + 1), Y (n + 1))), a method of obtaining the velocity vector of the image part.

  On the other hand, the gradient method is suitable for detecting the optical flow of a moving body whose outer surface is relatively smooth and whose outer surface has little saturation change, and the local density pattern of the image portion remains unchanged after movement (movement). The density pattern of an image portion (a part of the image of the Nth frame) in an arbitrary image (Nth frame) is spatially linearly approximated to obtain a plane (space) having a density gradient. The density value (image signal value) Spqn of the target pixel (Xpn, Yqn) is obtained by paying attention to an arbitrary pixel (Xpn, Yqn) on the density gradient plane. In the corresponding image portion (a part of the (N + 1) th frame image) of the image ((N + 1) th frame) time-sequentially following the image (Nth frame), spatially with the pixel of interest (Xpn, Yqn) Corresponding The density value Spq (n + 1) of the element (corresponding target pixel (Xp (n + 1), Yq (n + 1))) is obtained, and the corresponding target pixel (Xp ( n + 1), Yq (n + 1)) is obtained as a pixel position (Xp′n, Yq′n) having the same density value Sp′q′n as the density value Spq (n + 1), and this arbitrary image (Nth frame) Pixel (Xp′n, Yq′n) corresponds to the corresponding pixel of interest (Xp (n + 1), Yq) of the same density value (Sp′q′n = Spq (n + 1)) in the subsequent image ((N + 1) th frame). (N + 1)) according to the direction and magnitude of displacement of the density gradient plane of an arbitrary image (Nth frame) toward the density gradient plane of the subsequent image (N + 1) frame, The speed of the image part It is a method of obtaining the vector.

  Note that the displacement direction and size of the above-described density gradient plane cannot be specified in a single manner based only on the above-described coincidence of density values of one pixel. Provided to uniquely identify the velocity vector (Patent Documents 1 and 2).

Also, in these Patent Documents 1 and 2, various proposals such as hierarchization processing and correlation value interpolation processing have been made for the purpose of speeding up the calculation and increasing the detection accuracy.
JP 2005-209155 A Japanese Patent Laid-Open No. 2004-56763

  By the way, since the correlation method is a process of searching for a highly correlated image portion by template matching, it is effective to perform a search using a template at a spatially finer interval in order to obtain precise matching. is there.

  In this case, the image portion as a template and the entire search target image may be interpolated to form subpixels, and the matching pixel interval may be narrowed.

  However, if the entire image portion is converted into subpixels, the time required for the interpolation calculation process is too long, and it is difficult to obtain the optical flow of the moving object in substantially real time (less time lag).

  This is not a problem in which the moving body is limited to an automobile, and the same applies to steel materials and other objects. That is, for example, when a steel plate or a wooden plate transported by a crane or a conveyor is detected as a moving body, and its optical flow is obtained by a correlation method, there is a problem that can occur in the same manner.

  The present invention has been made in view of the above circumstances, and an object of the present invention is to provide a mobile monitoring apparatus capable of performing precise matching while suppressing an increase in time required for arithmetic processing.

  The mobile monitoring apparatus according to the present invention subtracts the entire subsequent image as a search range when subtracting an optical flow of a mobile by the correlation method, and a part (local part) of an image part used for a template. ) Only as subpixels, while suppressing an increase in the time required for arithmetic processing, and by making the search pixel interval fine, precise matching is performed.

  That is, the moving body monitoring device according to the present invention is a time-series continuous image forming the video in the moving body monitoring device that obtains the optical flow of the moving body approaching the camera based on the video captured by the camera. Moving body detecting means for detecting an image portion including the moving body based on a plurality of images, and using the image portion in an arbitrary image as a template, an image time-sequentially following the arbitrary image as a search range Correlation method calculating means for obtaining an optical flow of the moving body by performing template matching, wherein the correlation method calculating means has a predetermined difference in average brightness of the image portion of the image portion as the template. Sub-pixelization processing is performed on the local portion larger than the threshold and the entire subsequent image as the search range. It has between treatment means, the template matching, and performs the pixel interval obtained by the sub-pixelation.

  Here, the detection of an image portion (block for a template) including a moving body such as an automobile by the moving body detection unit corresponds spatially with respect to two images that are continuous in time series, for example. This can be realized by calculating a difference in signal value between pixels (at the same position).

  That is, the signal values of the image portions other than the moving body (image portions of the stationary object) are approximately equal between the two images. Therefore, the difference becomes substantially zero. On the other hand, the image portion of the moving body has a large signal value difference between the two images, particularly in the portion close to the contour.

  Therefore, it is possible to determine that the range where the difference exceeds the predetermined value is the image portion of the moving body, and it is possible to detect the image portion of the moving body.

  The image portion including the moving body does not mean an image portion of a large area including the entire moving body, but means a small block area as compared with the entire image. For example, the size of the entire image is about 1/30 pixels in the vertical direction and about 1/40 pixels in the horizontal direction. For example, in a block having the same vertical and horizontal pixel numbers. is there.

  The moving body detecting means detects such an image portion from a plurality of different positions in the entire image, but is arranged in a matrix, for example. However, since each image part always includes a moving object, an image part far from the moving object that does not include any moving object is not a detection target by the moving object detecting means.

  The correlation method calculation means performs a calculation process for obtaining an optical flow by the correlation method, and specifically, movement in an arbitrary image (image of the Nth frame) among a plurality of time-series continuous images constituting the video. An image portion including a body (a part of an image of the Nth frame) is used as a template, and an image (image of the (N + 1) th frame) following this arbitrary image (image of the Nth frame) in time series is set as a search range. By performing the template matching, a highly correlated image portion (a part of the image of the (N + 1) th frame) that is the same or similar to the template is detected from the subsequent image (image of the (N + 1) frame), and both images (the first image) N-frame image and (N + 1) -th frame image) corresponding to each other (one part of N-th frame image and one of (N + 1) -th frame image) Min)) (position (Xn, Yn) in a part (block) of an image of the Nth frame → position (X (n + 1), Y (n + 1)) in a part of the image of the (N + 1) th frame) ) To obtain the optical flow (velocity vector) of the image portion.

  When the optical flow is obtained by the correlation method, it is preferable to target the image portion of the moving object in which a characteristic texture exists, and the presence / absence of the texture is determined by calculating the variance value of the image portion in advance. You may do it.

  Subpixel conversion by the interpolation processing means, that is, pixel interpolation processing, is preferably performed by a bilinear (primary interpolation processing) method or a bicubic (tertiary interpolation processing) method.

  On the other hand, the method based on the nearest neighbor (nearest neighbor interpolation process) method is not a preferred application because it has little effect of performing template matching at the pixel interval obtained by subpixel conversion.

  According to the moving body monitoring apparatus according to the present invention, the moving body detecting means detects an image portion including the moving body based on a plurality of time-sequential images constituting a video.

  Then, the correlation method calculating means obtains the optical flow of the moving object by the correlation method using the image portion including the moving object detected by the moving object detecting means as a template. The interpolation processing means provided in the correlation method calculating means includes In the image portion as the template, a new pixel is interpolated by interpolation processing between the local portion whose difference from the average brightness of the image portion is larger than a predetermined threshold and the entire subsequent image as the search range. Compared to the case where the template matching is performed at the pixel interval before sub-pixelization by performing the template matching at the pixel interval obtained by the sub-pixel conversion. Compared with the case where subpixels are not used, the matching process can be performed more precisely. It is possible to obtain.

  In addition, since the entire image portion as a template is not subpixelized, only the local portion of the image portion whose difference from the average brightness of the entire image portion is larger than a predetermined threshold is subpixelized. The number of template matching is the same as when the entire image portion is subpixelized, but the number of interpolated pixels by subpixelization is greatly reduced compared to the case where the entire image portion is subpixelated. Therefore, the calculation time required to calculate the value of the interpolated pixel can be greatly reduced.

  In addition, in template matching of individual image parts, the number of pixels to be matched can be significantly reduced compared to the case where the entire image part is converted into sub-pixels, which also shortens the processing time. can do.

  The above effects are particularly remarkable when the bicubic method in which the time required for the arithmetic processing is relatively long is applied.

  DESCRIPTION OF EXEMPLARY EMBODIMENTS Hereinafter, a mobile object monitoring apparatus according to an embodiment of the invention will be described with reference to the drawings.

  FIG. 1 is a block diagram showing a mobile object monitoring apparatus 100 according to an embodiment of the present invention. FIG. 2 is a diagram showing another vehicle 200 (mobile object) on a host vehicle 90 equipped with the mobile object monitoring apparatus 100 shown in FIG. It is a schematic diagram showing the state which is approaching from the right direction as planar view from upper direction.

  The mobile monitoring apparatus 100 shown in the figure is based on an image taken by a camera 80 attached to the host vehicle 90, and approaches the camera 80, that is, the optical of the other vehicle 200 that approaches the host vehicle 90 from the right side. Other vehicles approaching based on a plurality of time-sequential images P (n−1), P (n), P (n + 1),... The moving body detecting means 10 for detecting a plurality of blocks Bij that are image parts including 200, and the pixel value of 4 vertical pixels × 4 horizontal pixels (= 16 pixels) constituting each block Bij detected by the moving body detecting means 10 The moving body feature amount calculation means 20 for obtaining the variance value (feature amount) of each (signal value such as brightness and density) and each block Bij in an arbitrary image (image of the (N) frame) P (n) As a template T, the block Bij of the other vehicle 200 is obtained by template matching using an image (image of the (N + 1) th frame) P (n + 1) following the arbitrary image P (n) in time series as a search range. Correlation method computing means 40 for obtaining the optical flow OF of the image, and an image succeeding in time series from the arbitrary image P (n) corresponding to the block Bij in the arbitrary image P (n) spatially. Based on the fluctuation of the local image signal in the corresponding image portion Cij in P (n + 1), the gradient method calculating means 50 for obtaining the optical flow OF for the block Bij of the approaching other vehicle 200 and the moving object feature value calculating means 30 Based on the obtained dispersion value, processing for obtaining the optical flow OF by the correlation method computing means 40 and the gradient method computing means 5 The operation switching means 30 for selectively switching the processing for obtaining the optical flow OF by the optical switching OF 30 and the optical flow OF obtained by the predetermined prediction filter processing using the predicted variance value and the measured variance value. The configuration includes a prediction filter 70 that corrects the flow OF and outputs the corrected optical flow OF to a monitor 85 attached to the interior of the host vehicle 90.

  Here, the camera 80 and the monitor 85 may be those attached in advance to the host vehicle 90 for a purpose different from the purpose of use of the mobile object monitoring device 100 (also used as a mobile object monitor). It may be a dedicated item for the intended use of the device 100. In the moving body monitoring apparatus 100 of the present embodiment, the camera 80 and the monitor 85 are described as those that are not constituent elements (combined items) of the moving body monitoring apparatus 100. However, the camera 80 and the monitor 85 are monitored by the moving body. When the device 100 is a dedicated product, it can be a component of the mobile object monitoring device 100.

  Further, as shown in FIG. 2, the camera 80 is fixed to, for example, the front right end of the host vehicle 90 so as to mainly view the right direction, and moves integrally with the host vehicle 90.

  Then, the camera 80 captures the video that has entered the field of view of the camera 80 in a time-series manner at a predetermined time interval, the still image of the (1) th frame, ..., the still image of the (N-1) th frame. An image P (n−1), a still image P (n) in the (N) th frame, a still image P (n + 1) in the (N + 1) th frame,... input.

  Here, FIGS. 3A, 3B, and 4 show the still image P (n−1) of the (N−1) th frame and the still image P of the (N) th frame that are in chronological order. (N), still image P (n + 1) in the (N + 1) th frame, and still image P (n) in the (N) th frame after the still image P (n-1) in the (N-1) th frame. ) And the still image P (n + 1) of the (N + 1) th frame is input next to the still image P (n) of the (N) th frame.

  It is assumed that the host vehicle 90 is in a temporarily stopped state before an intersection or the like. Accordingly, in each of the images P (n−1), P (n), and P (n + 1), the traffic sign 300 having no relative movement with respect to the camera 80, the road surface itself not shown (or characters / characters drawn on the road surface) The symbols / patterns and the like appear at the same position without any difference between the images P (n−1), P (n), and P (n + 1).

  On the other hand, the other vehicle 200 traveling in the direction of the arrow shown in FIG. 2 and approaching the host vehicle 200 has a relative movement with respect to the camera 80, so each image P (n−1), P ( n) and P (n + 1) appear at different positions.

  Therefore, the moving body detection means 10 is between two spatially corresponding pixels (at the same position) between two images P (n−1) and P (n) that are in chronological order. By calculating the difference between the signal values, the stationary object similar to the sign 300 and the moving object similar to the other vehicle 200 can be clearly distinguished.

  That is, since the image part other than the moving body (the image part of the stationary object) does not move between the two images P (n−1) and P (n), the signal values thereof are substantially equal. Therefore, the difference becomes substantially zero. On the other hand, the image portion of the moving body has a large signal value difference between the two images P (n−1) and P (n) particularly in the portion close to the contour.

  Therefore, it is possible to accurately detect (extract) the image portion of the moving body by determining that the image area whose difference exceeds a predetermined value is the image portion of the moving body.

  FIG. 5 is a schematic diagram in which a difference ΔP (n) = P (n + 1) −P (n) between two images P (n) and P (n + 1) is brightly and darkly inverted, such as a marker 300 indicated by a two-dot chain line. The stationary object is deleted, and only the image portion of the moving object, that is, the other vehicle 200 is lifted.

  Note that the light / dark reversal is merely to make the light / dark image of the difference image common with that in FIGS. 3 and 4 for easy understanding, and the moving object detection process itself by the moving object detecting means 10 is not affected. In addition, it is possible to detect even in a state where the inversion process is not performed.

  In addition, the images P (n) and P (n + 1) used for generating the difference image ΔP (n) are temporarily stored in a memory or the like for later processing.

  The moving body detection means 10 further includes a plurality (for example, for example) of blocks Bij each consisting of 4 vertical pixels × 4 horizontal pixels shown in FIG. 6 so as to be distributed in a matrix at equal intervals over the entire difference image (FIG. 5). (i = 1, 2,..., 9 and j = 1, 2,..., 9) (FIG. 7).

  Here, among all the blocks Bij, among the 16 pixels constituting the block Bij, a block Bij (in FIG. 7, B55, B65, B75, B85, B46, B56, which includes pixels having a pixel value equal to or larger than a predetermined value). Only B76, B86, B96, B47, B57, B67, B77, B87, B97) are detected as the target block.

  The detection of the target block Bij is a process of selecting only the block Bij in which the other vehicle 200 that is a moving body exists, and thereafter, only the selected block Bij is targeted for processing, and the remaining blocks Bij are Not subject to processing.

  Then, these detected blocks Bij are set at corresponding positions of the original image P (n) stored in the memory (FIG. 8).

  The moving object feature value calculating means 20 obtains a variance value of the pixel values of 4 vertical pixels × 4 horizontal pixels (= 16 pixels) constituting each block Bij detected by the moving object detecting means 10, respectively, Evaluate the presence or absence of texture.

  That is, when the variance value is larger than the predetermined value, it can be determined that the brightness variation in the block Bij is large and the texture exists, and when the variance value is smaller than the predetermined value, the brightness variation in the block Bij is It is gentle and it can be determined that there is no texture.

  In this way, the moving body feature value calculating unit 30 obtains a variance value for each block Bij, and the calculation switching unit 30 performs the optical flow OF on the block Bij having a variance value larger than a predetermined value by the correlation method calculating unit 40. The processing to be obtained is applied, and the processing is switched so that the processing for obtaining the optical flow OF by the gradient method computing unit 50 is applied to the block Bij having a variance value smaller than the predetermined value.

  The correlation method calculation means 40 performs calculation processing for obtaining the optical flow OF by the correlation method. Specifically, each block Bij in the image P (n) of the (N) th frame is set as a template T, and each template T Each time, a part of the image P (n + 1) in the (N + 1) th frame is highly correlated with the template by template matching using the image P (n + 1) in the (N + 1) th frame following in time series as the search range. ), And corresponding image portions (blocks of Nth frame image P (n)) between both images (Nth frame image P (n) and (N + 1) th frame image P (n + 1)). Spatial positional relationship between Bij and a part of the image P (n + 1) of the (N + 1) th frame (position of the image P (n) of the Nth frame in the block Bij) Xn, Yn) → (N + 1) -th position in a portion of the frame of the image P (n + 1) (X (n + 1), based on Y (n + 1))), determine the optical flow (velocity vector) of the block Bij.

  On the other hand, the gradient method computing means 50 includes three typical computation processes for obtaining the optical flow OF by the gradient method (a spatial global optimization method (SGO; a method assuming that the optical flow OF changes smoothly), a spatial local Optimization method (SLO; method assuming that the optical flow OF is the same among a plurality of blocks Bij), temporal local optimization method (TLO; method assuming that the optical flow OF at the optical flow OF detection point does not change over time)) Among them, the arithmetic processing for obtaining the optical flow OF is performed by the temporal local optimization method. Specifically, under the assumption that the lightness pattern of the block Bij is kept unchanged after the movement, A plane having a lightness gradient (spatial lightness gradient) is obtained by spatially linearly approximating the lightness pattern of the block Bij in the image P (n). The brightness value Spqn of the target pixel (Xpn, Yqn) is obtained by paying attention to an arbitrary pixel (Xpn, Yqn) on the brightness gradient plane, and this Nth frame image P (n) In the corresponding block Cij of the image P (n + 1) of the (N + 1) -th frame following in time series, a pixel (corresponding target pixel (Xp (n + 1), Yq) spatially corresponding to the target pixel (Xpn, Yqn). (N + 1))) brightness value Spq (n + 1) is obtained, and the brightness value Spq () of the corresponding pixel of interest (Xp (n + 1), Yq (n + 1)) on the brightness gradient plane in the image P (n) of the Nth frame. n + 1) is obtained as the position (Xp′n, Yq′n) of the pixel having the same brightness value Sp′q′n as the pixel (Xp′n, Yq′n) in the image P (n) of the Nth frame. , Following (N + 1) The image P of the Nth frame so as to coincide with the corresponding target pixel (Xp (n + 1), Yq (n + 1)) of the same lightness value (Sp′q′n = Spq (n + 1)) in the image P (n + 1) of the frame The optical flow OF of the block Bij is obtained based on the direction and the magnitude of displacing the lightness gradient plane of the block Bij of (n) toward the lightness gradient plane of the image P (n + 1) of the (N + 1) th frame.

  Correlation method computing means 40 also includes a local portion of block Bij serving as template T whose difference from the average brightness of block Bij is greater than a predetermined threshold, and image P of the (N + 1) th frame serving as the search range. Interpolation processing means 41 for performing subpixel conversion processing is provided for the entire (n + 1), and template matching by the correlation method calculation means 40 is performed at pixel intervals obtained by this subpixel processing.

  The subpixel conversion processing by the interpolation processing means 41, that is, the pixel interpolation processing, is preferably performed by a bilinear (primary interpolation processing) method or a bicubic (third order interpolation processing) method. This is because the nearest neighbor (nearest neighbor interpolation process) method has little effect of performing template matching at the pixel interval obtained by subpixel conversion.

  Taking the block B46 having a variance value larger than a predetermined value as an example among the detected blocks Bij in FIG. 8, before the subpixel conversion process, as shown in FIG. This is a block of pixels, and the upper left pixel b11 has higher brightness than the adjacent pixels b21 and b12, and its brightness change is steep, and the difference from the average brightness of the block Bij is larger than a predetermined threshold value. Yes.

  At this time, the interpolation processing means 41 determines only this local region so that the pixel interval of the local region (pixels b11, b21, b12, b22) including the pixel b11 where the brightness change is steep is halved. As shown in FIG. 9B, the subpixel processing is performed.

  For example, the first pixel b11 is divided into four new pixels b11 (1), b11 (2), b11 (3), and b11 (4) by applying a primary interpolation method, for example. It will be. Similarly, the other pixels b21, b12, and b22 are each divided into four pixels, and the spatial pixel interval is halved only in this local region.

  The remaining 12 pixels b31, b41, b32, b42, b13, b23, b33, b43, b14, b24, b34, b44 are not converted into subpixels.

  Therefore, as shown in FIG. 9C, the calculation time required for the interpolation processing can be shortened as compared with the case where all the pixels b11,..., B44 in the block B46 are converted into subpixels.

  On the other hand, as shown in FIG. 10, for the image P (n + 1) of the (N + 1) th frame as the search range, the entire image P (n + 1) is sub-pixel so that the pixel interval is ½ times. Process.

  As a result, the correlation method calculating means 40 spatially sets the matching interval of the template matching in which the search range is the entire image P (n + 1) and the block B46 is the template T as compared to the matching interval before subpixel conversion. Can be done finely.

  Here, since there are pixels that are not sub-pixelated in the template T, matching between the pixels that are not sub-pixelated and the pixels of the image P (n + 1) in which all pixels are sub-pixelated is shown in FIG. 11, for example, a region of the image P (n + 1) having a size corresponding to the large non-subpixel pixel b31 of the template T (the pixel c31 before being subpixelated) is converted into a single subpixel. Matching may be performed by representing the pixel c31 (1) as a representative.

  Similarly, the pixels b41, b13, and b14 of the template T may be matched with the pixels c41 (1), c13 (1), and c14 (1) of the image P (n + 1), respectively. The same applies to the pixels.

  Note that subpixel conversion processing may be performed by selectively switching between the primary interpolation processing method and the tertiary interpolation processing method according to the arithmetic processing capability of the interpolation processing means 41.

  The cubic interpolation processing method is preferable to the primary interpolation processing method in that smooth interpolation is possible and gives a natural impression even with the naked eye. On the other hand, the cubic interpolation processing method uses a cubic polynomial, Since it is necessary to perform an arithmetic processing that solves based on the continuity condition, an arithmetic processing time that is incomparable to the primary interpolation processing method is required.

  Therefore, particularly when trying to detect the other vehicle 200 that is an approaching vehicle, a calculation capability in substantially real time is required.

  Therefore, in a situation where the interpolation processing means 41 satisfies the calculation capability of the cubic interpolation processing method in substantially real time, sub-pixelization is performed by the cubic interpolation processing method, and the calculation capability of the cubic interpolation processing method is satisfied in substantially real time. However, in the situation where the calculation capability of the primary interpolation processing method is not satisfied, the calculation processing can be performed by switching between the primary interpolation processing method and the cubic interpolation processing method so as to perform subpixel conversion by the primary interpolation processing method. It is possible to accurately switch between time reduction and high interpolation accuracy according to the situation.

  Further, the correlation method calculating means 40 sets the search range for the subsequent image P (n + 1) based on the template T according to the arrangement position of the block Bij in the image P (n).

  That is, the moving body monitoring apparatus 100 of the present embodiment is not intended to detect all moving bodies, but approaches the camera 80 (the host vehicle 90 (an approached body) to which the camera 80 is fixed). Since the main purpose of use is to obtain the optical flow OF of the other vehicle 200 coming, the direction in which the other vehicle 200 approaches the camera 80 in the real space and the image P (n) taken by the camera 80 Corresponds to the moving direction of the other vehicle 200 in the vehicle, and the moving direction is limited to a predetermined range.

  For example, as in the present embodiment, in a state where the camera 80 is installed so as to capture a front right image of the host vehicle 90 to which the camera 80 is attached, the other vehicle approaching the camera 80 In the image P (n) photographed by the camera 80, 200 is displaced from the substantially right R to the left L in the left-right direction as shown in FIG. Therefore, in the left-right direction, the matching search range for the next image P (n + 1) by the template T can be limited to the range in the left L direction.

  Further, the portion of the other vehicle 200 in the block Bij (for example, block B55) existing in the region RU above the central portion C in the vertical direction of the entire image P (n) is moved in the vertical direction due to the approach to the camera 80. Since the image is magnified in the upper U direction, the vertical direction is displaced toward the upper U. Accordingly, the search range can be limited to a region L to the left and U above the position where the block Bij exists.

  On the other hand, the portion of the other vehicle 200 in the block Bij (for example, the block B47) existing in the region RD below the central portion C in the vertical direction is lowered in the vertical direction due to the approach to the camera 80. Since the image is magnified in the side D direction, the vertical direction is displaced toward the bottom D. Therefore, the search range can be limited to a region left L and below D from the position where the block Bij exists.

  Thus, the search time can be shortened by limiting the search range of template matching.

  It should be noted that the camera 80 is installed so as to shoot an image of the front left L of the host vehicle 90 to which the camera 80 is attached (for example, for a right-handed country or for both left and right approaches). In the case of a moving body monitoring device for monitoring a vehicle, etc., among the blocks Bij of the other vehicle 200 approaching the camera 80 from the left L, the upper portion U of the entire image P (n) above the central portion C in the vertical direction U The portion of the other vehicle 200 in the block Bij existing in the area RU is displaced to the right R in the left-right direction and upward U in the up-down direction. The search range in the subsequent image P (n + 1) can be limited to a region in the right R direction and above U from the position where the block Bij exists.

  On the contrary, the part of the other vehicle 200 in the block Bij existing in the region RD below the central portion C in the up-down direction of the entire image P (n) is downward R in the left-right direction and downward in the up-down direction. Therefore, when this block Bij is a template T, the search range in the subsequent image P (n + 1) in the time series is in the region in the right R direction and below D from the position where this block Bij exists. It can be limited.

  As the processing by the prediction filter 70, for example, Kalman filter processing or the like can be applied, but is not limited to Kalman filter processing, and other appropriate prediction filter processing can also be applied.

  In the video imaged by the camera 80, the time interval between frames is usually about 33 [msec], and the movement of the approaching other vehicle 200 (proximity moving body) is substantially constant during the period required for several frames. This assumption is easy to hold.

  As described above, under the assumption that the movement of the proximity moving body is substantially constant, it is possible to predict the current optical flow OF based on the optical flow OF before the present time.

  In this case, the accuracy of the processing result can be improved by performing the correction by the prediction filter processing.

  For example, since the prediction filter 70 to which the Kalman filter process is applied performs an averaging process using both the predicted variance value and the actually measured variance value, these values need to be determined in advance.

  The actually measured dispersion value can be determined in advance by experiment. Since the measured dispersion value differs depending on whether the correlation method calculating means 40 and the gradient method calculating means 50 are different from each other, the values corresponding to the measured dispersion values. Can be optimized by dynamically switching the measured variance value for the correlation method and the measured variance value for the gradient method according to the switching of the processing between the correlation method and the gradient method. it can.

  On the other hand, the predicted variance value gives the maximum value of time change as its effect. Specifically, the required number of frames is determined from the time when the approaching other vehicle 200 is detected for the first time until the correct value of the optical flow OF is obtained.

  As the variance value is larger than the experimental value, the convergence can be made faster, but the value of the optical flow OF tends to fluctuate. On the other hand, the closer the difference between the predicted variance value and the actually measured variance value is to zero, the easier it is to stabilize the optical flow OF value, but the longer it takes to converge.

  Therefore, this value needs to be determined so that the ratio to the experimentally obtained dispersion value is adjusted to meet the system requirements of the mobile monitoring device 100. In this case, the allowable delay time of the system of the mobile monitoring apparatus 100 can be used as a guide.

  The operation of the moving object monitoring apparatus 100 of the present embodiment configured as described above will be described with reference to FIGS. 13 and 14.

  First, the moving body detecting means 10 sequentially stores a plurality of time-sequential images P (n), P (n + 1),... Constituting a video taken by the camera 80 attached to the host vehicle 90. (# 1 in FIG. 13), and based on the two images P (n) and P (n + 1) that follow each other in time series, the difference image ΔP () between the images P (n) and P (n + 1) n) is obtained, and the other vehicle 200 as a moving body is detected based on the difference image ΔP (n) (# 2, FIG. 5). At this time, the original images P (n) and P (n + 1) are temporarily stored in the memory (# 3).

  The moving body detection means 10 sets the block Bij in the difference image ΔP (n) (# 4, FIG. 7), and selects only the block Bij corresponding to the other vehicle 200 based on the difference image ΔP (n) ( # 5, FIG. 8).

  Next, the moving body feature value calculating means 20 calculates the variance value of each selected block Bij (# 6). Then, when the variance value of the block Bij is equal to or greater than a predetermined value, the calculation switching unit 30 switches the process for obtaining the optical flow OF for the block Bij to the process by the correlation method calculation unit 40 (# 7, # 8). When the variance value of the block Bij is less than or equal to a predetermined value, the processing for obtaining the optical flow OF for the block Bij is switched to the processing by the gradient method computing means 50 (# 7, # 9).

  When the processing is switched to the processing by the correlation method calculating means 40 (# 8), the search area is set in the above-mentioned limited direction (FIG. 12) for the next image P (n + 1) that is the object of the search range (FIG. 12). 14), the block Bij is cut out as a template T from the image P (n) (# 22).

  Next, the interpolation processing means 41 of the correlation method calculating means 40 converts the template T into local subpixels and converts the entire next image P (n + 1) into subpixels (FIG. 9 (b), # 23), and the correlation method calculation. The means 40 performs template matching on the next image P (n + 1) in which the search area is limited using the template T converted into subpixels (FIGS. 10 and 11), and the optical for the block Bij. The flow OF is obtained (# 24).

  On the other hand, when the processing is switched to the processing by the gradient method computing means 50 (# 9), the gradient method computing means 50 performs the optical flow OF for this block Bij by the temporal local optimization method (TLO) of the gradient method. (# 9).

  Next, the prediction filter 70 corrects the optical flow OF of the block Bij obtained by the correlation method or the gradient method by Kalman filter processing (# 10), and the corrected optical flow OF obtained by this correction is output. (# 12). At the same time, the output corrected optical flow OF is recorded in a predetermined memory (# 11). The recorded optical flow OF after correction is used for correction of the Kalman filter processing from the next time by the prediction filter 70.

  After the optical flow OF is output, the same processing (# 5 to # 12) as described above is repeated for other blocks Bij (# 13), and the corrected optical is applied to all selected blocks Bij. When the flow OF is obtained (# 13), the processing between the two images P (n) and P (n + 1) that follow each other in time series is finished.

  As described above in detail, according to the moving body monitoring apparatus 100 of the present embodiment, the moving body detection means 10 is based on a plurality of time-sequential images P (n) constituting a video. The block Bij including the other vehicle 200 approaching the host vehicle 90 is detected, and the calculation switching unit 30 uses the correlation method calculation unit 40 according to the variance value of the block Bij calculated by the moving body feature amount calculation unit 20. Since the process for obtaining the optical flow OF and the process for obtaining the optical flow OF by the gradient method computing means 50 are selectively switched, the optical flow OF can be obtained by a processing method suitable for the feature of the outer surface of the other vehicle 200. .

  Further, according to the moving object monitoring apparatus 100 of the present embodiment, the moving object feature amount calculating unit 20 obtains the variance value of the lightness value of the block Bij as the feature amount of the block Bij, and this variance value is a predetermined value. The correlation method calculating means 40 and the gradient method calculating means 50 are selected so that the processing by the correlation method calculating means 40 is selected when the value is larger than the value, and the processing by the gradient method calculating means 50 is selected when the value is smaller than the predetermined value. Therefore, the outer surface of the block Bij of the moving body is similar to the block Bij other than the contour portion of the automobile (other vehicle 200) to be monitored by the moving body monitoring apparatus 100 of the present embodiment. Since the variance value for the block Bij is small in the case of relatively smooth and little change in the saturation of the outer surface (the characteristic texture does not exist), the operation switching means 30 is , The process for obtaining the optical flow OF is selected by the gradient method computing means 50. Such a block Bij can be assumed to be a plane having a density gradient by spatially linearly approximating the lightness pattern. The optical flow OF can be determined appropriately.

  On the other hand, since the variance value obtained by the moving body feature value calculation means 20 is large for the case where a characteristic texture exists such as the block Bij of the contour portion of the automobile (other vehicle 200), the calculation is performed. The switching unit 30 selects the processing for obtaining the optical flow OF by the correlation method calculating unit 40, but the correlation unit can appropriately detect the image portion of the moving object such as a characteristic texture by template matching. The optical flow OF can be obtained appropriately.

  Further, the correlation method calculating means 40 includes an interpolation processing means 41 that performs subpixel conversion processing locally, and template matching is performed at a pixel interval obtained by subpixel conversion. In the block Bij, a local portion having a difference from the average brightness of the block Bij larger than a predetermined threshold and the entire subsequent image P (n + 1) as a search range are interpolated by interpolation. Compared to the case where the correlation method calculating means 40 performs template matching at the pixel interval obtained by the subpixel conversion, and performs template matching at the pixel interval before the subpixel conversion. Compared with the case where subpixels are not used, it is possible to perform matching processing more finely in space. Matching results can be obtained.

  In addition, since the entire block Bij as the template T is not converted into subpixels, but only the local portion is converted into subpixels, the number of times of template matching is the same as when the entire image portion is converted into subpixels. Since the number of interpolated pixels by subpixel conversion can be greatly reduced compared to the case where the entire block Bij is subpixelated, the calculation time required to calculate the value of the interpolated pixel is greatly reduced. be able to.

  This effect is particularly remarkable when the bicubic method in which the time required for arithmetic processing is relatively long is applied.

  In addition, in individual template matching, the number of pixels to be matched can be greatly reduced compared to the case where the entire block Bij is converted into sub-pixels. From this point, the calculation processing time can be shortened. it can.

  Furthermore, the mobile monitoring apparatus 100 of the present embodiment can reduce the template matching search time by the correlation method computing means 40 by limiting the template matching search range, and can reduce the calculation time of the optical flow OF by the correlation method. It can be shortened.

  In addition, although the mobile body monitoring apparatus 100 of this embodiment is the example which applied the other vehicle 200 which approaches as a mobile body, the mobile body detection apparatus concerning this invention is not limited to a motor vehicle. For example, the present invention can also be applied to a case where a steel plate or a wooden plate conveyed by a crane or a conveyor is detected as a moving body and its optical flow is obtained.

  The moving body monitoring apparatus 100 of the present embodiment selectively switches between the correlation method calculating means 40 and the gradient method calculating means 50, but the moving object monitoring apparatus according to the present invention is limited to this form. Instead, any mobile monitoring device including at least a correlation method calculation unit 40 having an interpolation processing unit 41 for locally sub-pixelizing the block Bij may be used. The gradient method calculation unit 50, the calculation switching unit 30, and the prediction filter 70 may be used. May not be provided.

  Moreover, the correlation method calculating means 40 may not perform the process of limiting the search range (FIG. 12).

It is a block diagram which shows the moving body monitoring apparatus which concerns on one Embodiment of this invention. It is a schematic diagram showing the state which the other vehicle approaches from the right direction to the own vehicle carrying the moving body monitoring apparatus shown in FIG. 1 as a plan view from above. (A) is a still image P (n-1) of the (N-1) th frame preceding in time series, and (b) is a still image of the (N) frame following the still image P (n-1). It is a figure showing P (n), respectively. It is a figure showing the still image P (n + 1) of the (N + 1) th frame following a still image P (n). FIG. 6 is a schematic diagram in which a difference ΔP (n) = P (n + 1) −P (n) between two images P (n) and P (n + 1) is bright and dark inverted. FIG. It is a figure which shows the some block Bij which consists of vertical 4 pixels x horizontal 4 pixels distributed in the matrix form at equal intervals over the whole area of an image. FIG. 6 is a diagram illustrating a state in which a plurality of blocks Bij illustrated in FIG. 6 are set over the entire difference image illustrated in FIG. 5. It is a figure which shows the state which selected only the block Bij in which another vehicle exists. (A) is a diagram showing a pixel arrangement of the original block Bij, (b) is a diagram showing a pixel arrangement of the block Bij that is locally subpixelized, and (c) is a pixel of the block Bij that is a subpixel of the whole It is a schematic diagram which shows the figure which shows an arrangement | sequence, respectively. It is a schematic diagram which shows a mode that the next image P (n + 1) is template-matching-processed by making the block Bij made into a local subpixel into the template T. FIG. It is a figure explaining the matching process with the next image P (n + 1) by which the pixel not subpixel-ized and the whole subpixel-ized. It is a figure explaining limitation of the search range of a template matching process. It is a flowchart explaining the process sequence of the effect | action by the moving body monitoring apparatus shown in FIG. It is a flowchart explaining the process sequence which calculates | requires the optical flow by the correlation method by a correlation method calculating means.

Explanation of symbols

DESCRIPTION OF SYMBOLS 10 Moving body detection means 20 Moving body feature-value calculation means 30 Calculation switching means 40 Correlation method calculation means 41 Interpolation processing means 50 Gradient method calculation means 70 Prediction filter 80 Camera 85 Monitor 100 Moving body monitoring apparatus

Claims (8)

In the moving object monitoring device for obtaining the optical flow of the moving object approaching the camera based on the video imaged by the camera,
A moving body detecting means for detecting an image portion including the moving body based on a plurality of time-sequential images constituting the video;
Correlation method calculating means for obtaining the optical flow of the moving body by template matching with the image portion in an arbitrary image as a template, and an image following the arbitrary image in time series as a search range,
The correlation method calculating means, for the image portion as the template, a local portion having a difference from the average brightness of the image portion larger than a predetermined threshold, and the entire subsequent image as the search range, An apparatus for monitoring a moving body, comprising: interpolation processing means for performing subpixel processing, and performing the template matching at a pixel interval obtained by the subpixel conversion.   2. The moving object monitoring apparatus according to claim 1, wherein the interpolation processing means performs the subpixel processing by primary interpolation processing or tertiary interpolation processing.   The moving body monitoring apparatus according to claim 1, wherein the interpolation processing unit selectively switches between a primary interpolation processing method and a cubic interpolation processing method to perform the subpixel conversion processing. The correlation method calculating means includes:
4. The search range for the subsequent image based on the template is set in accordance with an arrangement position of the image portion in the arbitrary image. 5. Mobile monitoring device. In a state where the camera is installed so as to capture a front right image of an approaching object to which the camera is attached,
The component along the vertical direction of the search direction of the template matching in which the image portion arranged in the upper portion from the center in the vertical direction in the arbitrary image is the template is a direction facing the upper side, The component along the vertical direction of the search direction of the template matching with the image portion arranged in the lower part from the center as the template is the direction facing the lower side,
The mobile body monitoring apparatus according to claim 4, wherein a component along the left-right direction of the search direction of the template matching is a direction facing left. In a state where the camera is installed so as to take a front left image of an approaching object to which the camera is attached,
The component along the vertical direction of the search direction of the template matching in which the image portion arranged in the upper portion from the center in the vertical direction in the arbitrary image is the template is a direction facing the upper side, The component along the vertical direction of the search direction of the template matching with the image portion arranged in the lower part from the center as the template is the direction facing the lower side,
The mobile body monitoring apparatus according to claim 4, wherein a component along the left-right direction of the search direction of the template matching is a direction facing right.   7. A prediction filter that corrects the optical flow by a predetermined prediction filter process using a predicted variance value and an actual variance value for the optical flow is provided. The moving body monitoring apparatus of any one of Claims. The moving body monitoring apparatus according to claim 1, wherein the moving body is a traveling vehicle.

Images