JP6188592B2 - Object detection apparatus, object detection method, and object detection program - Google Patents

Description

  The present invention relates to an object detection device, an object detection method, and an object detection program for detecting an object region from an image photographed by a camera.

  In order to support driving of a vehicle such as an automobile, for example, there is a technique in which a camera is installed in a vehicle and an object that can be an obstacle of the vehicle is detected from an image captured by the camera.

  In Patent Document 1, a motion vector between a plurality of images captured by a camera is detected, a movement parameter and a background vector of the own vehicle are detected from the detected motion vector, and a motion vector that is not detected as a background vector Is detected as an object vector, an object is detected from the object vector, and a warning is disclosed to the driver according to the approaching state of the obtained object.

JP 2013-97390 A

  However, in a configuration in which a motion vector is detected from a plurality of images taken by a camera and an object region is detected based only on the detected motion vector, the object region cannot be accurately detected.

  It is an object of the present invention to provide an object detection device, an object detection method, and an object detection program that can detect an object region more accurately than when detecting an object region based only on a motion vector. .

An object detection device according to the present invention includes an image acquisition unit that acquires a plurality of images sequentially captured by a camera, and a target image and at least one reference image among the plurality of images acquired by the image acquisition unit, A motion vector detection unit that detects a motion vector of each part of the target image, a difference calculation unit that calculates a difference image between two images of the plurality of images acquired by the image acquisition unit, and the motion vector An object detection unit that detects an object region in which an object exists based on the motion vector detected by the detection unit and the difference image calculated by the difference calculation unit ; and the object detection unit includes the motion vector Based on the target image, a region detection unit that detects a region of the object, a contour extraction unit that extracts a contour of the object from the difference image, and a detection by the region detection unit Corresponding to the region, the region surrounded by the contour extracted by the contour extracting unit, and having a body area detection unit for detecting as the object region.

The object detection method according to the present invention includes an image acquisition step of acquiring a plurality of images sequentially captured by a camera, a target image and at least one reference image among the plurality of images acquired in the image acquisition step, A motion vector detection step for detecting a motion vector of each part of the target image, a difference calculation step for calculating a difference image between two images of the plurality of images acquired in the image acquisition step, and the motion vector a motion vector detected in the detection step, the based on the difference image calculated by the difference calculation step, seen including a object detection step of detecting an object area where an object exists, the object detection step, the motion A region detecting step of detecting a region of the object from the target image based on the vector; and extracting an outline of the object from the difference image. To a contour extraction step, corresponding to the detected in the area detecting step region, a region surrounded by the extracted at the contour extraction step contour, in that it has an object area detection step of detecting as said object area Features.

The object detection program according to the present invention includes an image acquisition step of acquiring a plurality of images sequentially captured by a camera, a target image and at least one reference image among the plurality of images acquired in the image acquisition step, A motion vector detection step for detecting a motion vector of each part of the target image, a difference calculation step for calculating a difference image between two images of the plurality of images acquired in the image acquisition step, and the motion vector Based on the motion vector detected in the detection step and the difference image calculated in the difference calculation step, the computer executes an object detection step for detecting an object region where the object exists , and the object detection step includes: A region detecting step of detecting a region of an object from the target image based on the motion vector; and the difference An object region for detecting, as the object region, a region surrounded by the contour extracted in the contour extraction step, which corresponds to the region detected in the region detection step and a contour extraction step for extracting the contour of the object from the image And a detecting step .

  According to the present invention, by detecting an object region based on a difference image in addition to a motion vector, the object region can be detected more accurately than when detecting an object region based only on a motion vector. .

1 is a block diagram illustrating an example of a configuration of an object detection system including an object detection device according to Embodiment 1. FIG. (A)-(c) is a figure for demonstrating the process which detects the motion vector of a pixel unit. It is a figure which shows an example of the detection result of a motion vector. It is a figure for demonstrating the calculation process of a difference image. It is a figure for demonstrating the process which detects an area | region based on the detection result of a motion vector. It is a figure for demonstrating the process which extracts an outline from a difference image. It is a figure for demonstrating the process which detects an object area | region. 6 is a flowchart illustrating an example of an obstacle detection process performed by the object detection device according to the first embodiment. It is a flowchart which shows the content of the motion vector detection process of step S3 of FIG. It is a flowchart which shows the content of the difference calculation process of step S5 of FIG. It is a flowchart which shows an example of the obstruction detection process by the object detection apparatus in a modification. 6 is a block diagram illustrating an example of a configuration of an object detection system including an object detection device according to Embodiment 2. FIG. 10 is a flowchart showing the contents of motion vector detection processing in the second embodiment. 10 is a block diagram illustrating an example of a configuration of an object detection system including an object detection device according to Embodiment 3. FIG. 14 is a flowchart illustrating an example of an obstacle detection process performed by the object detection device according to the third embodiment.

Hereinafter, embodiments of the present invention will be described with reference to the drawings.
Embodiment 1 FIG.
FIG. 1 is a block diagram illustrating an example of a configuration of an object detection system 1 including an object detection device 20 according to the first embodiment. The object detection system 1 is a system that detects an object from an image taken by a camera. In the present embodiment, the object detection system 1 is an obstacle detection system that detects an obstacle around the vehicle from an image taken by a camera mounted on a vehicle such as an automobile.

  As shown in FIG. 1, the object detection system 1 includes a camera 10, an object detection device 20, and an obstacle notification unit 30.

  The camera 10 sequentially captures the imaging target area and outputs a plurality of captured images to the object detection device 20. Here, the image is image data (or an image signal). The image data is data representing the pixel value of each pixel constituting the image, specifically, data representing each pixel value of a plurality of pixels arranged in a two-dimensional manner. In the present embodiment, the camera 10 outputs an image frame (hereinafter simply referred to as “frame”) that is image data. For example, the camera 10 captures images at a predetermined frame rate, and sequentially outputs the obtained frames to the object detection device 20. Moreover, the camera 10 is installed in the vehicle and images the surroundings of the vehicle. For example, the camera 10 is attached to the position of the side mirror of the vehicle in the direction of imaging the rear of the vehicle. By installing at the position of the side mirror, the side mirror can be replaced with a camera. In this case, it is possible to eliminate the blind spot on the rear side by using a camera having a wider angle than the side mirror.

  The object detection device 20 acquires an image taken by the camera 10 and detects an object from the acquired image. In this example, the object detection device 20 is an obstacle detection device that detects an obstacle around the vehicle from the acquired image. In FIG. 1, the object detection device 20 includes an image acquisition unit 21, a motion vector detection unit 22, a difference calculation unit 23, and an object detection unit 24.

  The image acquisition unit 21 acquires a plurality of frames taken sequentially by the camera 10. The image acquisition unit 21 may capture all frames output from the camera 10 or may capture by thinning out. For example, the image acquisition unit 21 may acquire a plurality of frames shot at a frame rate of 30 fps at the same frame rate as that at the time of shooting (that is, a frame interval of 1/30 seconds), or a frame rate different from that at the time of shooting. (For example, it may be acquired at a rate of one frame to three frames, that is, a frame interval of 1/10 seconds). The image acquisition unit 21 outputs the acquired plurality of frames to the motion vector detection unit 22 and the difference calculation unit 23.

  The motion vector detection unit 22 detects a motion vector of each part of the image from the plurality of images acquired by the image acquisition unit 21. Specifically, the motion vector detection unit 22 detects a motion vector of each part of the target image from the target image and at least one reference image among the plurality of input images. The motion vector detection unit 22 outputs the detection result to the object detection unit 24.

  The difference calculation unit 23 calculates a difference image between two images among the plurality of images acquired by the image acquisition unit 21. The difference calculation unit 23 outputs the calculation result to the object detection unit 24.

  The object detection unit 24 detects an object region where an object exists based on the motion vector detected by the motion vector detection unit 22 and the difference image calculated by the difference calculation unit 23. Further, when detecting the object area, the object detection unit 24 determines whether the object existing in the object area is an obstacle to the vehicle based on the position of the object area and the motion vector in the object area. Thus, obstacles around the vehicle are detected. When the object detection unit 24 detects an obstacle, the object detection unit 24 notifies the obstacle notification unit 30 to that effect. At this time, the object detection unit 24 may notify the object area of the object determined to be an obstacle, or may notify the position or shape of the object specified from the object area.

  When the obstacle notification unit 30 receives the notification from the object detection unit 24, the obstacle notification unit 30 notifies or warns the driver of the vehicle that an obstacle exists. At this time, the obstacle notification unit 30 may notify the driver of the object region, position, or shape of the obstacle based on the notification from the object detection unit 24. The obstacle notification unit 30 performs notification by, for example, display by a display device or sound output by a sound output device.

  In the above configuration, the image acquisition unit 21 may include an image storage unit 21a that stores an image from the camera 10, and the motion vector detection unit 22 and the difference calculation unit 23 acquire an image used for processing from the image storage unit 21a. May be.

  Hereinafter, the configuration and processing of the motion vector detection unit 22, the difference calculation unit 23, and the object detection unit 24 will be specifically described.

  The motion vector detection unit 22 detects a motion vector of each part of the target frame from the target frame and the reference frame among the plurality of frames input from the image acquisition unit 21. For example, the target frame is a current frame (hereinafter referred to as “current frame”), and the reference frame is a frame (hereinafter referred to as “previous frame”) that is captured in time before the current frame. In the present embodiment, the target frame is divided into a plurality of blocks, and the motion vector detection unit 22 detects a motion vector for each block of the target frame, and uses the detected motion vector for each block to detect the target frame. A motion vector is detected in units of pixels. Each block is composed of a plurality of pixels of the target frame, and is, for example, a rectangular pixel region. The block size may be determined as appropriate, but here it is 16 × 16 pixels.

  In the example of FIG. 1, the motion vector detection unit 22 includes a block motion vector detection unit 22a and a pixel motion vector detection unit 22b.

  The block motion vector detection unit 22a detects a motion vector of each block of the target frame. As a method for detecting a motion vector, various methods including a known method can be used. Here, a block matching method is used. When obtaining a motion vector of a target block among a plurality of blocks, the block motion vector detection unit 22a sets a search range in the reference frame, searches for a block most similar to the target block within the search range, and is searched. A vector from the position of the block to the position of the target block is obtained as a motion vector of the target block. The block motion vector detection unit 22a outputs the motion vector of each block of the target frame to the pixel motion vector detection unit 22b as a detection result.

  The pixel motion vector detection unit 22b detects the motion vector of each pixel of the target frame based on the motion vector of each block of the target frame from the block motion vector detection unit 22a. The pixel motion vector detection unit 22b refines the motion vector in units of blocks into a motion vector in units of pixels. Specifically, the pixel motion vector detection unit 22b detects a motion vector in units of blocks while gradually reducing the block of the target frame, and finally detects a motion vector in units of pixels. In this case, when detecting a motion vector in units of blocks or pixels, a search range can be determined based on the motion vector detected in the previous stage. For example, the pixel motion vector detection unit 22b divides each block of 16 × 16 pixels into four blocks of 8 × 8 pixels, and obtains a motion vector of each block of 8 × 8 pixels. At this time, the search range for obtaining the motion vector is narrowed down based on the motion vector of the 16 × 16 pixel block to which the 8 × 8 pixel block for which the motion vector is obtained belongs. Similarly, each block of 8 × 8 pixels is divided into four blocks of 4 × 4 pixels to obtain a motion vector of each block of 4 × 4 pixels, and each block of 4 × 4 pixels is converted to a block of 2 × 2 pixels. The motion vector of each block of 2 × 2 pixels is obtained by dividing into four. Based on the motion vector of each block of 2 × 2 pixels, the motion vector of each pixel of the target frame is obtained. Hereinafter, this process will be described with reference to FIGS.

Here, a case where the motion vector of the target pixel 41 of the target frame 40 shown in FIG. As shown in FIG. 2B, a search range 44 is set in the reference frame 43 based on the motion vector 42 of the 2 × 2 pixel block to which the target pixel 41 belongs. Specifically, the pixel position 45 on the reference frame 43 corresponding to the target pixel 41 is obtained from the motion vector 42, and an area of a predetermined size centered on the position 45 is set as the search range 44. Then, in the reference frame 43, the reference pixel 46 is moved to each pixel position in the search range 44. At this time, the sum of the difference values between the pixel value of the reference pixel 46 and its surrounding pixels and the pixel value of the target pixel 41 is calculated at each pixel position. For example, as shown in FIG. 2C, when the pixel value of the target pixel 41 is p and the pixel values of each pixel of the 3 × 3 pixel group 47 centering on the reference pixel 46 are p1 to p9, the difference The sum S of values is calculated by the following equation (1). Then, the position 48 of the reference pixel having the minimum sum S in the search range 44 is specified, and the vector 49 from the position 48 of the specified reference pixel to the position of the target pixel 41 is determined as the motion vector of the target pixel 41. To do.
S = (p1−p) + (p2−p) +... + (P9−p) (1)

  FIG. 3 is a diagram illustrating an example of a detection result of the motion vector detection unit 22. FIG. 3 shows a result when a motion vector of each pixel of the target frame 51 is detected with reference to the reference frame 50. In one example, the target frame 51 is the current frame, and the reference frame 50 is the previous frame. In FIG. 3, in the target frame 51, a part of the motion vector of each pixel that is a detection result is indicated by an arrow. The length of the arrow indicates the magnitude of the motion vector, the direction of the arrow indicates the direction of the motion vector, and the starting point of the arrow indicates the position of the pixel corresponding to the motion vector. An object is shown in the area 52 of the reference frame 50, and the same object is shown in the area 53 of the target frame 51. For the pixel at the coordinate (x2, y2) of the target frame 51, when the coordinate (x1, y1) is calculated as the position of the reference pixel that minimizes the sum S of difference values, the coordinate (x2) from the coordinate (x1, y1) is calculated. , Y2) is detected as the motion vector 54 (x2-x1, y2-y1). The coordinates (x2, y2) correspond to the position of the object, and the motion vector 54 represents the direction and magnitude of the movement of the object. Similar to the motion vector 54 in FIG. 3, the other motion vectors indicated by solid arrows also indicate the direction and magnitude of the motion of the object. The motion vectors indicated by these solid arrows have the same size and direction as each other, and can be called object motion vectors. On the other hand, in FIG. 3, a plurality of motion vectors indicated by a one-dot chain line arrow including the motion vector 55 represents the direction and magnitude of the background motion of the object, and have a magnitude and direction different from the motion vector of the object. The motion vectors indicated by these alternate long and short dash arrows can be called background motion vectors.

  Note that in order to detect a motion vector, at least two frames of a target frame and a reference frame are necessary, but more reference frames may be used. For example, the motion vector detection unit 22 may detect a motion vector using a total of three frames including the target frame and the reference frames before and after the target frame. By using frames before and after the target frame, it is possible to detect a motion vector with higher accuracy.

  The interval between the target frame and the reference frame used for motion vector detection may be determined as appropriate. A frame one frame before the target frame may be used as a reference frame, or a frame N frames (N is an integer of 2 or more) before the target frame may be used as a reference frame. For example, in the case of shooting at 30 fps, the interval between adjacent frames is 1/30 second. However, at this interval, the change in the image is too small and it may be difficult to detect the motion vector. In such a case, the interval between frames used for motion vector detection may be 1/15 seconds instead of 1/30 seconds. However, if the interval between frames is too large, there is a possibility that a motion vector of a fast moving object such as an obstacle with a very fast approach speed cannot be detected properly. It is desirable that the interval between frames used for motion vector detection be optimized in consideration of the above situation.

  In addition, the interval between target frames for which motion vectors are to be detected may be determined as appropriate. For example, the motion vector detection unit 22 may detect a motion vector for each captured frame, or may detect a motion vector at a rate of one frame in N frames (N is an integer of 2 or more). For example, when shooting at 30 fps, if a motion vector is to be detected for each frame, the motion vector must be detected every 1/30 seconds, and a high calculation speed is required. Therefore, the interval for detecting the motion vector may be 1/15 seconds instead of 1/30 seconds. However, if the frame interval for detecting the motion vector is too large, the danger notification may be delayed when the approach speed of the obstacle is very high. It is desirable that the frame interval for detecting the motion vector is optimized in consideration of the above situation.

  The difference calculation unit 23 calculates a difference between two frames of the plurality of frames input from the image acquisition unit 21 to generate a difference image. Specifically, the difference calculation unit 23 calculates a difference between pixel values of each pixel between two frames, and generates a difference image having the calculated difference between the pixels as a pixel value. In one aspect, one of the two frames is the same frame as the target frame used for motion vector detection. The other of the two frames is, for example, the same frame as a reference frame used for motion vector detection, but may be a different frame. For example, the difference calculation unit 23 calculates a difference image between two frames including the target frame for each target frame of motion vector detection.

FIG. 4 is a diagram illustrating an example of processing of the difference calculation unit 23. In the example of FIG. 4, the difference calculation unit 23 generates a difference image 63 between the first frame 61 and the second frame 62. For example, the first frame 61 is the previous frame, and the second frame 62 is the current frame. Each frame is composed of a total of X × Y pixels located at coordinates (x, y) (x = 1, 2,..., X, y = 1, 2,..., Y). The pixel value of the pixel at the coordinates (x, y) of the first frame 61 is represented by p1 (x, y), and the pixel value of the pixel at the coordinates (x, y) of the second frame 62 is p2 (x, y). ). At this time, the difference image 63 is a total of X × Y pixels located at coordinates (x, y) (x = 1, 2,..., X, y = 1, 2,..., Y). The pixel value p3 (x, y) of the pixel having the coordinates (x, y) of the difference image 63 is expressed by the following equation (2).
p3 (x, y) = p1 (x, y) -p2 (x, y) (2)

  In FIG. 4, an object is shown in the area 61 a of the first frame 61, and the same object is shown in the area 62 a of the second frame 62. Between the two frames, the position of the object in the frame moves, and the contour of the object is shifted. Due to the deviation of the contour, the contour of the object appears in the difference image 63 between the two frames. Specifically, in the difference image 63, the contour L1 of the region 61a of the first frame 61 and the contour L2 of the region 62a of the second frame 62 appear. More specifically, in the difference image 63, the region 63a is a difference region between the object region 61a of the first frame 61 and the background region 62b of the second frame 62, and the absolute value of the pixel value of this region Is relatively large. The region 63b is a difference region between the background region 61b of the first frame 61 and the object region 62a of the second frame 62, and the absolute value of the pixel value of this region is relatively large. The region 63c is a difference region between the object region 61a of the first frame 61 and the object region 62a of the second frame 62, and the absolute value of the pixel value of this region is relatively small. The region 63d is a difference region between the background region 61b of the first frame 61 and the background region 62b of the second frame 62, and the absolute value of the pixel value of this region is relatively small. Due to the difference in pixel values between the regions, contours L1 and L2 appear at the boundary between the regions.

  The interval between two frames used for calculating the difference image may be determined as appropriate, a difference image between two adjacent frames may be calculated, or between two frames sandwiching one or more frames in between. The difference image may be calculated. For example, in the case of shooting at 30 fps, the interval between adjacent frames is 1/30 second. However, at this interval, the change in the image is too small and the contour may not be detected. In such a case, the interval between two frames may be 1/15 seconds instead of 1/30 seconds. However, if the interval between frames is too large, there is a possibility that the contour of an object that moves fast, such as an obstacle with a very fast approach speed, cannot be detected properly. It is desirable that the interval between the two frames used for calculating the difference image be optimized in consideration of the above situation.

  In addition, the frame interval for calculating the difference image (for example, the interval between the frames serving as the second frames) may be determined as appropriate. For example, the difference calculation unit 23 may calculate a difference image using each captured frame as the second frame, or one frame as the second frame every N frames (N is an integer of 2 or more). A difference image may be calculated. For example, in the case of shooting at 30 fps, if a difference image is to be calculated for each frame, the calculation must be performed every 1/30 seconds, and a high calculation speed is required. Therefore, the interval for calculating the difference image may be 1/15 seconds instead of 1/30 seconds. However, if the frame interval for calculating the difference image is too large, the danger notification may be delayed if the approach speed of the obstacle is very fast. It is desirable that the frame interval for calculating the difference image be optimized in consideration of the above situation.

  The object detection unit 24 detects an object region where an object in the frame exists based on the detection result of the motion vector detection unit 22 and the calculation result of the difference calculation unit 23. For example, the object detection unit 24 detects an object region for each target frame of motion vector detection. As will be described in detail below, the object detection unit 24 is a differential image corresponding to a region (or a region where the motion vectors in the target frame are uniform) composed of a plurality of portions having the same motion vector in the target frame. A region surrounded by the inside contour is detected as an object region.

  In the example of FIG. 1, the object detection unit 24 includes a region detection unit 24a, a contour extraction unit 24b, an object region detection unit 24c, and an obstacle determination unit 24d.

  The region detection unit 24 a detects the region of the object from the target frame based on the motion vector detected by the motion vector detection unit 22. Specifically, the region detection unit 24a detects a region including a plurality of pixels having the same motion vector in the target frame as an object region. Specifically, the region detection unit 24a detects a single region composed of a plurality of pixels having the same magnitude and direction of motion vectors as an object region. The above “same” is not limited to the same in a strict sense, but may be the same as long as it can be regarded as a motion vector of the same object, and may be substantially the same.

  For example, when the detection result shown in FIG. 3 is obtained by the motion vector detection unit 22, the region detection unit 24a is formed by a plurality of pixels having the same magnitude and direction of motion vectors as shown in FIG. The lump area 70 is detected as an object area. In FIG. 5, the area 70 of the detected object substantially coincides with the area 53 where the object is actually shown, but the outline is shifted.

  The contour extraction unit 24 b extracts the contour of the object from the difference image calculated by the difference calculation unit 23. For example, when the difference calculation unit 23 obtains the difference image 63 illustrated in FIG. 4, the contour extraction unit 24 b identifies the regions 63 a to 63 d from the difference image 63 as illustrated in FIG. 6 by, for example, labeling processing. Then, the boundary lines B1 to B4 between the specified regions are extracted as the contours of the object. The boundary lines B1 and B3 correspond to the contour L1, and the boundary lines B2 and B4 correspond to the contour L2. In addition, as a method for extracting a contour from an image, various methods including a known method can be used.

  The object region detection unit 24c detects a region surrounded by the contour extracted by the contour extraction unit 24b corresponding to the region detected by the region detection unit 24a as an object region. Specifically, the object region detection unit 24c superimposes the region detected by the region detection unit 24a and the contour extracted by the contour extraction unit 24b (or superimposes the target frame and the difference image). ), An area surrounded by the extracted contour corresponding to the detected area is detected as an object area.

  When the region 70 in FIG. 5 is detected by the region detection unit 24a and the contours B1 to B4 in FIG. 6 are extracted by the contour extraction unit 24b, the object region detection unit 24c adds the region 70 to the region 70 as shown in FIG. The contours B1 to B4 are overlapped, and a region 80 surrounded by the contours B2 and B4 (that is, the contour L2) corresponding to the region 70 is detected as an object region.

In one example, the object region detection unit 24c sets the region that is most consistent with the detected region 70 (here, the region configured by the contours B2 and B4) among the regions that can be configured by the extracted contours B1 to B4. Detect as a region. For example, for each candidate region that can be configured by the contours B1 to B4, the overlapping rate D with the region 70 is obtained, and the candidate region with the highest overlapping rate D is detected as the object region. The overlap rate D is calculated by the following equation (3), for example, when the area of the region 70 is S1, the area of the candidate region is S2, and the area of the region where the region 70 and the candidate region overlap is S3. Note that when the highest overlap rate among the calculated overlap rates is equal to or less than a predetermined threshold, the object region detection unit 24c may not detect the object region corresponding to the region 70.
D = S3 / (S1 + S2-S3) (3)

  In another example, the object region detection unit 24c calculates the degree of coincidence with the contour of the region 70 for each of the extracted contours B1 to B4, identifies the contour having a coincidence degree equal to or greater than a threshold, and identifies the identified contour A region surrounded by (here, contours B2 and B4) is detected as an object region. Note that if there is no contour having a matching degree equal to or higher than the threshold value, or if the region cannot be configured only by the specified contour, the object region detection unit 24c may not detect the object region corresponding to the region 70. .

  When the object region is detected by the object region detection unit 24c, the obstacle determination unit 24d is based on the position of the object region, the magnitude and direction of the motion vector in the object region, and the moving direction and speed of the vehicle. Then, it is determined whether an object in the object area is an obstacle to the vehicle. Here, the obstacle to the vehicle is an object that obstructs the vehicle, and may include an object that can obstruct the vehicle. The motion vector detected by the motion vector detection unit 22 is used as the motion vector in the object region. Specifically, when the obstacle determination unit 24d determines that the object may collide with the vehicle based on the relationship between the relative speed between the object and the vehicle and the moving direction, the object is determined to be an obstacle. judge. For example, an object approaching the vehicle at a certain speed or more is determined to be an obstacle.

  FIG. 8 is a flowchart illustrating an example of an obstacle detection process (or an obstacle detection method including the object detection method according to Embodiment 1) by the object detection device 20. The process of FIG. 8 is repeatedly executed by the object detection device 20.

  The object detection device 20 detects the traveling state of the host vehicle (S1). The traveling state includes the moving direction and speed of the own vehicle.

  Next, the object detection apparatus 20 acquires a frame shot from the camera 10 and stores the acquired frame in the image storage unit 21a (S2).

  Next, the object detection device 20 performs a motion vector detection process using the frame acquired in step S2 as a target frame, detects a motion vector of each pixel of the target frame (S3), and uses the detected motion vector. A region where an object in the target frame is present is detected (S4).

  Next, the object detection apparatus 20 performs a difference calculation process on the frame acquired in step S2 to calculate a difference image (S5), and extracts the contour of the object from the obtained difference image (S6).

  Next, the object detection device 20 detects the object region by combining the region detection result in step S4 and the contour extraction result in step S6 (S7). For example, by superimposing the contour of the object extracted in step S6 on the region of the object detected in step S4, the region surrounded by the contour of the extracted object corresponding to the detected region of the object Specify as an area.

  Next, the object detection device 20 uses the traveling state of the vehicle detected in step S1, the position of the object region detected in step S7, and the motion vector detected in step S3 in the object region. It is determined whether an object in the object area is an obstacle of the own vehicle (S8). For example, when the object detection device 20 determines that there is a risk of collision from the relationship between the moving direction and speed of the object in the object area and the moving direction and speed of the own vehicle, it determines that the object is an obstacle. . In addition, even when the moving direction and speed of the object in the object area and the own vehicle are equal and the object and the own vehicle are moving in parallel with each other, if the moving direction or speed of the own vehicle or the object changes, the object May be an obstacle for the vehicle, so it is determined that the vehicle is an obstacle. An object moving in the direction opposite to the moving direction of the host vehicle is determined not to be an obstacle.

  When the object detection device 20 determines that the object is an obstacle (S8: Yes), the object detection device 20 notifies the obstacle notification unit 30 to that effect (S9), and when the object detection device 20 determines that the object is not an obstacle (S8). : No), the process ends.

  FIG. 9 is a flowchart showing the contents of the motion vector detection process in step S3 of FIG. Hereinafter, the motion vector detection process will be described with reference to FIG.

  First, the object detection apparatus 20 acquires a plurality of frames used for motion vector detection from the image storage unit 21a (S11). Specifically, the object detection device 20 takes out the frame acquired in the current step S2 as a target frame, and takes out the frame acquired in the previous step S2 as a reference frame.

  Next, the object detection device 20 performs resolution conversion processing for reducing the resolution of the image for each frame acquired in step S11, and generates a low-resolution image (or a reduced image) for each frame (S12).

  Next, the object detection device 20 detects a motion vector from the low-resolution image of each frame generated in step S12. Specifically, first, the low-resolution image of the target frame is divided into a plurality of blocks, and a motion vector for each block is detected (S13). Then, the motion vector for each pixel is detected using the motion vector detection result for each block (S14).

  In the above processing, the reason why the resolution of each frame is lowered and the low-resolution image is used is to suppress the calculation amount of motion vector detection. If the resolution of the shot frame is large, using the frame as it is will increase the amount of motion vector detection, but the motion vector can be detected without any problem even if the resolution is lowered. Is effective. On the other hand, when the resolution of the original frame is small, the process of step S12 may be omitted.

  Alternatively, in step S12, a plurality of low-resolution images may be generated, and motion vectors may be detected for the low-resolution images at each level in order from the lowest resolution (steps S13 and S14). Good. In this case, in the motion vector detection for the low-resolution image after the second stage, the search range of the motion vector detection can be narrowed down by using the result of the motion vector detection for the low-resolution image one stage before. The amount can be reduced.

  FIG. 10 is a flowchart showing the contents of the difference calculation process in step S5 of FIG. Hereinafter, the difference calculation process will be described with reference to FIG.

  First, the object detection apparatus 20 acquires a plurality of frames used for calculating a difference image from the image storage unit 21a (S21). For example, the object detection device 20 acquires the same frame as a plurality of frames used in the motion vector detection process in step S3 of FIG. Specifically, the object detection device 20 takes out the frame acquired in the current step S2 and the frame acquired in the previous step S2. However, the object detection apparatus 20 may use a frame acquired at a timing different from the frame used for the motion vector detection process. For example, the object detection apparatus 20 may extract the frame acquired in step S2 this time and the frame acquired in step S2 the previous time.

  Next, the object detection device 20 calculates a difference between the pixels of the two frames acquired in step S21, and generates a difference image between the two frames (S22).

  As described above, in the present embodiment, the object detection device 20 detects the object region based on the motion vector and the difference image. Thereby, compared with the case where an object region is detected based only on a motion vector, an object region can be detected more accurately. Specifically, in the configuration in which the object region is detected based only on the motion vector, the motion vector may indicate a different direction within one object, and even the background portion is erroneously detected as the object region. In some cases, the contour of the object region cannot be accurately detected, and the object region cannot be detected with high accuracy. In contrast, in the present embodiment, the object region can be accurately detected up to the contour by superimposing the object region detected from the motion vector and the contour of the object extracted from the difference image. That is, the object region can be detected with high accuracy by roughly detecting the region of the object based on the motion vector and determining the contour of this region in detail using the difference image. In addition, there is a method of performing edge detection in order to grasp the contour of an object, but this method increases the amount of calculation. On the other hand, the calculation of the difference image is a very simple calculation that only calculates the difference between the pixel values of the image, and the calculation amount is very small. For this reason, according to the present embodiment, it is possible to increase the detection accuracy of the object region without greatly increasing the amount of calculation.

  Further, in the present embodiment, when the object region is detected, the object detection device 20 determines that an object existing in the object region is relative to the vehicle based on the position of the object region and the motion vector in the object region. Determine if it is an obstacle. For this reason, an obstacle can be detected accurately.

  Note that the order of the above processes is an example, and may be changed as appropriate. For example, it is determined whether or not the object area detected from the motion vector is an obstacle, and the object area is determined from the object area determined to be an obstacle and the contour extracted from the difference image. May be detected. Hereinafter, this modification will be described.

  The obstacle determination unit 24d exists in the detected region based on the position of the detected region and the motion vector in the detected region when the region of the object is detected by the region detection unit 24a. It is determined whether the object to be performed is an obstacle to the vehicle.

  The object region detection unit 24c is extracted by the contour extraction unit 24b corresponding to the detected region when the obstacle determination unit 24d determines that the object existing in the detected region is an obstacle. A region surrounded by a contour is detected as an object region.

  FIG. 11 is a flowchart illustrating an example of an obstacle detection process performed by the object detection device 20 according to the modification. The process of FIG. 11 is repeatedly executed by the object detection device 20.

  The object detection device 20 detects the running state of the vehicle (S31), acquires a frame from the camera 10 (S32), performs a motion vector detection process (S33), similarly to steps S1 to S4 in FIG. An object region is detected using the detected motion vector (S34).

  Then, similarly to step S8 in FIG. 8, the object detection device 20 uses the traveling state of the vehicle detected in step S31, the position of the area detected in step S34, and the motion vector in the area. It is determined whether or not the object in the area is an obstacle of the own vehicle (S35).

  If the object detection device 20 determines that the object is an obstacle (S35: Yes), the process proceeds to step S36. If the object detection device 20 determines that the object is not an obstacle (S35: No), the process ends.

  In step S36, the object detection apparatus 20 calculates a difference image similarly to step S5 of FIG. And the object detection apparatus 20 extracts the outline of an object from a difference image similarly to step S6-S7 of FIG. 8 (S37), and detects an object area | region (S38). The object area detected in this case is an area where an obstacle exists. And the object detection apparatus 20 notifies the object area | region where an obstruction exists to the obstruction notification part 30 (S39).

Embodiment 2. FIG.
FIG. 12 is a block diagram illustrating an example of the configuration of the object detection system 2 including the object detection device 220 according to the second embodiment. Hereinafter, the object detection system 2 will be described with reference to FIG. In the following description, the description of the same parts as those in the first embodiment is omitted or simplified, and the same or corresponding elements as those in the first embodiment are denoted by the same reference numerals.

  As shown in FIG. 12, the object detection system 2 has substantially the same configuration as the object detection system 1 of FIG. 1, but the pixel motion vector detection unit 22b is omitted.

  In the present embodiment, the motion vector detection unit 222 detects a motion vector for each block of the target frame. Specifically, in the motion vector detection unit 222, the block motion vector detection unit 222 a detects the motion vector of each block of the target frame and outputs the detected motion vector of each block to the object detection unit 24. Detection of motion vectors in pixel units is omitted.

  The object detection unit 24 uses the block-based motion vector input from the motion vector detection unit 222 to detect an object region and determine an obstacle. The processing of the object detection unit 24 is the same as that of the first embodiment except that a block-unit motion vector is used instead of a pixel-unit motion vector.

  The operation of the object detection device 220 is substantially the same as the operation of the object detection device 20 shown in FIG. 8, but the motion vector detection process in step S3 is different.

  FIG. 13 is a flowchart showing the contents of motion vector detection processing in the second embodiment. Hereinafter, the motion vector detection process will be described with reference to FIG.

  The object detection apparatus 220 acquires a plurality of frames used for motion vector detection (S41) and performs resolution conversion processing for reducing the resolution of each frame (S42), as in steps S11 to S12 of FIG.

  Next, the object detection device 220 detects a motion vector for each block of the target frame from the low-resolution image of each frame generated in step S42 (S43). In the first embodiment, the motion vector for each pixel is detected by refining the motion vector for each block. However, in this embodiment, the motion vector for each pixel is not detected and the motion vector for each block is detected. Subsequent processing is performed using the detection result.

  As described above, in the present embodiment, the object detection device 220 detects a motion vector in units of blocks, and detects an object region using the motion vectors in units of blocks. According to this configuration, the processing load or calculation amount of motion vector detection can be reduced and the processing speed can be increased compared to the case of detecting a motion vector in units of pixels. In addition, when using a motion vector in units of blocks, it is not possible to detect the detailed shape of the object from only the motion vectors, but the detailed shape of the object is detected by using a difference image having a difference value in units of pixels. It becomes possible. Therefore, according to the present embodiment, it is possible to reduce the processing load while maintaining the accuracy of object detection.

Embodiment 3 FIG.
FIG. 14 is a block diagram illustrating an example of the configuration of the object detection system 3 including the object detection device 320 according to the third embodiment. Hereinafter, the object detection system 3 will be described with reference to FIG. In the following description, the description of the same parts as those in the first embodiment is omitted or simplified, and the same or corresponding elements as those in the first embodiment are denoted by the same reference numerals.

  As shown in FIG. 14, the object detection system 3 has substantially the same configuration as the object detection system 1 of FIG. 1, but a motion vector detection unit 322 and a difference calculation unit 323 are connected.

  In the present embodiment, the difference calculation unit 323 calculates a difference between pixel values of each pixel between two frames, generates an image having the calculated difference between each pixel as a pixel value, and generates a difference image. The difference image is output to the object detection unit 24 and is also output to the motion vector detection unit 322.

  The motion vector detection unit 322 receives the difference image from the difference calculation unit 323 and detects a motion vector except for a portion corresponding to a portion of the target frame in which the absolute value of the pixel value in the difference image is equal to or less than a threshold value. Do.

  FIG. 15 is a flowchart illustrating an example of an obstacle detection process performed by the object detection device 320 according to the third embodiment. The process of FIG. 15 is repeatedly executed by the object detection device 320.

  The object detection device 320 detects the traveling state of the vehicle (S51) and acquires a frame from the camera 10 (S52), similarly to steps S1 and S2 of FIG.

  Next, similarly to steps S5 to S6 in FIG. 8, the object detection device 320 performs a difference calculation process on the frame acquired in step S52 to calculate a difference image (S53), and the obtained difference image. The outline of the object is extracted from (S54).

  Next, the object detection device 320 performs background filter processing (or distant view filter processing) on the frame acquired in step S52 using the difference image obtained in step S53 (S55). In this background filter processing, a portion of the difference image in which the absolute value of the pixel value is equal to or less than a predetermined threshold value is detected as a background portion (or a distant view portion). Then, the background portion is removed from the frame acquired in step S52, and a frame after background removal is generated.

  Next, the object detection device 320 performs a motion vector detection process using the frame after background removal generated in step S55 as a target frame (S56). That is, the object detection device 320 detects a motion vector for the remaining part of the frame acquired in step S52 from which the background part has been removed. In the motion vector detection process in step S56, up to a pixel-unit motion vector may be calculated as in the first embodiment, or the processing may be stopped by calculating a block-unit motion vector as in the second embodiment. .

  Next, the object detection device 320 detects an area where an object exists in the target frame using the motion vector detected in step S56 (S57).

  Next, the object detection device 320 detects the object region by combining the region detection result in step S57 and the contour extraction result in step S54 (S58).

  Next, the object detection device 320 uses the traveling state of the vehicle detected in step S51, the position of the object region detected in step S58, and the motion vector detected in step S56 in the object region. It is determined whether an object in the object area is an obstacle of the own vehicle (S59).

  If the object detection device 320 determines that the object is an obstacle (S59: Yes), it notifies the obstacle notification unit 30 to that effect (S60), and if it determines that the object is not an obstacle (S59). : No), the process ends.

  As described above, in the present embodiment, the object detection device 320 detects the background portion from the difference image, and performs motion vector detection by removing the background portion from the target frame. Thereby, the load of the motion vector detection process can be reduced.

  In the first to third embodiments, the function of the object detection apparatus may be realized only by hardware resources such as an electronic circuit, or may be realized by cooperation of hardware resources and software. In the latter case, the function of the object detection device is realized by, for example, an object detection program being executed by a computer, more specifically, an object detection program recorded on a recording medium such as a ROM (Read Only Memory). Is read out to the main memory and executed by a central processing unit (CPU). The object detection program may be provided by being recorded on a computer-readable recording medium such as an optical disk, or may be provided via a communication line such as the Internet.

  In addition, this invention is not limited to the said embodiment, In the range which does not deviate from the summary of this invention, it can implement with a various aspect. For example, one of the two frames used for calculating the difference image is preferably the same frame as the target frame for motion vector detection, but the frame is captured at substantially the same timing as the target frame or sufficiently close to the target frame. It may be. Further, the object detection device may be mounted on a moving body other than the vehicle, and may detect an obstacle on the moving body. Further, the object detection device may be applied to uses other than the detection of an obstacle.

  1 to 3 object detection system, 10 cameras, 20, 220, 320 object detection device, 21 image acquisition unit, 22, 222, 322 motion vector detection unit, 23, 323 difference calculation unit, 24 object detection unit, 24a region detection unit 24b Outline extraction unit 24c Object region detection unit 24d Obstacle determination unit 30 Obstacle notification unit

Claims (9)

An image acquisition unit for acquiring a plurality of images sequentially taken by the camera;
A motion vector detection unit that detects a motion vector of each part of the target image from the target image and at least one reference image of the plurality of images acquired by the image acquisition unit;
A difference calculating unit that calculates a difference image between two images of the plurality of images acquired by the image acquiring unit;
An object detection unit for detecting an object region where an object exists based on the motion vector detected by the motion vector detection unit and the difference image calculated by the difference calculation unit ;
The object detection unit is
An area detection unit for detecting an area of an object from the target image based on the motion vector;
A contour extraction unit that extracts a contour of an object from the difference image;
An object region detection unit for detecting, as the object region, a region surrounded by the contour extracted by the contour extraction unit corresponding to the region detected by the region detection unit;
An object detection apparatus comprising:   The object detection unit detects, as the object region, a region surrounded by a contour in the difference image corresponding to a region composed of a plurality of portions having the same motion vector in the target image. The object detection apparatus according to claim 1. The camera is provided on a moving body,
When the object area is detected, the object detection unit is configured such that an object existing in the object area is an obstacle to the moving body based on the position of the object area and the motion vector in the object area. object detection apparatus according to claim 1 or 2 characterized by having a determining obstacle determining unit that determines whether. The camera is provided on a moving body,
The object detection unit is
An area detection unit for detecting an area of an object from the target image based on the motion vector;
A contour extraction unit that extracts a contour of an object from the difference image;
When an area is detected by the area detection unit, an object present in the detected area is an obstacle to the moving body based on the position of the detected area and the motion vector in the detected area. An obstacle determination unit for determining whether the object is an object,
A region surrounded by the contour extracted by the contour extraction unit corresponding to the detected region when the obstacle determination unit determines that the object existing in the detected region is an obstacle. the object detection apparatus according to claim 1 or 2, characterized in that it has a object area detection unit for detecting as the object region. The motion vector detecting section, the object detection apparatus according to claim 1, any one of 4 and detects the motion vector in units of pixels of the target image. The motion vector detecting section, the object detection apparatus according to claim 1, any one of 4 and detects the motion vector in units of blocks composed of a plurality of pixels of the target image. The difference calculation unit calculates a difference between pixel values of each pixel between the two images, generates an image having the calculated difference between the pixels as a pixel value as the difference image,
The said motion vector detection part detects the said motion vector except the part corresponding to the part whose absolute value of the pixel value in the said difference image is below a threshold value among the said target images. Item 7. The object detection device according to any one of Items 1 to 6 . An image acquisition step of acquiring a plurality of images sequentially taken by the camera;
A motion vector detection step of detecting a motion vector of each part of the target image from the target image and at least one reference image of the plurality of images acquired in the image acquisition step;
A difference calculating step of calculating a difference image between two images of the plurality of images acquired in the image acquiring step;
A motion vector detected by said motion vector detection step, the based on the difference image calculated by the difference calculation step, seen including a object detection step of detecting an object area where an object is present,
The object detection step includes
A region detecting step of detecting a region of an object from the target image based on the motion vector;
A contour extracting step of extracting a contour of the object from the difference image;
An object region detection step for detecting, as the object region, a region surrounded by the contour extracted in the contour extraction step corresponding to the region detected in the region detection step;
An object detection method characterized by comprising: An image acquisition step of acquiring a plurality of images sequentially taken by the camera;
A motion vector detection step of detecting a motion vector of each part of the target image from the target image and at least one reference image of the plurality of images acquired in the image acquisition step;
A difference calculating step of calculating a difference image between two images of the plurality of images acquired in the image acquiring step;
Based on the motion vector detected in the motion vector detection step and the difference image calculated in the difference calculation step, causing the computer to execute an object detection step of detecting an object region where an object exists ,
The object detection step includes
A region detecting step of detecting a region of an object from the target image based on the motion vector;
A contour extracting step of extracting a contour of the object from the difference image;
An object region detection step for detecting, as the object region, a region surrounded by the contour extracted in the contour extraction step corresponding to the region detected in the region detection step;
An object detection program characterized by comprising:

Images