JP6384082B2 - Moving object detection apparatus, moving object detection method, and program - Google Patents

Description

  The present invention relates to a moving object detection device, a moving object detection method, and a program.

  Providing “cars that do not cause accidents” in the automobile industry is one of the ultimate goals. In order to achieve this goal, there has been proposed a vehicle equipped with a monitor device for confirming a blind spot on the front or left and right when starting at a parking lot or an intersection with poor visibility. In addition, a vehicle equipped with a back camera has been proposed as a monitor device for confirming the rear of the vehicle when starting backward in a parking lot.

  The monitor device for confirming the blind spot uses a prism mirror or a fisheye camera to capture an image of the front lower part or the side of the vehicle that is difficult for the driver to see and displays the image on the screen. The back camera also captures a wide-angle image behind the vehicle using a fisheye lens or the like and displays it on the screen.

  FIG. 1 shows an example of a monitor device using a back camera. When the vehicle 101 starts moving backward, the back camera 102 images the imaging range 103 behind the vehicle 101 and displays it on the screen as shown in FIG. Thereby, the driver can recognize the vehicle 104 entering the imaging range 103 from the rear left hand and avoid a collision.

  Furthermore, a moving object detection system that suppresses an accident by notifying a driver of the danger of a collision with another object from the situation around the vehicle has been proposed (see, for example, Non-Patent Document 1). In order to support safety confirmation by the driver, this moving object detection system performs image processing on the image captured by the camera, and notifies the driver to call attention when a moving object is detected.

  An optical flow is often used for detecting a moving object. In the moving object detection method based on the optical flow, the feature amount of each pixel in the image is calculated, the pixel having the feature amount that meets a predetermined condition is recognized as a feature point, and the temporal variation of the feature point is detected in the image. It is calculated as the movement of the feature point. As the predetermined condition, for example, that the feature amount is equal to or greater than a certain value is used.

  When the vehicle is stationary, all the feature points of the stationary object in the image are stationary. Therefore, when the movement of the feature point (flow) is detected, the feature point can be regarded as the feature point of the moving object. it can. The flow of a moving object that moves away from the vehicle is detected as an outward flow that goes outward from the center of the image, and the flow of a moving object that approaches the vehicle is detected as an inward flow that goes from the outside of the image to the center. By detecting such a flow, it can be determined whether or not a moving object is shown in the video.

  On the other hand, when the vehicle is moving, the feature point of the stationary object is moved accordingly. For example, when the vehicle is traveling straight ahead of the camera, all flows of a stationary object, a moving object moving away from the vehicle, and a moving object approaching the vehicle may be detected as an outward flow. Even in this case, it is possible to determine whether the feature point is a stationary object or a moving object by determining whether the movement of the feature point is equal to that caused by the movement of the vehicle.

  There is also known an image processing system in which a mask area is set as a non-detection area in an image and image processing is not performed in the mask area (see, for example, Patent Document 1).

  There is also known a collision prevention device that searches for a limited detection area of an image obtained by imaging the situation around a moving body, detects other moving bodies, predicts the possibility of a collision between both moving bodies, and issues an alarm. (For example, refer to Patent Document 2).

  A vehicle image processing apparatus that determines that an object corresponding to a stereoscopic feature area is a planar body when the stereoscopic feature impossible area includes a stereoscopic feature area in which pixels having common movement information are continuous in the vertical direction is known. (For example, see Patent Document 3).

JP 2010-9134 A JP 2000-285245 A JP 2009-187181 A

“Moving object detection”, [online], Nissan Motor Co., Ltd., [Search on February 14, 2014], Internet <URL: http://www.nissan-global.com/JP/TECHNOLOGY/OVERVIEW/mod. html>

The conventional moving object detection method described above has the following problems.
According to the moving object detection method based on the optical flow, even when the vehicle is moving, if the direction of movement of the vehicle (forward, backward, right turn, left turn, etc.) and the speed are known, it can be roughly determined. The background motion vector is known. If the background motion vector is known, it is possible to separate the background movement and other moving objects as the vehicle moves.

However, if the tracking of the feature points fails, even the feature points of the stationary object may show a motion different from the background motion vector, and the background stationary object may be erroneously detected as a moving object. For example, in a scene with a wire mesh lid (grating) on the background ground, if the vehicle goes straight or turns, mistracking may occur for the following reasons.
(1) At the intersection where the vertical line and the horizontal line of the mesh intersect, the edge feature amount in both the vertical direction and the horizontal direction is large, so that it is easily extracted as a feature point by the corner extraction operator.
(2) The net is a geometric pattern in which the same shape is repeated, and there are a plurality of similar intersections in the vicinity of an intersection. Therefore, when tracking a feature point, another intersection is mistakenly identified as the same intersection. Mistaken tracking is likely to occur.

  As described above, when a feature point is mistracked, a flow different from the background motion vector is generated, so that it is determined that a moving object exists even though the moving object does not exist.

  Such a problem occurs not only when the background includes a grating but also when another geometric pattern is included. In addition, such a problem occurs not only in a vehicle traveling on a road surface but also in other moving bodies that move in space.

  In one aspect, an object of the present invention is to suppress erroneous detection when a moving object is detected from an image captured by an imaging device.

In one plan, the moving object detection device includes a storage unit and a determination unit.
The storage unit stores an image captured by the imaging device. The determination unit refers to the storage unit to determine whether or not the first region in the image is correlated with the second region existing around the first region. When the first area is correlated with the second area, the determination unit distinguishes the first area from the third area other than the first area in the image in detecting the moving object based on the image. Control information to output.

  According to the moving object detection device of the embodiment, it is possible to suppress erroneous detection when a moving object is detected from an image captured by the imaging device.

It is a figure which shows the monitor apparatus using a back camera. It is a figure which shows the image of a monitor apparatus. It is a flowchart of an optical flow calculation process. It is a figure which shows the mistracking of a feature point. It is a functional block diagram of a moving object detection apparatus. It is a flowchart of an image process. It is a figure which shows the 1st specific example of a moving object detection apparatus. It is a functional block diagram of a determination part. It is a flowchart of a 1st moving object detection process. It is a flowchart of a tracking difficult area | region specific process. It is a figure which shows the vicinity area | region of an object pixel. It is a figure which shows the difference degree of a near region. It is a figure which shows the 2nd specific example of a moving object detection apparatus. It is a flowchart of a 2nd moving object detection process. It is a figure which shows the 3rd specific example of a moving object detection apparatus. It is a flowchart of a 3rd moving object detection process. It is a figure which shows the 4th specific example of a moving object detection apparatus. It is a flowchart of a 4th moving object detection process. It is a hardware block diagram of information processing apparatus.

Hereinafter, embodiments will be described in detail with reference to the drawings.
According to the moving object detection method based on the optical flow, even when the vehicle is moving, if the direction of movement of the vehicle (forward, backward, right turn, left turn, etc.) and the speed are known, it can be roughly determined. The background motion vector is known. If the background motion vector is known, it is possible to separate the background movement and other moving objects as the vehicle moves.

  FIG. 3 is a flowchart illustrating an example of an optical flow calculation process performed by the image processing apparatus. First, the image processing apparatus extracts a corner pixel of an object from an image of a target frame included in a video using a corner extraction operator, and records the extracted pixel as a feature point (step 301).

  Next, the image processing apparatus compares the region including the feature point in the image of the target frame with the image of the subsequent frame, and calculates where the extracted feature point is located in the image of the subsequent frame. The feature points are tracked (step 302). For example, when the target frame is the i-th frame (i is an integer equal to or greater than 1), frames captured after a predetermined interval from the target frame, such as the i + 1-th frame and the i + 2-th frame, are used as subsequent frames. Used.

  Next, the image processing apparatus determines whether or not the tracking end condition is satisfied (step 303). When the tracking end condition is not satisfied (step 303, NO), the subsequent frame is used as a new target frame. Step 302 is repeated. When the tracking end condition is satisfied (step 303, YES), the image processing apparatus ends the process.

  By tracking the position of the feature point in the image at a predetermined interval, the motion of the object having the feature point can be obtained based on the temporal variation of the position of the feature point. Then, the moving object can be detected by separating the movement of the feature point of the stationary object and the movement of the feature point of the moving object based on the motion vector of the background.

However, if the tracking of the feature points fails, even the feature points of the stationary object may show a motion different from the background motion vector, and the background stationary object may be erroneously detected as a moving object. For example, in a scene with a wire mesh lid (grating) on the background ground, if the vehicle goes straight or turns, mistracking may occur for the following reasons.
(1) At the intersection where the vertical line and the horizontal line of the mesh intersect, the edge feature amount in both the vertical direction and the horizontal direction is large, so that it is easily extracted as a feature point by the corner extraction operator.
(2) The net is a geometric pattern in which the same shape is repeated, and there are a plurality of similar intersections in the vicinity of an intersection. Therefore, when tracking a feature point, another intersection is mistakenly identified as the same intersection. Mistaken tracking is likely to occur.

  FIG. 4 shows an example of such mistracking. When the image of the grating 411 is included in the target frame 401, a plurality of intersections of the grating 411 are extracted as feature points. When one intersection point 412 that is separated from the lower end of the target frame 401 by the distance D among the intersection points becomes a tracking target, a neighboring region 414 at a position that is separated from the lower end by the distance D in the image of the subsequent frame 402. The corresponding intersection point is searched for.

  As a result, instead of the intersection 412, another intersection 413 existing in the neighborhood 414 may be erroneously recognized as the intersection 412. In this case, instead of the correct motion vector 415 at the intersection 412, a motion vector 416 toward the intersection 413 is generated as a flow of the intersection 412.

  As described above, when a feature point is mistracked, a flow different from the background motion vector is generated, so that it is determined that a moving object exists even though the moving object does not exist.

  Such a problem occurs not only when the background includes a grating but also when another geometric pattern is included. In addition, such a problem occurs not only in a vehicle traveling on a road surface but also in other moving bodies that move in space.

  FIG. 5 illustrates a functional configuration example of the moving object detection device of the embodiment. A moving object detection device 501 in FIG. 5 includes a storage unit 511 and a determination unit 512. The storage unit 511 stores an image captured by the imaging device. The determination unit 512 performs image processing according to the embodiment with reference to the storage unit 511.

  FIG. 6 is a flowchart illustrating an example of image processing performed by the moving object detection device 501 in FIG. First, the determination unit 512 refers to the storage unit 511 to determine whether or not the first region in the image correlates with the second region existing around the first region (step 601). . When the first region is correlated with the second region, the determination unit 512 determines the first region and the third region other than the first region in the image in the detection of the moving object based on the image. Control information for distinguishing is generated, and the control information is output (step 602).

  According to such image processing, it is possible to suppress erroneous detection when a moving object is detected from an image captured by the imaging device.

  The moving object detection device 501 performs image processing using, for example, an image captured by an imaging device such as a camera mounted on a moving body. The moving body on which the imaging device is mounted may be an object that moves on the road surface, such as a vehicle, or may be a robot, a railway vehicle, an aircraft, a ship, or an animal.

  The moving object detection device 501 may be a device mounted on a moving body such as an in-vehicle device, or may be a device not mounted on a moving body such as a server.

  FIG. 7 shows a first specific example of the moving object detection device 501 of FIG. The moving object detection device 701 in FIG. 7 includes a storage unit 511, a determination unit 512, a feature point extraction unit 711, a tracking unit 712, and a detection unit 713.

  The storage unit 511 stores video captured by the imaging device. The video includes a plurality of images corresponding to a plurality of frames.

  As shown in FIG. 8, the determination unit 512 includes a region extraction unit 801, a neighborhood region setting unit 802, a correlation calculation unit 803, and a region determination unit 804. The area extracting unit 801 extracts pixels in the image of the target frame from among a plurality of frames included in the video, and extracts an image area including the pixels as a template area. The neighborhood area setting unit 802 sets an image area including the template area as a neighborhood area of the extracted pixel.

  The correlation calculation unit 803 calculates an index indicating whether or not the region in the neighborhood region and the template region are correlated. The region determination unit 804 determines a region correlated with the template region as a difficult tracking region based on the calculated index, and generates control information for distinguishing the difficult tracking region from a region other than the difficult tracking region. Output.

  Mistracking is likely to occur when another image area having similar image information exists in the vicinity of the image area including the feature point to be tracked. There is a high probability that tracking will fail. Therefore, it is possible to suppress the occurrence of mistracking by determining such a region as a difficult-to-track region and excluding it from the feature point extraction target.

  Based on the control information output from the determination unit 512, the feature point extraction unit 711 extracts feature points from regions other than the difficult-to-track region among the regions in the image of each frame included in the video. The tracking unit 712 tracks a feature point extracted from a subsequent frame at a time later than the target frame, and calculates a temporal variation in the position of the feature point. The detection unit 713 detects an image region including a moving object from the time variation of the position of the feature point, and outputs a detection result.

  The hard-to-track area is only a part of the image of the target frame and is often an image area of a background stationary object. Therefore, even if the hard-to-track area is excluded from the detection target, the detection accuracy of the moving object is remarkably high. It is not expected to decrease.

  FIG. 9 is a flowchart illustrating an example of a moving object detection process performed by the moving object detection device 701 in FIG. First, the determination unit 512 identifies a difficult-to-track region in the image of each frame included in the video, and outputs control information for distinguishing between the difficult-to-track region and a region other than the difficult-to-track region (step 901).

  Next, based on the control information, the feature point extraction unit 711 extracts feature points from regions other than the hard-to-track regions among the regions in the image of each frame included in the video (step 902). For the feature point extraction processing, a corner extraction operator such as a Harris operator or Kanade-Lucas-Tomasi (KLT) method can be used.

  Next, the tracking unit 712 tracks the feature points by calculating where each feature point extracted from the target frame is located in the image of the subsequent frame (step 903). As the subsequent frame, for example, a frame captured after a predetermined interval from the target frame is used.

  Next, the tracking unit 712 determines whether or not the tracking end condition is satisfied (step 904), and when the tracking end condition is not satisfied (step 904, NO), the subsequent frame is used as a new target frame. Step 903 is repeated. An optical flow or the like can be used for the tracking process, and the tracking end condition may be that the number of tracked frames has reached a predetermined value or that a predetermined frame has been processed as a subsequent frame. it can.

  When the tracking end condition is satisfied (step 904, YES), the tracking unit 712 ends the tracking of the feature points. Note that the processing in step 903 and step 904 is performed on all extracted feature points. The detection unit 713 detects an image area including a moving object from the time variation of the position of each feature point, and outputs a detection result (step 905).

  For example, when the time variation of the position of the feature point represents a predetermined vector, the detection unit 713 can determine that the feature point is included in the image area of the moving object. As the predetermined vector, for example, a vector that is different from the movement vector of the moving body itself on which the imaging device is mounted can be used.

  A control device for preventing danger may be provided at the subsequent stage of the moving object detection device 701. Based on the detection result output from the moving object detection device 701, this control device can notify the driver of the danger by voice or video, or can activate the emergency brake of the vehicle.

  FIG. 10 is a flowchart illustrating an example of the difficult-to-track region specifying process performed by the determination unit 512 in step 901 of FIG. First, as shown in FIG. 11, the region extraction unit 801 selects one pixel in the image of the target frame as the target pixel 1101 (step 1001). Then, the area extraction unit 801 scans pixels around the target pixel 1101 and extracts an image area of m × n pixels (m and n are integers of 1 or more) including the target pixel 1101 as the template area 1102. . For example, raster scanning can be used for pixel scanning.

  Next, the neighborhood area setting unit 802 sets the neighborhood area 1103 of the target pixel including the template area 1102 (step 1002). Then, the correlation calculation unit 803 calculates the difference X between the image area 1104 having the same shape as the template area 1102 in the neighboring area 1103 and the template area 1102. As the degree of difference X, for example, an index indicating a difference in image information between two regions, such as a sum of absolute differences (SAD) and a sum of squared differences (SSD), can be used.

  When SAD is used, the dissimilarity X can be obtained by the following equation.

  Ax, y in Expression (1) represents pixel values of pixels at coordinates (x, y) in the image area 1104 (x = 1 to m, y = 1 to n). Bx, y represents the pixel value of the pixel at the same position in the template area 1102. As the pixel value, for example, luminance, color (RGB, etc.), color difference signal, or the like can be used.

  When SSD is used, the dissimilarity X can be obtained by the following equation.

  The degree of difference X increases as the image information in the image area 1104 differs from the image information in the template area 1102, and approaches 0 as the two image information are similar. Therefore, the larger the difference degree X, the lower the correlation between the image area 1104 and the template area 1102, and the smaller the difference degree X, the higher the correlation between the image area 1104 and the template area 1102.

  A plurality of dissimilarities X are calculated by repeating the calculation of the dissimilarity X using each pixel in the neighborhood area 1103 as the central pixel 1105 of the image area 1104. However, when the target pixel 1101 is used as the central pixel 1105, the image area 1104 matches the template area 1102, and in this case, the difference X is excluded from the calculation.

  In addition, an area close to the target pixel 1101 is likely to be similar to the template area 1102 regardless of whether or not it is an image area of a geometric pattern. Therefore, such an area is also excluded from the calculation of the dissimilarity X. It is desirable. Therefore, for example, the dissimilarity X is calculated using a pixel that is a predetermined distance or more away from the target pixel 1101 as the central pixel 1105.

  Next, the region determination unit 804 compares each of the plurality of calculated dissimilarities X with a threshold (step 1003), and if any dissimilarity X is smaller than the threshold (step 1003, YES), the target pixel 1101 Is determined to belong to a difficult-to-follow area (step 1004). On the other hand, when all the dissimilarities X are equal to or greater than the threshold (NO in step 1003), the region determination unit 804 determines that the target pixel 1101 does not belong to the difficult-to-track region.

  Improving the detection accuracy of a difficult-to-track region by determining whether or not it is a difficult-to-track region based on the difference X between the image region 1104 and the template region 1102 that are separated from the template region 1102 by a predetermined distance or more. Can do.

  Next, the area determination unit 804 checks whether or not the processing of all the pixels in the image of the target frame has been completed (step 1005). When there is an unprocessed pixel (step 1005, NO), the determination unit 512 repeats the processing after step 1001 for the next pixel.

  Then, when the processing of all the pixels is completed (step 1005, YES), the region determination unit 804 generates and outputs control information indicating a set of pixels determined to belong to the difficult-to-track region (step 1006). A set of pixels indicated by the control information represents a difficult-to-track area in the image of the target frame, and is used to distinguish the difficult-to-track area from areas other than the difficult-to-track area.

  For example, when an image of a geometric pattern such as a grating is included in the neighborhood area 1103, since the same shape exists around the target pixel 1101, the dissimilarity X tends to be small. For this reason, there is a high possibility that the pixels in the image area of the geometric pattern belong to the difficult tracking area. In addition to the grating, there is a high possibility that a region including an image such as a fence, a brick road surface, and a pedestrian crossing belongs to a difficult-to-follow region.

  FIG. 12 shows an example of the degree of difference X calculated when the size of the template area 1102 is 5 × 5 pixels and the size of the neighboring area 1103 is 15 × 11 pixels. In this example, an area of 5 × 5 pixels close to the target pixel 1101 is excluded from the calculation of the dissimilarity X, and 140 dissimilarities X are calculated. When the threshold is 440, the difference X indicated by the mark * among the calculated differences X is smaller than the threshold. Therefore, it is determined that the target pixel 1101 belongs to the difficult tracking area.

  By the way, the neighborhood area 1103 does not need to be an area centered on the template area 1102, and may be an area biased in a predetermined direction such as left, right, up, down, or the like. For example, when a vehicle on which an imaging device is mounted is moving, an area corresponding to the moving destination is included in the image of the subsequent frame and is likely to be a tracking target. Therefore, it is desirable to set the neighborhood region 1103 based on the moving direction of the imaging device.

  For example, the neighborhood area 1103 is set so that it includes a large area above the template area 1102 when the vehicle is traveling straight, and includes a large area below the template area 1102 when the vehicle is moving backward. Also good. In addition, a neighborhood area 1103 is set so that it includes many areas on the right side of the template area 1102 when the vehicle is turning right, and includes many areas on the left side of the template area 1102 when the vehicle is turning left. Also good.

  In this way, by setting the neighborhood area 1103 based on the moving direction of the imaging apparatus, it is possible to suppress the occurrence of erroneous tracking in the area corresponding to the movement destination.

  FIG. 13 shows a second specific example of the moving object detection device 501 of FIG. A moving object detection device 1301 of FIG. 13 includes a storage unit 511, a determination unit 512, a feature point extraction unit 711, a tracking unit 712, and a detection unit 713.

  FIG. 14 is a flowchart illustrating an example of a moving object detection process performed by the moving object detection device 1301 of FIG. First, the determination unit 512 identifies a difficult-to-track region in the image of each frame stored in the storage unit 511, and outputs control information for distinguishing the difficult-to-follow region from regions other than the difficult-to-follow region (step 1401). ). Next, the feature point extraction unit 711 extracts feature points from the image of each frame (step 1402).

  Next, based on the control information, the tracking unit 712 tracks feature points extracted from regions other than the hard-to-track region among the regions in the image of the target frame by the first tracking process. The extracted feature points are tracked by the second tracking process (step 1403). At this time, the tracking unit 712 calculates the time variation of each feature point by calculating where each feature point is located in the image of the subsequent frame.

  As the second tracking process, for example, a tracking process that is less likely to cause erroneous tracking of feature points than the first tracking process is used. For example, when the first tracking process performs tracking by template matching using the first template, the second tracking process tracks by template matching using the second template having a larger area than the first template. It can be performed. By using the second template having a larger area when tracking the feature points of the hard-to-track region, a region in which no geometric pattern exists is included in the second template, which is shown in FIG. Such mistracking is less likely to occur.

  As described above, it is possible to suppress the occurrence of mistracking by switching the tracking process according to whether or not the feature point is included in the difficult-to-track region.

  Next, the tracking unit 712 determines whether or not the tracking end condition is satisfied (step 1404). If the tracking end condition is not satisfied (NO in step 1404), the subsequent frame is used as a new target frame. Step 1403 is repeated.

  When the tracking end condition is satisfied (step 1404, YES), the tracking unit 712 ends the feature point tracking. Note that the processing of step 1403 and step 1404 is performed on all extracted feature points. The detection unit 713 detects an image area including a moving object from the time variation of the position of each feature point, and outputs a detection result (step 1405).

  FIG. 15 shows a third specific example of the moving object detection device 501 of FIG. 15 includes a storage unit 511, a determination unit 512, a feature point extraction unit 711, a tracking unit 712, a detection unit 713, and a selection unit 1511.

  FIG. 16 is a flowchart illustrating an example of a moving object detection process performed by the moving object detection device 1501 of FIG. First, the feature point extraction unit 711 extracts feature points from each frame image stored in the storage unit 511 (step 1601). Next, the tracking unit 712 tracks the feature points by calculating where each feature point extracted from the target frame is located in the image of the subsequent frame (step 1602).

  Next, the tracking unit 712 determines whether or not the tracking end condition is satisfied (step 1603). When the tracking end condition is not satisfied (step 1603, NO), the subsequent frame is used as a new target frame. Step 1602 is repeated.

  If the tracking end condition is satisfied (step 1603, YES), the tracking unit 712 ends the feature point tracking. Note that the processing in step 1602 and step 1603 is performed on all extracted feature points. The detection unit 713 detects an image area including a moving object from the time variation of the position of each feature point, and outputs it as a detection area (step 1604).

  Next, the determination unit 512 specifies a difficult tracking area among the detection areas, and outputs control information for distinguishing the difficult tracking area from a region other than the difficult tracking area (step 1605). At this time, the determination unit 512 can identify the difficult tracking region by performing the difficult tracking region specifying process of FIG. 10 using the pixels in the detection region output from the detection unit 713 as target pixels.

  Based on the control information, the selection unit 1511 selects a region other than the hard-to-track region from the detection regions, and outputs the selected detection region as a detection result. By excluding the difficult tracking area from the detection result, it is possible to suppress erroneous detection.

  According to the moving object detection device 1501 in FIG. 15, the target of the difficult tracking area specifying process can be limited to the detection region of the moving object. Therefore, compared with the moving object detection device 701 in FIG. 7 or the moving object detection device 1301 in FIG. Thus, the load of the difficult tracking area specifying process is reduced.

  FIG. 17 shows a fourth specific example of the moving object detection device 501 of FIG. A moving object detection apparatus 1701 in FIG. 17 includes a storage unit 511, a determination unit 512, a feature point extraction unit 711, a tracking unit 712, a detection unit 713, and a selection unit 1711.

  FIG. 18 is a flowchart illustrating an example of a moving object detection process performed by the moving object detection device 1701 in FIG. First, the feature point extraction unit 711 extracts feature points from each frame image stored in the storage unit 511 (step 1801).

  Next, the determination unit 512 identifies feature points belonging to the hard-to-track region among the extracted feature points, and outputs control information for distinguishing the hard-to-track region from regions other than the hard-to-track region (step 1802). ). At this time, the determination unit 512 uses the pixel corresponding to each feature point output from the feature point extraction unit 711 as a target pixel, and performs the difficult tracking region specifying process in FIG. Can be specified.

  Based on the control information, the selection unit 1711 selects a feature point belonging to a region other than the hard-to-track region from the extracted feature points, and outputs the selected feature point to the tracking unit 712.

  Next, the tracking unit 712 tracks the feature point by calculating where each feature point other than the difficult-to-track region in the target frame is located in the image of the subsequent frame (step 1803). By excluding the difficult tracking area from the target of the tracking process, it becomes possible to suppress the occurrence of erroneous tracking.

  Next, the tracking unit 712 determines whether or not the tracking end condition is satisfied (step 1804). If the tracking end condition is not satisfied (NO in step 1804), the subsequent frame is used as a new target frame. Step 1803 is repeated.

  When the tracking end condition is satisfied (step 1804, YES), the tracking unit 712 ends the tracking of the feature points. Note that the processing in step 1803 and step 1804 is performed for all feature points belonging to a region other than the hard-to-track region. The detection unit 713 detects an image region including a moving object from the time variation of the position of each feature point, and outputs a detection result (step 1805).

  According to the moving object detection device 1701 in FIG. 17, since the target of the difficult tracking area specifying process can be limited to the feature points, tracking is performed as compared with the moving object detection device 701 in FIG. 7 or the moving object detection device 1301 in FIG. 13. The load of the difficult area specifying process is reduced. Furthermore, it can be expected that the load is reduced as compared with the moving object detection device 1501 of FIG.

  The moving object detection device 501 in FIG. 5, the moving object detection device 701 in FIG. 7, the moving object detection device 1301 in FIG. 13, the moving object detection device 1501 in FIG. 15, and the moving object detection device 1701 in FIG. Not too much. Depending on the application and conditions of the moving object detection apparatus, some components may be omitted or changed.

  For example, in the moving object detection device 701, the moving object detection device 1301, or the moving object detection device 1501, the feature point extraction unit 711, the tracking unit 712, and the detection unit 713 may be mounted as hardware circuits outside the device. It is done. In this case, the feature point extraction unit 711, the tracking unit 712, and the detection unit 713 can be omitted.

  In the case where the moving object detection device 701, the moving object detection device 1301, the moving object detection device 1501, or the moving object detection device 1701 is mounted on a moving object, an imaging device that captures an image stored in the storage unit 511 is used. May be included.

  The flowcharts shown in FIGS. 6, 9, 10, 14, 16, and 18 are merely examples, and some processes may be omitted or changed depending on the configuration and conditions of the moving object detection device. Good. For example, in step 1002 of FIG. 10, the correlation calculation unit 803 obtains a correlation value X smaller than the threshold value instead of calculating the dissimilarity X using each pixel in the neighboring area 1103 as the central pixel 1105. The calculation for the remaining pixels can be omitted. In this case, it is determined that the target pixel 1101 belongs to the difficult tracking area.

  In step 1002, the correlation calculation unit 803 can calculate the similarity between the two areas instead of calculating the degree of difference X between the image area 1104 and the template area 1102.

  The similarity increases as the image information in the image area 1104 and the image information in the template area 1102 are similar, and decreases as the two image information differ. Therefore, the larger the similarity is, the higher the correlation between the image region 1104 and the template region 1102 is. The smaller the similarity is, the lower the correlation between the image region 1104 and the template region 1102 is.

  In this case, in step 1003, the region determination unit 804 compares each of the plurality of calculated similarities with a threshold value, and determines that the target pixel 1101 belongs to a difficult-to-track region if any of the similarities is greater than the threshold value. To do. On the other hand, when all the similarities are equal to or less than the threshold value, the area determination unit 804 determines that the target pixel 1101 does not belong to the difficult-to-track area.

  Further, in step 1003, instead of comparing a plurality of dissimilarities X or a plurality of similarities with a threshold, a statistical value of the dissimilarity X or the similarity may be compared with a threshold. As the statistical value, for example, an average value, a maximum value, a minimum value, a mode value, or the like can be used.

  The template area 1102, the neighborhood area 1003, and the image area 1104 shown in FIG. 11 are merely examples, and may be changed according to the configuration and conditions of the moving object detection apparatus. For example, instead of a rectangular area, an area having another shape such as a polygon, a circle, or an ellipse can be used.

  The moving object detection device 501 in FIG. 5, the moving object detection device 701 in FIG. 7, the moving object detection device 1301 in FIG. 13, the moving object detection device 1501 in FIG. 15, and the moving object detection device 1701 in FIG. Can be implemented as Also, some or all of the components of each moving object detection device can be realized using an information processing device (computer).

  For example, the moving object detection apparatus 701 in FIG. 7 includes an information processing apparatus including a storage unit 511 and a determination unit 512, and a hardware circuit including a feature point extraction unit 711, a tracking unit 712, and a detection unit 713. Also good. In this case, the difficult-to-follow area specifying process by the information processing apparatus is executed as a pre-process of the moving object detection process by the hardware circuit. Similarly, the moving object detection device 1301 of FIG. 13 may include an information processing device and a hardware circuit.

  15 includes an information processing apparatus including a storage unit 511, a determination unit 512, and a selection unit 1511, and a hardware circuit including a feature point extraction unit 711, a tracking unit 712, and a detection unit 713. May be included. In this case, the difficult-to-follow area specifying process by the information processing apparatus is executed as a post-process of the moving object detection process by the hardware circuit.

  When the feature point extraction unit 711, the tracking unit 712, and the detection unit 713 are implemented as a graphics processing unit (GPU), it is difficult to distinguish and process pixels in a difficult tracking region and pixels in a region other than the difficult tracking region. There are cases. In such a case, it is effective to adopt the configuration shown in FIG. 15 and implement the difficult tracking area specifying process as a post-process.

  The moving object detection device 1701 in FIG. 17 may be realized using only an information processing device. When the feature point extraction unit 711, the tracking unit 712, and the detection unit 713 are realized using an information processing device, it is effective to employ the configuration of FIG. 17 because the moving object detection process can be customized.

  For example, an information processing apparatus as shown in FIG. 19 is used as a part or all of the components of the moving object detection apparatus 501 in FIG. 5, the moving object detection apparatus 701 in FIG. 7, and the moving object detection apparatus 1301 in FIG. Is feasible. Some or all of the components of the moving object detection device 1501 in FIG. 15 and the moving object detection device 1701 in FIG. 17 can also be realized using the information processing apparatus in FIG.

  19 includes a Central Processing Unit (CPU) 1901, a memory 1902, an input device 1903, an output device 1904, an auxiliary storage device 1905, a medium drive device 1906, and a network connection device 1907. These components are connected to each other by a bus 1908.

  The memory 1902 is, for example, a semiconductor memory such as a read only memory (ROM), a random access memory (RAM), or a flash memory, and stores programs and data used for processing. The memory 1902 can be used as the storage unit 511.

  The CPU 1901 (processor) operates as the determination unit 512, the feature point extraction unit 711, the tracking unit 712, or the detection unit 713, for example, by executing a program using the memory 1902, and performs a moving object detection process. The CPU 1901 also operates as the region extraction unit 801, the neighborhood region setting unit 802, the correlation calculation unit 803, the region determination unit 804, the selection unit 1511, or the selection unit 1711.

  The input device 1903 is, for example, a keyboard, a pointing device, or the like, and is used for inputting an instruction or information from a user or an operator. The output device 1904 is, for example, a display device, a printer, a speaker, or the like, and is used to output an inquiry to a user or an operator or a processing result. The output processing result includes, for example, information indicating the detection result of the moving object.

  The auxiliary storage device 1905 is, for example, a magnetic disk device, an optical disk device, a magneto-optical disk device, a tape device, or the like. The auxiliary storage device 1905 may be a hard disk drive. The information processing apparatus can store programs and data in the auxiliary storage device 1905 and load them into the memory 1902 for use.

  The medium driving device 1906 drives a portable recording medium 1909 and accesses the recorded contents. The portable recording medium 1909 is a memory device, a flexible disk, an optical disk, a magneto-optical disk, or the like. The portable recording medium 1909 may be a compact disk read only memory (CD-ROM), a digital versatile disk (DVD), a universal serial bus (USB) memory, or the like. A user or an operator can store programs and data in the portable recording medium 1909 and load them into the memory 1902 for use.

  As described above, a computer-readable recording medium that stores a program and data used for the moving object detection processing includes physical (non-temporary) such as the memory 1902, the auxiliary storage device 1905, and the portable recording medium 1909. Recording medium).

  The network connection device 1907 is a communication interface that is connected to a communication network such as a local area network (LAN) or the Internet and performs data conversion accompanying communication. The information processing apparatus can receive a processing request and video from the user terminal via the network connection apparatus 1907, and can transmit information indicating the detection result of the moving object to the user terminal. The information processing apparatus can also receive a program and data from an external apparatus via the network connection apparatus 1907 and load them into the memory 1902 for use.

  Note that the information processing apparatus does not have to include all the components illustrated in FIG. 19, and some components may be omitted depending on the application and conditions. For example, when the information processing apparatus receives a processing request and video from a user terminal via a communication network, the input device 1903 and the output device 1904 may be omitted. Further, if the information processing apparatus does not access the portable recording medium 1909, the medium driving apparatus 1906 may be omitted, and if the information processing apparatus is not connected to the communication network, the network connection apparatus 1907 may be omitted. .

  Although the disclosed embodiments and their advantages have been described in detail, those skilled in the art can make various modifications, additions and omissions without departing from the scope of the present invention as explicitly set forth in the claims. Let's go.

DESCRIPTION OF SYMBOLS 101 Vehicle 102 Back camera 103 Imaging range 401 Target frame 402 Subsequent frame 411 Grating 412, 413 Intersection 414, 1103 Neighborhood area 415, 416 Motion vector 501, 701, 1301, 1501, 1701 Moving object detection device 511 Storage unit 512 Determination unit 711 Feature point extraction unit 712 Tracking unit 713 Detection unit 801 Region extraction unit 802 Neighboring region setting unit 803 Correlation calculation unit 804 Region determination unit 1101 Target pixel 1102 Template region 1104 Image region 1105 Central pixel 1511, 1711 Selection unit 1901 CPU
1902 Memory 1903 Input device 1904 Output device 1905 Auxiliary storage device 1906 Medium drive device 1907 Network connection device 1908 Bus 1909 Portable recording medium

Claims (10)

A storage unit for storing an image captured by the imaging device;
Whether a target pixel is selected from the pixels in the image, a first region including the target pixel is extracted, and the first region correlates with a second region existing around the first region If the first region is correlated with the second region, it is determined that the target pixel belongs to the third region, and the first region is the second region. If not correlated, the target pixel is determined to belong to a fourth region other than the third region , and a plurality of targets determined to belong to the third region in detection of a moving object based on the image a determination unit which outputs control information for distinguishing a plurality of target pixels which are determined to belong to the pixel fourth region,
A moving object detection apparatus comprising:   The determination unit determines that the first area is correlated with the second area when the image information of the first area is similar to the image information of the second area. The moving object detection device according to claim 1.   The moving object detection device according to claim 1, wherein the determination unit determines the second region based on a moving direction of the imaging device.   4. The moving object detection device according to claim 1, wherein the second region is separated from the first region by a predetermined distance or more. 5. A feature point extraction unit that extracts feature points from the fourth region based on the control information;
A detection unit that detects the moving object from time variation of the feature point and outputs a detection result of the moving object;
The moving object detection apparatus according to claim 1, further comprising: A feature point extraction unit for extracting feature points from the image;
A detection unit that detects the moving object from time variation of the feature point and outputs a detection area including the detected moving object;
A selection unit that selects the fourth region of the detection regions based on the control information and outputs a detection result of the moving object including the fourth region;
The moving object detection apparatus according to claim 1, further comprising: A feature point extraction unit for extracting feature points from the image;
A selection unit that selects a feature point extracted from the fourth region among the feature points based on the control information;
A detection unit that detects the moving object from time variation of the selected feature point and outputs a detection result of the moving object;
The moving object detection apparatus according to claim 1, further comprising: A feature point extraction unit for extracting feature points from the image;
Based on the control information, feature points extracted from the fourth area are tracked by a first tracking process, and feature points extracted from the first area are different from the first tracking process. A tracking unit that performs tracking by two tracking processes and obtains temporal variation of feature points based on the first tracking process or the second tracking process;
A detection unit that detects the moving object from time variation of the feature point and outputs a detection result of the moving object;
The moving object detection apparatus according to claim 1, further comprising: A moving object detection method executed by a moving object detection device,
With reference to a storage unit that stores an image captured by the imaging device, a target pixel is selected from the pixels in the image, and a first region including the target pixel is extracted,
It determines whether the first region is correlated with a second region which exists in the periphery of the first region,
When the first region is correlated with the second region, the target pixel is determined to belong to the third region, and when the first region is not correlated with the second region, Determining that the target pixel belongs to a fourth region other than the third region;
Generates control information for distinguishing a plurality of target pixels determined to belong to the third region and a plurality of target pixels determined to belong to the fourth region in detection of a moving object based on the image To
A moving object detection method characterized by the above. With reference to a storage unit that stores an image captured by the imaging device, a target pixel is selected from the pixels in the image, and a first region including the target pixel is extracted,
Determines whether the first region is correlated with a second region which exists in the periphery of the first region,
When the first region is correlated with the second region, the target pixel is determined to belong to the third region, and when the first region is not correlated with the second region, Determining that the target pixel belongs to a fourth region other than the third region;
Generates control information for distinguishing a plurality of target pixels determined to belong to the third region and a plurality of target pixels determined to belong to the fourth region in detection of a moving object based on the image To
A program that causes a computer to execute processing.