JP2008276307A - Video image processing apparatus, video image processing system, and navigation device - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To solve the problem that detecting accuracy is insufficient in detecting an optical flow from an image picked up with an on-vehicle camera. <P>SOLUTION: An optical flow detecting part 20 detects an optical flow from a moving image picked up with an image pickup element installed in a moving element. Depending on the moving speed of the moving element, a reference frame selecting part 18 adaptively varies the frame interval between a subject frame and a reference frame when detecting the optical flow. For example, as the moving speed increases, the frame interval increases, the moving speed is reduced, the frame interval is reduced. <P>COPYRIGHT: (C)2009,JPO&INPIT

Description

  The present invention relates to a moving image processing apparatus, a moving image processing system, and a navigation apparatus that detect an object that may collide from within a moving image.

  In recent years, automobile safety technology has been developed. As one of the safety technologies, a camera that captures an image of the front or rear of the vehicle is mounted on the vehicle, and the optical flow is detected by comparing frames that are captured by the camera in the time direction. Techniques have been proposed for extracting objects that may collide from within.

Patent Document 1 discloses an optical flow detection system, and discloses a method for controlling a frame rate according to an evaluation amount associated with the magnitude of an optical flow or a distance to a target object.
JP 2004-355082 A

  When detecting an optical flow from an image captured by an in-vehicle camera, it is difficult to optimally set a reference frame, and a low-speed moving object located at a short distance from the vehicle cannot be detected correctly, or a long distance from the vehicle In some cases, the object located in the location could not be detected. For example, when detecting a pedestrian or the like located at a short distance during high-speed traveling, the movement direction and movement speed may not be detected correctly. Further, there are cases where pedestrians and the like located at a long distance cannot be detected during low-speed traveling.

  The present invention has been made in view of such circumstances, and an object thereof is to provide a moving image processing apparatus, a moving image processing system, and a navigation apparatus that can detect a detection target from a moving image with high accuracy.

  An aspect of the present invention includes a detection unit that detects an optical flow from a moving image captured by an image sensor mounted on a moving body, and a target frame when detecting an optical flow according to the moving speed of the moving body. A reference frame selection unit that adaptively changes a frame interval with the reference frame.

  It should be noted that any combination of the above-described constituent elements and a conversion of the expression of the present invention between a method, an apparatus, a system, a recording medium, a computer program, etc. are also effective as an aspect of the present invention.

  According to the present invention, it is possible to accurately detect a detection target from within a moving image.

  First, an outline of the present invention will be described before describing the embodiments of the present invention in detail. In the embodiment, a camera that images the front of a vehicle is mounted, and a moving image captured by the camera is processed to detect pedestrians, obstacles, vehicles ahead, and the like as an optical flow. Based on the optical flow, it becomes possible to notify the driver of the danger of collision with those objects. Here, the optical flow is a vector indicating in what direction and how much distance a certain point or figure in the image moves at the next moment.

  FIG. 1 is a diagram showing a relationship L1 between the host vehicle speed and the safe inter-vehicle distance, and an inter-vehicle distance L2 that contracts when the preceding vehicle suddenly brakes. The vertical axis in FIG. 1 represents the inter-vehicle distance (m), and the horizontal axis represents the vehicle speed (km / h). The first line L1 indicates the safe inter-vehicle distance corresponding to the own vehicle speed. In FIG. 1, the region above the first line L1 is a region where the distance between the vehicles is sufficiently high with respect to the speed and the safety is high. The second line L2 indicates an inter-vehicle distance that the driver senses a danger when the preceding vehicle applies a sudden brake and contracts when the brake is applied. In FIG. 1, the area below the second line L2 is an area where there is an extremely high possibility of a collision, even if the brake is applied. A region between the first line L1 and the second line L2 is a region where a collision can be avoided if the driver appropriately copes with it.

  Therefore, it is basically too late to detect objects such as the previous vehicle, pedestrians and obstacles at a distance in the region below the second line L2. On the other hand, when such an object is detected at a distance in a region far above the first line L1, there is a relatively large margin. If such an object is detected at a distance in the area in the vicinity of the first line L1 and a distance in the area between the first line L1 and the second line L2, a collision will occur if the driver recognizes it early. It can be avoided. Therefore, it can be said that it is the most urgent and requires the highest detection accuracy. Hereinafter, embodiments will be described using this knowledge.

  First, an outline of the first embodiment will be described. In the first embodiment, when detecting an optical flow, the frame interval between the target frame and the reference frame is adaptively switched according to the speed of the vehicle. Specifically, the frame interval is narrowed when the speed of the vehicle is slower than the current speed, and the frame interval is widened when the speed is high. Since the camera is fixed to the vehicle, the vehicle speed corresponds to the moving speed of the camera.

  FIG. 2 is a diagram schematically illustrating the frame interval between the target frame and the reference frame. As shown on the left side of FIG. 2, the target frame is usually a frame adjacent in the time direction as a reference frame. That is, the frame interval T1 is fixed at one. On the other hand, as shown on the right side of FIG. 2, in the present embodiment, the frame intervals T1, T2, and T3 between the target frame and the reference frame may be one, two, or three. is there. Of course, there may be more.

  FIG. 3 is a diagram for explaining a frame interval determination criterion. FIG. 3 shows an image captured by an image sensor 12 of FIG. 4 (described later) mounted on the vehicle. As shown in FIG. 1, the distance that can be stopped without colliding with the object and the distance that the collision cannot be avoided differ depending on the speed of the vehicle. In the first embodiment, processing is performed so that an object at a distance near a line that divides the distance between the former and the latter can be detected with high accuracy.

  In the first embodiment, the frame interval is set narrow so that the optical flow in the region A can be detected with the highest accuracy when the vehicle is stopped or at a low speed. The frame interval is set to a medium level so that the optical flow in the region B can be detected with the highest accuracy at the medium speed. The frame interval is set wide so that the optical flow in region C can be detected with the highest accuracy at high speed. Here, the low speed, the medium speed, and the high speed are set, for example, as 0 to 40 km / h (low speed), 40 to 80 km / h (medium speed), and 80 km / h <(high speed). A frame interval is set for each division, and when the acquired speed crosses the division, the frame interval is switched to the frame interval of the transition destination division.

  As shown in FIG. 3, the frame interval is set to be widened so that the front is detected with higher accuracy as the speed increases. In other words, the attention area is set to switch to the area A, the area B, and the area C according to the speed. More specifically, when the speed of the vehicle is increased, the position of the attention area is moved in the direction of the area where the image of a long distance is captured in the frame, and when the speed is decreased, the position of the attention area is changed to a short distance image. Move in the direction of the area to be copied. Hereinafter, a configuration and operation for specifically executing this control will be described.

  FIG. 4 shows a configuration of a moving image processing system 500 according to Example 1 of the first embodiment. The moving image processing system 500 includes the imaging unit 10 and the moving image processing apparatus 100. The imaging unit 10 includes an imaging element 12 and a signal processing unit 14. The moving image processing apparatus 100 includes a frame buffer 16, a reference frame selection unit 18, a lookup table 19, an optical flow detection unit 20, and a post-processing unit 26. The optical flow detection unit 20 includes a feature point extraction unit 22 and an optical flow calculation unit 24. The configuration of the moving image processing apparatus 100 can be realized by an arbitrary processor, memory, or other LSI in terms of hardware, and is realized by a program loaded in the memory in terms of software. Depicts functional blocks realized by. Therefore, those skilled in the art will understand that these functional blocks can be realized in various forms by hardware only, software only, or a combination thereof.

  The imaging unit 10 is mounted in front of the vehicle, captures a moving image in front of the vehicle, and outputs the moving image to the moving image processing apparatus 100. The image sensor 12 uses a CCD (Charge Coupled Devices) sensor, a CMOS (Complementary Metal-Oxide Semiconductor) image sensor, or the like, converts incident light into an electrical signal, and outputs it to the signal processing unit 14.

  The signal processing unit 14 converts an analog signal input from the image sensor 12 into a digital signal. Further, a smoothing filter may be provided, and smoothing processing may be performed on the image data output to the moving image processing apparatus 100. Further, other noise removal processing may be performed.

  The moving image processing apparatus 100 detects an optical flow from image data input from the imaging unit 10. The frame buffer 16 has an area for storing m (m is a natural number) frames, and temporarily stores image data input from the imaging unit 10. When the area overflows, it is discarded in order from the previously input frame. The image frames stored in the frame buffer 16 are sequentially output to the optical flow detection unit 20.

  The reference frame selection unit 18 acquires speed information from a speed sensor in the vehicle and refers to the lookup table 19 to specify a frame interval associated with the acquired speed information. Based on the frame interval, the frame I (t−nT) that is n frames past or the frame I (t + nT) that is n frames in advance that should be the reference frame of the target frame I (t) is specified, and the frame buffer 16 get. T represents the frame interval, and n is an integer that takes a value in the range of (1 ≦ n ≦ m−1).

  The lookup table 19 manages the speed information and the frame interval in association with each other. The relationship between the speed information and the frame interval is set to a relationship determined by experiments or simulations based on the above-described knowledge.

  The optical flow detection unit 20 detects an optical flow using the target frame I (t) and the reference frame I (t ± nT). An algorithm for detecting an optical flow includes a gradient method, a block matching method, and the like. However, the present embodiment is not limited to any one, and any algorithm may be used. In the following description, a pyramid type LK (Lucas-kanade) method, which is one of gradient methods, is used. The LK method is one of spatial local optimization methods that assume that the optical flow is constant within a local region of the same object.

  The feature point extraction unit 22 extracts a predetermined number of feature points having a large change in shading based on the LK method from the target frame I (t) input from the frame buffer 16. Here, it is performed in a plurality of layers having different resolutions. The feature point extraction unit 22 outputs the extracted feature point coordinates and luminance information to the optical flow calculation unit 24.

  The optical flow calculation unit 24 includes the target frame I (t) input from the frame buffer 16, the feature point coordinates and luminance information input from the feature point extraction unit 22, and the reference frame input from the reference frame selection unit 18. An optical flow is calculated based on I (t ± nT). More specifically, a point corresponding to the feature point in the target frame I (t) is searched for in the reference frame I (t ± nT).

  The post-processing unit 26 performs predetermined post-processing on the optical flow output from the optical flow detection unit 20. For example, you may perform the process which subtracts the component by the movement of imaging part 10 itself from the input optical flow. In addition, in order to remove noise, the length of the optical flow to be validated may be limited by comparing the length of the input optical flow with a predetermined threshold. Further, an average value of the lengths of the optical flows output from the optical flow detection unit 20 may be calculated. The threshold value for noise removal may be a preset value, or an average value or a value obtained by adjusting the average value.

  An arbitrary user interface (not shown) allows the user to recognize the appearance of a dangerous target based on the optical flow output from the post-processing unit 26. For example, the optical flow may be simply displayed on the display unit, or only the optical flow having a length that is determined to be dangerous according to the distance from the vehicle may be displayed. Further, when an optical flow having a length that is considered to be dangerous occurs, a warning sound may be emitted from a speaker. Further, a control signal may be output to the vehicle braking system.

Hereinafter, Example 1 according to Embodiment 1 will be described. The first embodiment is an example in which optical flow detection is performed for all frames.
FIG. 5 is a diagram schematically illustrating the relationship between the target frame and the reference frame according to Example 1 of the first embodiment. In FIG. 5, frame intervals T3a, T3b, and T3c between the target frame and the reference frame are set to three. Here, the optical flow is calculated for all frames.

  FIG. 6 is a diagram illustrating an example of an optical flow detected by the moving image processing system 500 according to Example 1 of the first embodiment. In FIG. 6, an arrow indicates an optical flow. For simplicity, one is drawn for each object. In practice, a plurality of objects may be detected for each object, and may be detected radially around the object.

  FIG. 6A shows an optical flow at a stop time or at a low speed detected by the moving image processing system 500 according to Example 1 of the first embodiment. The optical flow of the person in the region A is detected, and the optical flow of the person in the region B and the region C is not detected. This is because, at low speed, the distance between the target frame and the reference frame is set to be narrow, so that the motion of the target located at a short distance is detected with high accuracy.

  FIG. 6B shows an optical flow at medium speed detected by the moving image processing system 500 according to Example 1 of the first embodiment. The optical flow of the person in the area A and the area B is detected, and the optical flow of the person in the area C is not detected. This is because, at medium speed, the interval between the target frame and the reference frame is set to a medium level, so that the movement of the target located at the medium distance is detected with high accuracy. The length of the optical flow of the person in the area A in FIG. 6B is longer than that in FIG. The direction is also shifted. This indicates that in FIG. 6B, the detection accuracy in the region A is lower than that in FIG.

  FIG. 6C shows an optical flow at a high speed detected by the moving image processing system 500 according to Example 1 of the first embodiment. Optical flows of persons in the areas A, B, and C are detected. This is because a target located at a long distance is also detected because the interval between the target frame and the reference frame is set wide at high speed. The length of the optical flow of the person in the area A and the area B in FIG. 6C is longer than that in FIG. Also, the direction is further shifted. This indicates that the detection accuracy in the region A and the region B is lower than that in FIG. 6B in FIG.

  Therefore, an optical flow exceeding a predetermined length may be invalidated as noise in order to reduce a noise component appearing in a target optical flow located at a short distance at medium speed or high speed. The post-processing unit 26 described above can perform this process.

  FIG. 7 is a flowchart showing the operation of the moving image processing apparatus 100 according to Example 1 of the first embodiment. First, the frame buffer 16 stores the image data input from the imaging unit 10 (S10). When the optical flow detection process is not completed by the moving image processing apparatus 100 (N in S12), the reference frame selection unit 18 acquires the current speed information of the vehicle (S14). The reference frame selection unit 18 refers to the lookup table 19 and determines a reference frame I (t ± nT) based on the speed information (S16).

  The feature point extraction unit 22 extracts a predetermined number of feature points of the target frame I (t) (S18). The optical flow calculation unit 24 calculates an optical flow based on the target frame I (t), the feature points extracted from the frame, and the determined reference frame I (t ± nT) (S20). Next, the target frame I (t) is incremented (S22). And it changes to step S12. Until the optical flow detection process ends (Y in S12), the processes from step S14 to step S22 are repeated.

  As described above, according to the first example of the first embodiment, the detection target is accurately detected from the moving image by adaptively controlling the frame interval between the target frame and the reference frame according to the speed of the vehicle. Can be detected well.

  For example, when the optical flow is detected using the above-described LK method, it is possible to detect an object that is moving about 20 m away at a horizontal speed of 15 km / h, but based on the above-described knowledge of FIG. For example, it is impossible for a vehicle actually traveling at 45 km / h or more to detect a dangerous object and stop safely.

  On the other hand, in this embodiment, by adaptively switching the frame interval between the target frame and the reference frame, both a target moving at a low speed located at a short distance and a target moving at a high speed located at a long distance can be used. It can be detected with high accuracy. That is, an object that moves at a low speed located in a short distance can accurately detect the movement by narrowing the frame interval. Even if an optical flow of a target located at a long distance is detected at this frame interval, it is difficult to detect the movement. On the contrary, the movement of the object located at a long distance can be accurately detected by widening the frame interval. If an optical flow of a target located at a short distance is detected at this frame interval, the detection process becomes rough, and the detection result becomes noisy.

  Further, as a result of the optimization of the optical flow detection process, the processing amount can be reduced and the power consumption can also be reduced. Furthermore, when the length of the optical flow exceeds a predetermined threshold value, noise can be further reduced by adding processing for invalidating the optical flow.

  Hereinafter, Example 2 of Embodiment 1 will be described. The second embodiment is an example in which the optical flow is not detected for all frames included in the moving image captured by the image sensor 12 but intermittently.

  FIG. 8 is a diagram schematically illustrating a relationship between a target frame and a reference frame according to Example 2 of the first embodiment. In FIG. 8, frame intervals T3a, T3d, and T3g between the target frame and the reference frame are set to three. Optical flow is calculated every three frames.

  FIG. 9 shows a configuration of a moving image processing system 510 according to Example 2 of the first embodiment. The configuration of the moving image processing system 510 according to the second embodiment is basically the same as the configuration of the moving image processing system 500 according to the first embodiment. Hereinafter, differences will be described.

  The configuration of the moving image processing system 510 according to the second embodiment is a configuration in which the target frame selection unit 21 is added to the configuration of the moving image processing system 500 according to the first embodiment. The target frame selection unit 21 thins out the frames in the frame buffer 16 and outputs them to the optical flow detection unit 20. For example, one sheet is output for N (N is an integer). N is a value set by the designer through experiments and simulations. Frames not selected by the target frame selection unit 21 may be discarded.

  The reference frame selection unit 18 operates in conjunction with the target frame selection unit 21. That is, only the reference frame of the target frame selected by the target frame selection unit 21 may be specified and acquired from the frame buffer 16.

  FIG. 10 is a flowchart illustrating the operation of the moving image processing apparatus 110 according to the second example of the first embodiment. The flowchart shown in FIG. 10 is basically the same as the flowchart shown in FIG. Hereinafter, differences will be described. After performing the end determination in step S12, the target frame selection unit 21 determines whether or not the order of the target frame I (t) corresponds to a multiple of N (S13). If it is a multiple of N (Y in S13), the process proceeds to step S14, and the optical flow of the target frame I (t) is calculated. If it does not correspond to a multiple of N (N in S13), the process shifts to step S22 without calculating the optical flow of the target frame I (t), and increments the target frame I (t) (S22).

  As described above, according to the second embodiment of the first embodiment, the same effects as the first embodiment are obtained. Furthermore, the amount of calculation can be reduced by intermittently performing the optical flow detection process on all the frames. In the case of processing one sheet per N sheets, the amount of calculation can be reduced to about 1 / N. Further, if the value of N is suitably set, the influence on the detection accuracy is also limited.

  For example, when the vehicle speed is 60 km / h, the frame rate of the input image is 30 fps, and N = 2, the distance the vehicle travels during the time required for the optical flow detection process is 2 × 60 × 10 ^ 3 ÷ 3600 ÷ 30 ≈ 1.1 (m), and the influence is slight.

  Hereinafter, Example 3 of the first embodiment will be described. In the third embodiment, the reference frame is normally set to a frame that is temporally continuous with the target frame, and the frame that is not temporally continuous with the target frame at a rate of one frame per N frames, that is, two or more frames apart in time. Set to the specified frame.

  The configuration of the moving image processing system according to Example 3 of the first embodiment is basically the same as the configuration of the moving image processing system 500 according to Example 1. The reference frame selection unit 18 selects a reference frame having a fixed frame interval as a basic operation, and selects a reference frame having a frame interval wider than the frame interval for each predetermined period.

  For example, as a basic operation, the reference frame of the target frame I (t) is determined as a frame I (t−T) that is one frame past or a frame I (t + T) that is one frame in the future. One frame is determined as N (excluding 1) frames, past frame I (t-nT), or n frames future frame I (t + nT). At least one of the value of N and the value of the number of frame intervals n may be set to a value associated with the input speed information with reference to the lookup table 19.

  When the post-processing unit 26 calculates the average value of the length of the optical flow, the length of the optical flow using the frame I (t−nT) in the past n frames or the frame I (t + nT) in the future n frames as a reference frame. It is desirable to remove from the basis for calculating the average value.

  FIG. 11 is a flowchart showing the operation of the moving image processing apparatus 100 according to Example 3 of the first embodiment. The flowchart shown in FIG. 11 is basically the same as the flowchart shown in FIG. Hereinafter, differences will be described. Prior to the reference frame determination process in step S16, the reference frame selection unit 18 determines whether or not the order of the target frame I (t) corresponds to a multiple of N (S15). When it corresponds to a multiple of N (Y in S15), the reference frame selection unit 18 determines the frame n frames before the target frame I (t) as the reference frame I (t−nT) (S16b). When it does not correspond to the multiple of N (N of S15), the reference frame selection unit 18 determines the previous frame of the target frame I (t) as the reference frame I (t−T) (S16a). Thereafter, the same processing as in FIG. 7 is performed.

  As described above, according to the third embodiment of the first embodiment, the same effects as the first embodiment can be obtained. In addition, by mixing frames that are detected with a large frame interval and frames that are detected in continuous frames, the accuracy of optical flow detection in a medium to long distance target is improved, and the optical flow of a short distance target is improved. Detection can be detected with low noise.

  Next, a second embodiment will be described. First, an outline of the second embodiment will be described. The second embodiment divides an image into a plurality of regions in a different manner depending on the speed and steering angle of the vehicle when detecting an optical flow. Further, the optical flow is calculated under the conditions set for each region.

  FIG. 12 is a diagram schematically illustrating an aspect in which an image is divided into a plurality of regions. As described above, the optical flow is calculated using frames that move back and forth in the time direction. The left side of FIG. 12 shows a normal mode that is not divided into a plurality of regions. The right side of FIG. 12 shows a mode in which the region is divided into two regions, region D and region W. In this division mode, the region D becomes the attention region, and the remaining region W becomes the non-attention region. Optical flows are detected for the attention area D and the non-attention area W, respectively. The division mode, here, the position and size of the attention area D are adaptively switched according to the speed and steering angle of the vehicle. The attention area D corresponds to the area A, the area B, or the area C shown in FIGS. 3 and 6, and is an area that changes according to the speed of the vehicle as a main detection area. That is, this is an area where highly accurate detection is required. The frame intervals TD and TW are also adaptively switched according to the vehicle speed. The frame intervals TD and TW may be different between the region D and the region W, respectively.

  FIG. 13 is a diagram for explaining criteria for determining an image division mode. The upper left image in FIG. 13 shows how the position of the attention area D in the front-rear direction is determined according to the speed of the vehicle. That is, when the speed is fast, the attention area D is moved upward in the image, and when the speed is slow, the attention area D is moved downward. Accordingly, the size of the attention area D may be controlled. That is, when the speed is high, the size of the attention area D is reduced while moving the attention area D upward in the image. On the contrary, when the speed is low, the size of the attention area D is increased while moving the attention area D downward in the image. The method of determining the attention area D is based on the knowledge described with reference to FIGS.

  The upper right image in FIG. 13 shows how the left and right positions of the attention area D are determined according to the steering angle. When the handle is turned to the right, the attention area D is moved to the right in the image. When the handle is turned to the left, the attention area D is moved to the left. The lower image in FIG. 13 shows the attention area D and the non-attention area W separated. The attention area D and the non-attention area W may have different parameters for calculating the optical flow. The parameter includes at least one of a frame interval between the target frame and the reference frame, a frame resolution, and a noise removal threshold value for determining that the calculated optical flow is invalid as noise.

  FIG. 14 shows a configuration of a moving image processing system 520 according to the second embodiment. The configuration of the moving image processing system 520 according to the second embodiment has a common part with the configuration of the moving image processing system 500 according to the first embodiment. Hereinafter, a description will be made mainly on the different portions.

  First, the configuration of the imaging unit 10 and the frame buffer 16 is the same as the configuration of the moving image processing system 500 according to the first embodiment. The reference frame selection unit 18 acquires speed information from a vehicle speed sensor in the vehicle and refers to the lookup table 19 to identify a frame interval associated with the acquired speed information. At that time, a different frame interval may be specified for each divided area. In the second embodiment, it is not always necessary to adaptively control the frame interval, and the frame interval may be fixed.

  The look-up table 19 manages at least one vehicle state of speed information and steering angle information in association with the following parameters. The parameters include the position of the attention area D, the size of the attention area D, the frame interval between the target frame and the reference frame, the resolution of the image when calculating the optical flow, and the optical flow to be invalidated. At least one of the threshold values is included. These associations are set to relationships determined by experiments and simulations based on the above-described knowledge and the knowledge shown below.

  The criteria for determining the frame interval, position and size of the attention area D have been described above. That is, as the vehicle speed increases, the frame interval of the attention area D is increased, the position of the attention area D is moved upward in the image, and the size of the attention area D is decreased. In addition, the threshold value of the attention area D is reduced as the vehicle speed increases. These are based on the property that the size of the optical flow detected in the attention area D is relatively small, that is, there is little noise. Note that the resolution of the attention area D may be fixed regardless of the speed.

  The frame interval of the non-attention area W may be fixed regardless of the speed. The shape and size of the non-attention area W is the remaining area from which the attention area D is separated, and thus changes according to the position and size of the attention area D. The resolution of the non-attention area W is increased as the speed decreases. The threshold for noise removal in the non-attention area W is increased as the speed increases. The threshold value is based on the knowledge that a shorter distance should be set to reduce noise. When the speed increases, the attention area D moves upward in the image, and when the speed decreases, the attention area D moves downward. Therefore, the short distance becomes the non-attention area W when the speed is high, and the short distance becomes the attention area D when the speed is low.

  Regarding the relationship between the attention area D and the non-attention area W, the frame interval of the attention area D is set larger than that of the non- attention area W, the resolution of the attention area D is set higher than that of the non-attention area W, and noise removal of the attention area D is performed. The threshold is made smaller than the non-attention area W. This is because the attention area D needs to be more accurate than the non-attention area W. Another reason is to reduce the total amount of calculation while suppressing a decrease in accuracy.

  The optical flow detection unit 20 includes a feature point extraction unit 22, a parameter determination unit 23, a first optical flow calculation unit 24a, and a second optical flow calculation unit 24b. Two optical flow calculation units 24 are provided. This corresponds to dividing the region into two, and when dividing into three or more, three or more optical flow calculation units 24 are provided.

  The feature point extraction unit 22 is the same as that in the first embodiment. The parameter determination unit 23 acquires speed information and steering angle information from a speed sensor and a steering angle sensor in the vehicle, refers to the look-up table 19, and various parameters associated with the acquired speed information and steering angle information. To get.

  When the position and size of the attention area D are acquired, the first optical flow calculation unit 24a is set and information corresponding to the first optical flow calculation unit 24b is set. When the resolutions of the attention area D and the non-attention area W are acquired, they are set in the first optical flow calculation section 24a and the second optical flow calculation section 24b. When the noise removal threshold values of the attention area D and the non-attention area W are acquired, they are set in the first threshold determination section 28a and the second threshold determination section 28b.

  The first optical flow calculation unit 24 a receives the target frame I (t) input from the frame buffer 16, the coordinates and brightness of the feature points input from the feature point extraction unit 22, and the reference input from the reference frame selection unit 18. Based on the frame I (t ± nT), the optical flow of the attention area D is calculated. Similarly, the second optical flow calculation unit 24b calculates the optical flow of the non-attention area W. When the resolution is lowered, the second optical flow calculation unit 24b may use an average value of a plurality of pixels located in the vicinity instead of simply thinning out the pixels.

  The post-processing unit 26 includes a first threshold determination unit 28a and a second threshold determination unit 28b. The first threshold determination unit 28a compares the threshold set by the parameter determination unit 23 with the size of the optical flow input from the first optical flow calculation unit 24a, and invalidates the optical flow exceeding the threshold. The second threshold determination unit 28b compares the threshold set by the parameter determination unit 23 with the magnitude of the optical flow input from the second optical flow calculation unit 24b, and invalidates the optical flow exceeding the threshold.

  FIG. 15 is a diagram illustrating an example of an optical flow detected by the moving image processing system 520 according to the second embodiment. FIG. 15A shows an optical flow at the time of stoppage or low speed detected by the moving image processing system 520 according to the second embodiment. The optical flow in the attention area D is calculated using normal parameters. The optical flow in the non-attention area W is calculated with the frame interval wider than usual. The threshold for noise removal is set large in the attention area D and small in the non-attention area W.

  FIG. 15B shows an optical flow at medium speed detected by the moving image processing system 520 according to the second embodiment. The optical flow in the attention area D is calculated with the frame interval wider than usual. The optical flow in the non-attention area W is calculated with a low resolution. The threshold for noise removal is set small in the attention area D and large in the non-attention area W.

  FIG. 15C shows an optical flow at a high speed detected by the moving image processing system 520 according to the second embodiment. The optical flow in the attention area D is calculated by further widening the frame interval from that at the medium speed. The optical flow in the non-attention area W is calculated with the resolution further lower than that at the medium speed. The threshold for noise removal is set to be smaller in the attention area D and larger in the non-attention area W than at the medium speed.

  FIG. 16 is a flowchart showing the operation of the moving image processing apparatus 120 according to the second embodiment. The flowchart shown in FIG. 16 is basically the same as the flowchart shown in FIG. Hereinafter, differences will be described. After the feature point is extracted in step S18 or in parallel therewith, the parameter determination unit 23 acquires speed information from a speed sensor in the vehicle, and refers to the lookup table 19 to acquire the acquired speed. Various parameters associated with information and the like are acquired, and parameters are determined for each divided area (S19).

  The first optical flow calculation unit 24a performs the first optical flow based on the target frame I (t), the feature points extracted from the frame, the determined reference frame I (t ± nT), and the determined parameters. Calculate (S20a). In parallel with this, the second optical flow calculation unit 24b similarly calculates the second optical flow (S20b). Thereafter, the same processing as in FIG. 7 is performed.

  As described above, according to the second embodiment, the same effects as those of the first embodiment can be obtained. Furthermore, by dividing an image into a plurality of regions and setting optimum parameters for each region, it is possible to further reduce the amount of computation while maintaining accuracy. For example, the amount of calculation can be reduced by reducing the resolution of the non-attention area W, that is, the number of pixels to be processed. Since the resolution of the attention area D is not lowered, the detection accuracy is maintained.

  The present invention has been described based on some embodiments. It should be understood by those skilled in the art that these embodiments are exemplifications, and that various modifications can be made to combinations of the respective components and processing processes, and such modifications are within the scope of the present invention. is there.

The moving image processing systems 500, 510, and 520 according to the above-described embodiments can be used in cooperation with the car navigation system.
FIG. 17 is a diagram illustrating a configuration of a car navigation system 700 that is linked to the moving image processing system according to the embodiment. As an aspect of cooperation, the moving image processing apparatus 100 can notify the detection result to the outside using the user interface of the car navigation apparatus 600. For example, the moving image processing apparatus 100 can display an optical flow using the display unit of the car navigation apparatus 600 and can emit a warning sound from a speaker. Further, the moving image processing apparatus 100 may be installed in a housing of the car navigation apparatus 600.

  Further, the processing performed by the moving image processing apparatus 100 may be performed using a processor and a memory mounted on the car navigation apparatus 600. In this case, hardware resources unique to the moving image processing apparatus 100 are not necessary. According to these aspects, it is possible to detect the optical flow by effectively utilizing the hardware resources of the car navigation device having a high penetration rate.

  In the embodiment, an automobile has been described as an example. However, as long as it is a moving body, it can be applied to a motorcycle, a bicycle, a train, a ship, an airplane, and the like. Moreover, although the example which image | photographs the front image of a motor vehicle was demonstrated, it is applicable also to the form which images the image of back and right and left.

It is a figure which shows the relationship between the own vehicle speed and a safe inter-vehicle distance, and the inter-vehicle distance which shrink | contracts when the front vehicle applies sudden braking. It is the figure which showed typically the flame | frame space | interval of an object frame and a reference frame. It is a figure for demonstrating the determination criteria of a frame space | interval. 1 is a diagram illustrating a configuration of a moving image processing system according to Example 1 of Embodiment 1. FIG. 6 is a diagram schematically showing a relationship between a target frame and a reference frame according to Example 1 of Embodiment 1. FIG. FIG. 6A shows an optical flow at the time of stoppage or low speed detected by the moving image processing system according to Example 1 of the first embodiment. FIG. 6B shows an optical flow at medium speed detected by the moving image processing system according to Example 1 of the first embodiment. FIG. 6C shows an optical flow at a high speed detected by the moving image processing system according to Example 1 of the first embodiment. 6 is a flowchart showing the operation of the moving image processing apparatus according to Example 1 of Embodiment 1. 6 is a diagram schematically showing a relationship between a target frame and a reference frame according to Example 2 of Embodiment 1. FIG. It is a figure which shows the structure of the moving image processing system which concerns on Example 2 of Embodiment 1. FIG. 6 is a flowchart showing the operation of the moving image processing apparatus according to Example 2 of the first embodiment. 10 is a flowchart showing the operation of the moving image processing apparatus according to Example 3 of Embodiment 1. It is the figure which showed typically the aspect which divides | segments an image into a some area | region. It is a figure for demonstrating the determination criteria of the division | segmentation aspect of an image. 6 is a diagram illustrating a configuration of a moving image processing system according to Embodiment 2. FIG. FIG. 15A shows an optical flow at the time of stopping or at a low speed detected by the moving image processing system according to the second embodiment. FIG. 15B shows an optical flow at medium speed detected by the moving image processing system according to the second embodiment. FIG. 15C shows an optical flow at a high speed detected by the moving image processing system according to the second embodiment. 10 is a flowchart illustrating an operation of the moving image processing apparatus according to the second embodiment. It is a figure which shows the structure of the car navigation system cooperated with the moving image processing system which concerns on embodiment.

Explanation of symbols

  DESCRIPTION OF SYMBOLS 10 Image pick-up part, 12 Image pick-up element, 14 Signal processing part, 16 Frame buffer, 18 Reference frame selection part, 19 Look-up table, 20 Optical flow detection part, 21 Target frame selection part, 22 Feature point extraction part, 23 Parameter determination part 24 optical flow calculation unit, 24a first optical flow calculation unit, 24b second optical flow calculation unit, 26 post-processing unit, 28a first threshold value determination unit, 28b second threshold value determination unit, 100 moving image processing device, 500 moving image Image processing system, 600 car navigation system, 700 car navigation system.

Claims (6)

A detection unit that detects an optical flow from a moving image captured by an image sensor mounted on a moving body;
A reference frame selection unit that adaptively changes a frame interval between a target frame and a reference frame for detecting the optical flow according to a moving speed of the moving body;
A moving image processing apparatus comprising: The reference frame selection unit
The moving image processing apparatus according to claim 1, wherein the frame interval is widened when the moving speed is high, and the frame interval is narrowed when the moving speed is low.   The moving image processing apparatus according to claim 1, wherein the detection unit intermittently detects an optical flow for a plurality of frames included in the moving image. A detection unit that detects an optical flow from a moving image captured by an image sensor mounted on a moving body;
A reference frame selection unit for determining a frame interval between a target frame and a reference frame for detecting the optical flow,
The moving picture processing apparatus, wherein the reference frame selection unit selects a reference frame having a fixed frame interval and selects a reference frame having a frame interval wider than the frame interval for each predetermined period. An image sensor that is mounted on a moving body and captures a moving image;
The moving image processing apparatus according to any one of claims 1 to 4, wherein a moving image captured by the image sensor is processed.
A moving image processing system comprising:   5. The moving image processing apparatus according to claim 1, which processes a moving image picked up by an image pickup device that is mounted on a vehicle and picks up a moving image around the vehicle, and notifies a detection result to the outside. A navigation device characterized by that.

Images