JP5724859B2 - Image processing device - Google Patents

Description

  The present invention relates to an image processing apparatus that generates an optical flow based on image data captured by an imaging apparatus installed in a vehicle.

  Conventionally, as an example of an image processing apparatus that performs image processing of image data captured by an imaging apparatus, there is an image information correction apparatus described in Patent Document 1.

  According to this image information correction device, pixel information detected by the imaging device is sequentially input to the abnormality determination processing unit, normality determination processing unit, and interpolation processing unit for each pixel. In the abnormality determination processing unit, all input pixel information becomes pixel information to be corrected in the pixel arrangement order. The abnormality determination processing unit sequentially determines whether or not the correction target pixel information is abnormal information (information including noise) by comparing the pixel information to be corrected with the pixel information of the surrounding pixels. When it is determined that the error is detected, an abnormality determination output signal is output.

  In addition, in the normal determination processing unit, when the abnormality determination is performed on the pixel information to be corrected by the abnormality determination processing unit, peripheral pixels in the correction target pixel information are normal information (not including noise). Information) is sequentially determined, and when it is determined that the information is normal, a normal determination output signal is output. Then, when an abnormality determination output signal is output from the abnormality determination processing unit and a normal determination output signal is output from the normality determination processing unit, an interpolation control signal is output, and the interpolation processing unit interpolates the pixel information to be corrected. Processing is executed, and the interpolated pixel information is output.

JP 2005-284355 A

  As an image processing apparatus, there is an image processing apparatus that performs image processing of image data including data of a plurality of pixels imaged by an imaging device mounted on a vehicle. For example, there is one that generates an optical flow from a plurality of temporally continuous image data captured by an imaging device mounted on a vehicle.

  By the way, the vehicle state may suddenly change against the driver's intention due to the surrounding environment of the vehicle (foreign matter on the road (stone etc.), road shape, tunnel, etc.). For example, when a vehicle steps on a foreign object such as a stone on a road to jump the foreign object (stone jump) or travels on a depressed road, the vehicle body may tilt suddenly (instantly). Further, when the vehicle passes through the tunnel, the brightness around the vehicle body changes suddenly (instantaneously).

  In this way, when the vehicle state changes suddenly, the image data captured by the imaging device mounted on the vehicle changes suddenly exceeding a predetermined value (that is, the luminance data of each pixel in the image data is changed). (Suddenly changing beyond a predetermined value). The image data captured when the vehicle state suddenly changes is regarded as noise when generating an optical flow from a plurality of temporally continuous image data captured by the imaging device. That is, the data of all pixels included in the image data captured when the vehicle state suddenly changes becomes noise (noise image data). Therefore, when noise image data is included in a plurality of temporally continuous image data, the optical flow is affected by the noise image data (the noise image data is reflected).

  However, in the above-described image information correction device, the pixel information to be corrected is compared with the pixel information of the surrounding pixels to determine whether or not the pixel information is abnormal, and the abnormal pixel information is complemented. It is. Therefore, as described above, noise image data whose entire image data is noise cannot be complemented. That is, the influence of noise image data cannot be suppressed when generating an optical flow.

  The present invention has been made in view of the above problems, and provides an image processing apparatus capable of suppressing the influence of noise image data caused by a sudden change in the vehicle state when generating an optical flow. The purpose is to do.

In order to achieve the above object, an image processing apparatus according to claim 1,
An image processing device that generates an optical flow based on image data composed of data of a plurality of pixels imaged by an imaging device (20) installed in a vehicle,
Image data acquisition means (S10) for acquiring a plurality of image data imaged in time series by the imaging device (20);
Vehicle state estimating means (S20) for estimating that the vehicle state has suddenly changed;
Based on the estimation result of the vehicle state estimation means (S20), the image data obtained when the vehicle state is estimated to be suddenly changed in the image data acquired by the image data acquisition means (S10) is represented as noise image data. Sorting means (S30) for classifying image data when it is not estimated that the vehicle state has suddenly changed into normal image data;
Generating complementary image data based on the normal image data and replacing the noise image data with the complementary image data (S50);
Feature point extraction means (S60) for extracting feature points from normal image data and complementary image data;
Optical flow generation means (S70) for generating a plurality of optical flows by tracking the positions of feature points in temporally continuous image data in normal image data and complementary image data, To do.

  In this way, even when noise image data is included in a plurality of acquired image data, when generating an optical flow, complementary image data supplemented with noise image data and a normal image Data is used. Therefore, when generating an optical flow, it is possible to suppress the influence of noise image data that accompanies a sudden change in the vehicle state.

Further, as shown in claim 2, the vehicle state estimating means (S20)
First acquisition means (S201) for acquiring acceleration information indicating the acceleration in the vertical direction of the vehicle;
Second acquisition means (S202) for acquiring sound information outside the vehicle,
When the acceleration is equal to or higher than a predetermined value and there is no collision sound, it may be estimated that the vehicle state has suddenly changed.

  When the vehicle bouncing stones or when the vehicle undulates when traveling on a slope, there is no collision sound outside the vehicle and the vertical acceleration becomes a predetermined value or more. Therefore, it is possible to estimate that the vehicle state has suddenly changed when the vehicle body jumps stones or when the vehicle body undulates when traveling on a slope. it can. The vertical direction is acceleration in a direction perpendicular to the ground in the vehicle.

As shown in claim 3,
The vehicle state estimating means (S20)
Third acquisition means (S203) for acquiring acceleration information indicating vertical acceleration of the vehicle;
Fourth acquisition means (S204) for acquiring vehicle speed information indicating the vehicle speed of the vehicle,
When the acceleration is equal to or higher than a predetermined value and the vehicle speed is within a predetermined range, it may be estimated that the vehicle state has suddenly changed.

  In this way, even when sound information outside the vehicle cannot be acquired, the vehicle state may suddenly occur when the vehicle body jumps stones or when the vehicle body undulates when traveling on a slope. Can be estimated to have changed.

As shown in claim 4,
The vehicle state estimating means (S20)
Fifth acquisition means (S205) for acquiring solar radiation information indicating the amount of solar radiation irradiated to the vehicle;
Sixth acquisition means (S206) for acquiring vehicle speed information indicating the vehicle speed of the vehicle,
When the vehicle speed is equal to or higher than a predetermined value, the amount of change in solar radiation is equal to or higher than a predetermined value, and the solar radiation amount returns to a value before changing after a certain time, it may be estimated that the vehicle state has suddenly changed. .

  When the vehicle passes through the tunnel, the vehicle speed is equal to or higher than a predetermined value, the change amount of the solar radiation amount is equal to or higher than the predetermined value, and the solar radiation amount returns to the value before the change after a certain time. Therefore, by making it as shown in Claim 4, when a vehicle passes a tunnel, it can be estimated that the vehicle state changed suddenly.

As shown in claim 5,
The vehicle state estimating means (S20)
Seventh acquisition means (S207) for acquiring acceleration information indicating the vertical direction of the vehicle, the traveling direction, and the acceleration in the direction orthogonal to the vertical direction and the traveling direction;
An eighth acquisition means (S208) for acquiring vehicle speed information indicating the vehicle speed of the vehicle,
When at least one acceleration is equal to or higher than a predetermined value and the vehicle speed is zero, it may be estimated that the vehicle state has suddenly changed.

  When an impact is applied to the imaging device (20) or the vehicle body while the vehicle is stopped (for example, an impact is applied due to a collision with the vehicle body or the vehicle or the imaging device (20) being stolen), the vehicle speed is zero. In addition, at least one acceleration in the vertical direction, the traveling direction, and the direction perpendicular to the traveling direction is equal to or greater than a predetermined value. Therefore, according to the fifth aspect, it can be estimated that the vehicle state suddenly changes when an impact is applied to the imaging device (20) or the vehicle body while the vehicle is stopped.

As shown in claim 6,
The vehicle state estimating means (S20)
Ninth acquisition means (S209) for acquiring acceleration information indicating the vertical direction of the vehicle, the traveling direction, and the acceleration in the direction perpendicular to the vertical direction and the traveling direction;
Tenth acquisition means (S210) for acquiring sound information outside the vehicle,
When at least one acceleration is equal to or higher than a predetermined value and there is a collision sound, it may be estimated that the vehicle state has suddenly changed.

  When a vehicle collides with another vehicle while traveling, there is a collision sound outside the vehicle, and at least one acceleration in the vertical direction, the traveling direction, and the vertical direction and the direction orthogonal to the traveling direction is greater than or equal to a predetermined value. Become. Therefore, according to the sixth aspect, it is possible to estimate that the vehicle state has suddenly changed when the vehicle is collided with another vehicle or the like while the vehicle is traveling.

As shown in claim 7,
The plurality of pixel data in the image data includes luminance value data indicating the luminance of each pixel,
The complementing means (S50) generates complementary image data using normal image data captured immediately before the noise image data and normal image data captured immediately after the noise image data. The average luminance value data, which is the average value of the luminance value data of the pixels corresponding to each other in the data and the normal image data captured immediately thereafter, is calculated, and the average luminance value data is used as the luminance value data of each pixel to obtain a complementary image. Data may be generated.

  By doing in this way, complementary image data can be generated without using special data other than the image data acquired by the image data acquisition means (S10).

It is a block diagram which shows schematic structure of the control apparatus in embodiment. It is a flowchart which shows the processing operation of the control apparatus in embodiment. It is a flowchart which shows the estimation process operation | movement of the control apparatus in embodiment. It is an image figure which shows an example of the image data continuous in time. (A) is a graph which shows the relationship between each imaging timing and applied acceleration, (b) is a graph which shows the relationship between each imaging timing and luminance value in Fig.5 (a), (c) is after a complement. It is a graph which shows the relationship between each imaging timing of this, and a luminance value. 10 is a flowchart showing an estimation processing operation of a control device in Modification 1; 10 is a flowchart illustrating an estimation processing operation of a control device according to Modification 2. 10 is a flowchart illustrating an estimation processing operation of a control device according to Modification 3. 10 is a flowchart illustrating an estimation processing operation of a control device according to Modification 4.

  Hereinafter, embodiments of the present invention will be described with reference to the drawings.

  A control device 10 shown in FIG. 1 corresponds to the image processing device in the claims. The control device 10 receives image data from a vehicle-mounted camera 20 (simply described as “camera” in the following description and drawings) that is installed in a vehicle and electrically connected to itself (the control device 10) every predetermined time. To get to. Based on the acquired image data, an optical flow (hereinafter also referred to as OF) is generated. The optical flow is a vector representing the motion of an object in a temporally continuous digital image. The camera 20 corresponds to the imaging device in the claims.

  Further, the control device 10 performs pedestrian detection and various vehicle controls based on the generated OF. For example, as vehicle control, VSC (Vehicle Stability Control) control or the like is performed. Note that pedestrian detection and vehicle control based on the OF are well-known techniques and will not be described in detail.

  In addition, each image data which the control apparatus 10 acquires every predetermined time consists of data of a plurality of pixels. The data of a plurality of pixels in each image data includes position data indicating the position of each pixel and luminance value data indicating the luminance of each pixel. The position data and the brightness value data are associated with each other.

  For example, the control device 10 is connected to an in-vehicle LAN (Local Area Network) and can be configured to be able to communicate with a control device such as a first ECU (Electronic Control Unit) 30, a second ECU 40, and a third ECU 50. Then, the control device 10 may perform various vehicle controls using the first ECU 30, the second ECU 40, and the third ECU 50. In addition, the control device 10 acquires acceleration information indicating acceleration in the vertical direction of the vehicle and sound information outside the vehicle from any of the first ECU 30 to the third ECU 50. Here, the control apparatus 10 employs an example in which acceleration information is acquired from the first ECU 30 and sound information is acquired from the second ECU 40.

  As shown in FIG. 1, the control device 10 is mainly composed of a microcomputer including a CPU (Central Processing Unit) 11, a RAM (Random Access Memory) 12, a ROM (Read Only Memory) 13, an I / F (interface) 14, and the like. It is configured. In the control device 10, the CPU 11 reads out a program stored in advance in the ROM 13 or the like to the RAM 12 and executes it, thereby executing a predetermined processing operation such as acquisition of image data captured by the camera 20. The camera 20 is mounted on the vehicle together with the control device 10, and images a predetermined range (predetermined imaging range) such as the front of the host vehicle.

  Here, the processing operation of the control device 10 will be described with reference to FIGS. The control device 10 executes the flowchart shown in FIG. 2 every predetermined time while power is supplied from a power source (not shown, for example, a battery) mounted on the host vehicle.

  First, in step S10 of FIG. 2, an image data acquisition process is executed. The control device 10 (CPU 11) acquires a plurality of image data imaged by the camera 20 in time series (for example, every predetermined imaging interval Δt) (image data acquisition means). In other words, the image data (image signal) from the camera 20 is taken in every fixed time Δt. The CPU 11 stores the image data in the RAM 12 or the like in time series. The camera 20 is mounted on the vehicle together with the control device 10 and captures a predetermined range (predetermined imaging range) such as the front of the host vehicle. That is, the camera 20 captures the front of the host vehicle at predetermined imaging intervals Δt and outputs the image data (captured image information) to the control device 10. Therefore, each image data acquired by the control device 10 is an image of a frame every certain time.

  Next, in step S20 of FIG. 2, a vehicle state estimation process is executed. The control device 10 (CPU 11) estimates that the vehicle state has suddenly changed (vehicle state estimation means). Thus, it is determined whether or not the vehicle state has suddenly changed is determined by whether or not the image data acquired in step S10 is image data captured when the vehicle state has suddenly changed. This is to determine whether the image data is noise image data or normal image data.

  An example of this vehicle state estimation process will be described with reference to the flowchart shown in FIG. The sudden change in the vehicle state indicates a sudden change in the vehicle state contrary to the driver's intention due to the surrounding environment of the vehicle (foreign matter on the road (stone etc.), road shape). is there.

  In step S201 in FIG. 3, an acceleration information acquisition process is executed in order to estimate whether or not the vehicle state has suddenly changed. The control device 10 (CPU 11) acquires acceleration information from the first ECU 30 (first acquisition means).

  In step S202 of FIG. 3, sound information acquisition processing is executed in order to estimate whether or not the vehicle state has suddenly changed. The control device 10 (CPU 11) acquires sound information from the second ECU 40 (second acquisition means).

  In step S21 of FIG. 3, it is determined whether or not the acceleration is equal to or greater than a predetermined value and there is no collision sound. The control device 10 (CPU 11) determines whether or not the acceleration indicated by the acceleration information acquired in step S201 is equal to or greater than a predetermined value (predetermined value (threshold value) stored in the ROM 13), and the sound information acquired in step S202. It is determined whether or not a collision sound is included. If the control device 10 (CPU 11) determines that the acceleration is equal to or greater than the predetermined value and no collision sound, the control device 10 (CPU 11) proceeds to step S22, and if it is not determined that the acceleration is equal to or greater than the predetermined value and no collision sound, the step. Proceed to S23.

  The predetermined value here is based on the acceleration in the vertical direction (direction perpendicular to the ground) applied to the vehicle body when the stone jumps, and the acceleration in the vertical direction applied to the vehicle body when traveling on a slope. This is the value that is set. It is a value by which it can be determined that the vehicle has bounced off a stone or traveled on a slope.

  In step S22 of FIG. 3, the control device 10 (CPU 11) determines that the vehicle state has suddenly changed. In other words, it is determined that the vehicle state has suddenly changed when acceleration higher than the acceleration applied when the vehicle jumps on a stone or travels on a slope is applied, and there is no collision sound at that time. . On the other hand, in step S23 of FIG. 3, the control device 10 (CPU 11) determines that the vehicle state has not changed suddenly. Thus, when it is determined whether or not the vehicle state has suddenly changed, the process returns to step S30 in the flowchart of FIG.

  In addition, when the vehicle body is rocked or when the vehicle body undulates when traveling on a slope, there is no collision sound outside the vehicle, and the vertical acceleration may be a predetermined value or more. Therefore, in this way, the vehicle state suddenly changes when the vehicle bouncing stones or when the vehicle body undulates vertically and the vertical acceleration exceeds a predetermined value when traveling on a slope. It can be estimated that it has changed.

  In step S30 of FIG. 2, filter processing (sorting means, sorting process) is executed. The control device 10 (CPU 11) classifies the image data acquired in step S10 into noise image data and normal image data based on the estimation result in step S20. In other words, the process here is a process of filtering the image data into noise image data and normal image data.

  Then, in step S20, the control device 10 (CPU 11) classifies the image data when it is estimated that the vehicle state has suddenly changed into noise image data, and it is not estimated that the vehicle state has suddenly changed. The current image data is divided into normal image data. That is, the noise image data indicates image data when it is estimated that the vehicle state has suddenly changed. On the other hand, normal image data indicates image data when it is not estimated that the vehicle state has suddenly changed.

  In other words, the noise image data in the present invention is a variation in luminance over the entire image caused by a vehicle state that is unnecessary for performing vehicle control, and is non-stationary, and is one of time-series image data. Part (some frames). For example, the camera 20 is shaken when the vehicle travels on a step, a slope, or the like. Further, when the vehicle jumps a stone, the entire vehicle body instantaneously moves up and down, and the camera 20 also moves up and down. The noise image data indicates image data captured at such a time. That is, the data of all pixels included in the image data (frame) captured when the vehicle state suddenly changes becomes noise. In other words, the noise image data is different from that in which any pixel data in each image data is abnormal. Further, it is different from that in which any pixel data in one image data is abnormal due to the influence of electromagnetic waves or electrical failure (circuit failure in the camera 20 or the like).

  In step S40 of FIG. 2, it is determined whether complementation is necessary. The control device 10 (CPU 11) determines that complementation is necessary when it is classified into noise image data in step S30, and determines that complementation is not necessary when it is classified into normal image data in step S30. . In other words, the image data divided into noise image data needs to be complemented, and the image data divided into normal image data need not be complemented.

  In step S50 of FIG. 2, a complement process is executed. The control device 10 (CPU 11) complements the image data (that is, noise image data) determined to be necessary for complementation in step S40, and generates complement image data (complementing means). Note that, as an example of the method of generating the complementary image data, the complementary image data is generated using the normal image data captured immediately before the noise image data and the normal image data captured immediately after the noise image data. For example, it is an average value of luminance value data of pixels corresponding to each other (pixels at corresponding positions) in normal image data captured immediately before the noise image data and normal image data captured immediately after the noise image data. Average brightness value data is calculated. Then, the noise image data is deleted, and complementary image data is generated using the average luminance value data as the luminance value data of each pixel. That is, the noise image data is replaced with complementary image data. By doing in this way, complementary image data can be generated without using special data other than the image data acquired in step S10.

  As described above, the control device 10 (CPU 11) sequentially classifies the noise image data and the normal image data with respect to the acquired image data, and replaces the noise image data with complementary image data. That is, the control device 10 (CPU 11) specifies the noise image data in real time and replaces the noise image data with complementary image data.

  Next, in step S60, feature point extraction processing is executed. The control device 10 (CPU 11) extracts feature points from normal image data and complementary image data (feature point extraction means). In other words, the control device 10 (CPU 11) extracts feature points in the normal image data and the complementary image data from among the image data captured by the camera 20.

  When extracting the feature points, a known SIFT (Scale-Invariant Feature Transform) feature amount is used for the feature points, and the feature points are featured within a specific image range so that the density of the feature points is uniform. Decide the number of quantities as one. In other words, a known SIFT is used for extracting the feature points. In addition, the image data is divided into a plurality of regions having the same area so that the density of the feature points in the image data is uniform, and one feature point is extracted from each region (that is, all of the image data is extracted). Extract feature points evenly from the range). Specifically, first, key points are detected and the scale is determined. At this time, the smoothing parameter maximum is used as a scale (scale invariant). Next, feature amount description (feature vector) is described. At this time, the feature amount is robust against a change in illumination by describing with a multidimensional feature vector. Therefore, it is possible to extract feature points that are invariant to scale changes and robust to illumination changes.

  Next, in step S70, an optical flow generation process is executed. The control device 10 (CPU 11) generates OF by tracking the position of the feature point in temporally continuous image data (a plurality of image data) in the normal image data and the complementary image data (optical flow generation). means). With this OF, the motion of each feature point in the entire image data can be acquired as a vector.

  The optical flow generation process is a well-known technique and will not be described in detail. Further, when generating the OF, the control device 10 (CPU 11) obtains the distance from the own vehicle (camera 20) to each feature point by stereo matching (normalization means). Then, it is preferable to normalize the OF (normalization means) based on the distance to the feature point and the direction of the vanishing point.

  In step S80, vehicle control is executed. The control device 10 (CPU 11) executes vehicle control based on the OF generated in step S70. In this step S80, pedestrian detection may be executed based on the OF generated in step S70.

  Here, as shown in FIG. 4 and FIG. 5, an image captured at timing t = 0, timing t = 1 after Δt of timing t = 0, timing t = 2 after Δt of timing t = 1, and A description will be given using applied acceleration and luminance value data at each timing. FIG. 4 shows images captured by the camera 20 at the imaging timings t = 0, 1, and 2. In FIG. 4, the feature point 60 and the OF 70 are shown in each image for easy understanding.

  As shown in FIG. 5A, the image at the imaging timing t = 0, 2 is an image captured by the camera 20 when the vehicle state has not changed suddenly. On the other hand, the image at timing t = 1 is captured by the camera 20 when the vehicle state suddenly changes. The data in FIG. 5 is taken from point 1 in FIG.

  As shown in FIG. 5B, the image data (t = 1) captured by the camera 20 when the vehicle state suddenly changes is obtained when the vehicle state is not suddenly changed. Compared with the image data (t = 0, 2) when imaged at 20, the luminance (luminance value data) is clearly abnormal. In addition, as shown in FIG. 4, when the vehicle state suddenly changes (t = 1), compared to when the vehicle state does not suddenly change (t = 0, 2), the OF 70 The direction will be different.

  Therefore, as described in step S50, complementary image data is created, and noise image data that is image data captured by the camera 20 when the vehicle state suddenly changes is replaced with complementary image data.

  As described so far, the control device 10 divides the acquired plurality of image data into noise image data and normal image data. Even when noise image data is included in a plurality of acquired image data, complementary image data supplemented with noise image data and normal image data are used when generating an OF. Therefore, when generating OF, the influence by noise image data accompanying the sudden change in the vehicle state can be suppressed. As described above, by suppressing the influence of the noise image data when generating the OF, it is possible to suppress malfunction of the vehicle control executed based on the OF. Similarly, erroneous detection of pedestrian detection can be suppressed.

  Furthermore, since the control device 10 identifies noise image data in real time and complements the noise image data before extracting feature points, the image processing algorithm from feature point extraction to OF generation can be simplified. . Therefore, even if noise image data is generated due to a sudden change in the vehicle state, the control device 10 can be robust with respect to the noise image data and can be processed at high speed.

  As mentioned above, although preferable embodiment of this invention was described, this invention is not restrict | limited to the embodiment mentioned above at all, and various deformation | transformation are possible in the range which does not deviate from the meaning of this invention.

(Modification 1)
In the above-described embodiment, an example in which acceleration information and sound information are used when estimating the vehicle state (vehicle state estimation processing) is adopted, but the present invention is not limited to this. For example, acceleration information and vehicle speed information can also be used (Modification 1). Note that the configuration and basic processing operations of Modification 1 are substantially the same as those in the above-described embodiment, and thus detailed description thereof is omitted. Therefore, the control apparatus 10 of the modification 1 performs the process shown to the flowchart of FIG. 2 similarly to the above-mentioned embodiment. The control device 10 in Modification 1 is different from the above-described embodiment in that it acquires vehicle speed information indicating the vehicle speed of the vehicle instead of acquiring sound information. For example, the control device 10 acquires vehicle speed information from the third ECU 50.

  Here, based on FIG. 6, the vehicle state estimation process which the control apparatus 10 of the modification 1 performs is demonstrated. When the image data acquisition process in step S10 of FIG. 2 is completed, the control device 10 of Modification 1 executes the vehicle state estimation process shown in FIG.

  In step S203 of FIG. 6, an acceleration information acquisition process is executed in order to estimate whether or not the vehicle state has suddenly changed. The control device 10 (CPU 11) acquires acceleration information from the first ECU 30 (third acquisition means).

  In step S204 in FIG. 6, a vehicle speed information acquisition process is executed to estimate whether or not the vehicle state has suddenly changed. The control device 10 (CPU 11) acquires vehicle speed information from the third ECU 50 (fourth acquisition means).

  In step S21a of FIG. 6, it is determined whether or not the acceleration is equal to or greater than a predetermined value and the vehicle speed is within a predetermined range. The control device 10 (CPU 11) determines whether or not the acceleration indicated by the acceleration information acquired in step S203 is equal to or greater than a predetermined value (predetermined value (threshold value) stored in the ROM 13 in advance), and the vehicle speed information acquired in step S204. Is within a predetermined range (predetermined range stored in the ROM 13 in advance). Then, when the control device 10 (CPU 11) determines that the acceleration is equal to or greater than the predetermined value and the vehicle speed is within the predetermined range, the control device 10 (CPU 11) proceeds to step S22 and does not determine that the acceleration is equal to or greater than the predetermined value and the vehicle speed is within the predetermined range. If so, the process proceeds to step S23.

  The predetermined value that is the comparison target of the acceleration information is the same as that in the above-described embodiment. On the other hand, the predetermined range that is a comparison target of the vehicle speed information is a range in which it can be determined that the vehicle is traveling.

  In step S22 of FIG. 6, the control device 10 (CPU 11) determines that the vehicle state has suddenly changed. On the other hand, in step S23 of FIG. 6, the control device 10 (CPU 11) determines that the vehicle state has not changed suddenly. Thus, when it is determined whether or not the vehicle state has suddenly changed, the process returns to step S30 in the flowchart of FIG.

  In this way, even when sound information outside the vehicle cannot be acquired (for example, a microphone that circulates the sound is not provided or only engine sound can be acquired), the vehicle body jumps off the stone. It can be estimated that the vehicle state has suddenly changed when the vehicle body undulates in a vertical direction when traveling on a slope.

(Modification 2)
Moreover, in the above-mentioned embodiment, when estimating a vehicle state (vehicle state estimation process), the example which uses acceleration information and sound information was employ | adopted, However, This invention is not limited to this. For example, solar radiation information and vehicle speed information can also be used (Modification 2). Note that the configuration and basic processing operations of the modified example 2 are substantially the same as those of the above-described embodiment, and thus detailed description thereof is omitted. Therefore, the control apparatus 10 of the modification 2 performs the process shown to the flowchart of FIG. 2 similarly to the above-mentioned embodiment. The control device 10 in Modification 2 is different from the above-described embodiment in that it acquires solar radiation information indicating the amount of solar radiation irradiated on the vehicle and vehicle speed information indicating the vehicle speed of the vehicle. For example, the control device 10 acquires solar radiation information from the second ECU 40 and acquires vehicle speed information from the third ECU 50. Note that when the solar radiation information is acquired from the second ECU 40, the control device 10 may store the solar radiation information in a RAM or the like for a predetermined time.

  Here, based on FIG. 7, the vehicle state estimation process which the control apparatus 10 of the modification 2 performs is demonstrated. The control apparatus 10 of the modification 2 performs the vehicle state estimation process shown in FIG. 7 after the image data acquisition process in step S10 of FIG.

  In step S205 in FIG. 7, a solar radiation information acquisition process is executed to estimate whether or not the vehicle state has suddenly changed. The control device 10 (CPU 11) acquires solar radiation information from the second ECU 40 (fifth acquisition means).

  In step S206 in FIG. 7, a vehicle speed information acquisition process is executed to estimate whether or not the vehicle state has suddenly changed. The control device 10 (CPU 11) acquires vehicle speed information from the third ECU 50 (sixth acquisition means).

  In step S21b of FIG. 7, it is determined whether the vehicle speed is equal to or higher than a predetermined value and the amount of solar radiation satisfies a condition. The condition of the amount of solar radiation is a value before the amount of change in solar radiation is a predetermined value (for example, the difference between the amount of solar radiation at sunrise and the amount of solar radiation after sunset) and before the amount of solar radiation changes after a certain time ( (Steady value). The control device 10 (CPU 11) determines whether or not the vehicle speed information acquired in step S206 is greater than or equal to a predetermined value (predetermined value (threshold value) stored in the ROM 13 in advance). Further, the control device 10 (CPU 11) determines that the amount of change in the amount of solar radiation indicated by the solar radiation information acquired in step S205 is greater than or equal to a predetermined value (predetermined value (threshold value) stored in the ROM 13 in advance) and the amount of solar radiation is after a certain time. It is determined whether or not the value has returned to the value before the change. When the control device 10 (CPU 11) determines that the vehicle speed is equal to or greater than the predetermined value and the solar radiation amount satisfies the condition, the process proceeds to step S22, and when the vehicle speed is equal to or greater than the predetermined value and the solar radiation amount does not determine that the condition is satisfied. Advances to step S23.

  When the vehicle passes through the tunnel, the vehicle speed is equal to or higher than a predetermined value, the change amount of the solar radiation amount is equal to or higher than the predetermined value, and the solar radiation amount returns to the value before the change after a certain time. In addition, obstacles ahead of the host vehicle can be excluded because the amount of solar radiation returns to a steady value.

  In this way, when the vehicle passes through the tunnel, it can be estimated that the vehicle state (brightness around the vehicle) has suddenly changed. In addition, this makes it possible to prevent erroneous tracking of feature points from luminance deviation due to changes in the amount of solar radiation (light quantity) when passing through the tunnel, and to prevent malfunction of vehicle control. Here, as a condition for the amount of solar radiation, the amount of change in the amount of solar radiation is a predetermined value (for example, the difference between the amount of solar radiation at sunrise and the amount of solar radiation after sunset), and the value before the amount of solar radiation changes after a certain period of time. The return to (steady value) was adopted as an example. However, in addition to this, any conditions that can determine that the vehicle has passed through the tunnel can be adopted.

(Modification 3)
In the above-described embodiment, an example in which acceleration information and sound information are used when estimating the vehicle state (vehicle state estimation processing) is adopted, but the present invention is not limited to this. For example, triaxial acceleration information and vehicle speed information can also be used (Modification 3). Note that the configuration and basic processing operations of Modification 3 are substantially the same as those in the above-described embodiment, and thus detailed description thereof is omitted. Therefore, the control apparatus 10 of the modification 3 performs the process shown to the flowchart of FIG. 2 similarly to the above-mentioned embodiment. The control device 10 in Modification 3 acquires acceleration information indicating acceleration in the vertical direction, the traveling direction, and the orthogonal direction (three-axis direction) with respect to the vertical direction and the traveling direction, and vehicle speed information indicating the vehicle speed of the vehicle. Thus, this embodiment is different from the above-described embodiment. For example, the control device 10 acquires acceleration information from the first ECU 30 and also acquires vehicle speed information from the third ECU 50.

  Here, based on FIG. 8, the vehicle state estimation process which the control apparatus 10 of the modification 3 performs is demonstrated. The control apparatus 10 of the modification 3 will perform the vehicle state estimation process shown in FIG. 8, after the image data acquisition process in step S10 of FIG.

  In step S207 of FIG. 8, an acceleration information acquisition process is executed to estimate whether or not the vehicle state has suddenly changed. The control device 10 (CPU 11) acquires each acceleration information in the three-axis directions from the first ECU 30 (seventh acquisition means).

  In step S208 of FIG. 8, vehicle speed information acquisition processing is executed to estimate whether or not the vehicle state has suddenly changed. The control device 10 (CPU 11) acquires vehicle speed information from the third ECU 50 (eighth acquisition means).

  In step S21c of FIG. 8, it is determined whether or not the acceleration is equal to or higher than a predetermined value (for example, Earth G) and the vehicle speed is 0 (zero). The control device 10 (CPU 11) determines whether or not the acceleration indicated by the acceleration information acquired in step S207 is equal to or greater than a predetermined value (predetermined value (threshold value) stored in the ROM 13 in advance), and the vehicle speed information acquired in step S208. Whether or not is zero. When the controller 10 (CPU 11) determines that any one of the accelerations in the three-axis directions is equal to or greater than the predetermined value and the vehicle speed is zero, the control device 10 (CPU 11) proceeds to step S22 and selects one of the accelerations in the three-axis directions. If it is not determined that one acceleration is equal to or greater than a predetermined value and the vehicle speed is zero, the process proceeds to step S23. The predetermined value, which is a comparison target of acceleration information, is a value that can determine that an impact has been applied due to a rear-end collision with the vehicle body or the vehicle or the camera 20 being stolen.

  In step S22 of FIG. 8, the control device 10 (CPU 11) determines that the vehicle state has suddenly changed. On the other hand, in step S23 of FIG. 8, the control device 10 (CPU 11) determines that the vehicle state has not changed suddenly. Thus, when it is determined whether or not the vehicle state has suddenly changed, the process returns to step S30 in the flowchart of FIG.

  When the vehicle is stopped, if an impact is applied to the camera 20 or the vehicle body (for example, an impact is applied due to a collision with the vehicle body or the vehicle or the camera 20 being stolen), the vehicle speed is zero and the vehicle travels in the vertical direction. The acceleration in the direction and at least one direction perpendicular to the vertical direction and the traveling direction is a predetermined value or more. Therefore, by doing in this way, when an impact is applied to the camera 20 or the vehicle body while the vehicle is stopped, it can be estimated that the vehicle state has suddenly changed. In addition, this makes it possible to suppress malfunction of vehicle control while the vehicle is stopped.

(Modification 4)
In the above-described embodiment, an example in which acceleration information and sound information are used when estimating the vehicle state (vehicle state estimation processing) is adopted, but the present invention is not limited to this. For example, triaxial acceleration information and sound speed information can also be used (Modification 4). Note that the configuration and basic processing operations of the modification 4 are substantially the same as those in the above-described embodiment, and thus detailed description thereof is omitted. Therefore, the control apparatus 10 of the modification 4 performs the process shown to the flowchart of FIG. 2 similarly to the above-mentioned embodiment. The control device 10 according to the modified example 4 obtains acceleration information indicating the acceleration in the vertical direction of the vehicle, the traveling direction, and each of the vertical direction and the direction orthogonal to the traveling direction (three-axis direction). Is different. For example, the control device 10 acquires acceleration information from the first ECU 30.

  Here, based on FIG. 9, the vehicle state estimation process which the control apparatus 10 of the modification 3 performs is demonstrated. The control apparatus 10 of the modification 4 will perform the vehicle state estimation process shown in FIG. 9, if the image data acquisition process in step S10 of FIG. 2 is completed.

  In step S209 of FIG. 9, an acceleration information acquisition process is executed in order to estimate whether or not the vehicle state has suddenly changed. The control device 10 (CPU 11) acquires each acceleration information in the three-axis directions from the first ECU 30 (ninth acquisition means).

  In step S210 of FIG. 9, sound information acquisition processing is executed to estimate whether or not the vehicle state has suddenly changed. The control device 10 (CPU 11) acquires sound information from the second ECU 40 (tenth acquisition means).

  In step S21d of FIG. 9, it is determined whether or not the acceleration is equal to or greater than a predetermined value and there is a collision sound. The control device 10 (CPU 11) determines whether or not the acceleration indicated by the acceleration information acquired in step S209 is greater than or equal to a predetermined value (predetermined value (threshold value) previously stored in the ROM 13), and the sound information acquired in step S210. It is determined whether or not a collision sound is included. If the control device 10 (CPU 11) determines that any one of the accelerations in the three-axis directions is equal to or greater than the predetermined value and that there is a collision sound, the control device 10 (CPU 11) proceeds to step S22, and any one of the accelerations in the three-axis directions. If it is not determined that one acceleration is equal to or greater than the predetermined value and there is a collision sound, the process proceeds to step S23. The predetermined value, which is a comparison target of acceleration information, is a value that can determine that the vehicle has collided with the vehicle body.

  In step S22 of FIG. 9, the control device 10 (CPU 11) determines that the vehicle state has suddenly changed. On the other hand, in step S23 of FIG. 9, the control device 10 (CPU 11) determines that the vehicle state has not changed suddenly. Thus, when it is determined whether or not the vehicle state has suddenly changed, the process returns to step S30 in the flowchart of FIG.

  When a vehicle collides with another vehicle while traveling, there is a collision sound outside the vehicle, and acceleration in a vertical direction, a traveling direction, and at least one direction perpendicular to the traveling direction is predetermined. More than the value. Therefore, by doing in this way, when the vehicle is collided with another vehicle or the like while traveling, it can be estimated that the vehicle state has suddenly changed. In addition, this makes it possible to suppress malfunction of vehicle control while the vehicle is stopped.

  In the fourth modification, when the control device 10 determines that only the acceleration information indicating the acceleration in the direction orthogonal to the vertical direction and the traveling direction of the vehicle is equal to or greater than the predetermined value and that there is a collision sound, the vehicle state suddenly occurs. It may be determined that the change has occurred. This makes it possible to stratify the inclination of the vehicle by the roll.

  In the above-described embodiment, an example in which noise image data is replaced with complementary image data is used. However, feature points may be extracted using only image data before and after the deleted noise image data.

  Also, when the acceleration in any one of the vertical direction with respect to the vehicle, the traveling direction, the orthogonal direction with respect to the vertical direction and the traveling direction, the pitch direction, the roll direction, and the yaw direction reaches a predetermined value, the vehicle state suddenly occurs. You may make it determine with having changed to. The pitch direction is acceleration in the rotational direction with the axis (pitch axis) extending in the left-right direction (vehicle width direction) of the vehicle as the central axis. The roll direction is acceleration in a rotational direction with an axis (roll axis) extending in the traveling direction of the vehicle (the full width direction of the vehicle) as the central axis. The yaw direction is an acceleration in a rotational direction with an axis (yaw axis) extending in a direction perpendicular to the ground (vehicle height direction) as a central axis. The predetermined value here is an acceleration of a magnitude that is not applied when the vehicle is traveling normally or is stopped (in any one of the above directions). Acceleration). Normal travel is a state where the vehicle is traveling without stepping on foreign objects (stones, steps, dents), swelling up and down on a slope, colliding with another vehicle, or colliding with an obstacle.

  DESCRIPTION OF SYMBOLS 10 Control apparatus, 11 CPU, 12 RAM, 13 ROM, 14 I / F, 20 Camera, 30 1st ECU, 40 2ECU, 50 3ECU

Claims (7)

An image processing device that generates an optical flow based on image data composed of data of a plurality of pixels imaged by an imaging device (20) installed in a vehicle,
Image data acquisition means (S10) for acquiring a plurality of the image data imaged in time series by the imaging device (20);
Vehicle state estimation means (S20) for estimating that the vehicle state has suddenly changed;
Based on the estimation result of the vehicle state estimation means (S20), image data when it is estimated that the vehicle state has suddenly changed in the image data acquired by the image data acquisition means (S10). Noise image data, and classification means (S30) for classifying image data when it is not estimated that the vehicle state has suddenly changed into normal image data;
Complementary means (S50) for generating complementary image data based on the normal image data and replacing the noise image data with the complementary image data;
Feature point extraction means (S60) for extracting feature points from normal image data and complementary image data;
Optical flow generation means (S70) for generating an optical flow by tracking the position of the feature point in the temporally continuous image data in the normal image data and the complementary image data;
An image processing apparatus comprising: The vehicle state estimating means (S20)
First acquisition means (S201) for acquiring acceleration information indicating vertical acceleration of the vehicle;
Second acquisition means (S202) for acquiring sound information outside the vehicle,
The image processing apparatus according to claim 1, wherein when the acceleration is equal to or greater than a predetermined value and there is no collision sound, it is estimated that the vehicle state has suddenly changed. The vehicle state estimating means (S20)
Third acquisition means (S203) for acquiring acceleration information indicating the vertical acceleration of the vehicle;
Fourth acquisition means (S204) for acquiring vehicle speed information indicating the vehicle speed of the vehicle,
The image processing apparatus according to claim 1, wherein when the acceleration is equal to or greater than a predetermined value and the vehicle speed is within a predetermined range, it is estimated that the vehicle state has suddenly changed. The vehicle state estimating means (S20)
Fifth acquisition means (S205) for acquiring solar radiation information indicating the amount of solar radiation irradiated to the vehicle;
Sixth acquisition means (S206) for acquiring vehicle speed information indicating the vehicle speed of the vehicle,
When the vehicle speed is equal to or higher than a predetermined value, the amount of change in the solar radiation amount is equal to or higher than the predetermined value, and the solar radiation amount returns to a value before changing after a certain time, it is estimated that the vehicle state has suddenly changed. The image processing apparatus according to claim 1. The vehicle state estimating means (S20)
Seventh acquisition means (S207) for acquiring acceleration information indicating the vertical direction of the vehicle, the traveling direction, and the acceleration in the direction perpendicular to the vertical direction and the traveling direction;
And eighth acquisition means (S208) for acquiring vehicle speed information indicating the vehicle speed of the vehicle,
The image processing apparatus according to claim 1, wherein when at least one acceleration is equal to or greater than a predetermined value and the vehicle speed is zero, it is estimated that the vehicle state has suddenly changed. The vehicle state estimating means (S20)
Ninth acquisition means (S209) for acquiring acceleration information indicating the vertical direction of the vehicle, the traveling direction, and the acceleration in the direction orthogonal to the vertical direction and the traveling direction;
Tenth acquisition means (S210) for acquiring sound information outside the vehicle,
The image processing apparatus according to claim 1, wherein when at least one acceleration is equal to or greater than a predetermined value and there is a collision sound, it is estimated that the vehicle state has suddenly changed. The data of a plurality of pixels in the image data includes luminance value data indicating the luminance of each pixel,
The complementary means (S50) generates complementary image data using normal image data captured immediately before the noise image data and normal image data captured immediately after, and was captured immediately before. Average luminance value data, which is an average value of luminance value data of pixels corresponding to each other in normal image data and normal image data captured immediately after, is calculated, and the average luminance value data is used as luminance value data of each pixel. The image processing apparatus according to claim 1, wherein complementary image data is generated.

Images