JP2006163607A - Image processor and processing method - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide an image processor for accurately calculating a correction quantity of image data taken by a camera 200 mounted on a mobile body while eliminating the influence of pitching/bouncing of the mobile body. <P>SOLUTION: This processor comprises an image data acquisition means 10 for successively acquiring image data continuously taken at predetermined time intervals from an imaging device 200 for imaging the circumference of the mobile body; a characteristic extraction means for extracting a characteristic part from each of the acquired image data; an acceleration detection means 30 for detecting the acceleration of each extracted characteristic part; a reference position determination means 40 for determining a reference state position of each characteristic part; and a correction quantity calculation means 50 for calculating a slippage on image of each characteristic part extracted by the extraction means 20. <P>COPYRIGHT: (C)2006,JPO&NCIPI

Description

  The present invention relates to an image processing apparatus and an image processing method for correcting image shift caused by a change in posture of a moving body.

  With respect to this type of image processing apparatus and image processing method, a lane marker represented by a white line is linearly approximated to detect a vanishing point (FOE) in the image, and the pitch angle or imaging height is determined based on the position of the vanishing point. Some measure change.

  In Patent Document 1, coordinate conversion is performed on a lane marker extracted from a captured image so that the pitch angle and imaging height can be measured even when the lane marker is a curve, and two lane markers are mutually connected in a predetermined plane coordinate system. An apparatus for detecting a change in the pitch angle of a vehicle on the observer side based on being parallel is described.

  Further, in Patent Document 2, when an outside camera is photographed with a vehicle-mounted camera, if the visual axis change occurs in the camera due to pitching or the like, the same amount of y coordinate position displacement occurs in all objects in the image. Describes an apparatus that extracts a plurality of objects from an image and corrects the pitching amount by an average value of y-coordinate displacement amounts of the extracted objects.

However, the method of Patent Document 1 has a problem that the pitch angle and the imaging height cannot be measured when there is no reference lane marker or the like. In the method of Patent Document 2, since the pitch angle is measured using the y-coordinate displacement amount of the object, the position change of the y-coordinate that does not depend on the visual axis change such as pitching is caused by the occurrence of the visual axis change. There was a problem in that it was wrongly determined and the visual axis was corrected incorrectly.
Japanese Patent Laid-Open No. 11-203445 JP 2001-84497 A

The present invention has been made in view of the above problems, and an object of the present invention is to eliminate an influence caused by a change in posture on the observer side in image processing.
According to the present invention, image data is sequentially acquired from an imaging device that continuously captures the periphery of a moving body at a predetermined timing, and feature portions are extracted from each acquired image data. The speed or acceleration is detected, the reference state position of the feature is determined based on the detected speed or acceleration of each feature, and based on the determined reference state position of the feature, due to the behavior of the moving body An image processing apparatus or an image processing method for calculating a correction amount of image data is provided.
The image processing apparatus and the image processing method of the present invention do not require the presence of a lane marker or the like, and even when a relative position change between the moving object and the observation target occurs, an image resulting from a change in posture of the self (observer side) The amount of data correction can be calculated accurately.

Hereinafter, an image processing apparatus 100 according to an embodiment of the present invention will be described with reference to the drawings. The image processing apparatus 100 according to the present embodiment is mounted on a vehicle or other moving body together with the imaging apparatus 200. A block configuration of an in-vehicle device including the image processing apparatus 100 of the present embodiment is shown in FIG. As shown in FIG. 1, the image processing apparatus 100 according to the present embodiment can exchange information with a vehicle posture detection device 300, a travel support device 400, or an obstacle detection device 500 mounted on the same vehicle via an in-vehicle LAN. It is connected like that. The correction amount of the image data calculated by the image processing apparatus 100 or the corrected image data is output toward these, and is used for vehicle posture detection, driving support, obstacle detection, and the like.
The image processing apparatus 100 according to this embodiment includes an image data acquisition unit 10, a feature extraction unit 20, an acceleration detection unit 30, a reference position determination unit 40, a correction amount calculation unit 50, and an image data correction unit 60. Have. Specifically, at least a ROM (Read Only Memory) or the like that stores a program for calculating a correction amount based on image data and a program stored in the ROM or the like to execute the reference position determination means 40 and correction A CPU (Central Processing Unit) that functions as the amount calculation means 50 and the image data correction means 60, and a RAM (Random Access Memory) that functions as an accessible storage device 70 are provided. The storage device 70 only needs to be accessible by the image processing apparatus 100 and is provided inside or outside the image processing apparatus 100. Of course, an LSI such as an application specific integrated circuit (ASIC) or a field programmable gate array (FPGA) that realizes these functions may be mounted.

Hereinafter, the image processing apparatus 100 will be described.
The “image data acquisition means 10” sequentially acquires image data continuously captured at a predetermined timing from the imaging device 200 that captures the periphery of the host vehicle. The imaging apparatus 200 includes an imaging unit 210 including an image sensor such as a complementary metal oxide semiconductor (CMOS) image sensor and a charge coupled device (CCD), and image data captured by the imaging unit 210 in time series for each frame. And a frame memory 220 for storage.

  The “feature extraction unit 20” extracts a feature portion that can observe the movement on the image to be captured from each image data (frame) acquired by the image data acquisition unit 10. The feature extraction unit 20 can perform normal image processing such as tracking area setting, edge detection, and image conversion.

  The “acceleration detection means 30” detects the speed and / or acceleration of the feature portion extracted by the feature extraction means 20. The characteristic part in the image data corresponds to the observation target existing around the host vehicle. The characteristic part in this image data moves with the relative position change by the movement of the own vehicle or the movement of the observation target. The acceleration detection means 30 detects the moving speed or moving acceleration of the feature in the image data. Specifically, the acceleration detection means 30 compares the position of the feature part included in two image data with different imaging timings, and performs the first-order differentiation or the second-order differentiation of the position change amount, whereby the speed of the feature part, Find the acceleration. The obtained velocity or acceleration of the characteristic portion is stored in the storage device 70 in association with the imaging timing identifier or the frame identifier. When there are a plurality of feature parts in one image data, either one or two or more feature parts may be selected to calculate the speed or acceleration, or the speed or acceleration may be calculated for all the feature parts. May be.

  The “reference position determination unit 40” determines the reference state position of the feature based on the speed or acceleration of the feature detected by the acceleration detection unit 30. Specifically, the reference position determination unit 40 extracts image data in which the host vehicle on which the imaging apparatus 200 is mounted is in the reference state, and determines the position on the image data of the feature portion included in the extracted image data as the reference state position. Judge as. The determined reference state position of the feature is stored in the storage device 70 in association with the imaging timing. The stored reference state position of the characteristic portion is read out when calculating the correction amount of the image data captured thereafter.

  The present embodiment focuses on the relationship between the behavior of the vehicle (moving body) and the behavior of the feature (image data) corresponding to the observation target, and determines the state of the vehicle based on the behavior of the feature on the image data. Then, it is detected that the vehicle is in a reference state (a state where the most accurate image processing can be performed). In the present embodiment, a “reference state” in which pitching and / or bouncing has not occurred in the vehicle, which is defined with respect to a “vibration state” in which pitching and / or bouncing has occurred in the vehicle, is detected.

  Although not particularly limited, the reference position determination unit 40 of the present embodiment determines that the moving body is in the reference state when the movement acceleration of each specific part of the image data is less than a predetermined threshold. That is, the reference position determination unit 40 extracts image data in which the acceleration of the feature detected by the acceleration detection unit 30 is less than a predetermined threshold, and determines the position on the image of the feature included in the image data in the reference state. Judge as position. The reference state position that is the determination result is stored in the storage device 70 in association with the imaging timing of the image data. In this case, the threshold value for evaluating the acceleration of the feature part is preferably a value near zero.

  Further, the reference position determination unit 40 of the present embodiment determines that the moving body is in the reference state when the moving speed of each specific part of the image data is substantially maximum. That is, the reference position determination unit 40 extracts image data in which the speed of the feature detected by the acceleration detection unit 30 is approximately maximum, and uses the position of the feature included in the image data on the image as the reference state position. to decide. The reference state position that is the determination result is stored in the storage device 70 in association with the imaging timing of the image data.

  Further, the reference position determination unit 40 of the present embodiment determines that the moving body is in the reference state when the sign of acceleration of each specific part of the image data is inverted. That is, the reference position determination unit 40 extracts image data in which the sign of the acceleration of the feature portion detected by the acceleration detection unit 30 is inverted (captured first to initially after the inversion), and the feature included in the image data. The position of the image on the image is determined as the reference state position. The reference state position that is the determination result is stored in the storage device 70 in association with the imaging timing of the image data.

  Here, the “vehicle reference state” defined by the present embodiment will be described with reference to FIGS. 2 to 6 while being compared with the “vehicle vibration state”.

When the host vehicle is not pitched or bouncing and the host vehicle is in the reference state, the position change of the feature portion on the image is mainly caused by the relative position change between the host vehicle and the imaging target. 2A shows the relative positional relationship when the distance between the host vehicle R and the observation target Q is x (t ₀ ), and FIG. 2B shows the distance between the host vehicle R and the observation target Q. The relative positional relationship in the case of x (t ₁ ) (t ₁ > t ₀ ) is shown. FIG. 3A shows the image data of the observation target Q imaged in the relative positional relationship shown in FIG. 2A, and FIG. 3B shows the imaged in the relative positional relationship shown in FIG. The image data of the observed object Q is shown. Since the depression angle (elevation angle) changes when the relative position of the host vehicle and the observation target Q changes, the position of the observation target Q on the image picked up by the imaging means 210 installed in the host vehicle changes. The change of the y coordinate on the image in time series is as shown in FIG.

  Next, the positional change of the characteristic part when the own vehicle is pitched and / or bouncing and the own vehicle is in a vibration state will be described. For example, when the distance from the pitching center to the imaging means 210 (camera) is sufficiently small with respect to the distance from the pitching center to the observation target, it is considered that the movement of the feature on the image data is governed by the pitching behavior. be able to. That is, it can be assumed that the rotation center of the imaging means 210 (camera) is the pitching center.

  Pitching and bouncing occur when vibrations input from the road surface are transmitted from the tire to the coil spring damper to the body (chassis). That is, the vibration frequency of pitching and bouncing is the natural frequency of the spring-mass-damper system formed by them, and is generally about 1 to 2 Hz.

  4A shows the imaging direction P1 when pitching and / or bouncing has not occurred in the host vehicle, and FIG. 4B shows the imaging direction when pitching and / or bouncing has occurred in the host vehicle. P2 is shown. 5A shows the image data of the observation target Q imaged in the reference state shown in FIG. 4A, and FIG. 5B is imaged in the vibration state shown in FIG. 4B. The image data of the observed object Q is shown. When the attitude of the host vehicle changes due to pitching and / or bouncing, the depression angle (elevation angle) changes, so that movement due to pitching and bouncing is detected in the entire image, and the position of the observation target Q on the image changes. The change of the y coordinate on the image in time series is as shown in FIG.

  As described above, the amount of change in the feature on the image data includes the amount of displacement caused by the change in the relative position between the host vehicle and the observation target Q, and the pitching or bouncing that occurs in the host vehicle, that is, the imaging means (camera) 219 The amount of displacement due to is superimposed and detected. The change in the image in the time series that occurs in a complex manner due to such factors is as shown in FIG.

  In order to accurately measure the change amount of the pitch angle and the change amount of the imaging height, it is necessary to separate the change amounts caused by the respective factors and analyze them. In this embodiment, the influence of pitching / bouncing is detected by detecting a static equilibrium state of the spring / mass / damper system in which the influence of pitching / bouncing is negligible, and setting the state as a “vehicle reference state”. Perform the excluded image processing. The reference state without the influence of pitching / bouncing means that the vehicle's spring / mass / damper system is in an equilibrium state. It is possible to measure the position of the feature on the image.

As shown in FIG. 6A, the state in which the displacement amount due to pitching / bouncing is not superimposed (observation timings at times t ₀ and t ₁ in FIG. 2) is the image position y (t _{Since 0} ) and y (t ₁ ) are determined by the relative positional relationship between the host vehicle and the observation target Q, the displacement amount of the feature portion on the image data according to the relative positional relationship can be obtained. On the other hand, the amount of displacement of the feature due to the pitching / bouncing shown in FIG. 6B cannot be directly measured from the position of the image data of the observation target Q. Therefore, in the present embodiment, the amount of displacement is obtained using the behavior of the feature on the image. That is, in this embodiment, a reference state that is not affected by pitching / bouncing is detected by observing the behavior of the feature portion on the image, and the feature portion image data corresponding to the observation target in the detected reference state is detected. A reference state position (for example, y (t ₀ ) and y (t ₁ )) is measured, and using this as a reference position, the position on the image data of the feature corresponding to the actually imaged observation target is evaluated. Thus, the shift amount or the correction amount on the image data is obtained.

Since the vibration of about 1 to 2 Hz, which is the frequency of pitching / bouncing, is sufficiently higher than the dynamics in the change in the relative positional relationship between the observation target and the host vehicle, the motion due to pitching / bouncing is dominant in the motion on the image Can be considered to be a major factor. In general, in an image of a characteristic part to be observed imaged in a vibration state, the first-order derivative of the displacement with respect to time is the speed, and the second-order derivative is the acceleration. When the speed is the maximum value, when the acceleration is zero, or when the sign of the acceleration is reversed (changes from plus to minus / minus to plus) (t ₀ , t _{1 in} FIG. 6B), the spring It can be considered that the mass damper system is in an equilibrium state, and the displacement due to pitching / bouncing becomes substantially zero.

  For this reason, in the present embodiment, a reference state that is not affected by pitching / bouncing is detected based on moving speed information, moving acceleration information, or moving acceleration sign information on the image data of the feature corresponding to the observation target. Specifically, the reference position determination unit 40 according to the present embodiment automatically determines that the movement acceleration of the feature on the image data detected by the acceleration detection unit 20 is less than a predetermined threshold value near zero set in advance. It is determined that the vehicle is in the reference state. As another method, the reference state position determination unit 40 determines that the host vehicle is in the reference state when the moving speed of the feature on the image data detected by the acceleration detection unit 20 is the maximum value. As yet another method, the reference state position determination unit 40 compares the sign of the acceleration of the feature on the image data detected by the acceleration detection unit 20 with the acceleration of the feature extracted at the previous timing, If the sign of acceleration changes, it is determined that the host vehicle is in the reference state.

  Then, the reference position determination unit 40 determines the position on the image of the characteristic part included in the image data when the host vehicle is in the reference state as the reference state position. Further, the reference position determination unit 40 stores the reference state position of the feature portion in association with the image data capturing timing.

  The “correction amount calculation means 50” refers to the reference state position on the feature image determined by the reference position determination means 40, and calculates the correction amount on the image of each feature portion extracted by the feature extraction means 20. To do. In other words, the correction amount calculating means 50 determines whether the feature portion in the image data has a characteristic acceleration at a timing at which the moving acceleration of the characteristic portion is less than a predetermined threshold or zero, a timing at which the moving speed is substantially maximum, or a timing at which the sign of the acceleration is reversed. The position (reference state position) is compared with the position of the feature portion (to be determined) at the current timing to calculate the shift amount (correction amount) of the feature portion on the image data.

  Although not particularly limited, the correction amount calculation unit 50 calculates the predicted position of the feature portion at a predetermined timing after the elapse of a predetermined time based on the reference state position on the image of the feature portion determined by the reference position determination unit 40, The calculated predicted position of the feature portion is compared with the observation position of the feature portion of the image data captured at the predetermined timing, and the correction amount on the image of each feature portion extracted by the feature extraction unit 20 is calculated. To do.

An example of a correction amount (deviation amount) calculation method will be described. When the position of the characteristic portion when the acceleration becomes zero is continuously observed in time series, the y coordinate position after Δt seconds from an arbitrary timing can be expressed as follows. Here, the position of the feature to be observed is y (t ₀ ) and y (t ₁ ), the acceleration on the image of the feature is zero (or the velocity is almost maximum), and then the acceleration is zero (the velocity is The time until reaching (approximately the maximum) is T, and the position y (t ₁ + Δt) of the feature portion on the image at the time t ₁ + Δt when the time has elapsed by Δt from the time t ₁ is shown.

Therefore, if the y coordinate position on the image to be observed is Y (t ₁ + Δt), the position displacement Δy on the image caused by pitching / bouncing can be obtained by the following equation.

In this way, based on the stored reference state position at the previous timing, the position on the image data of the feature portion at the predetermined timing is predicted under the situation where no pitching or bouncing has occurred, By comparing / converting the observed feature position on the image, the shift amount (correction amount) of the image data at a predetermined timing including the current time can be calculated. The calculated correction amount may be output to the image data correction unit 60 that corrects the image data, or output to the vehicle posture detection device 300, the travel support device 400, the obstacle detection device 500, and other external devices. Also good.

From the correction amount (deviation amount) calculated by the correction amount calculation means 50, it is also possible to derive the pitch angle change amount caused by pitching of the host vehicle and the imaging height change amount caused by bouncing. Information on the pitch angle and the imaging height derived from the correction amount can be used for distance measurement of the object and estimation of the collision time with the object (hereinafter referred to as TTC). For example, in the estimation of the collision time estimation (TTC) between the observation target and the own vehicle, when the own vehicle (camera) and the observation target do not change significantly in the vertical direction, the position of the feature corresponding to the observation target on the image From the moving speed, it can be calculated from the formula: collision time (TTC) = y / v _y . Here, y is the ordinate value on the observation target image when the processing origin is the vanishing point, and _vy is the vertical movement speed of the observation target.

The operation of the image processing apparatus 100 configured as described above will be described. The control procedure of the image processing apparatus 100 is shown in the flowchart of FIG.
First, in step S1, the imaging means (camera) 210 of the imaging apparatus 200 continuously captures the surroundings of the host vehicle (moving body) at a predetermined timing, and stores the captured image data in the frame memory 220 (S1). .

  In step S2, the image data acquisition means 10 of the image processing apparatus 100 sequentially acquires stored image data (S2). In step S3, a feature portion corresponding to the observation target, which is a pattern or region in which the movement of the observation target can be tracked, is extracted from the image data acquired by the feature extraction unit 20 (S3).

  In step S4, the feature portion extracted in S3 is tracked between successive frames, and the position of the feature portion on the image data is measured (S4).

  In step S5, the acceleration calculation means 30 calculates the acceleration or speed of the moving feature based on the time-series position information of the feature obtained in S4 (S5).

  In step S6, the reference position determination means 40 determines whether or not the acceleration calculated in S5 is equal to or less than a predetermined threshold (S6). The set acceleration threshold is zero or a value near zero. preferable. In S6, it may be determined whether or not the speed calculated in S5 is substantially maximum. Further, in S6, it may be determined whether or not the sign of the acceleration calculated in S5 is reversed.

  In step S6, when the reference position determination means 40 determines that the acceleration is less than the predetermined threshold (or the speed is substantially the maximum value, or the sign of the acceleration is reversed), the process proceeds to step S7 and is obtained by S4. The coordinate value of the feature is stored as “reference state position” (S7). The “reference state position” is preferably stored in association with the imaging timing at which this image data was captured. After storing in the storage device 70, the process proceeds to step S9.

  When the reference position determination means 40 determines in step S6 that the acceleration is not less than a predetermined threshold (or the speed is not substantially maximum or the sign of the acceleration has not changed), the “reference state” stored in step S7 "Position" is read (S8). In step S9, an estimated position when pitching / bouncing has not occurred at a predetermined timing after the elapse of a predetermined time Δt from the time (timing t1) when the latest reference state position is obtained is calculated (S9).

  The correction amount calculation means 50 compares the actual position of the feature observed in S4 with the estimated position calculated in S9 when no pitching / bouncing has occurred, and performs the pitching / bouncing at the current time point. The displacement of the generated feature on the image is calculated. Based on this displacement, a correction amount or deviation amount of the image data is obtained.

  The calculated correction amount (deviation amount) is sent to the image data correction means 60. The image data correction unit 60 corrects the image data based on the correction amount. Thereby, the image shift due to pitching or bouncing is corrected, and accurate image data can be obtained. The corrected image data can be sent to an external device (the vehicle attitude detection device 300, the travel support device 400, or the obstacle detection device 500). The calculated correction amount is sent to the vehicle posture detection device 300, the driving support device 400, or the obstacle detection device 500 (S11), and is used as a correction amount for image data, as well as a correction amount for pitch angle or imaging height. Used as

  According to the image processing apparatus 100 of the present embodiment, even when there is no reference or mark such as a lane marker, the position error on the image caused by pitching / bouncing can be calculated, and the correction amount on the image, The pitch angle of the host vehicle or the amount of change in imaging height can be measured.

  In addition, since the shift amount caused by pitching / bouncing can be obtained accurately, an appropriate correction amount can be calculated, and accurate image data that eliminates the influence of the vehicle behavior can be obtained. For example, although no pitching has occurred, an erroneous correction process is not performed in which pitching correction is performed based on a displacement caused by other factors.

The image processing apparatus 100 according to the present embodiment, when the acceleration of the feature on the image becomes substantially zero (below a predetermined threshold), when the speed becomes maximum, or when the sign of the acceleration changes.
Pitching is performed because it is determined that the moving body is in a reference state in which no pitching / bouncing has occurred and the moving amount of the image data is calculated based on the position of the feature of the image data captured at the timing when the moving body is in the reference state. Further, it is possible to calculate a correction amount that eliminates the influence of bouncing.

  In addition, pitching is performed by extrapolating temporally the position of the imaged feature on the moving object in the reference state and calculating the correction amount (deviation amount) of the actually observed image data. Further, it is possible to calculate a more accurate correction amount that eliminates the influence of bouncing.

  In the present specification, the image processing apparatus 100 has been described as an example. However, when the image processing method of the present application is used, the image processing apparatus 100 operates in the same manner and has the same effects.

  The embodiment described above is described for facilitating the understanding of the present invention, and is not described for limiting the present invention. Therefore, each element disclosed in the above embodiment is intended to include all design changes and equivalents belonging to the technical scope of the present invention.

It is a block block diagram of an image processing apparatus. (A) (B) is the 1st figure explaining the state of the own vehicle. (A) (B) is a figure which shows the observation object (preceding vehicle) n image data imaged in the state of FIG. 2 (A) (B). (A) (B) is the 2nd figure explaining the state of the own vehicle. (A) (B) is a figure which shows the observation object (preceding vehicle) n image data imaged in the state of FIG. 3 (A) (B). (A) (B) (C) is a figure which shows the time-dependent change of the y coordinate position on an image. It is a figure which shows the control procedure of an image processing apparatus.

Explanation of symbols

DESCRIPTION OF SYMBOLS 100 ... Image processing apparatus 10 ... Image data acquisition means 20 ... Feature extraction means 30 ... Acceleration detection means 40 ... Reference position judgment means 50 ... Correction amount calculation means 60 ... Image data correction means 70 ... Storage device 200 ... Imaging device 210 ... Imaging Means 220 ... Frame memory 300 ... Vehicle posture detection device 400 ... Driving support device 500 ... Obstacle detection device

Claims (14)

An image data acquisition unit that is mounted on a mobile body and sequentially acquires image data from an imaging device that continuously images the periphery of the mobile body at a predetermined timing;
Feature extraction means for extracting a feature from each image data acquired by the image data acquisition means;
Acceleration detecting means for detecting the speed or acceleration of each feature extracted by the feature extracting means;
Reference position determination means for determining a reference state position of the feature based on the speed or acceleration of each feature detected by the acceleration detection means;
An image processing apparatus comprising: a correction amount calculation unit that calculates and outputs a correction amount of image data caused by the behavior of the moving body based on the reference state position of the characteristic portion determined by the reference position determination unit.   The reference position determination means extracts image data in which the acceleration of each feature detected by the acceleration detection means is less than a predetermined threshold, and determines the position on the image of the feature included in the image data as a reference state position The image processing apparatus according to claim 1, wherein the reference state position is stored in association with the imaging timing of the image data.   The image processing apparatus according to claim 2, wherein the predetermined threshold is a value near zero.   The reference position determination unit extracts image data in which the speed of each feature detected by the acceleration detection unit is approximately maximum, and determines the position on the image of the feature included in the image data as a reference state position. The image processing apparatus according to claim 1, wherein a reference state position is stored in association with an imaging timing of the image data.   The reference position determination unit extracts image data in which the sign of acceleration of each feature detected by the acceleration detection unit is changed, and determines a position on the image of the feature included in the image data as a reference state position. The image processing apparatus according to claim 1, wherein a reference state position is stored in association with an imaging timing of the image data.   The correction amount calculating means calculates a predicted position of the feature at a predetermined timing after a predetermined time based on the reference state position of the feature determined by the reference position determining means, and predicts the calculated feature 2. The correction amount is calculated from a shift amount on the image of each feature portion extracted by the feature extraction unit by comparing the position and the observation position of the feature portion of the image data captured at the predetermined timing. The image processing device according to any one of?   The image processing apparatus according to claim 1, further comprising an image data correction unit that corrects the image data based on the correction amount calculated by the correction amount calculation unit. Sequentially acquiring image data from an imaging device that continuously images the periphery of the moving object at a predetermined timing;
Extracting a feature from each acquired image data;
Detecting the speed or acceleration of each extracted feature;
Determining a reference state position of the feature based on the detected speed or acceleration of each feature;
And calculating a correction amount of the image data resulting from the behavior of the moving body based on the determined reference state position of the characteristic portion. The step of determining the reference state position includes:
Extracting image data in which the detected acceleration of each feature is less than a predetermined threshold;
Determining the position on the image of the feature portion included in the extracted image data as a reference state position;
The image processing method according to claim 8, further comprising a step of storing the determined reference state position in association with the imaging timing of the image data.   The image processing method according to claim 9, wherein the predetermined threshold is a value near zero. The step of determining the reference state position includes:
Extracting image data in which the speed of each detected feature is approximately maximum;
Determining the position on the image of the feature portion included in the extracted image data as a reference state position;
The image processing method according to claim 8, further comprising a step of storing the determined reference state position in association with the imaging timing of the image data. The step of determining the reference state position includes:
Extracting image data in which the sign of acceleration of each detected feature has changed;
Determining the position on the image of the feature portion included in the extracted image data as a reference state position;
The image processing method according to claim 8, further comprising a step of storing the determined reference state position in association with the imaging timing of the image data.   The step of calculating the correction amount includes: a predicted position of the feature portion at a predetermined timing after a lapse of a predetermined time based on the determined reference state position of the feature portion; and a feature portion of image data captured at the predetermined timing. The image processing method according to claim 8, wherein a correction amount on the image of each extracted characteristic part is calculated by comparing with an observation position. The image processing method according to claim 8, further comprising a step of correcting the image data based on the calculated correction amount.

Images