EP1873716A1 - Calibration apparatus of on-vehicle camera, program, and car navigation system - Google Patents
Calibration apparatus of on-vehicle camera, program, and car navigation system Download PDFInfo
- Publication number
- EP1873716A1 EP1873716A1 EP07012828A EP07012828A EP1873716A1 EP 1873716 A1 EP1873716 A1 EP 1873716A1 EP 07012828 A EP07012828 A EP 07012828A EP 07012828 A EP07012828 A EP 07012828A EP 1873716 A1 EP1873716 A1 EP 1873716A1
- Authority
- EP
- European Patent Office
- Prior art keywords
- camera
- vehicle
- calibration
- road surface
- vehicle camera
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/80—Analysis of captured images to determine intrinsic or extrinsic camera parameters, i.e. camera calibration
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/30—Subject of image; Context of image processing
- G06T2207/30248—Vehicle exterior or interior
- G06T2207/30252—Vehicle exterior; Vicinity of vehicle
Definitions
- the present invention relates to a calibration apparatus of an on-vehicle camera (hereinafter referred to as "on-vehicle camera calibration apparatus" as needed) comprising an image processing device.
- an on-vehicle camera comprising an image processing device
- a role of the camera increasingly becomes important for the purpose of safe running of a vehicle, an advanced position calibration of a car navigation, and the like; and an image processing technology for enhancing an accuracy for recognizing an object on a road surface or on a road are variously proposed.
- an installation height and imaging angle of an on-vehicle camera attached to a vehicle change from an initial set condition over time due to such an up-down vibration, gravitation by a sudden acceleration and deceleration, and other impacts.
- An on-vehicle camera calibration apparatus of the present invention comprises an on-vehicle camera, a camera parameter calculation unit configured to calculate a camera parameter from a characteristic amount of a road surface sign, and a camera parameter calibration unit configured to perform optical axis calibration control of the on-vehicle camera.
- an on-vehicle camera calibration apparatus that enables an optical axis calibration of the camera from information obtained at a high frequency on a road.
- FIG. 1 is a drawing showing an example of a general configuration of an on-vehicle camera calibration apparatus CA relating to the embodiment, including a peripheral system.
- the configuration shown in FIG. 1 is already put into practical use as part of a car navigation system CNS (hereinafter referred to as "car-navi" as needed), and further, an ITS (Intelligent Transporting System) 3 including a collaboration with an on-vehicle camera 1, an image processing device 2, and an map DB (data base) 4.
- CNS car navigation system
- ITS Intelligent Transporting System
- the on-vehicle camera 1 and the image processing device 2 are put into practical use for a purpose of calibrating position information of the car navigation 3 by image information from such a road surface sign 5 and a road sign, and for a purpose of monitoring for a safe running; and are also usable for the embodiment of the present invention.
- the on-vehicle camera 1 is attached to a rear of a user's own vehicle as a rear view camera, the camera 1 may also be placed at another position if it is possible to capture an image of the road surface sign 5.
- the image processing device 2 extracts any one of a predetermined characteristic point and characteristic line (hereinafter generically referred to also as "characteristic amount”) from a captured image of the road surface sign 5; and from the extracted information, calculates a camera parameter with respect to an installation height and installation angle of the on-vehicle camera 1 and performs an optical axis calibration.
- characteristic amount a predetermined characteristic point and characteristic line
- the on-vehicle camera 1 and the image processing device 2 detect the road surface sign 5, of which position information (longitude and latitude) is known, and calculate a distance D between the user's own vehicle and the sign 5, for example, in order to calibrate the position information of the vehicle, it becomes possible to derive the distance D more accurately because the camera 1 and the device 2 similarly configured enable the optical axis calibration.
- the on-vehicle camera calibration apparatus CA comprises the on-vehicle camera 1 and a camera parameter calculation unit 23 and camera parameter calibration unit 25 of the image processing device 2.
- any of a characteristic point and a characteristic line is included in such the road surface sign 5 as shown in the image 1a, and the image is sent to the image processing device 2.
- the input image is firstly processed in a road surface sign detection unit 21; a kind of the road surface sign 5 detected from the image is determined; the image is sent to a characteristic point and characteristic line extraction unit 22.
- the unit 22 are extracted characteristic points (four, six, or eight points according to a kind) on a white line predetermined for every kind of the road surface sign 5; or in a case of a characteristic line, a point intersecting on an extension of two characteristic lines is made a road vanishing point, its coordinate position is derived, and the position is sent to the camera parameter calculation unit 23.
- the information of the obtained characteristic points is referred to that of a reference characteristic point stored in a road surface sign DB 24 for every kind of the road surface sign 5; and the installation height and installation angle of the on-vehicle camera 1 are calculated and sent to the camera parameter calibration unit 25 as a camera parameter.
- the information sent from the characteristic point and characteristic line extraction unit 22 is the coordinate information of the road vanishing point
- the installation height and installation angle of the on-vehicle camera 1 are calculated and sent to the camera parameter calibration unit 25 as the camera parameter.
- the camera parameter calibration unit 25 updates and memorizes the camera parameter sent from the camera parameter calculation unit 23 as a new camera parameter, performs an image processing (not shown), for example, by an image and a camera parameter for deriving the distance D for a car-navi position calibration, and send out the information to the car navigation 3.
- the camera parameter calibration unit 25 issues a notice of maintenance and manually prompts a modification of a camera attachment position, when a displacement of the optical axis of the on-vehicle camera 1 exceeds a predetermined range displacement of the optical axis of the on-vehicle camera 1 and the optical axis calibration is difficult.
- the on-vehicle camera 1 in attaching the on-vehicle camera 1, it is attached (not shown) at a place such as an inspection place before new vehicle shipping or a general maintenance shop where a reference marker for the calibration exists.
- the on-vehicle camera 1 attached to a vehicle captures the reference marker from a position with a definite distance from the marker, and obtains initial set values (height h, pitch angle ⁇ , roll angle ⁇ , yaw angle ⁇ ) with respect to the installation position of the camera 1.
- a shape of the road surface sign 5 seen from the position (the angles) of the on-vehicle camera 1 is converted to a screen coordinate system, based on the initial set values, and is stored in the road surface sign DB 24 as kind determination use.
- the initial set values are used when a position calibration (measurement of the distance D) in the car-navi 3 or an image processing for a safe running is performed.
- a period is set for performing the calibration.
- the period is assumed to be from a previous calibration performance to a first vehicle driving timing after one week from the performance, and to be changeable to an arbitrary period by a user.
- the instruction of the calibration is also assumed to be manually possible by a user; the instruction of the calibration is preferably adapted to be manually performed, for example: at a time when the user's own vehicle proceeds into a general road from mountain road running whose road condition is bad; and at a temporary stop at a position where a favorable road surface sign 5 exists after the user's own vehicle.
- FIG. 3 is the flowchart of the calibration being performed in any one of a default timing and a manual timing, using the road surface sign 5 at any one of a running and a stop.
- the timing is determined whether or not to be suitable for performing the calibration (step S301). A specific determination element will be described later.
- the processing proceeds to the next step (step S302); when No, the processing ends.
- step S302 the on-vehicle camera 1 captures an image of a road surface, and the image is input in the road surface sign detection unit 21.
- step S303 an image processing for a road surface sign detection is performed.
- FIG. 4 is the flowchart showing the detail (subroutine) of the image processing of the step S303 in the road surface sign detection unit 21.
- a pre-processing such as a contrast adjustment is performed (step S401); secondly, an identification of a white line (or yellow line) on a road surface is made easier by an edge extraction processing (step S402).
- the road surface sign detection unit 21 extracts some points (characteristic point or characteristic amount) representing a characteristic of the shape of the profile (step S403). Then the road surface sign detection unit 21 refers to the information of the screen coordinate system registered in advance in the road surface sign DB 24, determines which kind of the road surface sign 5 the characteristic amount corresponds to (step S404), and saves the detection result of the road surface sign 5 (step S405).
- the unit 21 determines that the two lines are white lines respectively showing roadside zones (characteristic lines), performs the detection processing of the white lines (step S406), and saves the result (step S407).
- the processing proceeds to a step S305 of the characteristic amount extraction for the camera parameter calculation (Yes in the step S304 of FIG. 3).
- the processing proceeds to a step S311 (No in the step S304 of FIG. 3) described later.
- FIG. 5 is the flowchart showing the detail of the characteristic amount extraction (step S305 of FIG. 3) for the camera parameter calculation in the characteristic point and characteristic line extraction unit 22.
- the characteristic point and characteristic line extraction unit 22 reads the detection result of the road surface sign 5 saved in the step S405 (step S501), and extracts a characteristic amount corresponding to that (an example will be described later) defined for every kind of the road surface sign 5 (step S502). Furthermore, the characteristic point and characteristic line extraction unit 22 reads the detection result of the characteristic lines of the roadside zones (two parallel lines) saved in the step S407 (step S503), and extracts the position information of the two points with a definite interval from the respective characteristic lines, that is, the position information of total four points, as the characteristic amount (step S504).
- the processing proceeds to a step S307 (Yes in a step S306 of FIG. 3); when the unit cannot extract the predetermined characteristic amount from any one of the steps 502 to 504, the processing proceeds to the step S311 (No in the step S306 of FIG. 3).
- the processing When proceeding to the step S311 from No in the step S304 or No in the step S306, the processing returns to the image input S302 (No in the step S311); when again retuning to the step S311 from the step S304 or the step S306, the processing returns to the step S302 and is repeated before a predetermined number of times (for example, total five times) or a predetermined time (for example, for five minutes) is reached; and when the predetermined number of the times or the predetermined time is reached, the processing ends as timeout.
- a predetermined number of times for example, total five times
- a predetermined time for example, for five minutes
- the camera parameter calculation unit 23 refers to the characteristic point and reference dimension information of a characteristic point according to the kind of the road surface sign 5 stored in the road surface sign DB 24, and calculates camera parameters (height h', pitch angle ⁇ ', roll angle ⁇ ', yaw angle ⁇ ') of the on-vehicle camera 1 in capturing the image. A detailed calculation example will be described later.
- the characteristic amount is obtained from the characteristic lines of two roadside zones (see the image la in FIG.
- the camera parameter calculation unit 23 derives coordinates on the image of a point (road vanishing point), which is an intersection of the characteristic lines on an extension thereof, and calculates the camera parameters of the on-vehicle camera 1.
- a method for calculating the camera parameters from the road vanishing point is introduced in the JP H07-77431 and other document.
- the calculated camera parameters are sent to the camera parameter calibration unit 25.
- the camera parameter calibration unit 25 determines whether or not the calculated camera parameters respectively exceed thresholds that can be calibrated; when any one of the parameters exceeds its threshold and the calibration is determined impossible (No in a step S309), a position displacement of the on-vehicle camera 1 is large; and therefore, the unit 25 notices a user of maintenance information and prompts to modify the installation position or angles of the camera 1 (step S312).
- the camera parameter calibration unit 25 determines that the calibration is possible (Yes in the step S309), updates (memorizes) the parameters (step S310), performs, for example, an image processing for calculating the distance D, using the parameters, and sends the image processing information (position information) after the optical axis calibration to the ITS (car-navi) 3.
- a kind and format of the road surface sign 5 are defined in the road traffic law; in the embodiment, selecting the sign 5 of which a dimension is nationally unified or specified according to an area, a characteristic point of a reference is defined for every kind of the sign 5. In the characteristic point four to eight points are needed in order to derive two pairs of parallel segments; in FIGS. 6A, 6B, and 6C are shown some road surface signs 5 and examples of dimensions and characteristic points thereof.
- FIG. 6A is the road surface sign 5 indicating that there exists a pedestrian crossing or a bicycle crossing zone ahead.
- a width of the white line either 20cm or 30cm is used according to an area, and in some cases a cut is provided in the white line in order that rainwater is not accumulated inside the sign 5; however, an outside dimension is unified as shown and the sign 5 is marked on a road surface symmetrically in left-right and front-rear; the dimension of four characteristic points of a black circle is nationally common.
- the maximum speed of 30km/hour in FIG. 6B even if the width of the white line (yellow line) is different, the dimension of eight characteristic points defined on the outer edges of the road surface sign 5 is not changed.
- an extracted characteristic point is calibrated into a state where there exists no lens distortion.
- a known method can be used.
- a camera parameter is calculated from a characteristic point where no lens distortion exists.
- two vanishing points are calculated from predetermined points (four, six, or eight points) and vanishing lines are derived from the vanishing points. Accordingly, two pairs of parallel segments are extracted from characteristic points defined in the kind of the road surface sign 5. The two pairs are preferably orthogonal with each other.
- a, b, c, and d are the characteristic points; segments L 1 and L 3 are parallel on a road plane; segments L 2 and L 4 are parallel on the road plane, and these two pairs of the parallel segments are in a position relationship near a right angle. Therefore, on the image a line Lv is the vanishing line, wherein an intersection P 1 of straight lines from which the segments L 1 and L 3 are extended and, an intersection P 2 of straight lines from which the segments L 2 and L 4 are extended.
- the vanishing line Lv can be derived from the intersections P 1 and P 2 by using the eight characteristic points, wherein the P 1 is assumed to be a point where an extension line of characteristic points a 1 and b 2 and that of characteristic points c 1 and d 2 intersect on the image, and the P 2 is assumed to be a point where an extension line of characteristic points b 1 and a 2 and that of characteristic points d 1 and c 2 intersect on the image.
- the vanishing line Lv can be derived from: an intersection of an extension line of characteristic points a 1 and a 2 and that of characteristic points b 1 and b 2 ; and an intersection of an extension line of characteristic points c 1 and c 2 and that of characteristic points d 1 and dz.
- the extension lines are not segments nearly orthogonal with each other, the intersections P 1 and P 2 are positioned near on the image, and therefore, it is difficult to obtain the vanishing line Lv of which an accuracy is higher.
- the roll angle ⁇ ' of the on-vehicle camera 1 is derived from the derived vanishing line Lv.
- draw a straight line Lh parallel to a horizontal axis of the screen and then an angle ⁇ made by the line Lh and the vanishing line Lv derived above is the roll angle ⁇ '.
- the pitch angle ⁇ ' of the on-vehicle camera 1 is derived.
- a point Pc to be the center of the screen, and drop a perpendicular line PcQ on the vanishing line Lv from the Pc.
- the focus distance f is known as a unique value of the on-vehicle camera 1, and the distance of the segment PcQ on the image can be derived because a unit cell size on the image is known.
- the yaw angle ⁇ ' of the on-vehicle camera 1 is derived.
- RS is an orbit between frames of the road surface sign 5, and means that a characteristic point positioned on a point S on the image of a current frame is positioned on a point R on the image before a few frames.
- a straight line connecting the points R and S is assumed to be Ly
- an intersection of the straight line Ly and the vanishing line Lv is assumed to be U.
- the point U is uniquely derived even if the point S (point R) is any characteristic point.
- the point Q in FIG. 9 is similar to that in FIG. 8.
- the line Lv is the vanishing line
- a line Lz is a straight line that passes the center point Pc of the screen and is in parallel to the vanishing line Lv.
- the characteristic points of the road surface sign 5 detected previously to be a 1 , a 2 , b 1 , and b 2 ; and consider a case of a quadrangle a 1 a 2 b 1 b 2 becoming a rectangle on the road (any one of such the road surface signs 5 of a pedestrian crossing and a maximum speed).
- EG A 1 ⁇ B 1 / cos tan - 1 tan ⁇ / sin ⁇ , where ⁇ ' is the pitch angle of the on-vehicle camera 1 previously derived, and A 1 B 1 is uniquely derived because the dimension of the road surface sign 5 is standardized.
- Ratio EG / length of the segment eg on the image
- h ⁇ f ⁇ Ratio ⁇ sin ⁇
- the installation height h' of the on-vehicle camera 1 can be derived.
- a representative point means a point corresponding to the position information (latitude and longitude) of the road surface sign 5, and may also be a characteristic point for calculating the camera parameters.
- FIG. 11A consider to derive the distance D from the user's own vehicle to a representative point K. Assume a screen coordinate system of which an origin is the screen center Pc to be (x, y), and coordinates of the point K to be (xk, yk). However, assume values of (xk, yk) to be on the image.
- the coordinates (xk, yk) of the representative point K are derived specifically as follows:
- a relative position of the representative point K inside the road surface sign 5 and a calculation method of the point K based on characteristic points are defined in advance for every road surface sign 5.
- the representative point K in FIG. 6A is defined in advance to be the center of the rhombus.
- the calculation method of the representative point K in FIG. 6A is defined in advance to make an intersection of segments where characteristic points facing each other are connected.
- a characteristic point itself may also be assumed to be the representative point K.
- a camera coordinate system consists of (x', y', z') and that: the z'-axis is an optical axis direction of the on-vehicle camera 1; the x'-axis is a lateral direction in an image of the camera 1; the y'-axis is a longitudinal direction in the image; and an origin is a center of a lens (center of the image).
- x ⁇ y ⁇ z ⁇ [ - cos ⁇ - sin ⁇ 0 sin ⁇ - cos ⁇ 0 0 0 1 ] 1 0 0 0 cos ⁇ sin ⁇ 0 - sin ⁇ cos ⁇ [ cos ⁇ 0 - cos ⁇ 0 1 0 sin ⁇ - sin ⁇ cos ⁇ ] X Y - h ⁇ Z , where in the on-vehicle camera 1, ⁇ ' is the roll angle, ⁇ ' is the pitch angle, ⁇ ' is the yaw angle, and h' is the installation height.
- the distance D from the user's own vehicle to the representative point K can be derived.
- the on-vehicle camera calibration apparatus CA relating to the embodiment of the present invention is realized by dedicated hardware; and additionally, may be performed by recording a program for realizing a function of the apparatus CA in a computer readable recording medium, and by making the program recorded in the medium to be read in a computer system.
- the computer readable recording medium includes any one of: a recording medium that dynamically holds a program as in a case of the program being transferred through a communication medium of such the Internet; and a recording medium that holds the program for a definite time like a volatile memory possessed by a server when the present invention is performed.
Landscapes
- Engineering & Computer Science (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Theoretical Computer Science (AREA)
- Image Processing (AREA)
- Image Analysis (AREA)
- Traffic Control Systems (AREA)
- Closed-Circuit Television Systems (AREA)
- Studio Devices (AREA)
Abstract
Description
- The present invention relates to a calibration apparatus of an on-vehicle camera (hereinafter referred to as "on-vehicle camera calibration apparatus" as needed) comprising an image processing device.
- In an on-vehicle camera comprising an image processing device, a role of the camera increasingly becomes important for the purpose of safe running of a vehicle, an advanced position calibration of a car navigation, and the like; and an image processing technology for enhancing an accuracy for recognizing an object on a road surface or on a road are variously proposed. However, in many cases an installation height and imaging angle of an on-vehicle camera attached to a vehicle change from an initial set condition over time due to such an up-down vibration, gravitation by a sudden acceleration and deceleration, and other impacts. In these cases, because accurate image processing information cannot be obtained, it becomes necessary to periodically calibrate an optical axis of the camera.
- Therefore, there are introduced a method of using a reference marker, of which a relative position relationship to a vehicle is known, and calibrating an on-vehicle camera (for example, see claim 1 in
Japanese Patent Laid-Open Publication No. 2004-200819 Japanese Patent Laid-Open Publication No. H07-77431 - In the
JP 2004-200819 JP H07-77431 - Consequently, there is a need for an on-vehicle camera calibration apparatus that enables an optical axis calibration of the camera from information obtained at a higher frequency on a road even if the vehicle is running.
- An on-vehicle camera calibration apparatus of the present invention comprises an on-vehicle camera, a camera parameter calculation unit configured to calculate a camera parameter from a characteristic amount of a road surface sign, and a camera parameter calibration unit configured to perform optical axis calibration control of the on-vehicle camera.
- In accordance with the present invention, even if a vehicle is running, an on-vehicle camera calibration apparatus that enables an optical axis calibration of the camera from information obtained at a high frequency on a road.
-
- FIG. 1 is a drawing showing an example of a general configuration of an embodiment of the present invention.
- FIG. 2 is a detailed drawing showing an on-vehicle camera calibration apparatus of relating to the embodiment.
- FIG. 3 is, in the embodiment, a drawing showing a procedure for performing a calibration, using a road surface sign, by means of a flowchart.
- FIG. 4 is a flowchart showing a detailed procedure of a road surface detection processing in FIG. 3.
- FIG. 5 is a flowchart showing a detailed procedure of a characteristic amount extraction for a camera parameter calculation in FIG. 3.
- FIGS. 6A, 6B, and 6C are drawings respectively showing examples of characteristic points of road surface signs relating to the embodiment.
- FIGS. 7A, 7B, and 7C are drawings respectively showing examples of deriving vanishing lines from the characteristic points of the road surface signs relating to the embodiment.
- FIG. 8 is a drawing showing an example of calculating a roll angle and pitch angle of an on-vehicle camera relating to the embodiment.
- FIG. 9 is a drawing showing an example of calculating a yaw angle of the on-vehicle camera relating to the embodiment.
- FIG. 10 is a drawing showing an example of calculating a height of the on-vehicle camera relating to the embodiment.
- FIGS. 11A and 11B are drawings showing an example of a position calibration of a user's own vehicle from a camera parameter of the on-vehicle camera relating to the embodiment.
- Here will be described an embodiment of the present invention in detail, referring to drawings.
- FIG. 1 is a drawing showing an example of a general configuration of an on-vehicle camera calibration apparatus CA relating to the embodiment, including a peripheral system. The configuration shown in FIG. 1 is already put into practical use as part of a car navigation system CNS (hereinafter referred to as "car-navi" as needed), and further, an ITS (Intelligent Transporting System) 3 including a collaboration with an on-vehicle camera 1, an
image processing device 2, and an map DB (data base) 4. - The on-vehicle camera 1 and the
image processing device 2 are put into practical use for a purpose of calibrating position information of the car navigation 3 by image information from such aroad surface sign 5 and a road sign, and for a purpose of monitoring for a safe running; and are also usable for the embodiment of the present invention. In the embodiment, although the on-vehicle camera 1 is attached to a rear of a user's own vehicle as a rear view camera, the camera 1 may also be placed at another position if it is possible to capture an image of theroad surface sign 5. Theimage processing device 2 extracts any one of a predetermined characteristic point and characteristic line (hereinafter generically referred to also as "characteristic amount") from a captured image of theroad surface sign 5; and from the extracted information, calculates a camera parameter with respect to an installation height and installation angle of the on-vehicle camera 1 and performs an optical axis calibration. - When the on-vehicle camera 1 and the
image processing device 2 detect theroad surface sign 5, of which position information (longitude and latitude) is known, and calculate a distance D between the user's own vehicle and thesign 5, for example, in order to calibrate the position information of the vehicle, it becomes possible to derive the distance D more accurately because the camera 1 and thedevice 2 similarly configured enable the optical axis calibration. - Next will be described the on-vehicle camera calibration apparatus CA of in detail based on FIG. 2. As shown in FIG. 2, the on-vehicle camera calibration apparatus CA comprises the on-vehicle camera 1 and a camera parameter calculation unit 23 and camera
parameter calibration unit 25 of theimage processing device 2. - In an image captured by the on-vehicle camera 1, any of a characteristic point and a characteristic line is included in such the
road surface sign 5 as shown in the image 1a, and the image is sent to theimage processing device 2. In thedevice 2 the input image is firstly processed in a road surfacesign detection unit 21; a kind of theroad surface sign 5 detected from the image is determined; the image is sent to a characteristic point and characteristicline extraction unit 22. Then in theunit 22 are extracted characteristic points (four, six, or eight points according to a kind) on a white line predetermined for every kind of theroad surface sign 5; or in a case of a characteristic line, a point intersecting on an extension of two characteristic lines is made a road vanishing point, its coordinate position is derived, and the position is sent to the camera parameter calculation unit 23. - In the camera parameter calculation unit 23 the information of the obtained characteristic points is referred to that of a reference characteristic point stored in a road
surface sign DB 24 for every kind of theroad surface sign 5; and the installation height and installation angle of the on-vehicle camera 1 are calculated and sent to the cameraparameter calibration unit 25 as a camera parameter. When the information sent from the characteristic point and characteristicline extraction unit 22 is the coordinate information of the road vanishing point, the installation height and installation angle of the on-vehicle camera 1 are calculated and sent to the cameraparameter calibration unit 25 as the camera parameter. - The camera
parameter calibration unit 25 updates and memorizes the camera parameter sent from the camera parameter calculation unit 23 as a new camera parameter, performs an image processing (not shown), for example, by an image and a camera parameter for deriving the distance D for a car-navi position calibration, and send out the information to the car navigation 3. - Furthermore, the camera
parameter calibration unit 25 issues a notice of maintenance and manually prompts a modification of a camera attachment position, when a displacement of the optical axis of the on-vehicle camera 1 exceeds a predetermined range displacement of the optical axis of the on-vehicle camera 1 and the optical axis calibration is difficult. - Here will be described an outline of the optical axis calibration (hereinafter referred to as "calibration") of the on-vehicle camera 1 in the embodiment.
- Firstly, in attaching the on-vehicle camera 1, it is attached (not shown) at a place such as an inspection place before new vehicle shipping or a general maintenance shop where a reference marker for the calibration exists. The on-vehicle camera 1 attached to a vehicle captures the reference marker from a position with a definite distance from the marker, and obtains initial set values (height h, pitch angle θ, roll angle φ, yaw angle ψ) with respect to the installation position of the camera 1. Then in the information of characteristic points described later, a shape of the
road surface sign 5 seen from the position (the angles) of the on-vehicle camera 1 is converted to a screen coordinate system, based on the initial set values, and is stored in the roadsurface sign DB 24 as kind determination use. In addition, at this timing, because the optical axis calibration of the on-vehicle camera 1 is not necessary, the initial set values are used when a position calibration (measurement of the distance D) in the car-navi 3 or an image processing for a safe running is performed. - Next, a period is set for performing the calibration. For example, in setting a default, the period is assumed to be from a previous calibration performance to a first vehicle driving timing after one week from the performance, and to be changeable to an arbitrary period by a user. Furthermore, the instruction of the calibration is also assumed to be manually possible by a user; the instruction of the calibration is preferably adapted to be manually performed, for example: at a time when the user's own vehicle proceeds into a general road from mountain road running whose road condition is bad; and at a temporary stop at a position where a favorable
road surface sign 5 exists after the user's own vehicle. - FIG. 3 is the flowchart of the calibration being performed in any one of a default timing and a manual timing, using the
road surface sign 5 at any one of a running and a stop. - Firstly, even in any one of the default timing and the manual timing, in cases of such a high speed running and a sudden acceleration, because any of an image and the optical axis of the on-vehicle camera 1 becomes unstable, the timing is determined whether or not to be suitable for performing the calibration (step S301). A specific determination element will be described later. When the determination is Yes, the processing proceeds to the next step (step S302); when No, the processing ends.
- In the next step (S302) the on-vehicle camera 1 captures an image of a road surface, and the image is input in the road surface
sign detection unit 21. In theunit 21 an image processing for a road surface sign detection is performed (step S303). - FIG. 4 is the flowchart showing the detail (subroutine) of the image processing of the step S303 in the road surface
sign detection unit 21. - Firstly, a pre-processing such as a contrast adjustment is performed (step S401); secondly, an identification of a white line (or yellow line) on a road surface is made easier by an edge extraction processing (step S402). Next, from a profile of any one of a corner and cut of the white line in the
road surface sign 5, the road surfacesign detection unit 21 extracts some points (characteristic point or characteristic amount) representing a characteristic of the shape of the profile (step S403). Then the road surfacesign detection unit 21 refers to the information of the screen coordinate system registered in advance in the roadsurface sign DB 24, determines which kind of theroad surface sign 5 the characteristic amount corresponds to (step S404), and saves the detection result of the road surface sign 5 (step S405). When the road surfacesign detection unit 21 detects two parallel straight lines where any one of a corner and a cut is not recognized, theunit 21 determines that the two lines are white lines respectively showing roadside zones (characteristic lines), performs the detection processing of the white lines (step S406), and saves the result (step S407). After the completion of the subroutine, if the road surfacesign detection unit 21 has detected any one of theroad surface sign 5 and the roadside zones, in the next step S304, according to the result of the step S303, the processing proceeds to a step S305 of the characteristic amount extraction for the camera parameter calculation (Yes in the step S304 of FIG. 3). When the road surfacesign detection unit 21 cannot detect from the image any one of theroad surface sign 5 and the roadside zones due to some reason, the processing proceeds to a step S311 (No in the step S304 of FIG. 3) described later. - FIG. 5 is the flowchart showing the detail of the characteristic amount extraction (step S305 of FIG. 3) for the camera parameter calculation in the characteristic point and characteristic
line extraction unit 22. - The characteristic point and characteristic
line extraction unit 22 reads the detection result of theroad surface sign 5 saved in the step S405 (step S501), and extracts a characteristic amount corresponding to that (an example will be described later) defined for every kind of the road surface sign 5 (step S502). Furthermore, the characteristic point and characteristicline extraction unit 22 reads the detection result of the characteristic lines of the roadside zones (two parallel lines) saved in the step S407 (step S503), and extracts the position information of the two points with a definite interval from the respective characteristic lines, that is, the position information of total four points, as the characteristic amount (step S504). - Then when the characteristic point and characteristic
line extraction unit 22 can extract a predetermined characteristic amount from any one of the steps 502 to 504, the processing proceeds to a step S307 (Yes in a step S306 of FIG. 3); when the unit cannot extract the predetermined characteristic amount from any one of the steps 502 to 504, the processing proceeds to the step S311 (No in the step S306 of FIG. 3). - When proceeding to the step S311 from No in the step S304 or No in the step S306, the processing returns to the image input S302 (No in the step S311); when again retuning to the step S311 from the step S304 or the step S306, the processing returns to the step S302 and is repeated before a predetermined number of times (for example, total five times) or a predetermined time (for example, for five minutes) is reached; and when the predetermined number of the times or the predetermined time is reached, the processing ends as timeout.
- Next, when the processing proceeds from the step S306 to the step S307, if a characteristic amount for the camera parameter calculation sent from the characteristic point and characteristic
line extraction unit 22 is a characteristic point of theroad surface sign 5, the camera parameter calculation unit 23 refers to the characteristic point and reference dimension information of a characteristic point according to the kind of theroad surface sign 5 stored in the roadsurface sign DB 24, and calculates camera parameters (height h', pitch angle θ', roll angle φ', yaw angle ψ') of the on-vehicle camera 1 in capturing the image. A detailed calculation example will be described later. When the characteristic amount is obtained from the characteristic lines of two roadside zones (see the image la in FIG. 2), the camera parameter calculation unit 23 derives coordinates on the image of a point (road vanishing point), which is an intersection of the characteristic lines on an extension thereof, and calculates the camera parameters of the on-vehicle camera 1. A method for calculating the camera parameters from the road vanishing point is introduced in theJP H07-77431 parameter calibration unit 25. - Then the camera
parameter calibration unit 25 determines whether or not the calculated camera parameters respectively exceed thresholds that can be calibrated; when any one of the parameters exceeds its threshold and the calibration is determined impossible (No in a step S309), a position displacement of the on-vehicle camera 1 is large; and therefore, theunit 25 notices a user of maintenance information and prompts to modify the installation position or angles of the camera 1 (step S312). When the camera parameters are not more than the thresholds, the cameraparameter calibration unit 25 determines that the calibration is possible (Yes in the step S309), updates (memorizes) the parameters (step S310), performs, for example, an image processing for calculating the distance D, using the parameters, and sends the image processing information (position information) after the optical axis calibration to the ITS (car-navi) 3. - Thus it is possible to perform the calibration of the on-vehicle camera 1, and here will be described the
road surface sign 5 that is an object of an imaging. - A kind and format of the
road surface sign 5 are defined in the road traffic law; in the embodiment, selecting thesign 5 of which a dimension is nationally unified or specified according to an area, a characteristic point of a reference is defined for every kind of thesign 5. In the characteristic point four to eight points are needed in order to derive two pairs of parallel segments; in FIGS. 6A, 6B, and 6C are shown someroad surface signs 5 and examples of dimensions and characteristic points thereof. - FIG. 6A is the
road surface sign 5 indicating that there exists a pedestrian crossing or a bicycle crossing zone ahead. In a width of the white line either 20cm or 30cm is used according to an area, and in some cases a cut is provided in the white line in order that rainwater is not accumulated inside thesign 5; however, an outside dimension is unified as shown and thesign 5 is marked on a road surface symmetrically in left-right and front-rear; the dimension of four characteristic points of a black circle is nationally common. Similarly, with respect to the maximum speed of 30km/hour in FIG. 6B, even if the width of the white line (yellow line) is different, the dimension of eight characteristic points defined on the outer edges of theroad surface sign 5 is not changed. Furthermore, like theroad surface sign 5 of a pedestrian crossing in FIG. 6C, when the dimension of thesign 5 is not nationally unified and is standardized per an area (prefecture), it is possible to utilize manyroad surface signs 5 by storing the information of six characteristic points in the roadsurface sign DB 24 per the area, referring to the position information from the car-navi 3 in performing the calibration, and reading the characteristic point information per the area. - Next will be described a calculation example of the camera parameters from the characteristic points of the
road surface signs 5 described above, and a calculation example for deriving the distance D, using the calculated camera parameters. - Firstly, an extracted characteristic point is calibrated into a state where there exists no lens distortion. In order to calibrate a lens distortion, it is possible to calibrate a characteristic point of an input image into a state where there exists no lens distortion by using data of a unique distortion amount of the lens used in a camera. As this method, a known method can be used.
- Next, a camera parameter is calculated from a characteristic point where no lens distortion exists.
- Firstly, two vanishing points are calculated from predetermined points (four, six, or eight points) and vanishing lines are derived from the vanishing points. Accordingly, two pairs of parallel segments are extracted from characteristic points defined in the kind of the
road surface sign 5. The two pairs are preferably orthogonal with each other. In a case of theroad surface sign 5 "Warning of Pedestrian Crossing or Bicycle Crossing Zone" shown in FIG. 7A, a, b, c, and d are the characteristic points; segments L1 and L3 are parallel on a road plane; segments L2 and L4 are parallel on the road plane, and these two pairs of the parallel segments are in a position relationship near a right angle. Therefore, on the image a line Lv is the vanishing line, wherein an intersection P1 of straight lines from which the segments L1 and L3 are extended and, an intersection P2 of straight lines from which the segments L2 and L4 are extended. - In a case of the
road surface sign 5 "Maximum Speed 30km/hour'' shown in FIG. 7B, the vanishing line Lv can be derived from the intersections P1 and P2 by using the eight characteristic points, wherein the P1 is assumed to be a point where an extension line of characteristic points a1 and b2 and that of characteristic points c1 and d2 intersect on the image, and the P2 is assumed to be a point where an extension line of characteristic points b1 and a2 and that of characteristic points d1 and c2 intersect on the image. In addition, the vanishing line Lv can be derived from: an intersection of an extension line of characteristic points a1 and a2 and that of characteristic points b1 and b2; and an intersection of an extension line of characteristic points c1 and c2 and that of characteristic points d1 and dz. However, the extension lines are not segments nearly orthogonal with each other, the intersections P1 and P2 are positioned near on the image, and therefore, it is difficult to obtain the vanishing line Lv of which an accuracy is higher. - In a case of the
road surface sign 5 "Pedestrian Crossing" shown in FIG. 7C, because intervals between characteristic points a1 and b1, and b1 and c1, and those between characteristic points a2 and b2, and b2 and c2 are same (45cm or 50cm), two pairs of parallel segments can be obtained from the six characteristic points, and thus the vanishing line Lv can be derived from the intersections P1 and P2 on the image. - Next, the roll angle φ' of the on-vehicle camera 1 is derived from the derived vanishing line Lv. In FIG. 8, draw a straight line Lh parallel to a horizontal axis of the screen, and then an angle α made by the line Lh and the vanishing line Lv derived above is the roll angle φ'.
- Next, the pitch angle θ' of the on-vehicle camera 1 is derived. In FIG. 8, assume a point Pc to be the center of the screen, and drop a perpendicular line PcQ on the vanishing line Lv from the Pc. Here, assume a focus distance of the on-vehicle camera 1 to be f; then, the pitch angle θ' is:
- Next, the yaw angle ψ' of the on-vehicle camera 1 is derived. In FIG. 9, assume the line Lv to be the vanishing line and the point Pc to be the center of the screen. Furthermore, RS is an orbit between frames of the
road surface sign 5, and means that a characteristic point positioned on a point S on the image of a current frame is positioned on a point R on the image before a few frames. Here, a straight line connecting the points R and S is assumed to be Ly, and an intersection of the straight line Ly and the vanishing line Lv is assumed to be U. In addition, the point U is uniquely derived even if the point S (point R) is any characteristic point. Furthermore, the point Q in FIG. 9 is similar to that in FIG. 8. The yaw angle ψ' of the on-vehicle camera 1 is expressed as:
where θ' is the pitch angle of the on-vehicle camera 1 derived above, the focus distance f is known as the unique value of the camera 1, and the distance of the segment QU on the image can be derived because a unit cell size on the image is known. - Next, the height h' of the on-vehicle camera 1 is derived. In FIG. 10, the line Lv is the vanishing line; a line Lz is a straight line that passes the center point Pc of the screen and is in parallel to the vanishing line Lv. Here, assume the characteristic points of the
road surface sign 5 detected previously to be a1, a2, b1, and b2; and consider a case of a quadrangle a1a2b1b2 becoming a rectangle on the road (any one of such theroad surface signs 5 of a pedestrian crossing and a maximum speed). Furthermore, assume an intersection of straight lines a1b1 and a2b2 to be W, and an angle on the screen made by a straight line PcW and the straight line Lz to be β. Firstly, when intersections of the quadrangle a1a2b1b2 and the straight line Lz are assumed be points e and g, a practical length EG is derived on the road surface. Then assume a length of the segment a1b1 to be A1B1; and thus the length EG is expressed as:
where θ' is the pitch angle of the on-vehicle camera 1 previously derived, and A1B1 is uniquely derived because the dimension of theroad surface sign 5 is standardized. Next, a ratio of the practical length EG on the road surface to a length of a segment eg on the image is derived: - Thus according to the equations (1) to (3), the installation height h' of the on-vehicle camera 1 can be derived.
- Thus from the characteristic points of the
road surface sign 5 detected can be calculated current camera parameters (height h', pitch angle θ', roll angle φ', yaw angle ψ' of the on-vehicle camera 1), and then by assuming these to be the camera parameters after calibration, the distance D from the user's own vehicle to theroad surface sign 5 can be derived. - Next will be described a method for deriving the distance D from the user's own vehicle to the
road surface sign 5. Here, a representative point means a point corresponding to the position information (latitude and longitude) of theroad surface sign 5, and may also be a characteristic point for calculating the camera parameters. In FIG. 11A, consider to derive the distance D from the user's own vehicle to a representative point K. Assume a screen coordinate system of which an origin is the screen center Pc to be (x, y), and coordinates of the point K to be (xk, yk). However, assume values of (xk, yk) to be on the image. - The coordinates (xk, yk) of the representative point K are derived specifically as follows:
- A relative position of the representative point K inside the
road surface sign 5 and a calculation method of the point K based on characteristic points are defined in advance for everyroad surface sign 5. For example, the representative point K in FIG. 6A is defined in advance to be the center of the rhombus. Then the calculation method of the representative point K in FIG. 6A is defined in advance to make an intersection of segments where characteristic points facing each other are connected. In someroad surface sign 5, if any, a characteristic point itself may also be assumed to be the representative point K. - In FIG. 11B, assume a coordinate system on the road surface to be (X, Y, Z). Then assume that: the Z-axis is parallel to a longitudinal direction of the user's own vehicle; the X-axis is parallel to a lateral direction of the vehicle; the Y-axis is vertical to the road surface; an XZ plane is the road surface; and an origin O is just below the on-vehicle camera 1. Assuming coordinates of the representative point K in an XYZ coordinate system to be (Xk, 0, Zk), then the distance D from the vehicle (just below the on-vehicle camera 1) to the point K is expressed according to the following equation (4):
- Next, in FIG. 11B, assuming that a camera coordinate system consists of (x', y', z') and that: the z'-axis is an optical axis direction of the on-vehicle camera 1; the x'-axis is a lateral direction in an image of the camera 1; the y'-axis is a longitudinal direction in the image; and an origin is a center of a lens (center of the image). Here, a relationship between the XYZ coordinate system and the x'y'z' coordinate system is calculated according to the following equation (5):
where in the on-vehicle camera 1, φ' is the roll angle, θ' is the pitch angle, ψ' is the yaw angle, and h' is the installation height. -
- Thus according to the equations (4) to (6), the distance D from the user's own vehicle to the representative point K can be derived.
- At the representative point K, because its position information (latitude and longitude) is known, it is possible to display more accurate position information of the user's own vehicle on the car navigation 3 according to the position information and the distance D derived above.
- As described above, by utilizing various
road surface signs 5 and roadside zones in extracting the characteristic amount, it is possible to perform the calibration at a higher frequency in any one of a running and stop of the user's own vehicle. - In addition, even if there exists a favorable
road surface sign 5 around the user's own vehicle, there is a situation not suitable for the calibration due to another reason. In this case it is requested that in the step S301 the situation be determined not to be suitable for performing the calibration, and that capturing an image be prohibited. Although an example of the situation and the determination is shown as follows, the prohibition is not limited to the example. Furthermore, if permitting an image in a state below to be captured, because an end due to a timeout is enabled in the step S311 ofFIG. 3, it is not necessary to strictly control all of the followings: In reality, it is preferable to individually define determination criteria according to the performances of the on-vehicle camera 1 and theimage processing device 2. - (1) When a vehicle is running at a high speed, the optical axis of the on-vehicle camera 1 becomes unstable and an image quality becomes bad due to such a vibration of the vehicle; therefore, when a running speed exceeds a definite range, capturing an image is prohibited.
- (2) When a vehicle is in any state of a sudden acceleration and deceleration, the optical axis of the on-vehicle camera 1 becomes unstable; and therefore, information of such a gyro sensor is referred and a threshold is provided in an acceleration rate (deceleration rate), and when a driver presses any one of an accelerator and a brake, capturing an image is prohibited.
- (3) If a weight of a fellow passenger and a baggage exceeds a definite range, and a vehicle height lowers, the height of the on-vehicle camera 1 similarly lowers. Accordingly, a sensor that can measure the vehicle weight or the vehicle height is provided and a threshold suitable for the calibration is set.
- (4) When there does not exist a parallel relationship between a proceeding direction of a vehicle and a road direction, the characteristic amount of the
road surface sign 5 and the roadside zones cannot be obtained accurately; therefore, the calibration is kept from being performed from a time when a driver operates a wheel until after a few seconds after he/she returns the wheel. - In addition, the on-vehicle camera calibration apparatus CA relating to the embodiment of the present invention is realized by dedicated hardware; and additionally, may be performed by recording a program for realizing a function of the apparatus CA in a computer readable recording medium, and by making the program recorded in the medium to be read in a computer system. In addition, other than recording media such as a flexible disk, a hard disk, and an optical disk, the computer readable recording medium includes any one of: a recording medium that dynamically holds a program as in a case of the program being transferred through a communication medium of such the Internet; and a recording medium that holds the program for a definite time like a volatile memory possessed by a server when the present invention is performed.
Claims (5)
- An on-vehicle camera calibration apparatus (CA) characterised by comprising:an on-vehicle camera (1);a camera parameter calculation unit (23) configured to calculate camera parameters (h', θ', Φ', Ψ') from a characteristic amount of a road surface sign (5) photographed by the on-vehicle camera and recognized by an image processing and to output the camera parameters, wherein the camera parameters include the installation height (h') and installation angles (θ', Φ', Ψ') of the on-vehicle camera in photographing; anda camera parameter calibration unit (25) configured to perform optical axis calibration control of the on-vehicle camera by the camera parameters output from the camera parameter calculation unit.
- The on-vehicle camera calibration apparatus according to claim 1, characterised in that when the road surface sign is different in any one of a dimension and a shape according to an area, the camera parameter calculation unit identifies the area based on vehicle position information acquired from outside, acquires a reference characteristic amount of the road surface sign matching with the dimension and the shape, and calculates the camera parameters.
- The on-vehicle camera calibration apparatus according to claim 1 or 2, characterised in that: the camera parameter calibration unit determines whether or not a user's own vehicle is in a state being suitable for a calibration of the on-vehicle camera; and when the user's own vehicle is not in the state being suitable for the calibration, the camera parameter calibration unit keeps the on-vehicle camera from photographing an image (1a).
- The on-vehicle camera calibration apparatus according to claim 1, 2 or 3 characterised in that: the camera parameter calibration unit further compares thresholds on the installation height and the installation angles memorized in a memory device (25) in advance with the calculated camera parameters; and when at least one of the calculated camera parameters is determined to exceed the threshold, the camera parameter calibration unit outputs the determination result outside.
- A car navigation system (CNS) characterised by comprising:an on-vehicle camera (1) ;an image processing device (2) configured to calculate camera parameters (h', θ', Φ', Ψ') from a characteristic amount of a road surface sign (5) photographed by the on-vehicle camera and recognized by an image processing and to perform optical axis calibration control of the on-vehicle camera, wherein the camera parameters include an installation height (h') and installation angles (θ', Φ', ψ') of the on-vehicle camera in photographing; andan ITS (Intelligent Transportation System) (3) configured to perform a current position calibration of a user's own vehicle, using a distance (D) from the user's own vehicle to the photographed and recognized road surface sign calculated by the image processing device.
Applications Claiming Priority (1)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
JP2006179540A JP4820221B2 (en) | 2006-06-29 | 2006-06-29 | Car camera calibration device and program |
Publications (2)
Publication Number | Publication Date |
---|---|
EP1873716A1 true EP1873716A1 (en) | 2008-01-02 |
EP1873716B1 EP1873716B1 (en) | 2010-08-18 |
Family
ID=38480628
Family Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
EP07012828A Active EP1873716B1 (en) | 2006-06-29 | 2007-06-29 | Calibration apparatus of on-vehicle camera, program, and car navigation system |
Country Status (4)
Country | Link |
---|---|
US (1) | US8212878B2 (en) |
EP (1) | EP1873716B1 (en) |
JP (1) | JP4820221B2 (en) |
DE (1) | DE602007008504D1 (en) |
Cited By (15)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
EP2079053A1 (en) * | 2008-01-10 | 2009-07-15 | Delphi Technologies, Inc. | Method and apparatus for calibrating a video display overlay |
WO2010111993A1 (en) * | 2009-03-30 | 2010-10-07 | Conti Temic Microelectronic Gmbh | Method and device for lane detection |
EP2293223A1 (en) * | 2009-08-24 | 2011-03-09 | Autoliv Development AB | Vision system and method for a motor vehicle |
EP2309746A1 (en) * | 2008-08-05 | 2011-04-13 | Aisin Seiki Kabushiki Kaisha | Target position identifying apparatus |
WO2013087759A1 (en) | 2011-12-16 | 2013-06-20 | Morpho | Method and system for auto-calibration of a device for measuring speed of a vehicle travelling in a three-dimensional space |
WO2013101800A1 (en) * | 2011-12-28 | 2013-07-04 | Pelco, Inc. | Camera calibration using feature identification |
FR2990279A1 (en) * | 2012-05-03 | 2013-11-08 | Thales Sa | Display system for displaying scene in front of helicopter used for rescue missions of injured persons in e.g. mountains, has image processing unit verifying whether camera functions and/or estimates performance of camera from image |
EP2528048A3 (en) * | 2011-05-26 | 2014-04-23 | JENOPTIK Robot GmbH | Method for aligning and controlling the alignment of a traffic monitoring device |
US8744125B2 (en) | 2011-12-28 | 2014-06-03 | Pelco, Inc. | Clustering-based object classification |
WO2015090553A1 (en) * | 2013-12-19 | 2015-06-25 | Continental Automotive France | Method and system for calibrating a camera of a vehicle |
DE102016117500A1 (en) | 2016-09-16 | 2018-03-22 | Connaught Electronics Ltd. | Calibration of a camera of a vehicle |
WO2018202259A1 (en) * | 2017-05-05 | 2018-11-08 | Conti Temic Microelectronic Gmbh | Assembly and method for calibrating the height of a monocular vehicle camera |
CN109782755A (en) * | 2018-12-27 | 2019-05-21 | 广东飞库科技有限公司 | Control AGV calibrated, the method for AGV calibrating position, computer storage medium and AGV |
CN111223150A (en) * | 2020-01-15 | 2020-06-02 | 电子科技大学 | Vehicle-mounted camera external parameter calibration method based on double vanishing points |
CN111376841A (en) * | 2018-12-28 | 2020-07-07 | 北汽福田汽车股份有限公司 | Calibration method and device of lane departure system and storage medium |
Families Citing this family (76)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
JPH10299655A (en) * | 1997-02-18 | 1998-11-10 | Calsonic Corp | Piston assembly for swash plate type compressor |
US20110115912A1 (en) * | 2007-08-31 | 2011-05-19 | Valeo Schalter Und Sensoren Gmbh | Method and system for online calibration of a video system |
US8373763B2 (en) * | 2008-05-22 | 2013-02-12 | GM Global Technology Operations LLC | Self calibration of extrinsic camera parameters for a vehicle camera |
US8311283B2 (en) * | 2008-07-06 | 2012-11-13 | Automotive Research&Testing Center | Method for detecting lane departure and apparatus thereof |
TWI336670B (en) * | 2008-12-03 | 2011-02-01 | Ind Tech Res Inst | Method for determining the angular magnitude of imaging acquiring apparatus and vehicle collision warning system using thereof |
JP5201356B2 (en) * | 2009-02-16 | 2013-06-05 | 株式会社リコー | On-vehicle camera device calibration support method, calibration support device, and on-vehicle camera device |
JP5425500B2 (en) * | 2009-03-19 | 2014-02-26 | 本田技研工業株式会社 | Calibration apparatus and calibration method |
JP5190712B2 (en) * | 2009-03-24 | 2013-04-24 | アイシン精機株式会社 | Obstacle detection device |
JP4690476B2 (en) * | 2009-03-31 | 2011-06-01 | アイシン精機株式会社 | Car camera calibration system |
JP4751939B2 (en) * | 2009-03-31 | 2011-08-17 | アイシン精機株式会社 | Car camera calibration system |
KR101023275B1 (en) * | 2009-04-06 | 2011-03-18 | 삼성전기주식회사 | Calibration method and apparatus for automotive camera system, and method and ecu for determining angular misalignments of automotive camera system |
JP4988786B2 (en) * | 2009-04-09 | 2012-08-01 | 株式会社日本自動車部品総合研究所 | Boundary line recognition device |
JP5375958B2 (en) | 2009-06-18 | 2013-12-25 | 富士通株式会社 | Image processing apparatus and image processing method |
FR2952264B1 (en) * | 2009-10-30 | 2012-03-23 | Valeo Vision | SYSTEM FOR CALIBRATING A CAMERA SUITABLE TO EQUIP A VEHICLE. |
AT509438A1 (en) * | 2010-01-18 | 2011-08-15 | Zeno Track Gmbh | Method and system for detecting the position of a vehicle in a defined area |
JP5491235B2 (en) | 2010-03-02 | 2014-05-14 | 東芝アルパイン・オートモティブテクノロジー株式会社 | Camera calibration device |
JP5208147B2 (en) * | 2010-03-09 | 2013-06-12 | 三菱電機株式会社 | Camera calibration device for in-vehicle camera |
JP5552892B2 (en) * | 2010-05-13 | 2014-07-16 | 富士通株式会社 | Image processing apparatus and image processing program |
JP5665458B2 (en) * | 2010-09-27 | 2015-02-04 | 京セラ株式会社 | Driving support device and display method of driving support display |
JP2012093872A (en) * | 2010-10-26 | 2012-05-17 | Fujitsu Ten Ltd | Image recognition device and image recognition method |
DE102010062589A1 (en) * | 2010-12-08 | 2012-06-14 | Robert Bosch Gmbh | Camera-based method for distance determination in a stationary vehicle |
JP5706190B2 (en) * | 2011-02-28 | 2015-04-22 | 綜合警備保障株式会社 | Calculation apparatus, calculation method, and calculation program |
US9357208B2 (en) | 2011-04-25 | 2016-05-31 | Magna Electronics Inc. | Method and system for dynamically calibrating vehicular cameras |
CN103582907B (en) * | 2011-06-13 | 2016-07-20 | 日产自动车株式会社 | Vehicle-mounted pattern recognition device and lane recognition method |
JP5883275B2 (en) * | 2011-11-18 | 2016-03-09 | 東芝アルパイン・オートモティブテクノロジー株式会社 | In-vehicle camera calibration device |
JP5836774B2 (en) * | 2011-11-28 | 2015-12-24 | 三菱電機株式会社 | Overhead video generation device, overhead video generation method, video display system, and navigation device |
JP5898475B2 (en) * | 2011-11-28 | 2016-04-06 | クラリオン株式会社 | In-vehicle camera system, calibration method thereof, and calibration program thereof |
JP5971939B2 (en) * | 2011-12-21 | 2016-08-17 | アルパイン株式会社 | Image display device, imaging camera calibration method in image display device, and calibration program |
JP2013160895A (en) * | 2012-02-03 | 2013-08-19 | Denso Corp | Camera exposure setting device |
JP6141601B2 (en) | 2012-05-15 | 2017-06-07 | 東芝アルパイン・オートモティブテクノロジー株式会社 | In-vehicle camera automatic calibration device |
JP5926645B2 (en) | 2012-08-03 | 2016-05-25 | クラリオン株式会社 | Camera parameter calculation device, navigation system, and camera parameter calculation method |
KR101957256B1 (en) * | 2012-08-09 | 2019-03-12 | 현대모비스 주식회사 | Apparatus and Method for Compensating Angle of Camera Mounting Vehicle |
CN104508706B (en) * | 2012-08-15 | 2017-03-22 | 国际商业机器公司 | Feature extraction method, program and system |
JP5974762B2 (en) * | 2012-09-19 | 2016-08-23 | 日産自動車株式会社 | Camera mounting error correction apparatus and camera mounting error correction method |
JP6009894B2 (en) * | 2012-10-02 | 2016-10-19 | 株式会社デンソー | Calibration method and calibration apparatus |
JP6107081B2 (en) * | 2012-11-21 | 2017-04-05 | 富士通株式会社 | Image processing apparatus, image processing method, and program |
TWI517670B (en) | 2012-12-28 | 2016-01-11 | 財團法人工業技術研究院 | Automatic calibration for vehicle camera and image conversion method and device applying the same |
JP6081216B2 (en) * | 2013-02-14 | 2017-02-15 | 富士通テン株式会社 | Parameter acquisition apparatus, parameter acquisition method and program |
US9092695B1 (en) | 2013-03-13 | 2015-07-28 | Google Inc. | High-accuracy real-time road sign detection from images |
WO2014142367A1 (en) * | 2013-03-13 | 2014-09-18 | 엘지전자 주식회사 | Electronic device and method for controlling same |
US9488483B2 (en) * | 2013-05-17 | 2016-11-08 | Honda Motor Co., Ltd. | Localization using road markings |
JPWO2015079533A1 (en) * | 2013-11-28 | 2017-03-16 | パイオニア株式会社 | Detection device, detection method, detection program, and recording medium |
JP6201148B2 (en) | 2013-12-20 | 2017-09-27 | パナソニックIpマネジメント株式会社 | CALIBRATION APPARATUS, CALIBRATION METHOD, MOBILE BODY CAMERA HAVING CALIBRATION FUNCTION, AND PROGRAM |
KR101574195B1 (en) | 2014-02-14 | 2015-12-03 | 브이앤아이 주식회사 | Auto Calibration Method for Virtual Camera based on Mobile Platform |
JP6326624B2 (en) | 2014-04-02 | 2018-05-23 | パナソニックIpマネジメント株式会社 | CALIBRATION DEVICE, CALIBRATION METHOD, CAMERA HAVING CALIBRATION FUNCTION, AND PROGRAM |
JP6299373B2 (en) * | 2014-04-18 | 2018-03-28 | 富士通株式会社 | Imaging direction normality determination method, imaging direction normality determination program, and imaging direction normality determination apparatus |
JP6210946B2 (en) * | 2014-07-29 | 2017-10-11 | 三菱電機ビルテクノサービス株式会社 | Angle-of-view adjustment device for camera in elevator car and method of angle-of-view adjustment for camera in elevator car |
JP6354425B2 (en) * | 2014-07-30 | 2018-07-11 | 株式会社デンソー | In-vehicle camera mounting attitude detection method and apparatus |
JP6507590B2 (en) * | 2014-11-26 | 2019-05-08 | 株式会社デンソー | Image conversion apparatus and image conversion method |
FR3030091B1 (en) * | 2014-12-12 | 2018-01-26 | Airbus Operations | METHOD AND SYSTEM FOR AUTOMATICALLY DETECTING A DISALLIATION IN OPERATION OF A MONITORING SENSOR OF AN AIRCRAFT. |
EP3046076B1 (en) * | 2015-01-16 | 2018-03-14 | Delphi Technologies, Inc. | Method for calibrating an image capturing device |
JP6037351B2 (en) * | 2015-01-29 | 2016-12-07 | 三菱電機株式会社 | Vehicle image processing apparatus and vehicle image processing method |
JP5901812B1 (en) * | 2015-03-05 | 2016-04-13 | 三菱電機株式会社 | Vehicle image processing device |
KR101666959B1 (en) * | 2015-03-25 | 2016-10-18 | ㈜베이다스 | Image processing apparatus having a function for automatically correcting image acquired from the camera and method therefor |
EP3104303A1 (en) | 2015-06-11 | 2016-12-14 | Conti Temic microelectronic GmbH | Signage detection on the ground using surround view cameras |
JP6638373B2 (en) * | 2015-12-17 | 2020-01-29 | 日本電気株式会社 | Road information detection device, road information detection system, road information detection method and program |
US10339390B2 (en) * | 2016-02-23 | 2019-07-02 | Semiconductor Components Industries, Llc | Methods and apparatus for an imaging system |
JP6316468B1 (en) * | 2017-03-03 | 2018-04-25 | 三菱電機株式会社 | Calibration system |
DE102017207614A1 (en) * | 2017-05-05 | 2018-11-08 | Conti Temic Microelectronic Gmbh | Device and method for calibrating a camera system of a vehicle |
US10466027B2 (en) | 2017-06-21 | 2019-11-05 | Fujitsu Ten Corp. Of America | System and method for marker placement |
US11158088B2 (en) | 2017-09-11 | 2021-10-26 | Tusimple, Inc. | Vanishing point computation and online alignment system and method for image guided stereo camera optical axes alignment |
US11089288B2 (en) * | 2017-09-11 | 2021-08-10 | Tusimple, Inc. | Corner point extraction system and method for image guided stereo camera optical axes alignment |
JP6886079B2 (en) * | 2017-09-26 | 2021-06-16 | 日立Astemo株式会社 | Camera calibration systems and methods using traffic sign recognition, and computer-readable media |
JP6898872B2 (en) * | 2018-01-18 | 2021-07-07 | 株式会社インテック | Imaging support system, imaging support method, and imaging support program |
JP6974192B2 (en) * | 2018-01-24 | 2021-12-01 | 株式会社デンソーテン | Camera misalignment detection device and camera misalignment detection method |
CN110246184B (en) * | 2018-03-07 | 2021-06-11 | 杭州海康威视数字技术股份有限公司 | Method, device, equipment and system for determining external parameters of vehicle-mounted camera |
KR102071418B1 (en) * | 2018-05-17 | 2020-01-30 | 에스케이텔레콤 주식회사 | Apparatus and method for providing camera calibration for vehicle |
WO2020028219A1 (en) * | 2018-07-30 | 2020-02-06 | Pony Ai Inc. | System and method for calibrating on-board vehicle cameras |
JP7314486B2 (en) | 2018-09-06 | 2023-07-26 | 株式会社アイシン | camera calibration device |
CN109389650B (en) * | 2018-09-30 | 2021-01-12 | 京东方科技集团股份有限公司 | Vehicle-mounted camera calibration method and device, vehicle and storage medium |
JP7148064B2 (en) | 2018-10-25 | 2022-10-05 | 株式会社アイシン | Camera parameter estimation device, camera parameter estimation method, and camera parameter estimation program |
JP7263807B2 (en) * | 2019-02-06 | 2023-04-25 | 株式会社デンソー | Information processing equipment |
JP7217577B2 (en) | 2019-03-20 | 2023-02-03 | フォルシアクラリオン・エレクトロニクス株式会社 | CALIBRATION DEVICE, CALIBRATION METHOD |
CN112639883B (en) * | 2020-03-17 | 2021-11-19 | 华为技术有限公司 | Relative attitude calibration method and related device |
JP7505909B2 (en) | 2020-04-20 | 2024-06-25 | フォルシアクラリオン・エレクトロニクス株式会社 | Calibration device and calibration method |
WO2022254630A1 (en) * | 2021-06-02 | 2022-12-08 | 三菱電機株式会社 | Elevator camera calibration system |
Citations (3)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US5220508A (en) * | 1989-12-28 | 1993-06-15 | Kabushiki Kaisha Toyota Chuo Kenkusho | Position and heading detecting device for self controlled vehicle |
EP0691599A2 (en) * | 1994-07-05 | 1996-01-10 | Hitachi, Ltd. | Environment recognition device provided with image pickup device |
US5638116A (en) * | 1993-09-08 | 1997-06-10 | Sumitomo Electric Industries, Ltd. | Object recognition apparatus and method |
Family Cites Families (16)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
JP2514779B2 (en) | 1993-09-08 | 1996-07-10 | 住友電気工業株式会社 | Camera posture parameter calculation method |
DE19507956C2 (en) * | 1995-03-07 | 2002-11-07 | Daimler Chrysler Ag | Device for determining the vehicle distance from a side lane marking |
DE19613386A1 (en) * | 1996-04-03 | 1997-10-09 | Fiat Om Carrelli Elevatori | Industrial truck, which can be operated either manually or automatically |
FR2770317B1 (en) * | 1997-10-24 | 2000-12-08 | Commissariat Energie Atomique | METHOD FOR CALIBRATING THE ORIGINAL POSITION AND ORIENTATION OF ONE OR MORE MOBILE CAMERAS AND ITS APPLICATION TO MEASURING THE THREE-DIMENSIONAL POSITION OF FIXED OBJECTS |
JP3280001B2 (en) * | 1999-09-16 | 2002-04-30 | 富士重工業株式会社 | Stereo image misalignment adjustment device |
JP3479006B2 (en) * | 1999-09-22 | 2003-12-15 | 富士重工業株式会社 | In-vehicle camera inspection method and device |
JP3599639B2 (en) * | 2000-05-26 | 2004-12-08 | 松下電器産業株式会社 | Image processing device |
JP3820874B2 (en) * | 2000-11-22 | 2006-09-13 | 日産自動車株式会社 | Image processing apparatus for vehicle |
JP2002259995A (en) * | 2001-03-06 | 2002-09-13 | Nissan Motor Co Ltd | Position detector |
US20040212484A1 (en) * | 2001-09-04 | 2004-10-28 | Exon Science Inc. | Control device and method for automatically adjusting view angle of rearview angle of outside camera in response to output of navigation system |
JP3986360B2 (en) * | 2002-05-14 | 2007-10-03 | 松下電器産業株式会社 | Camera calibration device |
JP2004200819A (en) | 2002-12-17 | 2004-07-15 | Clarion Co Ltd | Method of calibrating mounted camera and calibrator |
US7106421B2 (en) * | 2003-04-04 | 2006-09-12 | Omron Corporation | Method of adjusting axial direction of monitoring apparatus |
JP4370915B2 (en) * | 2004-01-14 | 2009-11-25 | オムロン株式会社 | In-vehicle application selection system and in-vehicle application selection device |
JP4681856B2 (en) * | 2004-11-24 | 2011-05-11 | アイシン精機株式会社 | Camera calibration method and camera calibration apparatus |
DE102004062275A1 (en) * | 2004-12-23 | 2006-07-13 | Aglaia Gmbh | Method and device for determining a calibration parameter of a stereo camera |
-
2006
- 2006-06-29 JP JP2006179540A patent/JP4820221B2/en active Active
-
2007
- 2007-06-28 US US11/769,986 patent/US8212878B2/en not_active Expired - Fee Related
- 2007-06-29 DE DE602007008504T patent/DE602007008504D1/en active Active
- 2007-06-29 EP EP07012828A patent/EP1873716B1/en active Active
Patent Citations (3)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US5220508A (en) * | 1989-12-28 | 1993-06-15 | Kabushiki Kaisha Toyota Chuo Kenkusho | Position and heading detecting device for self controlled vehicle |
US5638116A (en) * | 1993-09-08 | 1997-06-10 | Sumitomo Electric Industries, Ltd. | Object recognition apparatus and method |
EP0691599A2 (en) * | 1994-07-05 | 1996-01-10 | Hitachi, Ltd. | Environment recognition device provided with image pickup device |
Cited By (21)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
EP2079053A1 (en) * | 2008-01-10 | 2009-07-15 | Delphi Technologies, Inc. | Method and apparatus for calibrating a video display overlay |
EP2309746A1 (en) * | 2008-08-05 | 2011-04-13 | Aisin Seiki Kabushiki Kaisha | Target position identifying apparatus |
EP2309746A4 (en) * | 2008-08-05 | 2011-09-07 | Aisin Seiki | Target position identifying apparatus |
WO2010111993A1 (en) * | 2009-03-30 | 2010-10-07 | Conti Temic Microelectronic Gmbh | Method and device for lane detection |
US8842884B2 (en) | 2009-03-30 | 2014-09-23 | Conti Temic Microelectronic Gmbh | Method and device for lane detection |
EP2293223A1 (en) * | 2009-08-24 | 2011-03-09 | Autoliv Development AB | Vision system and method for a motor vehicle |
EP2528048A3 (en) * | 2011-05-26 | 2014-04-23 | JENOPTIK Robot GmbH | Method for aligning and controlling the alignment of a traffic monitoring device |
WO2013087759A1 (en) | 2011-12-16 | 2013-06-20 | Morpho | Method and system for auto-calibration of a device for measuring speed of a vehicle travelling in a three-dimensional space |
US8744125B2 (en) | 2011-12-28 | 2014-06-03 | Pelco, Inc. | Clustering-based object classification |
WO2013101800A1 (en) * | 2011-12-28 | 2013-07-04 | Pelco, Inc. | Camera calibration using feature identification |
US9286678B2 (en) | 2011-12-28 | 2016-03-15 | Pelco, Inc. | Camera calibration using feature identification |
FR2990279A1 (en) * | 2012-05-03 | 2013-11-08 | Thales Sa | Display system for displaying scene in front of helicopter used for rescue missions of injured persons in e.g. mountains, has image processing unit verifying whether camera functions and/or estimates performance of camera from image |
US10152794B2 (en) | 2013-12-19 | 2018-12-11 | Continental Automotive France | Method and system for calibrating a camera of a vehicle |
WO2015090553A1 (en) * | 2013-12-19 | 2015-06-25 | Continental Automotive France | Method and system for calibrating a camera of a vehicle |
FR3015660A1 (en) * | 2013-12-19 | 2015-06-26 | Continental Automotive France | METHOD, SUPPORT AND SYSTEM FOR CALIBRATING A CAMERA OF A VEHICLE |
DE102016117500A1 (en) | 2016-09-16 | 2018-03-22 | Connaught Electronics Ltd. | Calibration of a camera of a vehicle |
WO2018202259A1 (en) * | 2017-05-05 | 2018-11-08 | Conti Temic Microelectronic Gmbh | Assembly and method for calibrating the height of a monocular vehicle camera |
CN109782755A (en) * | 2018-12-27 | 2019-05-21 | 广东飞库科技有限公司 | Control AGV calibrated, the method for AGV calibrating position, computer storage medium and AGV |
CN109782755B (en) * | 2018-12-27 | 2022-05-31 | 广东飞库科技有限公司 | Method for controlling AGV to calibrate and AGV to calibrate position, computer storage medium and AGV |
CN111376841A (en) * | 2018-12-28 | 2020-07-07 | 北汽福田汽车股份有限公司 | Calibration method and device of lane departure system and storage medium |
CN111223150A (en) * | 2020-01-15 | 2020-06-02 | 电子科技大学 | Vehicle-mounted camera external parameter calibration method based on double vanishing points |
Also Published As
Publication number | Publication date |
---|---|
EP1873716B1 (en) | 2010-08-18 |
JP2008011174A (en) | 2008-01-17 |
DE602007008504D1 (en) | 2010-09-30 |
US20080007619A1 (en) | 2008-01-10 |
US8212878B2 (en) | 2012-07-03 |
JP4820221B2 (en) | 2011-11-24 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
EP1873716B1 (en) | Calibration apparatus of on-vehicle camera, program, and car navigation system | |
JP4557288B2 (en) | Image recognition device, image recognition method, position specifying device using the same, vehicle control device, and navigation device | |
EP2933790B1 (en) | Moving object location/attitude angle estimation device and moving object location/attitude angle estimation method | |
CN102208012B (en) | Landscape coupling reference data generation system and position measuring system | |
US20190392228A1 (en) | Integrated sensor calibration in natural scenes | |
EP1901259A1 (en) | Vehicle and lane recognizing device | |
EP3705371A1 (en) | Obstacle detecting device | |
JP4869745B2 (en) | Depression angle calculation device, depression angle calculation method, depression angle calculation program, and image processing apparatus | |
EP2034271A1 (en) | Positioning device | |
US20110216938A1 (en) | Apparatus for detecting lane-marking on road | |
US20050169501A1 (en) | Method and apparatus for determining driving lane of vehicle, and computer product | |
JP2011504248A (en) | Method and apparatus for creating lane information | |
KR20150038622A (en) | Correction of a vehicle position by means of characteristic points | |
EP2224743B1 (en) | Vehicle periphery monitoring device, vehicle, and vehicle periphery monitoring program | |
US12000711B2 (en) | Method for verifying a digital map of a higher-level automated vehicle, corresponding device and computer program | |
JP2002140696A (en) | Image pickup and image conversion device | |
JP4852905B2 (en) | Image processing apparatus and image processing method | |
CN110378836B (en) | Method, system and equipment for acquiring 3D information of object | |
US20170103271A1 (en) | Driving assistance system and driving assistance method for vehicle | |
WO2020064543A1 (en) | Vision system and method for a motor vehicle | |
JP5062091B2 (en) | Moving object identification device, computer program, and optical axis direction specifying method | |
JP7025293B2 (en) | Vehicle position estimation device | |
JP2007011994A (en) | Road recognition device | |
KR102414632B1 (en) | Method for determining the location of a fixed object using multiple observation information | |
EP3813020A1 (en) | Camera orientation estimation |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
PUAI | Public reference made under article 153(3) epc to a published international application that has entered the european phase |
Free format text: ORIGINAL CODE: 0009012 |
|
AK | Designated contracting states |
Kind code of ref document: A1 Designated state(s): AT BE BG CH CY CZ DE DK EE ES FI FR GB GR HU IE IS IT LI LT LU LV MC MT NL PL PT RO SE SI SK TR |
|
AX | Request for extension of the european patent |
Extension state: AL BA HR MK YU |
|
17P | Request for examination filed |
Effective date: 20080207 |
|
17Q | First examination report despatched |
Effective date: 20080305 |
|
AKX | Designation fees paid |
Designated state(s): DE FR |
|
GRAP | Despatch of communication of intention to grant a patent |
Free format text: ORIGINAL CODE: EPIDOSNIGR1 |
|
RIN1 | Information on inventor provided before grant (corrected) |
Inventor name: IRIE, KOTA Inventor name: MURAMATSU, SHOJI Inventor name: MONJI, TATSUHIKO Inventor name: SHIMA, TAKESHI Inventor name: OTSUKA, YUJI |
|
GRAS | Grant fee paid |
Free format text: ORIGINAL CODE: EPIDOSNIGR3 |
|
GRAA | (expected) grant |
Free format text: ORIGINAL CODE: 0009210 |
|
AK | Designated contracting states |
Kind code of ref document: B1 Designated state(s): DE FR |
|
REF | Corresponds to: |
Ref document number: 602007008504 Country of ref document: DE Date of ref document: 20100930 Kind code of ref document: P |
|
PLBE | No opposition filed within time limit |
Free format text: ORIGINAL CODE: 0009261 |
|
STAA | Information on the status of an ep patent application or granted ep patent |
Free format text: STATUS: NO OPPOSITION FILED WITHIN TIME LIMIT |
|
26N | No opposition filed |
Effective date: 20110519 |
|
REG | Reference to a national code |
Ref country code: DE Ref legal event code: R097 Ref document number: 602007008504 Country of ref document: DE Effective date: 20110519 |
|
REG | Reference to a national code |
Ref country code: FR Ref legal event code: PLFP Year of fee payment: 10 |
|
REG | Reference to a national code |
Ref country code: FR Ref legal event code: PLFP Year of fee payment: 11 |
|
REG | Reference to a national code |
Ref country code: FR Ref legal event code: PLFP Year of fee payment: 12 |
|
REG | Reference to a national code |
Ref country code: DE Ref legal event code: R081 Ref document number: 602007008504 Country of ref document: DE Owner name: HITACHI ASTEMO, LTD., HITACHINAKA-SHI, JP Free format text: FORMER OWNER: HITACHI AUTOMOTIVE SYSTEMS, LTD., HITACHINAKA-SHI, IBARAKI, JP Ref country code: DE Ref legal event code: R081 Ref document number: 602007008504 Country of ref document: DE Owner name: HITACHI ASTEMO, LTD., HITACHINAKA-SHI, JP Free format text: FORMER OWNER: HITACHI, LTD., TOKYO, JP |
|
PGFP | Annual fee paid to national office [announced via postgrant information from national office to epo] |
Ref country code: FR Payment date: 20230510 Year of fee payment: 17 Ref country code: DE Payment date: 20230502 Year of fee payment: 17 |