DE112015005317B4 - IMAGE CONVERSION DEVICE AND IMAGE CONVERSION METHOD - Google Patents

Abstract

An image converting device (10) which obtains a captured image from an in-vehicle camera (2) imaging the surroundings of a vehicle (1), converts the captured image into a bird's-eye view image showing a top view of the vehicle, and the image off outputs a bird's eye view on an on-vehicle monitor (3), the image converting device comprising:an image converting unit (12) which converts the captured image into the bird's eye view image based on a position and an angle of the vehicle camera by referring to a conversion table, wherein the conversion table obtained by calculating, based on the position and the angle of the vehicle camera, pixel positions in the captured image which correspond to pixel positions in the bird's-eye view image;a feature point extracting unit (15) extracting a plurality of feature points from the bird's-eye view image or from extrahi to the captured image ert;a calculation determination unit (16) which determines whether or not to calculate the position and the angle of the in-vehicle camera based on a distribution of the feature points extracted from the bird's-eye view image or the captured image;a calculation - a unit (17) which calculates the position and the angle of the in-vehicle camera based on the feature points when the calculation determination unit determines that the position and the angle of the in-vehicle camera are to be calculated;a change amount obtaining unit (18) which, by comparing the position and angle of the in-vehicle camera recalculated by the calculation unit with the previous position and angle acquires a change amount; anda conversion table updating unit (19) which, when the amount of change is equal to or smaller than a predetermined threshold amount of change, updates the conversion table based on the recalculated position and angle of the in-vehicle camera.

Description

TECHNICAL AREA

The present invention relates to a setting adjustment technology applied to a vehicle having an in-vehicle camera to convert a picked-up image obtained by the in-vehicle camera into a bird's-eye view image.

STATE OF THE ART

Various commercialized technologies enable a driver of a vehicle to view the surroundings of the vehicle by capturing an image of the surroundings of the vehicle with an in-vehicle camera and displaying the obtained captured image on an on-vehicle monitor. Furthermore, widespread technologies convert the picked-up image acquired by the in-vehicle camera into a bird's-eye view image and display the bird's-eye view image on the in-vehicle monitor instead of simply displaying the picked-up image acquired by the in-vehicle camera on the in-vehicle monitor. The bird's-eye view image refers to an image obtained by converting the captured image in such a manner as to show a plan view of the surroundings of the vehicle. When the bird's-eye view is obtained through accurate conversion, it shows the current positional relationship between the vehicle and, for example, another vehicle or a pedestrian. Consequently, viewing the obtained bird's-eye view image enables the driver to easily and accurately grasp the surroundings of the vehicle. In order to convert the captured image into an accurate bird's-eye view image, it is necessary to acquire the position and angle of the vehicle camera accurately.

Accordingly, before the vehicle is shipped from the factory, the position and angle of the in-vehicle camera are individually measured to position and align the in-vehicle camera based on the design requirements and reduce the influence of assembly errors. The position and angle of the vehicle camera can be calculated in a manner described below. First, a plurality of feature points are extracted from the captured image obtained by the vehicle camera. Second, coordinates on the captured image on which the feature points are located are associated with coordinates in a currently captured space. Third, the positions of the associated feature points are replaced in a relational formula with variables that represent the position and angle of the vehicle's camera. Finally, the resulting relational formula is solved to calculate the position and angle of the vehicle camera.

The position and angle of the vehicle camera attached to the vehicle can subsequently vary, for example due to fastening elements that have become loose after delivery from the factory. In the event of such unintentional variation, the position and angle of the vehicle camera will be different from those set before factory shipment. A technology proposed under such circumstances extracts feature points from an image captured while the vehicle is running, and recalculates the position and angle of the vehicle camera (see JP 2013 - 222 302 A ).

From the U.S. 2008/0 181 591 A1 a device for estimating a camera pose and a corresponding method are known. The ones from the U.S. 2008/0 181 591 A1 A known camera pose estimation apparatus comprises: a generator configured to generate bird's-eye view image data by transforming a viewpoint of captured image data obtained from the camera based on a pose parameter indicative of the pose of the camera; a calculator configured to calculate parallelism between lines in a bird's-eye view image indicated by the bird's-eye view image data generated by the generator; and a posture estimator configured to estimate the posture parameter from the parallelism calculated by the computer.

the JP 2008 - 131 250 A relates to a correction device for a vehicle camera that U.S. 2013/0 135 474 A1 teaches a vehicle camera system with its own calibration method, and the U.S. 2008/0007619A1 relates to a calibration device for a vehicle camera, a corresponding program and a vehicle navigation system.

SUMMARY OF THE INVENTION

However, although the feature points are extracted from a captured image to calculate the position and angle of the vehicle camera in the same manner as in a car repair shop, the technology proposed above does not always achieve sufficient accuracy.

It is an object of the present invention to provide a technology for accurately calculating the position and angle of a vehicle camera and storing a captured image in a battery rates to convert a bird's-eye view image without having to take a vehicle to an auto repair shop.

The object is achieved by an image conversion device according to claims 1 and 3 and an image conversion method according to claim 6, respectively. Advantageous developments are the subject of the subclaims.

The feature points can be arranged artificially, for example in a car repair shop. The position and the angle of the in-vehicle camera can be calculated by using the feature points having a distribution suitable for calculating the position and the angle of the in-vehicle camera. When the vehicle is running, the feature points are extracted from the current environment within an imaging area. As a result, feature points with various distributions are extracted. While the vehicle is running, the feature points suitable for calculating the position and angle of the vehicle camera are not necessarily extracted. Therefore, as long as the position and angle of the in-vehicle camera are calculated, the position and angle of the in-vehicle camera is accurately calculated even while the vehicle is running, and the extracted feature points have a distribution suitable for calculating the position and angle of the in-vehicle camera. As a result, the captured image is accurately converted to a bird's-eye view image.

character list

• 1 ein Diagramm, welches ein Fahrzeug zeigt, in welchem eine Bildumwandlungsvorrichtung entsprechend einem Ausführungsbeispiel der vorliegenden Erfindung angeordnet ist;
• 2 eine Reihe von schematischen Diagrammen, welche Prinzipien der Umwandlung eines aufgenommenen Bildes in ein Bild aus Vogelperspektive zeigen;
• 3 eine Reihe von Diagrammen, welche zeigen, wie eine Umwandlungstabelle zur Umwandlung eines aufgenommenen Bildes in ein Bild aus Vogelperspektive verwendet wird;
• 4 ein Blockdiagramm, welches die interne Struktur der Bildumwandlungsvorrichtung zeigt;
• 5 ein Flussdiagramm, welches die erste Hälfte eines Umwandlungstabellen-Aktualisierungsprozesses zeigt, der durch die Bildumwandlungsvorrichtung ausgeführt wird;
• 6 ein Flussdiagramm, welches die zweite Hälfte des Umwandlungstabellen-Aktualisierungsprozesses zeigt, der durch die Bildumwandlungsvorrichtung ausgeführt wird;
• 7 eine Reihe von Diagrammen, welche konkrete Beispiele für Umgebungswerte zeigen, die an die Werte-Sensoren gegeben werden;
• 8 eine Reihe von Diagrammen, welche zeigen, wie Merkmalspunkte aus dem aufgenommenen Bild und aus dem Bild aus Vogelperspektive extrahiert werden;
• 9 eine Reihe von Diagrammen, welche zeigen, wie die Verteilungen der Merkmalspunkte bestätigt werden;
• 10 eine Reihe von Diagrammen, welche ein Bild aus Vogelperspektive zeigen, das erhalten wird, wenn das Fahrzeug in Nick-Richtung geneigt ist;
• 11 eine Reihe von Diagrammen, welche ein Bild aus Vogelperspektive zeigen, das erhalten wird, wenn das Fahrzeug in Roll-Richtung geneigt ist;
• 12 eine Reihe von Diagrammen, welche ein Bild aus Vogelperspektive zeigen, das erhalten wird, wenn das Fahrzeug nach unten abgesenkt ist; und
• 13 ein Diagramm, welches die Auswahl einer Variablen aus dem Nick-Winkel, dem Roll-Winkel und der Vertikalstellung einer Fahrzeugkamera auf Basis der Verteilung von Merkmalspunkten zeigt.

The above and other aspects, features and advantages of the present disclosure will become apparent from the following description with reference to the accompanying drawings. In the drawing is:

• 1 14 is a diagram showing a vehicle in which an image converting device according to an embodiment of the present invention is mounted;
• 2 Fig. 12 is a series of schematic diagrams showing principles of converting a captured image into a bird's-eye view image;
• 3 Fig. 12 is a series of diagrams showing how a conversion table is used to convert a captured image into a bird's-eye view image;
• 4 Fig. 14 is a block diagram showing the internal structure of the image converting device;
• 5 Fig. 14 is a flowchart showing the first half of a conversion table update process performed by the image conversion device;
• 6 Fig. 14 is a flowchart showing the second half of the conversion table update process performed by the image conversion device;
• 7 a series of charts showing concrete examples of environmental values given to the value sensors;
• 8th a series of diagrams showing how feature points are extracted from the captured image and from the bird's-eye view image;
• 9 a series of graphs showing how feature point distributions are confirmed;
• 10 Fig. 14 is a series of diagrams showing a bird's-eye view image obtained when the vehicle is inclined in the pitch direction;
• 11 Fig. 14 is a series of diagrams showing a bird's-eye view image obtained when the vehicle is inclined in the roll direction;
• 12 Fig. 14 is a series of diagrams showing a bird's-eye view image obtained when the vehicle is lowered down; and
• 13 Fig. 12 is a diagram showing selection of one of pitch angle, roll angle and vertical position of a vehicle camera based on feature point distribution.

EMBODIMENT FOR CARRYING OUT THE INVENTION

An embodiment of an image converting apparatus according to the present invention will be described below.

A-1. Configuration of the device according to the present embodiment

1 Fig. 12 shows an outline structure of a vehicle 1 in which the image converting device 10 is mounted. As shown, the vehicle 1 has not only the image conversion device 10 but also an in-vehicle camera 2 and an in-vehicle monitor 3 . The vehicle camera 2 is arranged at the front of the vehicle 1 . The vehicle-side monitor 3 can be viewed from the driver's seat. The vehicle 1 further includes a vehicle speed sensor 41, a steering sensor 42, height sensors 43a-43d, a solar radiation sensor 44 and a rain sensor 45. the Vehicle speed sensor 41 acquires a running speed of the vehicle. The steering sensor 42 acquires a steering angle. The height sensors 43a-43d are arranged at the four corners of the vehicle 1 to detect vertical displacement. The solar radiation sensor 44 detects the sunlight intensity. The rain sensor 45 detects the amount of raindrops. The sensors detect information about the surroundings of the vehicle 1 or the status of the vehicle 1 and provide the image conversion device with the detection information. The sensors correspond to a detector.

The vehicle camera 2 is a wide-angle camera that has a fisheye lens. An image captured by the vehicle camera 2 shows the front surroundings of the vehicle 1 including a road surface. The image converting device 10 acquires a captured image from the in-vehicle camera 2 , converts the captured image into a bird's-eye view image, and outputs the bird's-eye view image to the on-vehicle monitor 3 . The captured image shows the road surface. Thus, the bird's-eye view image obtained by converting the picked-up image shows a plan view of the road surface in front of the vehicle 1. The conversion of the picked-up image to the bird's-eye view image will be described below.

2 Fig. 12 schematically shows the principles of converting a captured image into a bird's-eye view image. When the vehicle camera 2 captures an image of the road surface 21, the captured image 22 shows the road surface 21 within an imaging range of the vehicle camera 2 by projecting the road surface 21 onto an image plane perpendicular to the line of sight of the vehicle camera 2, as in FIG 2(a) shown. When the captured image is projected onto the road surface using the light of a light source arranged at the position of the in-vehicle camera 2, by reversing the above projection relationship, the projected image will match the actual road surface 21.

The road surface shown in the captured image is in 2 B) projected onto a road surface level 23 . The road surface plane 23 is a plane which is on the same plane as that in FIG 2(a) shown road surface 21 is arranged. Thus, the road surface level 23 can be considered as a projection surface. When an image of the road surface is displayed in the road surface plane 23 captured by a virtual camera 24 arranged to capture an image of the road surface in plan view, a bird's-eye view image 25 showing the plan view of the road surface in front of the Vehicle camera 2 shows.

Regarding the relationship between the imaging range and the position and the angle of the in-vehicle camera 2, the position and the shape of a trapezoidal area of the road surface 21 based on the position and the angle of the in-vehicle camera 2 are calculated as the imaging range shown in 2(a) is shown. In order to convert a captured image into a bird's-eye view image, the position and angle of the in-vehicle camera 2 must remain constant when an image has just been captured and when the captured image is projected onto the virtual road surface 23 (the projection screen).

The vehicle camera 2 is arranged on the vehicle 1 . Therefore, under normal conditions, the position and the angle of the vehicle camera 2 with respect to the road surface 21 can be regarded as constant. Thereby, the image conversion device 10 has a conversion table prepared, which is obtained by calculating the projection relationship under predetermined conditions that the position and the angle of the in-vehicle camera 2 are constant. The projection relationship is as with reference to 2 described. The conversion table is referred to when a captured image is to be converted into a bird's-eye view image, thereby reducing the processing load by eliminating the need for calculation of the above-mentioned projection relationship every image capture.

3 shows how to reference the conversion table for converting a captured image to a bird's-eye view image. The conversion table has data representing the relationship between the pixel positions of the bird's-eye view image and the pixel positions of the captured image. For example, the conversion table associates pixels located at the coordinates (Xe, Yf) of the bird's-eye view image in 3(b) lie with pixels located at the coordinates (Ag, Bh) of the captured image in 3(a) lie. Accordingly, the image data (brightness, saturation, etc.) of a pixel located at the coordinates (Ag, Bh) of the picked-up image is applied to a pixel located at the coordinates (Xe, Yf) of the bird's-eye view image. When the conversion table is referenced with respect to each pixel of the bird's-eye view image, the captured image can be converted into the bird's-eye view image.

As described above, the conversion table is prepared based on the position and the angle of the in-vehicle camera 2 . Therefore, the image conversion device 10 is capable of not only converting the captured image into the bird's-eye view image but also recalculating the position and angle of the in-vehicle camera 2 and updating the conversion table based on the recalculated position and angle.

4 12 shows an internal structure of the image converting device 10. As shown, the image converting device 10 has a captured image acquiring unit 11, an image converting unit 12, and a display unit 13. FIG. The captured image acquiring unit 11 acquires a captured image from the in-vehicle camera 2. The image converting unit 12 refers to the conversion table and converts the captured image into a bird's-eye view image as described above. The display unit 13 outputs the bird's-eye view image on the vehicle-side monitor 3 .

In order to calculate the position and angle of the in-vehicle camera 2 and update the conversion table, the image conversion device 10 further includes a value evaluation unit 14, a feature point extraction unit 15, a calculation determination unit 16, a calculation unit 17, a change amount acquisition unit 18 and a conversion table update unit 19.

The value evaluation unit 14 obtains the detection information obtained by the sensors that the vehicle 1 has, namely the vehicle speed sensor 41, the steering sensor 42, the height sensors 43a-43d, the solar radiation sensor 44 and the rain sensor 45. These sensors are generally referred to as value sensors 4 in the following. The value evaluation unit 14 indicates a value based on the information obtained from each of the value sensors 4 and evaluates this number of points in a comprehensive manner. The evaluation of the values will be described later in detail. In short: The values are evaluated by evaluating the information about the environment of the vehicle 1 or the state of the vehicle 1 and determines whether the current situation is suitable for feature point extraction. Feature points have distinctive features within a captured image of a scene and can be distinguished from other points. Furthermore, the feature points are such that their coordinates can be identified within a captured image or a bird's-eye view image showing the scene. The feature points are extracted to calculate the location and angle of the vehicle camera 2.

The value evaluation unit 14 determines whether or not the feature points are extracted as described above, and thus corresponds to an extraction determination unit.

The feature point extraction unit 15 extracts a plurality of feature points from a captured image or from a bird's-eye view image to calculate the position and the angle of the in-vehicle camera 2 .

The calculation determination unit 16 confirms the distribution of the feature points and determines whether or not to calculate the position and the angle of the in-vehicle camera 2 based on information indicating the width of the confirmed distribution.

The calculation unit 17 calculates the position and angle of the in-vehicle camera 2 based on the extracted feature points. The position and the angle of the in-vehicle camera 2 are calculated by using the correspondence of the positional relationship between a plurality of feature points extracted from a captured image or a bird's-eye view image and the positional relationship between a plurality of feature points from a currently captured image Scene. A method of calculating the position and the angle of the vehicle camera 2 is known and will not be described in detail.

The change amount obtaining unit 18 obtains a change amount by comparing the position and angle newly calculated by the calculation unit with the previous position and angle.

When the change amount obtained by the change amount obtaining unit 18 is equal to or smaller than a predetermined threshold change amount, the conversion table updating unit 19 updates the conversion table based on the recalculated position and angle of the in-vehicle camera 2. After updating the conversion table For example, the image converting unit 12 converts a captured image into a bird's-eye view image by referring to the updated conversion table.

The nine units from the captured image acquiring unit 11 to the conversion table updating unit 19 are concepts which correspond to the one shown in Fig conversion device contained functionalities are classified. Thus, the nine units do not imply that the image conversion device 10 is physically divided into nine units. The units are implemented by an executable computer program executable by a CPU, an electronic circuit having an LSI and a memory, or a combination thereof. A conversion table update process executed using the nine units will be described in detail below.

A-2. Bird's-eye view image generation process

the 5 and 6 12 are flowcharts showing the conversion table update process executed by the image conversion device 10 according to the present embodiment. The conversion table update process is performed to update the conversion used to convert a captured image into a bird's-eye view image, and is performed separately from a conversion process for displaying a bird's-eye view image on the on-vehicle monitor 3 . The bird's-eye view image conversion process is repeatedly executed at intervals corresponding to the image pickup cycle (e.g., 30 Hz) of the in-vehicle camera 2 . However, the conversion table updating process described below may be executed at the same intervals as the bird's-eye view image conversion process or at other suitable intervals (e.g., at 10-minute intervals).

The conversion table update process is started by calculating the environmental values (S101). The environmental values are indexes used for evaluating the surroundings of the vehicle 1 and the condition of the vehicle 1 by evaluating the information detected by the value sensors 4 .

7 shows concrete examples of the environmental values. As in 7(a) 1, the environmental values obtained by the value sensors 4 are given to the detection information of the vehicle speed sensor 41, the steering sensor 42, the height sensors 43a-43d, the solar irradiance sensor 44 and the rain - Sensor 45. The detection information related thereto is converted into numeric values. Thus, the environmental values are given based on the magnitude of the numeric value. The environmental values obtained from the value sensors 4, which have been given to the detection information, are denoted by the value signs S1 to S5, which respectively indicate the vehicle speed sensor 41, the steering sensor 42, the height sensors 43a- 43d, the solar radiation sensor 44 and the rain sensor 45 denote.

A method of providing an environmental value for the determination information obtained from the vehicle speed sensor 41, for example, will now be described. The detection information obtained from the vehicle speed sensor 41 is the running speed of the vehicle 1. Therefore, the surrounding value S1 decreases as the running speed increases. 7(b) shows a case where the environmental value S1 is given. As indicated in the example, the environmental value S1 is 100 at a vehicle speed of 0 to 20 (km/h), 80 at a vehicle speed of 20 to 80 (km/h), and 60 at a vehicle speed of 80 (km/h) or higher. The surrounding value S1 decreases as the vehicle speed increases because the increase in vehicle speed tends to blur a captured image of a subject and adversely affects the accuracy of feature point extraction.

For the value sensors 4 other than the vehicle speed sensor 41, namely the steering sensor 42, the height sensors 43a-43d, the solar radiation sensor 44 and the rain sensor 45, the environmental values are individually in the same manner given as for the vehicle speed sensor 41.

The steering sensor 42 determines the amount of the steering angle shift per unit of time. The environmental value S2 decreases as the amount of steering angle displacement increases. This is because when the amount of steering angle shift is large, the vehicle 1 tilts due to a load change, to the detriment of feature point extraction.

The height sensors 43a-43d detect the displacement amount of the inclination of the vehicle 1 per unit time by detecting the vertical displacement amounts in front, rear, left and right of the vehicle 1. When the vehicle 1 tilts, the surrounding value S2 decreases as the displacement amount increases like this is the case with ambient value S2.

The ambient value S4, which is to be assigned to an amount of solar radiation determined by the solar radiation sensor 44, falls when the amount of solar radiation decreases. This is because the captured image of a scene becomes darker as the amount of solar radiation decreases. If the captured image is dark, it is very likely that the difference in its brightness value becomes small. Thus, the accuracy of the feature point extraction is low.

The environmental value S5, which is to be assigned to rain detected by the rain sensor 45, decreases as the rainfall increases. This is because a captured image is likely to become blurry as precipitation increases. If the captured image is blurry, the accuracy of feature point extraction will be low.

After the environmental values S1 - S5 are individually associated with the detection information obtained by the vehicle speed sensor 41, the steering sensor 42, the height sensors 43a-43d, the solar radiation sensor 44 and the rain sensor 45, a Total Ambient Value S calculated as in 7(c) shown.

In 7(c) the overall environmental value S is the sum of the environmental values S1 to S5. However, an alternative calculation method can be used as long as the information determined by the value sensors 4 is taken into account. If it is determined that the vehicle speed exerts a greater influence than the other pieces of determination information, a special focus can be placed on the judgment of the surrounding value S1.

Also, sensors other than value sensors 4 may be used in addition to or in place of the sensors described above. An environmental value can be given by allowing a gyro sensor to detect the inclination of the vehicle 1 based on the same idea as the environmental value S3. In addition, an environmental value for detecting precipitation can be given by a wiper signal based on the same idea as the environmental value S5. Furthermore, the running speed and the inclination degree of the vehicle 1 can be predicted from traffic and road information obtained by a car navigation system to calculate an environmental value considering such predicted information.

As described above, the environmental values are used as indexes for evaluating the surrounding environment such as weather differences and surrounding brightness of the vehicle 1, or for evaluating the state of the vehicle 1 such as posture changes and vibration. When the surrounding values are high, clear captured images are highly likely to be obtained by reducing the influence of noise caused by the surrounding environment and the state of the vehicle. A situation in which clearly captured images are obtained is suitable for feature point extraction.

After the environmental value is calculated as described above (S101), it is determined whether the environmental value is equal to or greater than a predetermined threshold (S102). The above overall environmental value S is designed for determination, and the value of the threshold value for determination can be set as desired. When it is determined whether the total surrounding value S is equal to or greater than the predetermined threshold value, the surrounding environment of the vehicle 1 and the state of the vehicle 1 can be comprehensively known from the detection information obtained from the various value sensors 4 were obtained can be evaluated to determine whether the current situation is suitable for feature point extraction. However, the method of evaluating the environment is not limited to the one described above. If the number of surrounding values does not meet the predefined criteria of surrounding values S1 to S5, it can be determined that the current situation is not suitable for feature point extraction.

When it is determined that the environmental value does not match the predetermined threshold (S102: NO), the conversion table update process ( 6 ).

In contrast, when it is determined that the surrounding value is equal to or greater than a predetermined threshold (S102: YES), feature points are extracted. In the present embodiment, a captured image is acquired (S103), then the captured image is converted into a bird's-eye view image (S104), and feature points are extracted (S105).

8th Fig. 12 shows how a plurality of feature points are extracted from a captured image and from a bird's-eye view image. As already mentioned, the feature points have anomalies within a captured image of a scene and can be distinguished from other points. In addition, the feature points are such that their coordinates can be ascertained in a captured image or a bird's-eye view image showing the scene. A typical feature point would be a corner point where a straight line intersects with a straight line. However, the feature point can also be extracted from the boundary of a straight line. For example, the feature point can be extracted from road markings such as lane markings, pedestrian crossing markings, and stop line markings gene because there is a significant difference in brightness value at the boundary between such a marking and an exposed piece of asphalt.

(STOP) extrahiert. Das Ziel in 8 weist eine bekannte Form auf, weil die tatsächlichen Fahrbahnmarkierungen auf der linken und rechten Seite zueinander parallel sind, während das Fahrzeug 1 gerade weiterfährt und die japanische Straßenmarkierung

(STOP) eine Vielzahl an Eckpunkten aufweist, an denen sich Linien gegenseitig senkrecht kreuzen. Formen und Abmessungen von Straßenmarkierungen, welche zur Merkmalspunkt-Extraktion geeignet sind, können zur Vereinfachung der Merkmalspunkt-Extraktion in eine Datenbank aufgenommen werden, die von der Merkmalspunkt-Extraktionseinheit 15 referenziert wird.Black dots in the 8(a) and (b) indicate a location where a feature point is extracted. As already mentioned, feature points are extracted in order to calculate the position and the angle of the vehicle camera 2 . Furthermore, the position and the angle of the in-vehicle camera 2 are calculated using the correspondence of the positional relationship between a plurality of feature points extracted from an image and the positional relationship between a plurality of feature points on a currently captured image of the scene. The match can be used, for example, by extracting the feature points from a target that has a known shape and is shown in the currently captured image of the scene. In 8th become the feature points from the lane markings on the left and right sides of the vehicle 1 and from the Japanese road marking

(STOP) extracted. The target in 8th has a known shape because the actual lane markings on the left and right are parallel to each other while the vehicle 1 is running straight and the Japanese road marking

(STOP) has a large number of vertices where lines cross each other perpendicularly. Shapes and dimensions of road markings suitable for feature point extraction may be included in a database referred to by the feature point extraction unit 15 to facilitate feature point extraction.

The feature points can, as in 8(a) shown can be extracted from a captured image. The conversion table update process according to the present embodiment extracts the feature points as shown in FIG 8(b) shown from a bird's eye view image. The difference between feature point extraction from the captured image in 8(a) and feature point extraction from the bird's-eye view image in 8(b) is described below.

(STOP) gezeigt, sodass tatsächlich parallele Teile nicht parallel aussehen und aktuell sich senkrecht kreuzende Teile nicht senkrecht zueinander aussehen. Wenn die Merkmalspunkte (schwarze Punkte in 8(a)), die wie oben beschrieben gezeigt sind, aus der Fahrbahnmarkierung extrahiert sind, ist die Positionsbeziehung zwischen den im Bild gezeigten Merkmalspunkten nicht gleich der Positionsbeziehung zwischen den aktuellen Merkmalspunkten.The image captured in 8(a) was taken while the vehicle 1 was running straight ahead along a straight line. Actual lane markings are parallel, straight lines drawn along a lane. However, the left and right lane markings shown in the captured image in 8(a) are shown substantially in an inverted V-shape. Furthermore, the Japanese road marking

(STOP) shown so actually parallel parts don't look parallel and actually perpendicularly crossing parts don't look perpendicular to each other. If the feature points (black points in 8(a) ) shown as described above are extracted from the lane marker, the positional relationship between the feature points shown in the image is not equal to the positional relationship between the current feature points.

(STOP) sehen, wie in der realen Welt, parallel aus und die senkrecht kreuzenden Teile sehen senkrecht zueinander aus. Wenn die Merkmalspunkte (schwarze Punkte in 8(b)), die wie oben beschrieben gezeigt sind, aus der Fahrbahnmarkierung extrahiert sind, ist die Positionsbeziehung zwischen den im Bild gezeigten Merkmalspunkten gleich der Positionsbeziehung zwischen den aktuellen Merkmalspunkten.Meanwhile, see the picture from a bird's eye view in 8(b) , the left and right lane markings, as in the real world, parallel out, two parallel straight lines of Japanese road marking

(STOP) look parallel, as in the real world, and the perpendicularly crossing parts look perpendicular to each other. If the feature points (black points in 8(b) ) shown as described above are extracted from the lane marking, the positional relationship between the feature points shown in the image is equal to the positional relationship between the current feature points.

Accordingly, when the feature points extracted from a bird's-eye view image show the current positional relationship, the positional relationship between the feature points shown in an image is reconciled with the current positional relationship much more easily than when feature points are from a captured image not showing the current positional relationship , are extracted. Thus, the extraction of the feature points from the bird's-eye view image enables the position and angle of the in-vehicle camera 2 to be easily and accurately calculated.

For the above reasons, the feature points are extracted (S104 from 5 ) after converting a captured image into a bird's-eye view image (P105 off 5 ). The processing load can be reduced by preselecting lane and road markings as targets from which the feature points are to be extracted and converting only the targets of the captured image of a scene into a bird's-eye view image.

Subsequently, the distribution of the feature points is confirmed (S106). Based on the confirmed distribution of the feature points, it is determined whether to calculate the position and the angle of the in-vehicle camera 2 (S107). The determination is made to check whether the distribution of feature points is suitable for accurately calculating the position and angle of the in-vehicle camera 2 . Therefore, the determination can be made based on information indicating the width of the distribution of the feature points. The information that gives the width of the distribution relates to the size of an area represents the size of an image area that shows the distribution of the feature points. As described below, the suitability of the distribution can be comprehensively determined considering the width of the distribution, the position of a distribution area including the feature points, and the dispersion of the feature points within the distribution area.

9 Fig. 12 shows a case where the distribution of feature points in a bird's-eye view image is favorable and a case where the distribution of feature points in a bird's-eye view image is unfavorable. A captured image of a scene is turned into 9 omitted and the positions of extracted feature points are indicated by black dots. Further, the extracted feature points are integrally considered to connect peripheral feature points as vertices, and an area inside the resultant convex polygon is handled as a feature point distribution area (hatched portion). A feature point distribution is classified as favorable if the distribution area has a large area, which is located near the center of the image, as in 9(a) shown. The feature point distribution is judged to be unfavorable when the distribution area has a small area or is positioned in a distorted manner, occupying only part of an image, as in 9(b) shown.

Whether the feature point distribution is appropriate can be determined simply by checking the width, height and position of the feature point distribution area instead of examining the hatched area.

In 9(a) is the width of the feature point distribution area Ua and the height Va. The size of this feature point distribution area is sufficiently large compared to the size of an image. Also, the intermediate values of Ua and Va are near the center of the image. The feature point distribution in 9(a) can be classified as cheap.

In 9(b) Both the width Ub and the height Vb of the feature point distribution area are small compared to the size of an image. Thus, the feature point distribution from 9(b) be classified as unfavourable. Furthermore, the intermediate values of Ua and Va deviate significantly from the center of the image. The feature point distribution can be classified as unfavorable.

As already mentioned, the spread of the feature points can be taken into account in addition to the size and position of the feature point distribution area. The width Uc, the height Vc and the position of the feature point distribution area in 9(c) are exactly the same as in 9(a) . However, the feature point distribution within the distribution range is in 9(c) one-sided. In this case, not only the width (range) of the feature point distribution but also the spread of the feature points should be considered. In 9(c) a feature point which deviates significantly from, for example, an average distribution position is considered an outlier and will not be part of the feature point distribution range. Thus, a double hatched area is handled as a feature point distribution area. The feature point distribution in 9(c) is classified as unfavourable.

As mentioned above, the position and the angle of the in-vehicle camera 2 are calculated using the correspondence of the positional relationship between a plurality of feature points extracted from a bird's-eye view image and the positional relationship between a plurality of feature points in a currently captured image of a scene. Therefore, even if the feature point distribution is unfavorable as in Figs 9(b) and (c), the position and angle of the vehicle camera 2 can be calculated. However, for an area where no feature points exist, the correspondence of the positional relationship between a plurality of feature points on an image and the positional relationship between a plurality of feature points on an image actually captured cannot be confirmed. This makes it difficult to accurately calculate the position and angle of the in-vehicle camera 2 .

When the feature point distribution is unfavorable, it is determined that the position and the angle of the in-vehicle camera are not to be calculated (S107 from 5 : NO) and then the conversion table update process ( 6 ).

In contrast, when the feature point distribution is favorable on a bird's-eye view image, as in 9(a) shown, it is determined that the position and angle of the in-vehicle camera 2 are to be calculated (S107: YES), and thereafter the position and angle of the in-vehicle camera 2 are calculated (S108). In this way, the position and the angle of the vehicle camera 2 can only be calculated if sufficient accuracy is achieved. Furthermore, if the position and the angle of the in-vehicle camera 2 are not calculated because sufficient accuracy cannot be assumed, it is possible to prevent an additional processing load from being imposed on the CPU.

After the position and angle of the in-vehicle camera 2 are calculated (S108), the amount of change in position and angle, which result from the calculated position and the angle are confirmed (S109 from 6 ). The change amount to be confirmed is the amount of the difference between the recalculated position and angle of the vehicle camera 2 and the position and angle of the vehicle camera 2 used in the update of the currently used conversion table. When the change amount is equal to or smaller than a threshold change amount (S110: YES), the conversion table is updated (S111), and then the conversion table update process ends. In contrast, when the change amount is larger than the threshold change amount (S110: NO), the conversion table update process ends without updating the conversion table.

In the case of a contingency such as an abnormality in feature point extraction or in calculation of the position and angle of the in-vehicle camera 2, the position and angle of the in-vehicle camera 2 may become extreme. Under such circumstances, the conversion table is not updated when the amount of change presented by the calculated position and angle of the in-vehicle camera 2 is larger than a predetermined threshold amount of change. Therefore, the conversion table is updated only when it is determined that the position and angle of the in-vehicle camera 2 are obtained with sufficient accuracy. Further, as in the case of determining a feature point distribution, the position and angle of the in-vehicle camera 2 are not calculated when sufficient accuracy is unlikely to be obtained. Consequently, it is possible to prevent an additional process processing load from being imposed on the CPU.

As described above, when the conversion table update process is performed by the image conversion device 10 according to the present embodiment, the feature points are extracted based on the evaluation of the surrounding values with increased accuracy (S102), and the position and angle of the in-vehicle camera 2 with increased accuracy by confirmation of the distribution of the feature points is calculated (S107). In addition to such an accurate calculation of the position and angle of the in-vehicle camera 2, the amount of change presented by the calculated position and angle of the in-vehicle camera is confirmed (S110). Finally, after triple checking, the conversion table is updated. Thus, the image conversion device references the very accurate, very reliable conversion table obtained in the manner described above. An accurate bird's-eye view image can be obtained by conversion.

B. Modification

In the description of the above embodiment, the method of calculating the changed position and angle of the in-vehicle camera 2 has been explained on the assumption that the changes in the position and angle of the in-vehicle camera 2 are made with respect to the road surface 21 . In this regard, the position and angle of the vehicle camera relative to the road surface 21 may vary for two reasons. First, the mounted state (position and angle) of the in-vehicle camera 2 with respect to the vehicle 1 may vary, so that the mounted state with respect to the road surface 21 changes. Secondly, the position of the vehicle 1 in relation to the road surface can vary. The change in the first case is a permanent change, which occurs, for example, due to a loose fastener attached to the vehicle camera 2 . Meanwhile, in the second case, the change, that is, the inclination of the vehicle 1 with respect to the road surface, is a temporary change that occurs due to a change in payload weight or the acceleration or deceleration of the vehicle 1, for example. The above embodiment has been described without distinguishing between the two types of variations. However, the following description of a modification explains how the position and angle of the vehicle camera 2 are calculated in the case of the temporary change in the second case, that is, a change in the attitude of the vehicle 1 with respect to the road surface while focusing on the difference is placed between the modification and the preceding embodiment.

In general, the position of the in-vehicle camera 2 has three directional components, namely, a longitudinal position (the traveling direction of the vehicle 1), a lateral position (the left-right direction relative to the traveling direction of the vehicle 1), and a vertical position. The vehicle camera has three directional angles, namely roll, pitch and yaw. Therefore, in order to calculate the position and the angle of the in-vehicle camera 2, it is essentially necessary to calculate the three directional components and the three directional angles (six values in total). However, if the calculation of the position and the angle is required for the latter reason, i.e. because of the change in the attitude of the vehicle 1 in relation to the road surface, the calculation of three values will suffice, namely the vertical position, the roll angle and the Pitch angle of the vehicle camera 2. The reason will be described below.

First, assume that the attitude of the vehicle 1 with respect to the road surface is fixed (e.g., held in a horizontal position). Even if such a state is maintained, the longitudinal and lateral positions of the in-vehicle camera 2 vary as the vehicle 1 moves. Furthermore, when the orientation of the vehicle 1 varies, the yaw angle of the vehicle camera 2 varies. Therefore, a change in the posture of the vehicle 1 with respect to the road surface affects the vertical posture, roll angle, and pitch angle of the vehicle camera 2 and them does not affect the longitudinal position, lateral position and yaw angle of the vehicle camera 2. Accordingly, when the position of the vehicle 1 with respect to the road surface varies during the running of the vehicle 1, the intended purpose is achieved by calculating the three values, namely the vertical position, the roll angle and the pitch angle of the vehicle camera 2.

While the vehicle 1 is running, the posture of the vehicle 1 with respect to the road surface varies frequently, and accordingly, the frequency of calculations of the position and angle of the in-vehicle camera 2 may be high, and the calculation load is likely to be heavy. From this point of view, if it suffices to calculate the three values, namely, the vertical position, the roll angle, and the pitch angle of the in-vehicle camera 2, it is a great advantage because the calculation load can be reduced in half.

10 FIG. 12 shows an example of a bird's-eye view image obtained when the vehicle 1 is tilted in the pitch direction. When the front of the vehicle 1 is tilted down as in FIG 10(a) shown, the pitch angle of the vehicle camera 2 changes downward. A bird's-eye view image showing lane markings on the left and right sides of the vehicle 1 in the above state is shown in FIG 10(b) pictured. As in 10(b) illustrated, the lane markings on the left and right sides of the vehicle 1 are shown in an inverted V-shape. When the vehicle 1 is not tilted, the left and right lane markings should be imaged in parallel. Therefore, when feature points are extracted from the boundaries of the lane markers to make adjustments so that the lane markers inclined in an inverted V-shape are displayed in parallel, the pitch angle of the in-vehicle camera 2 can be recalculated based on the inclination of the vehicle 1 will. Black dots on the boundary lines of the lane markings in 10(b) are examples of feature points. When such feature points are connected, the resulting lines are just like the lane marking lines. Therefore, such feature points are omitted from examples of a bird's-eye view image presented below.

A bird's-eye view image showing a stop line in front of vehicle 1 in a, as in 10(a) shown condition is in 10(c) shown. As illustrated, the stop line shown has a trapezoidal shape such that two longitudinal straight lines have an inverted V-shape and two transverse straight lines are parallel to each other. When the vehicle 1 is not inclined, the stop line imaged should be shaped like a rectangle that is long in the lateral direction. Therefore, if adjustments are made so that the two straight lines shown as in the in 10(b) shown case are inclined in an inverted V-shape are parallel to each other, the pitch angle of the in-vehicle camera 2 can be recalculated based on the inclination of the vehicle 1 .

The accuracy of the pitch angle of the vehicle camera 2, which from the 10(b) and 10(c) are calculated are now compared. In 10(b) the slope of a line segment of length L1 can be examined. In case of 10(c) however, to examine the slope of a line segment with length L2, which is shorter than length L1. Obviously examining the slope of the line segment of length L2 produces a larger error than examining the slope of the line segment of length L1. Therefore, the calculated pitch angle of the vehicle camera 2 is from 10(c) more accurate than that of 10(b) .

Now, the difference in feature point distribution between the 10(b) and 10(c) examined. In 10(b) For example, the distribution of the feature points marked by the black points has a large width in the longitudinal direction of a bird's-eye view image. Meanwhile, since there are no feature points in the left and right end portions and in the middle portion, the distribution of the feature points marked by the black dots has a small lateral width.

Black dots indicating the feature points will be in 10(c) omitted. However, looking at the indicated range of the stop line, it is found that the longitudinal feature point distribution has a small width and the lateral feature point distribution has a large width.

Consequently, when the pitch angle of the in-vehicle camera 2 as in 10 shown, is to be recalculated, it is understood that a feature point distribution with a large width in longitudinal direction of a bird's-eye view image (in the traveling direction of the vehicle 1) is important for accurate calculation. Therefore, the pitch angle of the in-vehicle camera 2 should be calculated under the condition that the width of the feature point distribution in the longitudinal direction of a bird's-eye view image is larger than a predetermined longitudinal direction threshold value. The longitudinal threshold corresponds to a first threshold.

11 FIG. 12 shows an example of a bird's-eye view image obtained when the vehicle 1 is inclined in the roll direction. When the roll direction angle of the in-vehicle camera 2 is changed to the right since the right side of the vehicle 1 is tilted down when viewed from the front of the vehicle 1 as shown in FIG 11(a) shown, the lane markings on the left and right sides of the vehicle 1 are displayed in a different way as in FIG 11(b) shown. The left lane marking in 11(b) is depicted with a large width W1 and positioned away from the vehicle 1 . Meanwhile, the right lane marker is depicted with a small width W2 and positioned toward the vehicle 1 . In this case, the roll angle of the vehicle camera 2 can be recalculated based on the inclination of the vehicle 1 by, for example, making adjustments so that the left and right lane markings have the same width. However, actual pavement markings have a width of 15 cm to 20 cm. Furthermore, the displayed lane markers in a bird's-eye view image have a relatively small width compared to the image display area. Therefore, it is more difficult than in the case of 11(c) to make precise adjustments.

one inside 11(c) The stop line shown in FIG. Therefore, the roll angle of the in-vehicle camera 2 can be recalculated based on the inclination of the vehicle 1 by making adjustments so that the two straight lines extending laterally in an inverted V-shape are parallel to each other. The two lines, which extend transversely in an inverted V-shape, have a length L3, the length L3 apparently being greater than the widths W1 and W2 shown in FIG 11(b) are shown is. It is therefore conceivable that the calculated roll angle of the vehicle camera 2 in the case of 11(c) is more accurate than in the case of 11(b) .

Therefore, the roll angle of the in-vehicle camera 2 should be calculated under the condition that the width of the feature point distribution in the transverse direction of a bird's-eye view image is larger than a predetermined transverse direction threshold value. The cross-directional threshold corresponds to a second threshold.

12 FIG. 12 shows an example of a bird's-eye view image obtained when the vertical posture of the vehicle 1 is changed. When the vehicle 1 is fully submerged as in 12(a) shown, the vertical position of the vehicle camera 2 decreases. In this state, the displayed lane markings look like in 12(b) and a displayed stop line looks like in 12(c) . In the 12(b) and 12(c) the lane markings and the stop line, which are displayed before the vertical posture of the vehicle 1 changes, are represented by broken lines. Compared with the dashed lines, the left and right lane markings are in 12(b) each with an increased width and positioned further away from vehicle 1, and the stop line in 12(c) is similarly imaged with an increased size and positioned further away from the vehicle 1 .

When the vertical posture of the vehicle 1 is changed as described above, things displayed in a bird's-eye view image are enlarged or reduced. Therefore, the match of the feature points cannot be achieved by using the with respect to the 10 and 11 described forms are used. However, for example, the vertical position of the vehicle 1 can be calculated as described below. In case of 12(b) When data is prepared with the width of the actual lane markings, the vertical position of the in-vehicle camera 2 can be recalculated based on the vertical position of the vehicle 1 by making adjustments so that the width of the lane markings displayed in a bird's-eye view image of the width corresponds to the actual lane markings. Similarly in the case of 12(c) When data with the width of a stop line is prepared, the vertical position of the in-vehicle camera 2 can be recalculated based on the vertical position of the vehicle 1 . When the vertical position of the in-vehicle camera 2 is calculated as described above, matching of scales is achieved instead of matching of shapes. The scaling of an image cannot be accurately evaluated until the entire image is captured. So if the longitudinal scale of an image from a bird's-eye view is to be evaluated, at least the feature point distribution in the longitudinal direction must be favorable. If the transverse scale is to be evaluated, at least the feature point distribution in the transverse direction must be favorable.

When the vertical posture of the vehicle 1 is changed, things shown in a bird's-eye view image are enlarged or reduced as described above. However, when the vehicle 1 is tilted in the pitching direction, things shown in a bird's-eye view image may look enlarged or reduced in some cases. Further, these displayed image changes tend to be affected by the installation state of the in-vehicle camera 2, such as the mounting position (front, rear, left, or right of the vehicle 1) and the difference in lowering amount, which depends on the hardness of a bumper. Therefore, it is likely that an incorrect variable can be calculated. In such a situation, the data on the tendency of the posture change of the vehicle 1 and the display change of a bird's-eye view image should be obtained in advance while the in-vehicle camera 2 is mounted on the vehicle 1, and then the obtained data should be referred to to determine which of the three variables, namely the vertical position, the roll angle and the tilt angle of the vehicle camera 2 are to be calculated.

Furthermore, when the attitude of the vehicle 1 varies, in some cases the pitch angle, the roll angle, and the vertical attitude may change in a complex manner rather than individually. Even in such a situation, whether the feature point distribution in the longitudinal or lateral direction of a bird's-eye view image is favorable, similar to whether the pitch direction angle or roll direction angle of the vehicle camera 2 can be calculated with sufficient accuracy. Consequently, the variable to be calculated should be as in 13 shown can be selected.

13 shows an exemplary selection of a variable based on the feature point distribution. As described above, the feature point distribution is divided into the longitudinal direction and the transverse direction of a bird's-eye view image, and classification is made depending on whether the longitudinal feature point distribution and the transverse direction feature point distribution are large or small in width.

If the feature point distribution has a small width in the fore-and-aft direction and in the lateral direction, neither the pitch angle, the roll angle nor the vertical position of the in-vehicle camera 2 need to be calculated as described above in connection with the present embodiment.

If the feature point distribution has a large longitudinal width and a narrow transverse width, calculate the pitch angle. The reason for this is that it is difficult to calculate the roll angle accurately as referred to 11(b) has been explained, although the pitch angle can be accurately calculated as referred to in FIG 10(b) explained.

If the feature point distribution has a small longitudinal amplitude and a large transverse amplitude, calculate the roll angle. The reason for this is that it is difficult to calculate the pitch angle accurately as referred to 10(c) explained, although the roll angle can be calculated exactly as with reference to 11(c) explained.

When the feature point distribution has a large width in the longitudinal and lateral directions, all the variables, namely the pitch angle, the roll angle and the vertical position of the in-vehicle camera 1 can be calculated in the same manner as described in connection with the present embodiment . Besides, the vertical position of the in-vehicle camera 2 can be calculated when the feature point distribution has a large width in either the longitudinal direction or the lateral direction. Further, the vertical position of the in-vehicle camera 2 can be calculated when extracted feature points enable to achieve the agreement of magnitude, such as the widths of the lane markings and the stop line.

As referring to the 10 until 13 described, the accuracy of the calculated variables can be properly maintained when the feature point distribution in the longitudinal and lateral directions is confirmed and, based on the confirmation result, only one favorable variable, i.e., the pitch angle, the roll angle or the vertical position of the vehicle camera is calculated selectively. Furthermore, the time required for the calculation can be reduced by selecting the variable to be calculated. This makes it easy to cope with temporary changes in vehicle attitude. Unselected variables can be computed when favorable feature points are obtained from a bird's-eye view image derived from a subsequent image acquisition. Finally, after confirming the amount of change in a newly calculated variable (S109 and S110 in 6 ) updated as described in connection with the present embodiment.

An alternative is to calculate all variables regardless of the feature point distribution when the pitch angle, the roll angle and the vertical position of the vehicle camera 2 are to be calculated and the conversion using a as in 13 stated, selected variables if the conversion table is to be updated.

The image converting device 10 according to the modification described above calculates one of the pitch angle, roll angle and vertical position of the in-vehicle camera 2 when they are favorable and refers to the obtained conversion table based on the calculation result. Thus, a captured image is accurately converted into a bird's-eye view image.

Claims (6)

An image converting device (10) which obtains a captured image from an in-vehicle camera (2) imaging the surroundings of a vehicle (1), converts the captured image into a bird's-eye view image showing a top view of the vehicle, and the image off outputs a bird's-eye view on an on-board monitor (3), the image conversion device having: an image converting unit (12) which converts the picked-up image into the bird's-eye view image based on a position and an angle of the vehicle camera by referring to a conversion table, the conversion table being calculated by calculation based on the position and the angle of the vehicle camera from pixel positions in the captured image are obtained which correspond to pixel positions in the bird's-eye view image; a feature point extraction unit (15) which extracts a plurality of feature points from the bird's-eye view image or from the captured image; a calculation determination unit (16) which determines whether or not to calculate the position and the angle of the in-vehicle camera based on a distribution of the feature points extracted from the bird's-eye view image or the captured image; a calculation unit (17) that calculates the position and the angle of the in-vehicle camera based on the feature points when the calculation determination unit determines that the position and the angle of the in-vehicle camera are to be calculated; a change amount obtaining unit (18) which obtains a change amount by comparing the position and angle of the vehicle camera recalculated by the calculation unit with the previous position and angle; and a conversion table updating unit (19) which, when the amount of change is equal to or smaller than a predetermined threshold amount of change, updates the conversion table based on the recalculated position and angle of the in-vehicle camera. image conversion device claim 1 wherein the calculation determination unit determines whether to calculate the position and the angle of the in-vehicle camera based on the width of the distribution of the feature points extracted from the bird's-eye view image or the captured image. An image converting device (10) which obtains a captured image from an in-vehicle camera (2) imaging the surroundings of a vehicle (1), converts the captured image into a bird's-eye view image showing a top view of the vehicle, and the image off outputs a bird's-eye view on an on-board monitor (3), the image conversion device having: an image converting unit (12) that converts the captured image into the bird's-eye view image based on a position and an angle of the vehicle camera; a feature point extraction unit (15) which extracts a plurality of feature points from the bird's-eye view image or from the captured image; a calculation determination unit (16) which determines whether or not to calculate the position and the angle of the in-vehicle camera based on a distribution of the feature points extracted from the bird's-eye view image or the captured image; and a change amount obtaining unit (18) which obtains a change amount by comparing the position and angle of the vehicle camera recalculated by the calculation unit with the previous position and angle; whereby when the width of the distribution of the feature points in a longitudinal direction of the bird's-eye view image is larger than a predetermined first threshold value, the calculation determination unit determines that a pitch angle is to be calculated, the pitch angle causing an optical axis of the vehicle camera, to move in an up-down direction; and when the width of the distribution of the feature points in a lateral direction of the bird's-eye view image is greater than a predetermined second threshold, the calculation determination unit determines that a roll angle is to be calculated, the roll angle causing the vehicle camera to rotate the optical axis to rotate. Image conversion device according to one of Claims 1 until 3 , Further with an extraction determination unit (14), which on the basis of the surrounding environment of the vehicle or the Detection information obtained by a detector (4) detecting a condition of the vehicle determines whether or not the feature points are extracted. Image conversion device according to one of Claims 1 until 4 , wherein the feature point extraction unit extracts the feature points from the bird's-eye view image. Image conversion method applied to a vehicle having an in-vehicle camera showing surroundings of the vehicle, wherein an image picked up by the in-vehicle camera is acquired, the acquired picked-up image into a bird's-eye view image showing a top view of the vehicle shows is converted and the bird's-eye view image is displayed on an on-vehicle monitor, the image conversion method comprising: a step (S104) of converting the captured image into a bird's-eye view image based on the position and angle of the vehicle camera by referring to a conversion table, the conversion table obtained by calculating, based on the position and angle of the vehicle camera, pixel positions in the captured image obtaining which pixel positions in the bird's-eye view image correspond; a step (S105) of extracting a plurality of feature points from the bird's-eye view image or from the captured image; a step (S107) of determining, based on the distribution of the feature points extracted from the bird's-eye view image or the captured image, whether or not to calculate the position and the angle of the in-vehicle camera; a step (S108) of calculating the position and the angle of the in-vehicle camera based on the feature points when it is determined that the position and the angle of the in-vehicle camera are to be calculated; a step (S109) of obtaining a change amount by comparing the position and angle of the vehicle camera recalculated in step (S108) with the previous position and angle; and a step (S111) of, when the amount of change is equal to or smaller than a predetermined threshold amount of change, updating the conversion table based on the recalculated position and the angle of the in-vehicle camera.

Images