WO2016084308A1

WO2016084308A1 - Image transformation apparatus and image transformation method

Info

Publication number: WO2016084308A1
Application number: PCT/JP2015/005512
Authority: WO
Inventors: 田中　仁; 森下　洋司; 宗昭松本
Original assignee: 株式会社デンソー
Priority date: 2014-11-26
Filing date: 2015-11-03
Publication date: 2016-06-02
Also published as: DE112015005317T5; JP6507590B2; JP2016100887A; DE112015005317B4

Abstract

Provided is an image transformation apparatus (10) wherein an image captured by a vehicle-mounted camera (2) is transformed to a bird's eye view image, which is then displayed on a vehicle-mounted monitor (3). The image transformation apparatus comprises: an image transformation unit (12) that transforms the captured image to the bird's eye view image in accordance with the position and angle of the vehicle-mounted camera; a feature point extraction unit (15) that extracts a plurality of feature points from the bird's eye view image or from the captured image; a calculation determination unit (16) that determines, on the basis of the distribution of the plurality of feature points extracted from the bird's eye view image or from the captured image, whether to calculate the position and angle of the vehicle-mounted camera; and a calculation unit (17) that calculates the position and angle of the vehicle-mounted camera on the basis of the plurality of feature points if the calculation determination unit determines that the position and angle of the vehicle-mounted camera are to be calculated.

Description

Image conversion apparatus and image conversion method

Cross-reference of related applications

This application is based on Japanese Patent Application No. 2014-239377 filed on November 26, 2014, the disclosure of which is incorporated herein by reference.

This disclosure is applied to a vehicle including an in-vehicle camera, and relates to a technique for adjusting a setting value used when a captured image acquired from the in-vehicle camera is converted into a bird's-eye view image.

Various techniques for allowing the driver to check the situation around the vehicle have been put into practical use by photographing the surroundings of the vehicle with the in-vehicle camera and displaying the obtained captured image on the in-vehicle monitor. Furthermore, it is widely performed that a captured image acquired from an in-vehicle camera is not displayed as it is on the in-vehicle monitor but is converted into a bird's-eye view image and then displayed on the in-vehicle monitor. Here, the bird's-eye view image refers to an image obtained by converting a photographed image as if it was photographed by looking down at the scenery in the photographed image from above the vehicle. If the bird's-eye view image is correctly converted, the positional relationship between the other vehicle, pedestrian, etc. and the host vehicle is displayed while maintaining the same positional relationship as the actual vehicle. It can be easily and accurately grasped. In order to convert a captured image into an accurate bird's-eye view image, it is necessary to accurately acquire the position and angle of the in-vehicle camera.

Therefore, the in-vehicle camera is attached to the vehicle at the designed position and angle, and the position and angle of the in-vehicle camera are individually measured before the vehicle is shipped from the factory in order to reduce the influence of installation errors. The position and angle of the in-vehicle camera can be calculated as follows. First, a plurality of feature points are extracted from a photographed image by the in-vehicle camera, and the coordinates on the photographed image where each feature point is located are associated with the coordinates in the photographed real space. Then, the positions of the respective feature points associated with each other are substituted into a relational expression using the position and angle of the in-vehicle camera as variables. Thereafter, the position and angle of the in-vehicle camera can be calculated by solving the obtained relational expression.

However, the in-vehicle camera fixed to the vehicle may be displaced in position and angle due to loosening of the fastening part after shipment from the factory. In such a case, the position and angle of the vehicle are different from those at the time of factory shipment. Therefore, a technique has been proposed that enables feature points to be extracted from an image taken while the vehicle is running and recalculate the position and angle of the in-vehicle camera (Patent Document 1).

JP2013-222302A

However, with the proposed technique, sufficient accuracy cannot be obtained even though the feature point is extracted from the captured image and the position and angle of the in-vehicle camera are calculated in the same manner as when brought to the maintenance shop. There was a thing.

One of the objects of the present disclosure is to provide a technique that can accurately calculate the position and angle of the in-vehicle camera and convert it into an accurate bird's-eye view image without bringing the vehicle into a maintenance shop.

According to one aspect of the present disclosure, a captured image is acquired from a vehicle-mounted camera that captures the surroundings of the vehicle, and the captured image is converted into a bird's-eye image as if it was captured from a viewpoint above the vehicle. An image conversion apparatus for displaying on a monitor, wherein the image conversion unit converts the captured image into the bird's-eye image according to the position and angle of the in-vehicle camera, and extracts a plurality of feature points from the bird's-eye image or the captured image A feature point extraction unit, a calculation determination unit that determines whether to calculate the position and angle of the in-vehicle camera based on the distribution of the plurality of feature points extracted from the bird's-eye view image or the captured image, An image conversion device comprising: a calculation unit that calculates the position and angle of the in-vehicle camera based on the plurality of feature points when the calculation determination unit determines to calculate the position and angle of the in-vehicle camera. There is provided.

According to another aspect of the present disclosure, the present invention is applied to a vehicle equipped with an in-vehicle camera, acquires a captured image from the in-vehicle camera that captures the periphery of the vehicle, and the captured image is viewed from an upper viewpoint of the vehicle. An image conversion method for converting a photographed image into a bird's-eye image according to a position and an angle of the vehicle-mounted camera, wherein the image is converted into a bird's-eye image as if taken and then displayed on the vehicle-mounted monitor. Whether to calculate the position and angle of the in-vehicle camera based on the step of extracting a plurality of feature points from the image or the captured image and the distribution of the plurality of feature points extracted from the bird's-eye view image or the captured image And a step of calculating the position and angle of the in-vehicle camera based on the plurality of feature points when it is determined to calculate the position and angle of the in-vehicle camera. Conversion method is provided.

For example, since a feature point can be artificially arranged in a maintenance shop or the like, a position and an angle can be calculated using feature points having a distribution suitable for calculating the position and angle of the in-vehicle camera. On the other hand, when the vehicle is running, feature points are extracted from the scenery in the shooting range at that time, and thus feature points with various distributions are extracted. Therefore, when the vehicle is traveling, it is not always possible to extract feature points suitable for calculating the position and angle of the in-vehicle camera. Therefore, if the position and angle of the in-vehicle camera are calculated when the distribution of the extracted feature points is suitable for the calculation of the position and angle of the in-vehicle camera, even if the vehicle is running, The position and angle can be calculated with high accuracy. As a result, it is possible to accurately convert the captured image into a bird's-eye view image.

The above and other objects, features and advantages of the present disclosure will become more apparent from the following detailed description with reference to the accompanying drawings. In the attached drawings
FIG. 1 is an explanatory view showing a vehicle equipped with the image conversion apparatus of this embodiment. FIG. 2 is a schematic explanatory diagram of the principle of converting a captured image into a bird's-eye view image. FIG. 3 is an explanatory diagram showing a state in which a captured image is converted into a bird's-eye image with reference to a conversion table. FIG. 4 is a block diagram showing the internal structure of the image conversion apparatus. FIG. 5 is a flowchart of the first half of the conversion table update process executed by the image conversion apparatus. FIG. 6 is a flowchart of the latter half of the conversion table update process executed by the image conversion apparatus. FIG. 7 is a diagram showing a specific example of the environmental score to be given to the score sensor, FIG. 8 is an explanatory diagram showing how feature points are extracted from each of a captured image and a bird's-eye view image. FIG. 9 is an explanatory diagram showing how to check the distribution of feature points. FIG. 10 is a diagram illustrating a bird's-eye view image when the vehicle is tilted in the pitch direction. FIG. 11 is an example of a bird's-eye view image when the vehicle is tilted in the roll direction. FIG. 12 is a diagram illustrating a bird's-eye view image when the vehicle sinks downward; FIG. 13 is a diagram exemplifying selection of which variable of pitch angle, roll angle, and height of the in-vehicle camera is calculated according to the distribution of feature points.

Hereinafter, embodiments of the image conversion apparatus will be described.

A-1. Apparatus configuration of this embodiment:
FIG. 1 shows a rough structure of a vehicle 1 on which an image conversion device 10 is mounted. As shown in the drawing, the vehicle 1 includes an in-vehicle camera 2 attached to the front portion of the vehicle 1 and an in-vehicle monitor 3 that can be viewed from the driver's seat, in addition to the image conversion device 10. In addition to these, the vehicle 1 includes a vehicle speed sensor 41 that acquires a traveling speed, a steering sensor 42 that acquires a steering angle, height sensors 43a to 43d that are arranged at four corners of the vehicle 1 and detect vertical displacement, and solar radiation. A solar radiation sensor 44 that detects the intensity of the amount and a rain sensor 45 that detects the amount of raindrops are provided. These various sensors provide the image conversion apparatus 10 with detection information obtained by detecting the surrounding environment of the vehicle 1 or the state of the vehicle 1 itself, and correspond to detectors.

The in-vehicle camera 2 is a camera with a fish-eye lens and a wide angle of view, and captures a scene including the road surface around the front of the vehicle 1 in the captured image. The image conversion device 10 converts the captured image acquired from the in-vehicle camera 2 into a bird's-eye image and displays the bird's-eye image on the in-vehicle monitor 3. Since the photographed image shows the road surface, the bird's-eye view image obtained by converting the photographed image is an image as if the photograph was taken by looking down the road surface in front of the vehicle 1 in the photographed image from above. Hereinafter, an outline of conversion from a captured image to a bird's-eye view image will be described.

FIG. 2 schematically shows the principle of converting a captured image into a bird's-eye view image. As shown in FIG. 2A, when the in-vehicle camera 2 images the road surface 21, the captured image 22 is on the road surface 21 in the imaging area of the in-vehicle camera 2 on an image plane that intersects the line of sight of the in-vehicle camera 2. Becomes a projected image. Looking back at this projection relationship, if the photographed image is projected onto the road surface 21 with the light of the light source placed at the position of the in-vehicle camera 2, the projected image should match the actual situation of the road surface 21.

2B is projected on the road surface plane 23 in FIG. 2B. The road surface plane 23 is a plane at the same position as the road surface 21 in FIG. 2A, and may be considered as a screen. If a virtual camera 24 provided so that the road surface reflected on the road surface plane 23 is looked down from directly above is photographed, a bird's-eye view image 25 as if the road surface in front of the in-vehicle camera 2 was photographed from directly above is obtained. can get.

Here, looking at the relationship between the position and angle of the in-vehicle camera 2 and the imaging region, the position and shape of the area surrounded by the trapezoid of the road surface 21 shown as the imaging region in FIG. Is determined according to the position and angle. Therefore, in order to convert the captured image into a bird's-eye view image, the position and angle of the in-vehicle camera 2 are constant between when the image is actually captured and when the captured image is projected onto the virtual road surface plane 23 (screen). It is necessary to keep the state.

Since the actual in-vehicle camera 2 is fixedly attached to the vehicle 1, it may be considered that the position and angle of the in-vehicle camera 2 with respect to the road surface 21 are usually constant. Therefore, the image conversion apparatus 10 prepares in advance a conversion table in which the projection relationship as described in FIG. 2 is calculated under a certain position and angle of the in-vehicle camera 2. When converting a captured image into a bird's-eye view image, by referring to the conversion table, it is not necessary to calculate the above-described projection relationship for each shooting, thereby reducing the processing load.

FIG. 3 shows a state in which a captured image is converted to a bird's-eye image with reference to the conversion table. The conversion table is data representing the correspondence between the pixel position of the bird's-eye view image and the pixel position of the captured image. For example, the pixel at the position of the coordinates (Xe, Yf) of the bird's-eye image in FIG. 3B is associated with the pixel at the position of the coordinates (Ag, Bh) of the photographed image in FIG. It has been. Accordingly, the image data (luminance, saturation, etc.) of the pixel at the coordinate (Ag, Bh) of the captured image is applied to the pixel at the coordinate (Xe, Yf) of the bird's-eye view image. By referring to the conversion table for each pixel of the bird's-eye view image, the captured image can be converted to the bird's-eye view image.

As described above, since this conversion table is created according to the position and angle of the in-vehicle camera 2, the image conversion apparatus 10 has the function of converting a captured image into a bird's-eye image, the position of the in-vehicle camera 2, A function of recalculating the angle and updating the conversion table according to the new position and angle is provided.

FIG. 4 shows the internal structure of the image conversion apparatus 10. As illustrated, the image conversion apparatus 10 includes a captured image acquisition unit 11, an image conversion unit 12, and a display unit 13. The captured image acquisition unit 11 acquires a captured image from the in-vehicle camera 2. The image conversion unit 12 converts the captured image into a bird's-eye view image with reference to the conversion table as described above. The display unit 13 displays the bird's-eye view image on the in-vehicle monitor 3.

Further, the image conversion apparatus 10 calculates the position and angle of the in-vehicle camera 2 and updates the conversion table, so that the score evaluation unit 14, the feature point extraction unit 15, the calculation determination unit 16, and the calculation unit 17 The fluctuation amount acquisition unit 18 and the conversion table update unit 19 are provided.

The score evaluation unit 14 acquires detection information detected by the vehicle speed sensor 41, the steering sensor 42, the height sensors 43a to 43d, the solar radiation sensor 44, and the rain sensor 45 provided in the vehicle 1. Hereinafter, these various sensors are collectively referred to as a score sensor 4. Moreover, the score evaluation part 14 provides a score according to the detection information acquired from each of the score sensors 4, and comprehensively evaluates the score. The score evaluation will be described later with a specific example. The outline of the score evaluation is information on the environment around the vehicle 1 or the state of the vehicle 1 itself, and evaluates whether the situation is suitable for feature point extraction. Is to do. A feature point is a point that can be distinguished from other points by having a distinctive feature in the shooting scenery, and a point on the image coordinate can be specified by a shooting image or a bird's-eye view image of the shooting scenery. Say. The feature points are extracted in order to calculate the position and angle of the in-vehicle camera 2.

Note that the score evaluation unit 14 corresponds to the extraction determination unit because it determines whether or not to extract feature points in this way.

The feature point extraction unit 15 extracts a plurality of feature points from the photographed image or the bird's-eye view image in order to calculate the position and angle of the in-vehicle camera 2.

The calculation determination unit 16 confirms the distribution status of the plurality of feature points, and determines whether or not to calculate the position and angle of the in-vehicle camera 2 based on the information indicating the distribution width obtained from the confirmation.

The calculation unit 17 calculates the position and angle of the in-vehicle camera 2 based on the extracted feature points. In order to calculate the position and angle of the in-vehicle camera 2, the consistency between the positional relationship between the plurality of feature points extracted from the captured image or the bird's-eye view image and the positional relationship between the plurality of feature points in the actual captured scene is used. Since the calculation method of the position and angle of the vehicle-mounted camera 2 is well known, detailed description thereof will be omitted.

The fluctuation amount obtaining unit 18 obtains the fluctuation amount by comparing the position and angle after the calculation unit 17 newly calculates with the position and angle before the new calculation.

The conversion table update unit 19 updates the conversion table based on the newly calculated position and angle of the in-vehicle camera 2 when the fluctuation amount acquired by the fluctuation amount acquisition unit 18 is equal to or less than a predetermined fluctuation threshold. When the conversion table is updated, the image conversion unit 12 converts the captured image into a bird's-eye image with reference to the new conversion table.

Note that these nine units from the captured image acquisition unit 11 to the conversion table update unit 19 are the concepts in which the inside of the image conversion apparatus 10 is classified from the viewpoint of function, and the image conversion apparatus 10 is physically divided into nine parts. It does not represent being classified. Therefore, these units can be realized as a computer program executed by the CPU, can be realized as an electronic circuit including an LSI and a memory, or can be realized by combining them. Hereinafter, the conversion table update process executed by providing these nine units will be described in detail.

A-2. Bird's-eye view image generation processing:
5 and 6 show flowcharts of the conversion table update process executed by the image conversion apparatus 10 according to the present embodiment. This conversion table update process is a process for updating a conversion table used for converting a captured image into a bird's-eye view image, and is executed separately from the conversion process of the bird's-eye view image to be displayed on the in-vehicle monitor 3. Process. The bird's-eye image conversion processing is repeatedly executed at a cycle corresponding to the shooting cycle (for example, 30 Hz) of the in-vehicle camera 2, but the conversion table update processing described here is executed at the same cycle as the bird's-eye image conversion processing. Alternatively, it may be executed at an appropriate interval (for example, every 10 minutes).

When the conversion table update process is started, an environmental score is first calculated (S101). The environmental score is an index for scoring detection information acquired from various score sensors 4 to evaluate the surrounding environment of the vehicle 1 and the state of the vehicle 1 itself.

FIG. 7 shows a specific example of the environmental score. As shown in FIG. 7A, the environmental score is detected information obtained from each of the vehicle speed sensor 41, the steering sensor 42, the height sensors 43a to 43d, the solar radiation sensor 44, and the rain sensor 45 constituting the score sensor 4. Is granted to. Since the detection information handled here is digitized, a score is given according to the magnitude of those numeric values. Hereinafter, the environmental score given for each piece of detection information of the score sensor 4 will be referred to with the score symbols S1 to S5 in order from the vehicle speed sensor 41 to the rain sensor 45.

Here, a method for assigning an environmental score according to detection information will be described using the vehicle speed sensor 41 as an example. The detection information of the vehicle speed sensor 41 is the traveling speed of the vehicle 1, and the environmental score S1 is lowered as the traveling speed increases. FIG. 7B shows an example of giving the environmental score S1. As shown in the figure, the environmental score when the vehicle speed is 0 to 20 (km / h) is S1 = 100, and when the vehicle speed is 20 to 80 (km / h), S1 = 80 and the vehicle speed is 80 (km / H) S1 = 60 for more than The reason why the environmental score S1 is lowered as the vehicle speed increases is that subject blur tends to increase in the captured image, and the feature point extraction accuracy becomes unstable.

Similarly to the vehicle speed sensor 41, environmental scores are individually assigned to the score sensors 4 other than the vehicle speed sensor 41, that is, the steering sensor 42, the height sensors 43a to 43d, the solar radiation sensor 44, and the rain sensor 45.

The steering sensor 42 detects the amount of displacement per unit time of the steering angle. The environmental score S2 is lowered as the displacement amount increases. This is because when the amount of displacement of the steering angle is large, the vehicle 1 tilts due to a change in load, so that it is not suitable for extracting feature points.

The height sensors 43a to 43d detect the amount of displacement per unit time of the inclination of the vehicle 1 by detecting the amount of displacement in the vertical direction at the front, rear, left and right of the vehicle 1, respectively. Considering the same as S2 because the vehicle 1 is inclined, the environmental score S3 for the height sensors 43a to 43d is set to a lower score as the displacement amount is larger.

The environmental score S4 assigned to the amount of solar radiation detected by the solar radiation sensor 44 is set to a lower score as the amount of solar radiation is smaller. This is because, as the amount of solar radiation is smaller, the photographic scene is darker and the difference in luminance value of the photographed image is likely to be small, so the feature point extraction accuracy is considered to be low.

The environmental score S5 assigned to the rainfall detected by the rain sensor 45 is set to a lower score as the rainfall increases. This is because there is a high possibility that the photographed image becomes unclear as the amount of rain increases, so that the feature point extraction accuracy is considered to be low.

When individual environmental scores S1 to S5 are assigned to the detection information of the vehicle speed sensor 41, the steering sensor 42, the height sensors 43a to 43d, the solar radiation sensor 44, and the rain sensor 45, as shown in FIG. An environmental score S is calculated.

Note that, here, the total environmental score S is the sum of S1 to S5, but it is sufficient if information detected by each of the score sensors 4 can be taken into consideration, and other calculation methods may be used. For example, when it is determined that the influence of the vehicle speed is large among various types of detection information, the environmental score S1 may be evaluated with priority.

Further, the score sensor 4 may be other sensors in addition to or in place of the various sensors described above. For example, the environmental score can be given by detecting the inclination of the vehicle 1 from the gyro sensor in the same way as in S3. Moreover, it is also possible to detect an amount of rain from a wiper signal and give an environmental score based on the same idea as in S5. Further, the environmental score may be calculated by predicting the travel speed of the vehicle 1 and the inclination of the vehicle 1 from the traffic information and road information acquired from the car navigation system, and taking these information into consideration.

As described above, the environmental score evaluates the surrounding environment in which the vehicle 1 is placed, such as the difference in weather and ambient brightness, or the vehicle 1 itself in which the posture of the vehicle 1 changes or vibrations occur. It is an index for evaluating the state of. When such an environmental score is high, it can be considered that there is a high possibility of obtaining a clear photographed image by reducing the influence of disturbance caused by the surrounding environment and the vehicle state. The fact that a clear captured image is obtained means that the situation is suitable for extracting feature points.

When the environmental score is calculated in this way (S101), it is determined whether or not the environmental score is equal to or higher than a predetermined score threshold (S102). The object to be determined here is the above-mentioned total environmental score S, and this score threshold may be arbitrarily determined. By determining whether or not the total environmental score S is equal to or greater than a predetermined score threshold, the surrounding environment of the vehicle 1 and the state of the vehicle 1 itself are comprehensively determined from the detection information acquired from the various score sensors 4. Evaluation can be made to determine whether or not the situation is suitable for feature point extraction. Note that the environmental score evaluation method is not limited to this. For example, if a certain number of environmental scores among the individual environmental scores S1 to S5 do not satisfy a predetermined standard, it is determined that the environmental score is not suitable for feature point extraction. You may decide to do it.

If it is determined that the environmental score is less than the predetermined score threshold (S102: no), the conversion table update process is terminated (FIG. 6).

On the other hand, when it is determined that the environmental score is equal to or higher than the predetermined score threshold (S102: yes), feature points are extracted. In this embodiment, a captured image is acquired (S103), the captured image is converted into a bird's-eye image (S104), and feature points are extracted (S105).

FIG. 8 shows a state in which a plurality of feature points are extracted from each of the photographed image and the bird's-eye view image. As described above, a feature point is a point that can be distinguished from other points by having a distinctive feature in the shooting scenery, and the position on the image coordinate in the shooting image or the bird's-eye view image showing the shooting scenery. It is a point that can be identified. A typical feature point is a corner where a straight line and a straight line intersect, but it can also be extracted from the boundary of the straight line. For example, for road markings such as lane separators, pedestrian crossings, and stop lines, feature points can be extracted by using the fact that the difference in luminance value increases at the boundary between these display parts and asphalt parts.

8A and 8B, the black circle points represent the locations where feature points are extracted. As described above, the feature points are extracted in order to calculate the position and angle of the in-vehicle camera 2. The position and angle of the in-vehicle camera 2 are calculated using the consistency between the positional relationship between the plurality of feature points extracted on the image and the positional relationship between the plurality of feature points in the actual shooting scene. In order to use such consistency, for example, feature points may be extracted from an object whose shape in an actual shooting scene is known. In the example shown in FIG. 8, feature points are extracted from the lane breaks on the left and right sides of the vehicle 1 and the characters “stop” on the road marking. In this example, when the vehicle 1 is traveling in a lane that extends in a straight line, the actual lane separation is parallel on both the left and right sides, and there are a plurality of crossing at right angles to the characters “stop” on the actual road marking. It is a known shape that there is a corner. The shape and dimensions of road markings suitable for feature point extraction may be stored in a database in advance, and the feature point extraction unit 15 may simplify feature point extraction by referring to the database.

Although feature points can be extracted from the photographed image as shown in FIG. 8A, the feature points are extracted from the bird's-eye view image as shown in FIG. 8B in the conversion table update processing of this embodiment. Therefore, the difference between the case where feature points are extracted from the captured image in FIG. 8A and the case where feature points are extracted from the bird's-eye image in FIG. 8B will be described below.

The photographed image in FIG. 8A is an image photographed while the vehicle 1 is traveling straight in a straight lane. The actual lane separator is a straight line parallel to each other along the lane, but the left and right lane separators shown in the photographed image of FIG. The letters “Stop” on the road marking also indicate that the parts that are parallel to each other are not actually shown in parallel, or the parts that intersect at right angles are not shown at right angles. When feature points (locations indicated by black circles in the figure) are extracted from lane breaks and road markings displayed in this way, the positional relationship between those feature points on the image is the same as the actual feature points. Will not be.

On the other hand, in the bird's-eye view image of FIG. 8B, the left and right lane divisions are straight lines that are parallel to each other as in the real world, and the characters “stop” on the road marking are also in parallel as in the real world. The straight lines are parallel and the crossing at right angles is shown at right angles. When feature points (locations indicated by black circles in the figure) are extracted from lane breaks and road markings displayed in this way, the positional relationship between those feature points on the image is the same as the actual feature points. Will be.

Therefore, it is better to extract the feature points from the bird's-eye view image that is displayed while maintaining the actual positional relationship than the captured image that is displayed without maintaining the actual positional relationship. The position and angle of the in-vehicle camera 2 can be calculated more easily and accurately.

For this reason, after converting the photographed image into a bird's-eye view image (S104 in FIG. 5), feature points are extracted (S105). From the viewpoint of reducing the processing burden, an object for extracting feature points for lane delimiters and various road markings is determined in advance, and only the object is converted into a bird's-eye image in the scenery shown in the photographed image. May be.

Next, the distribution of the feature points is confirmed (S106), and based on the distribution of the feature points, it is determined whether or not to calculate the position and angle of the in-vehicle camera 2 (S107). This determination is a determination as to whether or not the distribution of feature points is a distribution suitable for calculating the position and angle of the in-vehicle camera 2 with high accuracy, and indicates, for example, the width of the distribution of feature points. Judgment can be made based on information. The information indicating the distribution width here is information indicating the size of the region in which the feature points are distributed with respect to the size of the image region. As described below, the suitability of the distribution may be comprehensively determined in consideration of the position of the distribution area of the feature points together with the width of the distribution and the degree of dispersion of the feature points in the distribution area.

FIG. 9 shows a bird's-eye view when the feature point distribution is good and when the feature point distribution is not good. Here, the scene where the shooting scene is omitted and the feature points are extracted is represented by black circles. Also, the extracted feature points are viewed as a whole, and the feature points located on the outer periphery are connected as vertices, and the region inside the convex polygon formed thereby is used as the feature point distribution region (shaded portion) . When the area of the distribution region is large as shown in FIG. 9A and the position of the distribution region is close to the center of the image, it is determined that the distribution of the feature points is good, and the distribution region is displayed as shown in FIG. When the area is small or when it is biased to a part of the image, it is determined that the distribution of the feature points is poor.

The suitability of the distribution of the feature points may be determined simply by examining the size and position of the horizontal and vertical widths of the region in which the feature points are distributed, instead of the region indicated by the shaded area.

In FIG. 9A, the horizontal width of the feature point distribution area is Ua, and the vertical width is Va. This is sufficiently larger than the image size, and since the intermediate values of Ua and Va are close to the center of the image, it can be determined that the feature point distribution is good.

On the other hand, in FIG. 9B, since the horizontal width Ub and the vertical width Vb of the feature point distribution region are both smaller than the image size, it can be determined that the distribution of the feature points is poor. Further, since the intermediate values of Ua and Va are greatly deviated from the center of the image, it can be determined that the distribution of the feature points is poor.

As described above, in addition to the size and position of the distribution area of feature points, it is also possible to consider the dispersion of feature points. In the example shown in FIG. 9C, the size and position of the horizontal width Uc and the vertical width Vc of the region where the feature points are distributed are the same as those in the example shown in FIG. Distribution is biased. Therefore, not only the width (range) in which the feature points are distributed but also the degree of dispersion should be considered. In the example shown in FIG. 9C, feature points that are far away from the average position of the distribution are not treated as outliers that form feature point distribution areas, and are indicated by double diagonal lines. The region is considered as a feature point distribution region. As a result, in the example shown in FIG. 9C, it is determined that the distribution of feature points is poor.

As described above, in order to calculate the position and angle of the in-vehicle camera 2, use the consistency between the positional relationship of the plurality of feature points extracted from the bird's-eye image and the positional relationship of the plurality of feature points in the actual shooting scene. To do. Therefore, the position and angle of the in-vehicle camera 2 can be calculated even when the distribution of feature points is poor as shown in FIGS. 9B and 9C. However, for areas where feature points do not exist, it is impossible to confirm the consistency between the positional relationship of multiple feature points on the image and the positional relationship of multiple feature points in the actual shooting scene. It is difficult to maintain the accuracy of the position and angle of the in-vehicle camera 2 to be used.

Therefore, if the distribution of the feature points is poor, it is determined that the position and angle of the in-vehicle camera are not calculated (S107: no in FIG. 5), and the conversion table update process is terminated (FIG. 6).

On the other hand, if the distribution of the feature points is good on the bird's-eye view image as shown in FIG. 9A, it is determined to calculate the position and angle of the in-vehicle camera 2 (S107: yes), and the position of the in-vehicle camera 2 is determined. The angle is calculated (S108). In this way, the position and angle of the in-vehicle camera 2 can be calculated only when good accuracy can be ensured. Further, since the position and angle of the in-vehicle camera 2 are not calculated when the accuracy cannot be ensured, it is possible to avoid an extra processing load on the CPU.

When the position and angle of the in-vehicle camera 2 are calculated (S108), the calculated fluctuation amount of the position and angle is confirmed (S109 in FIG. 6). What is confirmed here is the amount of variation between the newly calculated position and angle of the in-vehicle camera 2 and the position and angle of the in-vehicle camera 2 used for updating the conversion table currently used. If the fluctuation amount is equal to or less than the fluctuation threshold (S110: yes), the conversion table is updated (S111), and the conversion table update process is terminated. On the other hand, if the fluctuation amount is larger than the fluctuation threshold (S110: no), the process ends without updating the conversion table.

When the abnormality occurs due to the extraction of feature points, the calculation of the position and angle of the in-vehicle camera 2, and other unexpected situations, it is assumed that the position and angle of the in-vehicle camera 2 show extreme values. Therefore, when the calculated fluctuation amount of the position and angle of the in-vehicle camera 2 is larger than a predetermined fluctuation threshold, the position and angle of the in-vehicle camera 2 with good accuracy can be obtained by not updating the conversion table. The conversion table is updated only in the case where it is considered to have occurred. Further, as in the case of determining the distribution of feature points, the position and angle of the in-vehicle camera 2 are not calculated when good accuracy cannot be expected, so that it is possible to avoid an extra processing load on the CPU.

As described above, in the conversion table update process executed by the image conversion apparatus 10 according to the present embodiment, the accuracy of the feature points extracted by the evaluation of the environmental score is improved (S102), and the distribution of the feature points is confirmed. The accuracy of the position and angle of the in-vehicle camera 2 to be calculated is improved (S107). In this way, the position and angle of the in-vehicle camera 2 are calculated with high accuracy, and further, the calculated amount of variation in the position and angle of the in-vehicle camera is confirmed (S110). Will be updated. Since the image conversion apparatus 10 refers to the conversion table with high accuracy and reliability obtained in this way, it can be converted into an accurate bird's-eye view image.

B. Variations:
In the present embodiment described above, the method for calculating the changed position and angle on the assumption that the position and angle of the in-vehicle camera 2 change with respect to the road surface 21 has been described. However, there are two reasons why the position and angle of the in-vehicle camera 2 change with respect to the road surface 21. That is, the mounting state (position and angle) of the in-vehicle camera 2 with respect to the vehicle 1 changes, and as a result, the mounting state with respect to the road surface 21 changes, and the posture of the vehicle 1 changes with respect to the road surface. Conceivable. In the former case, this is a constant change that occurs when the fastening portion of the in-vehicle camera 2 is loosened. On the other hand, the fact that the vehicle 1 is inclined with respect to the road surface as in the latter case is caused by a change in the loaded weight, acceleration / deceleration of the vehicle 1 or the like and is a temporary change. In the above-described embodiment, the two have been described without particularly distinguishing between them. However, in the following modification, the in-vehicle camera is used for a temporary change in which the attitude of the vehicle 1 changes with respect to the road surface in the latter case. The difference from the present embodiment when calculating the position and angle of 2 will be described.

In general, the position of the in-vehicle camera 2 has components in three directions of vertical (traveling direction of the vehicle 1), lateral (horizontal direction with respect to the traveling direction of the vehicle 1), and height.・ There are three angles of yaw. Therefore, in principle, when calculating the position and angle of the in-vehicle camera 2, it is necessary to calculate the components in three directions and the angles in three directions (total of six values) for each in-vehicle camera 2. However, when the reason why the position and the angle need to be calculated is the latter, that is, because the posture of the vehicle 1 has changed with respect to the road surface, the height, the roll angle, and the pitch of the in-vehicle camera 2 What is necessary is just to calculate three values of an angle. This is due to the following reason.

First, a state in which the posture of the vehicle 1 with respect to the road surface is fixed (for example, a state in which the vehicle 1 is kept horizontal) is assumed. Even if such a state is maintained, if the vehicle 1 moves, the vertical and horizontal positions of the individual in-vehicle cameras 2 change. Moreover, if the direction of the vehicle 1 changes, the angle of the yaw direction of each vehicle-mounted camera 2 also changes. Therefore, the influence of the change in the posture of the vehicle 1 with respect to the road surface is not the vertical / horizontal position of the in-vehicle camera 2 or the change in the position in the height direction, the roll angle and the pitch angle, but the angle in the yaw direction. You can think that it will appear. From this, when the vehicle 1 is moving while the attitude of the vehicle 1 changes with respect to the road surface, three values of the height, roll angle, and pitch angle of the in-vehicle camera 2 may be calculated.

In addition, since the attitude | position of the vehicle 1 with respect to a road surface changes frequently during the driving | running | working of the vehicle 1, the frequency which calculates the position and angle of the vehicle-mounted camera 2 becomes high, and the calculation load for it tends to become large. From this point of view, if the three values of the height, the roll angle, and the pitch angle of the in-vehicle camera 2 only have to be calculated, the calculation load can be halved, which is a great advantage.

FIG. 10 shows an example of a bird's-eye view image when the vehicle 1 is tilted in the pitch direction. As illustrated in FIG. 10A, the angle in the pitch direction of the in-vehicle camera 2 changes downward as the front portion of the vehicle 1 tilts downward. FIG. 10B shows a bird's-eye view image in which lane breaks on the left and right sides of the vehicle 1 are displayed in this state. As shown in the figure, the lane divisions on the left and right of the vehicle 1 are displayed in a square shape. If the vehicle 1 is not tilted, the left and right lane divisions should be displayed parallel to each other. Therefore, if the feature points are extracted from the boundaries of the lane divisions and adjusted so that the lane divisions inclined in the letter C shape are displayed in parallel with each other, the pitch angle of the in-vehicle camera 2 corresponding to the inclination of the vehicle 1 can be set. It can be recalculated. In addition, the black circle shown on the boundary line of a lane division in FIG.10 (b) is an example of the extracted feature point. If this feature point is connected, it will be the same as the line separating the lanes, so the illustration of this feature point is omitted in the following bird's-eye view examples.

FIG. 10 (c) shows a bird's-eye view image displaying a stop line in front of the vehicle 1 in the state of FIG. 10 (a). As shown in the drawing, two straight lines extending in the vertical direction are formed in a square shape, and two straight lines extending in the horizontal direction are displayed as parallel trapezoids. If the vehicle 1 is not tilted, the stop line should be displayed in a rectangle that is long in the horizontal direction. Therefore, as in the case of FIG. 10B, the pitch angle of the in-vehicle camera 2 corresponding to the inclination of the vehicle 1 is calculated by adjusting the two straight lines inclined in the C shape so that they are parallel to each other. You can fix it.

Here, the accuracy of the calculated pitch angle of the in-vehicle camera 2 is compared between the case of FIG. 10B and the case of FIG. 10C. In the case of FIG. 10B, the inclination can be examined from the line segment having the length L1, whereas in the case of FIG. 10C, the inclination is examined from the line segment having the length L2 shorter than the length L1. It will be. Naturally, since the error becomes larger when the inclination is examined from the line segment of the length L2 than in the line segment of the length L1, the calculation in the case of FIG. 10B is calculated rather than the case of FIG. 10C. It is considered that the accuracy of the pitch angle of the in-vehicle camera 2 is good.

Subsequently, in the case of FIG. 10B and the case of FIG. 10C, looking at the difference in distribution of feature points, in the case of FIG. The width of distribution of the feature points indicated by is large. On the other hand, in the horizontal direction, feature points indicated by black circles do not exist in the left and right end regions and the central region, and the width in which the feature points are distributed is small.

In the case of FIG. 10C, the black circles of the feature points are not shown. However, when the display area of the stop line is confirmed, the width in which the feature points are distributed in the vertical direction is small, and the feature points in the horizontal direction. It can be seen that the width of the distribution is large.

From the above, when the pitch angle of the in-vehicle camera 2 is recalculated as shown in FIG. 10, the width in which the feature points are distributed in the vertical direction (the traveling direction of the vehicle 1) on the bird's-eye view image is large. It is understood that this is important for accurate calculation. Therefore, when calculating the pitch angle of the in-vehicle camera 2, it is preferable that the width in which the feature points are distributed in the vertical direction on the bird's-eye view image is larger than a predetermined vertical threshold. Note that the vertical threshold here corresponds to the first threshold.

FIG. 11 shows an example of a bird's-eye view image when the vehicle 1 is tilted in the roll direction. As shown in FIG. 11A, when the angle of the in-vehicle camera 2 in the roll direction changes to the right as the right side inclines downward toward the front of the vehicle 1, as shown in FIG. Therefore, the display mode of the left and right lane divisions of the vehicle 1 changes. In the left lane separation in FIG. 11B, the width W <b> 1 is wide and the display position is far from the vehicle 1. On the other hand, the right lane segment is displayed with a narrow width W2 and the display position is approaching from the vehicle 1. In such a case, for example, the roll angle of the in-vehicle camera 2 according to the inclination of the vehicle 1 can be recalculated by adjusting the width of the left and right lane divisions to be the same. However, the actual width of the lane division is 15 cm to 20 cm, and the width of the lane division on the bird's-eye view image is also displayed smaller than the display area of the image. Compared to the case of FIG. It is difficult to adjust accurately.

In the stop line shown in FIG. 11 (c), two straight lines extending in the vertical direction are parallel and two straight lines extending in the horizontal direction are trapezoidal. Therefore, the roll angle of the in-vehicle camera 2 according to the inclination of the vehicle 1 can be recalculated by adjusting the horizontal C-shaped two straight lines so that they are parallel to each other. Since the length L3 of the horizontal C-shaped two straight lines is clearly longer than the widths W1 and W2 shown in FIG. 11B, the length L3 in FIG. 11C is more than that in FIG. 11B. In this case, it is considered that the accuracy of the calculated roll angle of the in-vehicle camera 2 is better.

Therefore, when calculating the roll angle of the in-vehicle camera 2, it is preferable that the width in which the feature points are distributed in the lateral direction on the bird's-eye view image is larger than a predetermined lateral threshold. Note that the lateral direction threshold here corresponds to the second threshold.

FIG. 12 shows an example of a bird's-eye view image when the height of the vehicle 1 changes. As shown in FIG. 12 (a), the height of the in-vehicle camera 2 is lowered as the vehicle 1 sinks as a whole. The lane breaks displayed in this state are as shown in FIG. 12B, and the stop lines are as shown in FIG. In both FIG. 12B and FIG. 12C, the display before the height of the vehicle 1 changes is indicated by a broken line. Compared with the broken line, the left and right lane divisions in FIG. 12B are widened and displayed at positions away from the vehicle 1. The width of the stop line in FIG. 12C is also increased and is displayed at a position away from the vehicle 1.

As described above, when the height of the vehicle 1 changes, the display on the bird's-eye view image changes so as to be enlarged or reduced. Therefore, the feature points can be obtained using the shapes described with reference to FIGS. Although consistency cannot be obtained, for example, the height of the vehicle 1 can be calculated as follows. That is, in the case of FIG. 12B, if data regarding the actual lane separation width is prepared in advance, the lane separation width displayed in the bird's-eye view image matches the actual lane separation width. By adjusting, the height of the vehicle-mounted camera 2 according to the height of the vehicle 1 can be recalculated. In the case of FIG. 12C as well, if data relating to the width of the stop line is prepared in advance, the height of the in-vehicle camera 2 corresponding to the height of the vehicle 1 can be recalculated. Thus, when calculating the height of the in-vehicle camera 2, not the shape consistency but the scale consistency. The scale of the image can be evaluated accurately only after the situation of the entire image can be grasped. Therefore, if the vertical scale of the bird's-eye image is to be evaluated, the distribution of feature points in at least the vertical direction must be good, and if the horizontal scale is to be evaluated, at least in the horizontal direction. The distribution of feature points must be good.

As described above, when the height of the vehicle 1 changes, the display on the bird's-eye view image changes so as to be enlarged or reduced. However, even when the pitch direction of the vehicle 1 is inclined, it is enlarged or reduced. The display on the bird's-eye image may change as if it were. Further, the tendency of such display change is calculated because it is affected by the mounting position such as where the vehicle 1 is before, after, left and right, and the installation state of the in-vehicle camera 2 such as the difference in the amount of sinking due to the hardness of the bumper. It can be assumed that the variables are mistaken. In such a case, in a state where the vehicle-mounted camera 2 is mounted on the vehicle 1, data representing a tendency between the change in the posture of the vehicle 1 and the change in the display mode on the bird's-eye view image is acquired in advance, and the data is referred to. Therefore, it is preferable to determine which of the three variables of the height, roll angle, and pitch angle of the in-vehicle camera 2 is to be calculated.

Further, when the posture of the vehicle 1 changes, it is not limited to the case where the pitch angle, the roll angle, and the height change individually, and these may change in combination. Even in such a case, the pitch direction angle or the roll direction angle of the in-vehicle camera 2 is good with respect to whether the feature point distribution is good in the vertical direction or the horizontal direction on the bird's-eye view image. The relationship of whether or not it can be calculated with accuracy is the same. Therefore, it is preferable to select a variable to be calculated as shown in FIG.

FIG. 13 shows an example of selecting variables according to the distribution of feature points. As described above, the distribution of feature points is divided into a vertical direction and a horizontal direction on the bird's-eye view image, and the distribution of the feature points in the vertical direction and the horizontal direction is divided according to whether the width is large or small. I am doing.

When the feature point distribution width is small in both the vertical direction and the horizontal direction, as described above in the present embodiment, calculation is performed for any of the pitch angle, roll angle, and height variables of the in-vehicle camera 2. Do not do.

ピッチ If the feature point distribution width is large in the vertical direction and small in the horizontal direction, the pitch angle is calculated. In this case, the pitch angle can be calculated with good accuracy as described with reference to FIG. 10B, but the roll angle is difficult to obtain with accuracy as described with reference to FIG. 11B. .

¡If the distribution point of feature points is small in the vertical direction and large in the horizontal direction, the roll angle is calculated. In this case, the roll angle can be calculated with good accuracy as described with reference to FIG. 11C, but the pitch angle is difficult to obtain with accuracy as described with reference to FIG. 10C. .

If the feature point distribution width is large in both the vertical and horizontal directions, all variables of the pitch angle, roll angle, and height of the in-vehicle camera 2 are calculated in the same manner as in this embodiment. That's fine. In addition to this case, the height of the in-vehicle camera 2 may be calculated if the distribution width of the feature points is large in either the vertical direction or the horizontal direction. Further, the height of the in-vehicle camera 2 may be calculated on the condition that feature points that can achieve scale consistency such as lane separation and stop line width are extracted.

As described with reference to FIGS. 10 to 13, the distribution of the feature points is confirmed for each of the vertical direction and the horizontal direction, and the condition of the pitch angle, roll angle, and height of the in-vehicle camera 2 is determined according to the result. If only good variables are selected and calculated, the accuracy of the calculated variables can be kept good. Further, since the time required for the calculation can be reduced by selecting the variable to be calculated, it becomes easy to cope with a temporary change in the vehicle posture. Other variables that are not selected may be calculated when a feature point with good conditions is obtained from a bird's-eye image obtained by subsequent shooting. Then, as described in the present embodiment, after confirming the newly calculated variable fluctuation amount (S109 and S110 in FIG. 6), the conversion table is updated.

Further, when calculating the pitch angle, roll angle, and height of the in-vehicle camera 2, all variables are calculated regardless of the feature point distribution state, and when the conversion table is updated, it is selected as shown in FIG. The conversion table may be updated using the obtained variable.

The image conversion apparatus 10 according to this modification can calculate the pitch angle, the roll angle, and the height of the in-vehicle camera 2 when the conditions are good, and refers to the conversion table obtained based on this, so from the captured image It can be accurately converted into a bird's-eye view image.

As mentioned above, although an Example and a modification were illustrated, an Example and a modification are not restricted to said Example and a modification, It can be set as a various aspect in the range which does not deviate from the summary of this indication.

Claims

A captured image is acquired from an in-vehicle camera (2) that captures the surroundings of the vehicle (1), and the captured image is converted into a bird's-eye image as if it was captured from a viewpoint above the vehicle, and then the in-vehicle monitor (3 An image conversion device (10) to be displayed on
An image conversion unit (12) that converts the captured image into the bird's-eye view image according to the position and angle of the in-vehicle camera;
A feature point extraction unit (15) for extracting a plurality of feature points from the bird's-eye view image or the captured image;
A calculation determination unit (16) for determining whether or not to calculate the position and angle of the in-vehicle camera based on the distribution of the plurality of feature points extracted from the bird's-eye view image or the captured image;
An image conversion apparatus comprising: a calculation unit (17) that calculates the position and angle of the in-vehicle camera based on the plurality of feature points when the calculation determination unit determines to calculate the position and angle of the in-vehicle camera. .
The image conversion apparatus according to claim 1,
The calculation determination unit determines whether to calculate a position and an angle of the in-vehicle camera based on a distribution width of the plurality of feature points extracted from the bird's-eye view image or the captured image.
The image conversion apparatus according to claim 1 or 2, wherein
An image conversion apparatus comprising: an extraction determination unit (14) that determines whether or not to extract the feature points according to detection information acquired by a detector (4) that detects a surrounding environment of the vehicle or a state of the vehicle.
An image conversion device according to any one of claims 1 to 3,
The image conversion unit refers to the conversion table in which the pixel position in the captured image corresponding to the pixel position in the bird's-eye image is calculated based on the position and angle of the in-vehicle camera, and thereby the captured image is converted into the captured image. Converted to a bird's-eye view,
A fluctuation amount acquisition unit (18) that acquires a fluctuation amount by comparing the position and angle of the in-vehicle camera newly calculated by the calculation unit with the position and angle of the in-vehicle camera before the calculation;
An image conversion apparatus comprising: a conversion table update unit (19) that updates the conversion table based on a newly calculated position and angle of the in-vehicle camera when the amount of change is equal to or less than a predetermined change threshold. .
The image conversion device according to claim 4,
The calculation determination unit
When the width of the distribution of the feature points in the vertical direction of the bird's-eye view image is larger than a predetermined first threshold, it is determined to calculate an angle in the pitch direction that moves the optical axis of the in-vehicle camera in the vertical direction;
An image conversion apparatus that determines to calculate an angle in a roll direction that is rotated around the optical axis of the in-vehicle camera when a width of the distribution of the feature points in the horizontal direction of the bird's-eye view image is larger than a predetermined second threshold value.
An image conversion apparatus according to any one of claims 1 to 4, wherein
The feature point extraction unit is an image conversion apparatus that extracts the plurality of feature points from the bird's-eye view image.
Applied to a vehicle equipped with a vehicle-mounted camera, acquires a captured image from the vehicle-mounted camera that captured the surroundings of the vehicle, and converted the captured image into a bird's-eye image as if it was captured from a viewpoint above the vehicle Image conversion method for displaying on the vehicle monitor after
Converting the captured image into the bird's-eye view image according to the position and angle of the in-vehicle camera (S104);
Extracting a plurality of feature points from the bird's-eye view image or the captured image (S105);
Determining whether to calculate the position and angle of the in-vehicle camera based on the distribution of the plurality of feature points extracted from the bird's-eye view image or the captured image (S107);
A step (S108) of calculating the position and angle of the in-vehicle camera based on the plurality of feature points when it is determined to calculate the position and angle of the in-vehicle camera.