JP5455124B2 - Camera posture parameter estimation device - Google Patents

Description

  The present invention relates to a posture parameter estimation technique for a wide-angle camera.

  In recent years, for the purpose of improving the safety of driving operations, there are vehicles equipped with a camera system equipped with a plurality of cameras using wide-angle lenses such as fish-eye cameras and the like and monitoring the surroundings of the vehicle without blind spots. . Such a camera system generates a surrounding image of the vehicle using images taken by each camera installed at a plurality of locations of the vehicle, and displays the generated image on a display device. By viewing the peripheral image displayed on the display device, the driver can easily and safely drive the vehicle.

  Since the display area of the display device inside the vehicle is limited in size, it is important to display a peripheral image generated from a plurality of images photographed by each camera so that the driver can easily see and instantly understand. For example, Patent Document 1 below proposes a method for solving such a problem.

  In addition, in the camera system as described above, there is a system that generates a top view (hereinafter also referred to as a bird's-eye view) depicting a vehicle as viewed from above, as a surrounding image of the vehicle. According to the bird's-eye view, the driver can instantly grasp the situation around the vehicle without blind spots.

  However, in order to generate a bird's eye view in a camera system having a plurality of fisheye cameras, it is necessary to calibrate each fisheye camera in advance.

  Camera parameters that can be calibrated are generally classified into internal parameters and external parameters. Internal parameters include principal point, focal length, lens distortion, and the like. Such internal parameters do not change in the camera environment, and are measured in advance and set in a manufacturing process or the like.

On the other hand, the external parameters are the orientation and position of the camera with respect to the world coordinate system and are also called camera pose parameters. Various methods for calibrating such camera posture parameters have been proposed. For example, a technique for calibrating a stereo vision system in an automobile application using a plurality of markers placed on a hood of a car has been proposed (see Non-Patent Document 1 below). Further, it has been proposed to calibrate external parameters by using a plurality of circular patterns placed on the ground in front of the vehicle (see Non-Patent Document 2 below). Furthermore, a method for calibrating the direction of the front camera by using parallel lane markings detected by an in-vehicle camera has been proposed (see Non-Patent Document 3 below).

JP 2009-206747 A

A. Broggi, M. Bertozzi and A. Fascioli, “Self-calibration of a stereo system for automotive applications”, Proc. Of IEEE conference on Robotics and Automation, pp.3698-3703, 2001 S. Hold, C. Nunn, A. Kummert, and S. Muller-Schnerders, “Effi-cient and robust extrinsic camera calibration procedure for lane departure warning”, Proc. Of Intelligent Vehicle Symposium, pp.382-387, 2009 A. Catala-Part, J. Rataj and R. Reulke, “Self-calibration system for the orientation of a vehicle camera”, ISPR Commission V Sympo-sium on Image Engineering and Vision Metrology, pp.68-73, 2006

  However, the conventional camera calibration method as described above is a method applied to a general camera with a limited field of view (FOV (Field Of View)) based on the conventional pinhole camera model. It cannot be directly applied to a wide-angle camera having a wider field of view than a hemisphere such as a direction camera or a fish-eye camera.

  Therefore, in a vehicle-mounted camera system using a conventional wide-angle camera, as shown in FIG. 24, there is a method of calibrating camera posture parameters using a plurality of markers placed at predetermined accurate positions on the ground. It has been adopted. FIG. 24 is a diagram showing a conventional method for calibrating the attitude parameter of one wide-angle camera mounted on the rear part of the vehicle. According to the example of FIG. 24, one marker is installed at each position of 50 centimeters (cm), 1 meter (m), and 2 (m) from the rear of the vehicle on the center line in the width direction of the vehicle. Furthermore, it is necessary to install one marker at each position 1 (m) from the rear of the vehicle on two lines parallel to the surfaces on both sides of the vehicle and separated from each other by the vehicle width.

  As described above, in the conventional method, since a plurality of markers must be prepared and each marker must be arranged at a predetermined accurate position, a great burden is imposed on the calibration of the camera.

  In view of such a problem, an object of one embodiment of the present invention is to realize a technique for easily and accurately estimating a posture parameter of a wide-angle camera.

  Each aspect of the present invention employs the following configurations in order to solve the above-described problems.

  The first aspect relates to a camera posture parameter estimation device. The camera posture parameter estimation device according to the first aspect includes, from a wide-angle image obtained by a wide-angle camera having a field of view equal to or greater than a hemisphere, at least two first straight lines and second straight lines parallel to each other in real space; A pattern acquisition unit that acquires each line segment that can form a specific pattern composed of a straight line and a single third line that is perpendicular to the second line, and a specific pattern acquired by the pattern acquisition unit A determination unit that determines two vanishing point pairs for the wide-angle image using each possible line segment, and estimates a posture parameter of the wide-angle camera with respect to the ground using the two vanishing point pairs determined by the determination unit. And an estimation means.

  In the first aspect, each line segment corresponding to a specific pattern in the real space is acquired from the wide-angle image obtained from the wide-angle camera, and two vanishing point pairs for the wide-angle image are determined from each acquired line segment. The attitude parameter of the wide-angle camera is estimated from the vanishing point pair.

  Therefore, according to the first aspect, as long as the specific pattern is imaged by the wide-angle camera, the posture parameter of the wide-angle camera can be estimated easily and accurately.

  The specific pattern includes at least two first straight lines and second straight lines parallel to each other in real space, and one third straight line having a relationship perpendicular to the first straight lines and the second straight lines. Such a specific pattern may be a pattern that is generally present in real space, such as a parking lot line of a parking lot or a wall surface of a building. Can be greatly reduced.

  Preferably, in the first aspect, the determination unit uses the line segment corresponding to the first straight line and the line segment corresponding to the second straight line among the line segments corresponding to the specific pattern to generate a wide angle image. One vanishing point pair for the spherical image mapped using the internal parameters of the camera is determined as one vanishing point pair of the wide-angle image, and the determined one vanishing point and the center of the spherical image are determined. One vanishing line is determined from a plane whose normal is the connecting line, and another vanishing line is determined from the projection plane onto the spherical image of the third straight line that is parallel to the plane. A further vanishing point pair may be determined from the intersection of two vanishing lines.

  Thereby, two vanishing point pairs for the wide-angle image can be determined with a small amount of information, and the camera posture parameters can be estimated accurately.

  Preferably, in the first aspect, the pattern acquisition means is a fourth straight line different from the first straight line and the second straight line, and a third straight line having a relationship perpendicular to the first straight line and the second straight line. Further, each line segment that can form another specific pattern composed of the above is obtained, and the determining means determines a new pair of vanishing points using the fourth straight line in the same manner as the third straight line. One vanishing point pair may be determined by averaging the newly determined one vanishing point pair and the one vanishing point pair determined using the third straight line.

  In this way, two vanishing point pairs can be determined more accurately.

  In the first aspect, it is also possible to estimate the posture parameters of each wide-angle camera for a plurality of wide-angle cameras. In this case, the pattern acquisition unit acquires each line segment corresponding to the specific pattern from each wide-angle image obtained from a plurality of wide-angle cameras, and the determination unit selects two vanishing point pairs for each wide-angle image. The estimation means estimates the posture parameters of each wide-angle camera using each of the two vanishing point pairs for each wide-angle image. In this aspect, the camera posture parameter estimation apparatus according to the first aspect overlaps each other in a wide-angle image obtained from at least two adjacent wide-angle cameras having overlapping field areas among a plurality of wide-angle cameras. An image area corresponding to the visual field area is identified using each posture parameter of the adjacent wide-angle camera estimated by the estimation means, and each posture parameter of the adjacent wide-angle camera is identified using image information of each identified image area. You may make it further provide the correction means which corrects.

  According to this, each posture parameter regarding a plurality of wide-angle cameras is estimated, and each of these estimated posture parameters is further refined by relative information between at least two adjacent wide-angle cameras having overlapping visual field regions. Therefore, according to the first aspect, it is possible to accurately estimate the posture parameters of a plurality of wide-angle cameras.

  Further, in the aspect targeting the plurality of wide-angle cameras, each of the posture parameters of the adjacent wide-angle cameras is set so that the correction means has a value that maximizes the correlation value between the image data of each identified image area. You may make it correct.

A second aspect relates to a camera system including a plurality of wide-angle cameras, a parameter estimation device as described above, and an image generation device that generates image data from each wide-angle image obtained from each wide-angle camera. In this camera system, the image generation device uses the posture parameters of the plurality of wide-angle cameras estimated by the parameter estimation device as in the first aspect described above, and the plurality of wide-angle cameras are converted from a plurality of wide-angle images. Obtaining means for generating image data showing a bird's-eye view centered on the mounted vehicle and vehicle tilt information in the bird's-eye view inputted by the user with reference to the bird's-eye view corresponding to the image data generated by the generating means Tilt information obtaining means for reflecting the information, and reflecting means for reflecting the tilt information on the posture parameters of a plurality of wide-angle cameras.

  According to the second aspect, the inclination of the vehicle with respect to the ground can be reflected on each posture parameter of the plurality of wide-angle cameras based on the inclination information input by the user, and a more appropriate bird's-eye view can be output. Thus, a safe driving operation can be provided to the driver of the vehicle.

  As another aspect of the present invention, a camera posture parameter estimation method that realizes any one of the above-described configurations may be used, or a program that executes such a method may be used. It may also be a computer-readable storage medium that records various programs.

  According to each aspect of the present invention, it is possible to provide a technique for easily and accurately calibrating attitude parameters of a wide-angle camera.

1 is a schematic diagram illustrating a configuration of a camera system including a parameter estimation device in Embodiment 1. FIG. The schematic diagram which shows the top view of the vehicle by which the parameter estimation apparatus in Example 1 is mounted. FIG. 3 is a diagram illustrating an example of a wide-angle image in the first embodiment. FIG. 3 is a diagram illustrating an example of an output image (bird's eye view) according to the first embodiment. The figure which shows the model which shows the relationship between an image plane and a camera lens. The figure which shows a spherical image and spherical projection. The figure which shows the projection on the unit spherical surface of the straight line L. FIG. The figure which shows the example of detection of a line segment. The figure which shows the algorithm of vanishing point determination in a vanishing point determination part. The figure which shows the algorithm of vanishing point determination in a vanishing point determination part. The figure which shows the relationship between H pattern in a spherical image, and a vanishing point. 3 is a flowchart illustrating an operation example of the parameter estimation device according to the first embodiment. The figure which shows the schematic diagram which shows the top view of the vehicle by which the parameter estimation apparatus in Example 2 is mounted. FIG. 9 is a schematic diagram illustrating a configuration of a camera system including a parameter estimation device according to a second embodiment. The figure which shows the example of the wide angle image obtained from each vehicle-mounted camera. FIG. 10 is a diagram illustrating an example of an output image (bird's eye view) in the second embodiment. The figure which shows the example of a line segment detection from the wide-angle image acquired from each vehicle-mounted camera. 9 is a graph showing correlation values calculated with respect to the number of repetitions of camera posture parameter correction in the second embodiment. 6 is a flowchart illustrating an operation example of the parameter estimation device according to the second embodiment. The figure which shows the example of the bird's-eye view produced | generated by the attitude | position parameter before being corrected by the parameter correction part. FIG. 10 is a diagram illustrating an example of a bird's eye view output by the camera system according to the second embodiment. FIG. 6 is a schematic diagram illustrating a configuration of a camera system as a modification of the second embodiment. The figure which shows the example of the output image produced | generated using the attitude | position parameter finely adjusted in this modification. The figure which shows the conventional method for calibrating the attitude | position parameter of one wide angle camera mounted in the rear part of a vehicle.

  Hereinafter, a specific example of the parameter estimation apparatus as one embodiment of the present invention will be described. A parameter estimation device as an embodiment estimates a posture parameter of a wide-angle camera installed in a vehicle and installed in the vehicle. In addition, the parameter estimation apparatus in this embodiment does not limit the kind of vehicle mounted. Each Example given below is an illustration, respectively, and this invention is not limited to the structure of each following Example.

  In each of the following examples, an example is shown in which the posture parameter of the camera is estimated in a state where a vehicle equipped with the parameter estimation device in the present embodiment is parked in a parking lot where the parking line is attached to the ground. When parameter estimation is performed by the parameter estimation device, the vehicle is parked in an arbitrary parking space so that the traveling direction of the vehicle and the parking lines on both sides are substantially parallel. The term “substantially parallel” means that perfect parallelism is preferable, but it is acceptable even if not perfectly parallel. Hereinafter, the posture of the vehicle when parameter estimation is performed by the parameter estimation device in the present embodiment is also referred to as a basic posture. The parameter estimation apparatus according to the present embodiment estimates a posture parameter of an in-vehicle camera using a parking line photographed by a camera installed in a vehicle parked in a basic posture.

  Hereinafter, a first example of the parameter estimation apparatus as the embodiment will be described.

[System configuration]
FIG. 1 is a schematic diagram illustrating a configuration of a camera system including a parameter estimation device according to the first embodiment. FIG. 2 is a schematic diagram illustrating a top view of a vehicle on which the parameter estimation device according to the first embodiment is mounted. As shown in FIG. 1, the camera system 1 includes an in-vehicle camera 7, a parameter estimation device 10 according to the first embodiment, a peripheral image generation device 20, a display device 30, and the like. The camera system 1 is mounted on the vehicle 5 and provides the driver of the vehicle 5 with an image (video) that shows the periphery of the vehicle 5.

  As shown in FIG. 2, one in-vehicle camera 7 is installed in the rear part of the vehicle 5. The in-vehicle camera 7 is a wide-angle camera having a wide-angle lens having a field of hemisphere or more such as a fisheye lens. The vehicle-mounted camera 7 uses at least a region behind the rear portion of the vehicle 5 (a long chain line shown in FIG. 2) as a visual field region. A video signal captured by the in-vehicle camera 7 is sent to the peripheral image generation device 20. The present invention does not limit the exact position where the in-vehicle camera 7 is installed.

The peripheral image generation device 20 is configured by a CPU (Central Processing Unit), a ROM (Read Only Memory), a RAM (Random Access Memory), an input / output interface, and the like, for example, in response to a software or hardware activation instruction. Power is turned on and activated. The peripheral image generation apparatus 20 executes the following process by executing a program recorded in the ROM.

The peripheral image generation device 20 receives a video signal from the in-vehicle camera 7 and acquires image data from this video signal. Hereinafter, the image data obtained from the in-vehicle camera 7 is also referred to as a wide-angle image. The peripheral image generation device 20 generates image data that reflects the periphery of the vehicle 5 by converting the wide-angle image using internal parameters and external parameters of the in-vehicle camera 7. This image data is generated, for example, as a bird's eye view showing the field of view of the in-vehicle camera 7 as if looking down from above the vehicle 5. FIG. 3 is a diagram illustrating an example of a wide-angle image in the first embodiment, and FIG. 4 is a diagram illustrating an example of an output image (bird's eye view) in the first embodiment.

  The internal parameters of the in-vehicle camera 7 are measured in advance at the time of manufacture or the like, and are stored in advance in a memory that can be read by the peripheral image generation device 20. On the other hand, initial values of external parameters (posture parameters) are stored in advance in the memory, and are calibrated by the parameter estimation device 10 at an arbitrary timing. The arbitrary timing is, for example, when power is started or when a hardware or software operation is performed. The present invention does not limit the calibration timing of the attitude parameter of the in-vehicle camera 7 as described above.

  The display device 30 receives the image data generated by the peripheral image generation device 20 and displays a bird's eye view as shown in FIG. 4 on a display (display surface). The driver can accurately recognize the blind spot and the situation behind the vehicle which is difficult to see by referring to the image behind the vehicle displayed on the display surface of the display device 30, and as a result, performs a safe driving operation. be able to.

  Since the present invention does not limit the hardware configuration of the peripheral image generation apparatus 20 and the image data conversion method, the description is simplified as described above. 1 shows an example in which the parameter estimation device 10 and the peripheral image generation device 20 are realized as separate hardware, but both may be realized as software modules.

[Device configuration]
The parameter estimation device 10 includes a specific pattern acquisition unit 11, a vanishing point determination unit 12, a parameter estimation unit 13, and the like. Each of these processing units may be realized as a hardware circuit or a software module. When realized as a software module, it is realized by executing a program stored in a memory in the CPU of the parameter estimation apparatus 10. Note that the present invention does not limit the hardware configuration of such a parameter estimation apparatus 10.

  The parameter estimation device 10 is activated at an arbitrary timing as described above, estimates the posture parameter of the in-vehicle camera 7, and updates the posture parameter stored in the peripheral image generation device 20 with the estimated posture parameter. The parameter estimation device 10 receives a video signal from the in-vehicle camera 7 when activated.

  Since the pinhole camera model cannot handle an image captured by a wide-angle camera having a field of view larger than a hemisphere, the parameter estimation device 10 processes a wide-angle image acquired from the in-vehicle camera 7 by using a spherical camera model. . That is, the parameter estimation device 10 maps the wide-angle image to the spherical image.

FIG. 5 is a diagram showing a model showing the relationship between the image plane and the camera lens. In FIG. 5, the ray direction of the point P (X, Y, Z) in the camera coordinate system is indicated by spherical coordinates (θ, φ), and the projected coordinates on the image plane are indicated by (r _i , φ _i ). For example, the following projection types are used for general fisheye lenses.

Equidistant Projection: r _i = fθ
Orthogonal Projection: r _i = f sin θ
Stereographic Projection: r _i = 2f tan (θ / 2)
Equal Area Projection: r _i = 2f sin (θ / 2)
An actual camera does not physically follow an ideal projection model. The optical axis may not pass through the center of the image. In addition, the vertical and horizontal sizes of one pixel may not be equal, and there may be radial distortion and decentering distortion of the actual lens. Therefore, a camera distortion model as shown in the following (Expression 1) and (Expression 2) is used. In the following (Expression 1) and (Expression 2), θ, φ, r _i , and φ _i are as shown in FIG. 5, and parameters of radial strain and eccentric strain are (k ₁ , k ₃ , k ₅ , P ₁ , P ₂ ).

By using the shift parameter of the optical center (x _C , y _C ) and the aspect ratio α, the projection coordinates (r _i , φ _i ) on the image plane are projected points in the orthogonal coordinate system of the input wide-angle image. Are mapped to (x _f , y _f ). Accordingly, the correspondence relationship between the point P (X, Y, Z) in the camera coordinate system and the point (x _f , y _f ) on the wide-angle image is expressed as (Equation 3) below.

  According to (Expression 3), an arbitrary point in the wide-angle image is mapped to the unit spherical image by using the camera internal parameter. Such projection of spatial points on a spherical image is called spherical projection. FIG. 6 is a diagram showing a spherical image and a spherical projection.

If the coordinates of the point P (X, Y, Z) with respect to the spherical image coordinate system are M _C = [X Y Z] ^T , the projection of the point P in the unit spherical image is expressed as m = [sin θ cos φ sin θ sin φ cos θ] ^T. Can do. Thus, the relationship between the coordinates _{M C} and projected coordinates m in the spherical image coordinate system is m = λM _C ≒ _{M C.} This relationship, m means that equal to M _C except the scale factor.

  In such a spherical model, as shown in FIG. 7, the straight line L is projected as a great circular arc l on the unit spherical surface. FIG. 7 is a diagram illustrating the projection of the straight line L onto the unit sphere. 7 is a unit normal vector of the projection plane of the straight line L. Here, assuming that the point p is on the circular arc l, if the point p is expressed as p = [sin θ cos φ sin θ sin φ cos θ] and the unit normal vector n is expressed as n = [a b c], a straight line projection The equation of the plane is as follows (Equation 4). Here, θ and φ are a polar angle and an azimuth angle in the spherical coordinate system.

  Here, since the points belonging to the same straight line exist on the same projection plane, the Hough transform in the spherical space is performed by using the projection point of the spatial straight line in the spherical image, and the normal vector of the projection line of the spatial straight line. Can be used to determine

  Hereinafter, each processing unit of the parameter estimation apparatus 10 that performs processing using the spherical camera model as described above will be described.

  When the specific pattern acquisition unit 11 receives the video signal transmitted from the in-vehicle camera 7, the specific pattern acquisition unit 11 acquires a wide-angle image from the video signal. In this wide-angle image, as shown in FIG. 3, a plurality of parking lines and other scenery of the parking lot are projected. Hereinafter, when the vehicle 5 is parked in the basic posture, a parking line substantially parallel to the traveling direction of the vehicle 5 is referred to as a side line 41, and a parking line substantially perpendicular to the traveling direction of the vehicle 5 is referred to as a rear line 42. write. In a general parking lot, the parking line is drawn so that the side line 41 and the rear line 42 are orthogonal.

  The specific pattern acquisition unit 11 detects a line segment corresponding to a specific pattern in real space from such a wide-angle image. This specific pattern is composed of at least two parallel line segments (hereinafter referred to as a parallel line group) and one line segment perpendicular thereto (hereinafter referred to as a vertical line). This specific pattern is hereinafter referred to as an H pattern based on the shape of each line segment. The specific pattern acquisition unit 11 maps the wide-angle image to the unit spherical image by using the internal parameters, and uses the algorithm based on the spherical camera model as described above, and uses the algorithm in the wide-angle image corresponding to the parallel line group as described above. A line segment in the wide-angle image corresponding to each line segment and vertical line is detected. Note that a well-known method (for example, see Japanese Patent Application Laid-Open No. 2009-288885) may be used as a method for detecting a line segment from a wide-angle image, and thus the description is simplified here.

  The specific pattern acquisition unit 11 extracts a plurality of edge points from the wide-angle image, and calculates a gradient direction for each edge point. This gradient direction indicates the gradient of the brightness of the edge point. The specific pattern acquisition unit 11 classifies the edge points into two groups based on the calculated gradient directions. One group is a set of edge points having gradient directions from 0 degrees to 180 degrees, and the other group is a set of edge points having remaining gradient directions. By this classification process, for example, the opposing boundary lines of the parking lines included in the wide-angle image belong to different groups.

  The specific pattern acquisition unit 11 extracts line segments from a set of edge points belonging to each group. For example, the Hough transform technique is used for the extraction of the line segment as described above. The specific pattern acquisition unit 11 extracts a great circle on the spherical image corresponding to the spatial straight line using the Hough transform technique. FIG. 8 is a diagram illustrating an example of detecting a line segment. According to the example of FIG. 8, the line segments 41A and 41B of the boundary line extracted from the side line 41 are classified into different groups. Similarly, the line segments 42A and 42B extracted from the rear line 42 are classified into different groups.

The specific pattern acquisition unit 11 specifies a line segment that is close to the vehicle 5 and that is long in the vertical direction of the wide-angle image (the vertical direction in FIG. 8) as the line segment corresponding to the parallel line group. A line segment long in the horizontal direction (the horizontal direction in FIG. 8) is specified as the line segment corresponding to the vertical line. At this time, the specific pattern acquisition unit 11 specifies two or more line segments that are long in the vertical direction of the mouth corner image with respect to the one group. That is, a line segment obtained from each of two or more side lines is specified.

  According to FIG. 8, for example, the specific pattern acquisition unit 11 includes a line segment 41A extracted from the side line 41 on the left side of the drawing (right side of the vehicle 5) and the side line 41 on the right side of the drawing (left side of the vehicle 5). The line segment 41A extracted from is identified as each line segment corresponding to the parallel line group. The specific pattern acquisition unit 11 further specifies at least one of the line segments 42A and 42B as a line segment corresponding to the vertical line. The specific pattern acquisition unit 11 sends information about each line segment corresponding to the H pattern specified in this way to the vanishing point determination unit 12 together with the wide-angle image. The specific pattern acquisition unit 11 may send the normal vector of the projection plane of each line segment determined using (Equation 4) to the vanishing point determination unit 12 as information about each line segment. For the determination of the normal vector, the least square method is used.

  The vanishing point determination unit 12 determines two vanishing point pairs in the wide-angle image using information about each line segment corresponding to the H pattern obtained from the specific pattern acquisition unit 11. Specifically, the vanishing point determination unit 12 first determines one vanishing point pair from each line segment corresponding to the parallel line group. Next, the vanishing point determination unit 12 determines a pair of vanishing points from the relationship between the line segment corresponding to the vertical line and each line segment corresponding to the parallel line group. 9 and 10 are diagrams illustrating an algorithm for determining a vanishing point in the vanishing point determining unit 12.

  FIG. 9 shows a group of three parallel lines in real space observed by a virtual spherical camera. As shown in FIG. 9, these parallel line groups provide a pair of vanishing points in the spherical image.

  Vanishing lines are defined as the intersection of parallel plane groups in infinite space. In the spherical image, this vanishing line corresponds to a great circle that passes through the center of the spherical image, as shown in FIG. A pair of vanishing points of a line parallel to the parallel plane can be determined as an intersection between the line and the vanishing line of the plane. That is, a pair of vanishing points in one line can be determined by a parallel line group including the line or a vanishing line on a plane parallel to the line.

FIG. 11 is a diagram illustrating the relationship between the H pattern and the vanishing point in the spherical image. FIG. 11 shows an H pattern composed of two parallel lines l _S1 and l _S2 and another line l _R perpendicular to the two lines. The vanishing point determination unit 12 determines a pair of vanishing points V _S and V ′ _S from the two parallel lines l _S1 and l _S2 in the H pattern.

A plane 通 る that passes through the center of the spherical image is determined with a normal vector n from the origin to V _S. The intersection of this plane with the spherical image results in a great circle _CV . The great circle _CV is a vanishing line of parallel planes including a plane plane. Since the line l _R is parallel to the plane [pi, intersection V _R and V _'R of the projection plane and the plane [pi to the spherical image line l _R is a pair of vanishing point of the line l _R. Thereby, the vanishing point determination unit 12 can obtain two sets of vanishing points from the H pattern.

Hereinafter, the process of the vanishing point determination unit 12 will be described more specifically. The vanishing point determination unit 12 receives from the specific pattern determination unit 11 each normal vector of the projection plane of each line segment (parallel lines l _S1 and l _S2 and vertical line l _R ) forming the H pattern. Here, the normal vectors are denoted by n _S1 , n _S2 , and n _R , respectively.

The vanishing point determination unit 12 calculates a pair of vanishing point pairs (V _S and V ′ _S ) in the two parallel lines l _S1 and l _S2 using each normal vector as shown in the following (formula 5).

Here, a vanishing line (great circle C _V ) of a plane plane passing through the origin and having n (= n _S1 × n _S2 ) as a normal vector is expressed as in the following (formula 6). Further, the intersection of the unit sphere image projection plane perpendicular line l _R is shown as the following equation (7).

The vanishing point determination unit 12 calculates a pair of vanishing points V _R and V ′ _R related to the vertical line l _R as a solution of (Expression 6) and (Expression 7). Specifically, x = tan θ cos φ and y = tan θ sin φ are set, and both sides of (Expression 6) and (Expression 7) are divided by cos θ to obtain (Expression 8) below. Further, by solving the equation (Equation 8), the following (Equation 9) is obtained, and as a result, (Equation 10) is obtained.

Vanishing point determining unit 12 uses the value of φ to calculate the value of theta, and determines the pair of vanishing point pairs about vertical l _R. The position of the vanishing point pair is determined by polar coordinates (θ, φ) on the unit sphere, and the orthogonal coordinates thereof are obtained by (sin θ cos φ, sin θ sin φ, cos θ) and (−sin θ cos φ, −sin θ sin φ, −cos θ).

  When the vanishing point determination unit 12 obtains information of two vertical lines (the line segments 42A and 42B shown in FIG. 8) from the specific pattern acquisition unit 11, a pair of disappearances are respectively provided for each vertical line as described above. A point pair may be determined, and the average of these may be used as a pair of vanishing point pairs for the final vertical line.

  The parameter estimation unit 13 estimates posture parameters of the in-vehicle camera 7 with respect to the ground (world coordinate system) using the information on the two vanishing point pairs determined by the vanishing point determination unit 12. Since a known method may be adopted as a method for estimating camera posture parameters from two vanishing point pairs detected in a wide-angle image, the description is simplified here.

The parameter estimation unit 13 estimates a rotation matrix with respect to the ground from two vanishing point pairs. Further, the parameter estimation unit 13 sets the intersection of the side line and the rear line in the wide-angle image as the origin of one world coordinate system, and based on this origin and two pairs of vanishing points, a translational vector (translational vector) Is estimated. The estimated rotation matrix indicates the camera orientation, and the estimated translation vector indicates the camera displacement. The parameter estimation unit 13 sends posture parameters including the rotation matrix and the translation vector estimated in this way to the peripheral image generation apparatus 20.

[Operation example]
Hereinafter, an operation example of the parameter estimation apparatus 1 according to the first embodiment will be described with reference to FIG. FIG. 12 is a flowchart illustrating an operation example of the parameter estimation device 1 according to the first embodiment.

  When the specific pattern acquisition unit 11 receives the video signal transmitted from the in-vehicle camera 7, the specific pattern acquisition unit 11 acquires a wide-angle image from the video signal (S121). The specific pattern acquisition unit 11 detects each line segment corresponding to a specific pattern (H pattern) in real space from such a wide-angle image (122). Specifically, the specific pattern acquisition unit 11 detects each line segment corresponding to at least two parallel line groups constituting the H pattern and a line segment corresponding to a vertical line perpendicular to the parallel line group.

  The vanishing point determination unit 12 acquires each line segment corresponding to the H pattern from the specific pattern acquisition unit 11. The vanishing point determination unit 12 determines one vanishing point pair from each line segment corresponding to the parallel line group (S123). This is determined as one vanishing point pair for the spherical image in which the wide-angle image is mapped using the internal parameters of the wide-angle camera.

  Next, the vanishing point determination unit 12 determines another vanishing point pair from the relationship between the vanishing point pair determined from each line segment corresponding to the parallel line group and the line segment corresponding to the vertical line (S124). ). Specifically, one vanishing line is determined from a plane whose normal is a line connecting one vanishing point determined as described above and the center of the spherical image, and a vertical relationship that is parallel to the plane is determined. Another vanishing line is determined from the projection plane of the line onto the spherical image, and the other vanishing point pair is determined from the intersection of the two vanishing lines determined.

  The parameter estimation unit 12 estimates the posture parameter of the in-vehicle camera 7 using the two vanishing point pairs in the wide-angle image determined by the vanishing point determination unit 12 (S125). Specifically, the parameter estimation unit 13 estimates the rotation matrix with respect to the ground from the two vanishing point pairs, and further sets the intersection of the side line and the rear line in the wide-angle image as the origin of one world coordinate system. A translation vector is estimated based on the origin and two vanishing point pairs.

[Operation and Effect in Example 1]
In the parameter estimation apparatus 1 according to the first embodiment as described above, the specific pattern acquisition unit 11 is in real space from a wide-angle image obtained by imaging the parking line of the parking lot as the surrounding environment of the vehicle 5 by the in-vehicle camera 7. A line segment corresponding to the H pattern is detected. In Example 1, as an H pattern in the world coordinate system, each line corresponding to at least two parallel line groups and one vertical line perpendicular thereto among the parking lines attached to the ground of the parking lot. Minutes are detected from the wide-angle image.

  Subsequently, the vanishing point determination unit 12 determines two vanishing point pairs in the wide-angle image using each line segment corresponding to the H pattern in the real space thus detected. Subsequently, the parameter estimation unit 13 estimates the rotation matrix of the posture parameters of the in-vehicle camera 7 using the two vanishing point pairs determined in this way, and the vertical line and the parallel line group The translation vector of the posture parameters of the in-vehicle camera 7 is estimated using the origin set at the intersection and the two vanishing point pairs.

  Thereby, in the surrounding image generation device 20, the attitude parameter obtained from the parameter estimation unit 13 is used as an external parameter, and the image data that reflects the periphery of the vehicle 5 as shown in FIG. 4 from the wide-angle image obtained from the in-vehicle camera 7 is displayed. Is generated.

  As described above, according to the parameter estimation device 1 of the first embodiment, the external parameters of the in-vehicle camera 7 can be easily and accurately set as long as the target vehicle 5 is parked in the parking lot with the parking line having the specific pattern. Can be calibrated. Since an H pattern as described above exists in a normal parking lot, according to the first embodiment, a plurality of markers are prepared as in the conventional method, and each is set at a predetermined accurate position. There is no need for heavy work such as placing a marker. In addition, the H pattern of the parking line does not necessarily have to be imaged by the in-vehicle camera 7 with the parallel line group and the vertical line orthogonal to each other, and an obstacle such as a traffic cone is placed at the intersection. It goes without saying that the posture parameter can be estimated even if the line near the intersection is erased. In such a case, each line segment detected from the wide-angle image may be extended according to the direction so as to derive the intersection.

  Further, conventionally, in the spherical camera model, the vanishing point is determined as an intersection of two or more parallel line groups, but in Example 1, two vanishing point pairs can be determined from the H pattern. Therefore, it is possible to further relax the restriction of the scene for correcting the posture parameter.

  In the first embodiment, the posture parameter of one in-vehicle camera 7 is an estimation target. Therefore, as Example 2, an example in which each posture parameter of the three in-vehicle cameras 7 is an estimation target will be described below. Hereinafter, the parameter estimation apparatus according to the second embodiment will be described focusing on the content different from the first embodiment.

  FIG. 13 is a schematic diagram illustrating a top view of a vehicle on which the parameter estimation device according to the second embodiment is mounted. FIG. 14 is a schematic diagram illustrating a configuration of a camera system including the parameter estimation device according to the second embodiment. The camera system 1 according to the second embodiment includes three in-vehicle cameras 7, the parameter estimation device 10 according to the second embodiment, a peripheral image generation device 20, a display device 30, and the like.

  In the second embodiment, three in-vehicle cameras 7 are set on the vehicle 5 at the rear part (7B), the left side (7L), and the right side (7R) of the vehicle, respectively. Each in-vehicle camera 7 is a wide-angle camera as in the first embodiment. Hereinafter, the in-vehicle cameras 7B, 7R, and 7L are also referred to as a rear camera, a right camera, and a left camera, respectively. In addition, this invention does not limit each exact position which installs the vehicle-mounted cameras 7B, 7R, and 7L.

  FIG. 15 is a diagram illustrating an example of a wide-angle image obtained from each vehicle-mounted camera 7. Specifically, FIG. 15A shows an example of a wide-angle image obtained from the left camera 7L, FIG. 15B shows an example of a wide-angle image obtained from the rear camera 7B, and FIG. Shows an example of a wide-angle image obtained from the right camera 7R. As shown in FIGS. 15A and 15C, in each wide-angle image obtained from the left camera 7L and the right camera 7R, the rear line 42 extends long in the vertical direction with respect to the drawing, and the side line 41 has It extends long in the lateral direction with respect to the drawing. Conversely, in the wide-angle image obtained from the rear camera 7B shown in FIG. 15B, the rear line 42 extends in the horizontal direction with respect to the drawing, and the side line 41 extends in the vertical direction with respect to the drawing. ing.

As can be seen from FIGS. 13 and 15, the rear camera 7 </ b> B uses at least a region behind the rear portion of the vehicle 5 (long chain line 7 </ b> B-FOV shown in FIG. 13) as a visual field region. The right camera 7R uses at least a region in the right direction from the right side surface of the vehicle 5 (a long chain line 7R-FOV shown in FIG. 13) as a visual field region. The left camera 7L uses at least a region leftward from the left side surface of the vehicle 5 (a long chain line 7L-FOV shown in FIG. 13) as a visual field region. Therefore, in the second embodiment, an overlapping visual field region (region OL-FOV-R indicated by hatching in FIG. 13) exists between the right camera 7R and the rear camera 7B, and the left camera 7L and the rear camera 7B. There is an overlapping visual field region (region OL-FOV-L indicated by hatching in FIG. 13).

  The peripheral image generation device 20 receives video signals from the three in-vehicle cameras 7 and acquires each wide-angle image from each video signal. The peripheral image generation device 20 generates image data that reflects the periphery of the vehicle 5 by converting each wide-angle image using the internal parameter and the external parameter of each vehicle-mounted camera 7. FIG. 16 is a diagram illustrating an example of an output image (bird's eye view) according to the second embodiment. In addition, since a well-known method should just be utilized about the specific method of producing | generating image data as shown in FIG. 16 from the wide-angle image obtained from the some vehicle-mounted camera 7 like the example of FIG. 15, here, Description is omitted.

  The image data generated by the peripheral image generation device 20 is displayed on the display of the display device 30. Accordingly, the driver can accurately recognize the situation in each direction other than the front, which is easy to see, by referring to the video image displayed on the display surface of the display device 30. As a result, safe driving operation can be performed. It can be carried out.

  The peripheral image generation device 20 stores internal parameters and external parameters for each of the three in-vehicle cameras 7 in a readable memory. As in the first embodiment, the internal parameters of each camera are measured and stored in advance at the time of manufacture. The initial values of the external parameters (posture parameters) of each camera are stored in advance in the memory, and are calibrated by the parameter estimation device 10 at an arbitrary timing.

[Device configuration]
The parameter estimation device 10 according to the second embodiment includes a set of a specific pattern acquisition unit 11, a vanishing point determination unit 12, and a parameter estimation unit 13 for each in-vehicle camera 7. Specifically, the specific pattern acquisition unit 11B, the vanishing point determination unit 12B, and the parameter estimation unit 13B are provided as a set for processing a wide-angle image obtained from the rear camera 7B, and a set for processing a wide-angle image obtained from the right camera 7R. As a set for processing a wide-angle image obtained from the left camera 7L, the specific pattern acquisition unit 11L, the vanishing point determination unit 12L, and the parameter estimation unit 13L are provided. Is provided.

  The specific pattern acquisition unit 11B, the vanishing point determination unit 12B, and the parameter estimation unit 13B estimate the posture parameter of the rear camera 7B from the wide-angle image obtained from the rear camera 7B by the same method as in the first embodiment. Similarly, the specific pattern acquisition unit 11R, the vanishing point determination unit 12R, and the parameter estimation unit 13R estimate the posture parameter of the right camera 7R from the wide-angle image obtained from the right camera 7R by the same method as in the first embodiment. Similarly, the specific pattern acquisition unit 11L, the vanishing point determination unit 12L, and the parameter estimation unit 13L estimate the posture parameter of the rear camera 7L from the wide-angle image obtained from the left camera 7L by the same method as in the first embodiment.

  FIG. 17 is a diagram illustrating an example of line segment detection from a wide-angle image acquired from each in-vehicle camera. As described above, the extending direction of the rear line 42 and the side line 41 differs between the wide-angle image acquired from the rear camera 7B and the wide-angle image obtained from the other cameras 7L and 7B. Therefore, the specific pattern acquisition unit 11B may detect each line segment corresponding to the H pattern in the real space, as in the first embodiment, but the specific pattern acquisition units 11R and 11L are slightly different from the first embodiment. Each of the specific pattern acquisition units 11R and 11L specifies, as the vertical line, a line segment that is close to the vehicle 5 and is long in the vertical direction of the wide-angle image (the vertical direction in FIG. 17). A line segment long in the direction (the horizontal direction in FIG. 17) is specified as the parallel line group. The other processes of the specific pattern acquisition units 11R and 11L are the same as in the first embodiment.

  In the example of FIG. 14, an example in which a set of the specific pattern acquisition unit 11, the vanishing point determination unit 12, and the parameter estimation unit 13 is provided for each in-vehicle camera 7 is shown, but the present invention is limited to such a configuration. It is not a thing. One specific pattern acquisition unit 11, one vanishing point determination unit 12, and one parameter estimation unit 13 may be provided.

  The parameter estimation device 10 according to the second embodiment further includes a parameter correction unit 15. The parameter correction unit 15 sets the posture parameters estimated by the parameter estimation units 13B, 13L, and 13R between the right camera 7R and the rear camera 7B and between the left camera 7L and the rear camera 7B. Correct for posture.

  As described above, in the second embodiment, there are overlapping visual field areas OL-FOV-R and OL-FOV-L between two adjacent vehicle-mounted cameras 7. The parameter correction unit 15 uses each posture parameter estimated by each parameter estimation unit 13 to put the pixel on the wide-angle image obtained from one of the adjacent in-vehicle cameras 7 on the wide-angle image obtained from the other. Map. Accordingly, a region corresponding to the visual field region OL-FOV-R in the wide-angle image obtained from the right camera 7R and a region corresponding to the visual field region OL-FOV-L in the wide-angle image obtained from the left camera 7L. Are mapped to wide-angle images obtained from the rear camera 7B.

Hereinafter, the mapping process between wide-angle images obtained from the adjacent in-vehicle cameras 7 will be described in more detail. The parameter correction unit 15 acquires the posture parameters (R _lg , T _lg ) of the left camera 7L and the posture parameters (R _bg , T _bg ) of the rear camera 7B from the parameter estimation units 13L and 13B. At this time, the projection of the ground point P _g in the wide-angle image obtained from the wide-angle image and the rear camera 7B obtained from the left camera 7L is assumed to be, respectively x _l and x _bl, the following expression holds.

s (x _l ) and s (x _bl ) indicate the coordinates of the corresponding points x _l and x _bl in the unit spherical image, respectively, and are calculated using the above (Equation 3) with each internal parameter. lambda _l and lambda _b are scaling factor z coordinates of ground point P _g is determined by the constraints of the ground plane to become zero when x-axis and y-axis of the world coordinate system is set on the ground plane.

According to the above (Formula 11), the following (Formula 12) is obtained, and finally, the following (Formula 13) is obtained. R _bg ⁻¹ represents an inverse matrix of R _bg . Further, s ⁻¹ indicates an inverse operation of s, and means that a point of the wide-angle image is calculated from the point of the unit spherical image by an internal parameter of the camera.

The parameter correction unit 15 uses, for example, the camera posture parameter and the known internal parameter in the above (Equation 13), and thereby converts the point x _l corresponding to the ground point P _g in the wide-angle image obtained from the left camera 7L to the rear camera 7B. mapping the point x _bl corresponding to the ground point P _g in the resulting wide-angle image from. In the above description, the mapping process between the wide-angle images obtained from the left camera 7L and the rear camera 7B has been described, but the mapping process between the wide-angle images obtained from the right camera 7R and the rear camera 7B is the same. Therefore, the description is omitted here.

Hereinafter, in order to simplify the description, the mapping process using the above (Formula 13) is shown as the following (Formula 14). Similarly, the projection of the ground point P _g2 is assumed to be x _r and x _br respectively at the wide-angle image obtained from the wide-angle image and the rear camera 7B obtained from the right camera 7R, from right camera 7R and the rear camera 7B The mapping processing between the obtained wide-angle images is shown as (Equation 15) below.

Here, when the brightness of x _l , x _r , x _bl and x _br is indicated by L _l (x _l ), L _r (x _r ), L _b (x _bl ), L _b (x _br ), According to the following (Equation 16), the identity of the overlapping image region corresponding to the visual field region overlapping between two adjacent vehicle-mounted cameras can be calculated. Ω ₁ and Ω ₂ indicate overlapping image areas on the ground between the left camera 7L and the rear camera 7B, and between the right camera 7R and the rear camera 7B, and L ₁ (with an upper line), L _r (with an upper line) , L _b (with an overline) represents the average value of the pixel values of the overlapping region in each wide-angle image acquired from the left camera, the right camera, and the rear camera, respectively.

  The parameter correction unit 15 uses, for example, the above (Equation 16), so that overlapping image areas corresponding to overlapping field-of-view areas are among the wide-angle images respectively obtained from the rear camera 7B, the right camera 7L, and the rear camera 7B. The posture parameters of each camera are corrected so as to approximate. The parameter correction unit 15 corrects each posture parameter by, for example, maximizing the value of (Expression 16) by using a simplex method. Specifically, since each attitude parameter is included in Ψlb and Ψrb as shown in (Expression 13), each of the adjusted Ψlb and Ψrb obtained when (Expression 16) is maximized is corrected. Get attitude parameters.

  FIG. 18 is a graph showing correlation values calculated for the number of repetitions of camera posture parameter correction in the second embodiment. The correlation value is calculated using (Equation 16). The parameter correction unit 15 ends the process when, for example, the correlation value calculated in (Expression 16) is greater than 0.8. In the example of FIG. 18, the correlation value before correction is 0.69, and the correlation value after correction is 0.8. The parameter correction unit 15 sends each posture parameter thus refined to the peripheral image generation unit 20.

[Operation example]
Hereinafter, an operation example of the parameter estimation apparatus 1 according to the second embodiment will be described with reference to FIG. FIG. 19 is a flowchart illustrating an operation example of the parameter estimation device 1 according to the second embodiment.

  The specific pattern acquisition units 11R, 11B, and 11L specify parallel lines and vertical lines corresponding to the H pattern from the wide-angle images obtained from the in-vehicle cameras 7 by the same method as in the first embodiment (S121 and S122). Subsequently, the vanishing point determination units 12R, 12B, and 12L respectively determine two vanishing point pairs for each wide-angle image by using the identified line segments in the same manner as in the first embodiment (S123 and S124). ). The parameter estimation units 13R, 13B, and 13L respectively estimate the posture parameters of the in-vehicle cameras 7 based on these two determined vanishing point pairs by the same method as in the first embodiment (S125).

In the second embodiment, subsequently, the parameter correction unit 15 includes image data of image areas corresponding to the overlapping areas in the wide-angle images obtained from the in-vehicle cameras 7 having overlapping visual field areas among the plurality of in-vehicle cameras 7 ( (Pixel data) is associated with each estimated posture parameter (S191). Specifically, an association (mapping) process between the wide-angle images obtained from the left camera 7L and the rear camera 7B and an association process between the wide-angle images obtained from the right camera 7R and the rear camera 7B are performed. Each done.

  The parameter correction unit 15 calculates the correlation of the image data (pixel data) of the overlapping image region corresponding to the visual field region overlapping between the in-vehicle cameras 7 (S192). The parameter correction unit 15 corrects each posture parameter with a value that maximizes the correlation value (S193). Thereby, each attitude | position parameter is corrected regarding each relative attitude | position between the left camera 7L and the rear camera 7B between the right camera 7R and the rear camera 7B.

[Operation and Effect in Example 2]
In the second embodiment as described above, wide-angle images are acquired from the rear camera 7B, the right camera 7R, and the left camera 7L installed on the rear, right side, and left side of the vehicle 5, and parking lines included in each wide-angle image. The posture parameters of the rear camera 7B, the right camera 7R, and the left camera 7L are estimated by the same method as in the first embodiment using the line segments corresponding to the H pattern.

  In the parameter estimation apparatus 10 according to the second embodiment, each of the posture parameters of the three in-vehicle cameras 7 estimated by the parameter correction unit 15 by the same method as in the first embodiment is refined with respect to the relative posture between two adjacent cameras. Is done. Specifically, by using each posture parameter estimated by each parameter estimating unit 13 and a predetermined internal parameter, a point of a wide-angle image obtained from one of two adjacent cameras is obtained from the other. To a wide-angle image point. Each posture parameter is determined so that the correlation value of the points of each mapped wide-angle image is maximized in an image region corresponding to the overlapping visual field region of two adjacent cameras. As described above, according to the second embodiment, the relative posture between the cameras can be refined by using the overlapping area of the ground between the adjacent cameras.

  FIG. 20 is a diagram illustrating an example of a bird's eye view generated based on the posture parameters before being corrected by the parameter adjusting unit 15. FIG. 21 is a diagram illustrating an example of a bird's eye view output by the camera system according to the second embodiment. 20 and 21, enlarged views of the circled parts in the lower figure are respectively shown in the upper figures. In the bird's eye view of FIG. 20, there is a visual contradiction in a portion corresponding to the overlapping visual field region between adjacent cameras. However, it can be seen that the visual contradiction is resolved in the bird's eye view of FIG. Thus, according to the second embodiment, an output image having visual consistency can be generated.

[Modification]
As described above, according to the parameter estimation device 10 in the first and second embodiments, the posture parameter (external parameter) of the in-vehicle camera 7 can be estimated and calibrated easily and accurately. However, in the first and second embodiments, the camera posture with respect to the ground H pattern is estimated, but the camera posture with respect to the vehicle 5 is not necessarily estimated. However, since the final output image (bird's eye view) is an image centered on the vehicle 5, the side surface of the vehicle 5 may not be parallel to the side line of the parking line in the output image (the lower diagram in FIG. 21). reference). In this case, rotational displacement may occur in the camera posture with respect to the ground.

  Therefore, the camera system 1 in the first and second embodiments described above may be configured such that the camera posture around the vehicle 5 can be finely adjusted via an interactive interface with the user. FIG. 22 is a schematic diagram illustrating a configuration of a camera system as a modification of the second embodiment. Although FIG. 22 shows a modification of the second embodiment, it is needless to say that the same modification can be applied to the first embodiment.

In the present modification, the fine adjustment processing unit 25 is provided in the peripheral image generation apparatus 20. The fine adjustment processing unit 25 is realized, for example, by executing a program stored in a ROM in the peripheral image generation apparatus 20 by the CPU. The fine adjustment processing unit 25 adds an interface for fine adjustment as an operation menu of the user to an output image such as a bird's eye view generated by the peripheral image generation device 20.

  When a user such as a driver determines that the side surface of the vehicle 5 is not parallel to the side line of the parking line in the output image displayed on the display device 30, an interface for fine adjustment as the operation menu is displayed. Operation is performed via the input device 35. This fine-tuning interface rotates the output image clockwise or counterclockwise so that the vertical side of the rectangular output image is parallel to both sides of the car appearing in the image, with the center of the output image as the rotation center. It is an interface that can change the angle of rotation around. Note that the form of this interface is not limited.

  The fine adjustment processing unit 25 acquires the rotation angle corresponding to the user operation based on the data obtained from the input device 35, and reflects this rotation angle in the rotation matrix of the posture parameter estimated by the parameter estimation device 10. Accordingly, the peripheral image generation device 20 generates an output image again using the updated posture parameter and outputs the output image to the display device 30.

  While viewing the output image displayed on the display unit of the display device 30, the user adjusts the rotation angle using the interface so that the side surface of the vehicle 5 is parallel to the side line of the parking line.

  FIG. 23 is a diagram illustrating an example of an output image generated using the attitude parameter finely adjusted in the present modification. FIG. 23 shows that the inclination of the vehicle 5 in the output image shown in the lower diagram of FIG. 21 is adjusted.

  As described above, according to this modification, by providing such an interactive interface with the user, it is possible to accurately determine the camera posture parameter with respect to the vehicle 5 and to obtain a more accurate posture parameter. It becomes possible.

  In each of the above-described embodiments, the form using one and three in-vehicle cameras is shown, but the present invention does not limit the number of in-vehicle cameras. The form using four vehicle-mounted cameras including the camera installed ahead of the vehicle 5 may be sufficient, and the form using the camera of the number more than that may be sufficient.

  Moreover, although the example which utilizes the parking line attached | subjected on the ground as a specific pattern (H pattern) in the real world was shown in each above-mentioned Example, this invention is not limited to such a form. It is known in advance as being composed of at least two parallel line segments (hereinafter referred to as a parallel line group) and one line segment perpendicular thereto (hereinafter referred to as a vertical line). For example, a wall surface and a ground surface of a building, a wall surface and a ceiling surface, and a model having such a shape may be used as long as the vehicle-mounted camera 7 can capture an image.

1 Camera system 5 Vehicle 7, 7B, 7L, 7R Car-mounted camera (rear camera, left camera, right camera)
10 parameter estimation device 11, 11B, 11L, 11R specific pattern acquisition unit 12, 12B, 12L, 12R vanishing point determination unit 13, 13B, 13L, 13R parameter estimation unit 20 peripheral image generation device 25 fine adjustment processing unit 30 display device 35 Input device

Claims (8)

From a wide-angle image obtained by a wide-angle camera having a field of view larger than a hemisphere, at least two first straight lines and second straight lines that are parallel to each other in real space, and have a relationship perpendicular to the first straight lines and the second straight lines Pattern acquisition means for acquiring each line segment corresponding to a specific pattern composed of one third straight line;
Determining means for determining two vanishing point pairs for the wide-angle image using each line segment corresponding to the specific pattern acquired by the pattern acquiring means;
Estimating means for estimating posture parameters of the wide-angle camera with respect to the ground using the two vanishing point pairs determined by the determining means;
A camera posture parameter estimation apparatus comprising: The determining means uses the line segment corresponding to the first straight line and the line segment corresponding to the second straight line among the line segments corresponding to the specific pattern, so that the wide-angle image has an internal parameter of the wide-angle camera. A set of vanishing point pairs for the spherical image mapped by using is determined as one set of vanishing point pair of the wide-angle image, and a line connecting the determined one vanishing point and the center of the spherical image is a normal line. One vanishing line is determined from the plane to be determined, another vanishing line is determined from the projection plane of the third straight line that is parallel to the plane onto the spherical image, and the two vanishing lines are determined. Determine one more vanishing point pair from the intersection,
The camera posture parameter estimation apparatus according to claim 1. The pattern acquisition means includes the first straight line and the second straight line, and a fourth straight line different from the third straight line having a perpendicular relationship with the first straight line and the second straight line. Each line segment that can form a specific pattern is further acquired,
The determining means determines a new set of vanishing point pairs using the fourth straight line in the same manner as the third straight line, and determines the newly determined one set of vanishing point pair and the third straight line. Determine one vanishing point pair by averaging with one set of vanishing point pair determined using
The camera posture parameter estimation apparatus according to claim 1 or 2. The pattern acquisition means acquires each line segment corresponding to the specific pattern from each wide-angle image obtained from a plurality of wide-angle cameras,
The determining means determines two vanishing point pairs for each of the wide-angle images,
The estimation means estimates the attitude parameters of the wide-angle cameras using the two vanishing point pairs for the wide-angle images,
The camera posture parameter estimation device includes:
Among the plurality of wide-angle cameras, an image area corresponding to the overlapping field area in each wide-angle image obtained from at least two adjacent wide-angle cameras having overlapping field areas is estimated by the estimating unit. A correction means for specifying each posture parameter and correcting each posture parameter of the adjacent wide-angle camera using the image information of each specified image region;
The camera posture parameter estimation apparatus according to claim 1, further comprising:   The correction means corrects each posture parameter of the adjacent wide-angle camera so that a correlation value between image data of each specified image region becomes a value that is maximized. Camera posture parameter estimation device. A plurality of wide-angle cameras installed at least on the side and rear of the vehicle and having a field of view larger than a hemisphere, a parameter estimation device for estimating posture parameters of each wide-angle camera, and image data generated from each wide-angle image obtained from each wide-angle camera In a camera system including an image generation device,
The parameter estimation device includes:
From each wide-angle image respectively obtained by the plurality of wide-angle cameras, at least two first straight lines and second straight lines that are parallel to each other in a real space, and 1 that is perpendicular to the first straight lines and the second straight lines Pattern acquisition means for acquiring each line segment corresponding to a specific pattern composed of three third straight lines,
Determining means for determining two vanishing point pairs for each wide-angle image using each line segment corresponding to the specific pattern acquired by the pattern acquiring means;
Estimating means for estimating posture parameters of the wide-angle cameras with respect to the ground using the two vanishing point pairs determined by the determining means,
With
The image generation device includes:
Generation that generates image data indicating a bird's-eye view centered on a vehicle on which the plurality of wide-angle cameras are mounted, from the plurality of wide-angle images, using the posture parameters of the plurality of wide-angle cameras estimated by the parameter estimation device. Means,
Inclination information acquisition means for acquiring vehicle inclination information in the bird's-eye view input by the user with reference to the bird's-eye view corresponding to the image data generated by the generation means;
Reflecting means for reflecting the tilt information on each posture parameter of the plurality of wide-angle cameras;
A camera system comprising: Computer
From a wide-angle image obtained by a wide-angle camera having a field of view larger than a hemisphere, at least two first straight lines and second straight lines that are parallel to each other in real space, and have a relationship perpendicular to the first straight lines and the second straight lines Obtain each line segment corresponding to a specific pattern composed of one third straight line,
Determining two vanishing point pairs for the wide-angle image using each line segment that can form the acquired specific pattern;
Estimating the wide-angle camera attitude parameters with respect to the ground using the determined two vanishing point pairs;
A camera posture parameter estimation method characterized by the above. From a wide-angle image obtained by a wide-angle camera having a field of view larger than a hemisphere, at least two first straight lines and second straight lines that are parallel to each other in real space, and have a relationship perpendicular to the first straight lines and the second straight lines Obtain each line segment corresponding to a specific pattern composed of one third straight line,
Determining two vanishing point pairs for the wide-angle image using each line segment that can form the acquired specific pattern;
Estimating the wide-angle camera attitude parameters with respect to the ground using the determined two vanishing point pairs;
A camera posture parameter estimation program that causes a computer to execute

Images