JP4857143B2 - Camera posture calculation target device, camera posture calculation method using the same, and image display method - Google Patents

Description

  The present invention relates to a camera posture calculation target device, a camera posture calculation method using the same, and an image display method, and more particularly to a camera posture calculation target device suitable for calculating the posture of a camera relative to a reference plane and the same. The present invention relates to a used camera orientation calculation method and an image display method.

  Conventionally, as an image display method using captured images of a camera, a plurality of cameras attached to the vehicle are used to shoot a shooting target surface such as a road surface or a parking scene around the vehicle, and based on the captured image A method of generating a vehicle periphery monitoring image for monitoring the periphery of the vehicle and displaying it on a display has been adopted.

  Examples of the vehicle periphery monitoring image include a vehicle called a top view image and a bird's-eye view image of the periphery of the vehicle viewed from above (hereinafter the same), a side view image for monitoring the side of the vehicle, and the like. It was known.

  In such an image display method, a captured image of a surface to be captured by a camera is subjected to coordinate conversion (viewpoint conversion) using a mapping table in which the correspondence relationship between the captured image and the vehicle periphery monitoring image is described. Generally, a vehicle periphery monitoring image is generated.

  This mapping table functions correctly if the camera posture (ie, position and direction) with respect to the shooting target plane is appropriate (in other words, as designed), and image distortion is based on the shot image of the shooting target plane by each camera. It was possible to generate a good vehicle periphery monitoring image with little or no chipping.

  However, it may not always be possible to attach the camera to the vehicle with high accuracy as designed. In such a case, the posture of the camera with respect to the surface to be photographed will deviate from an appropriate posture, and a good vehicle periphery There is a risk of hindering the generation of the monitoring image.

  Therefore, conventionally, in an in-vehicle device, before displaying a vehicle periphery monitoring image based on a captured image of a surface to be imaged, the posture of the camera with respect to the mounting surface of the vehicle that can be regarded as the same as the surface to be imaged in advance. Was supposed to be calculated. When the calculated posture deviates from an appropriate posture, the mapping table is corrected according to the deviation, and the vehicle periphery monitoring image is generated and displayed using the corrected mapping table. It was.

  As an example of a camera posture calculation method for correcting such a mapping table, for example, as shown in FIG. 21, a pattern in which a plurality of circular patterns 1 are arranged at predetermined alignment intervals vertically and horizontally. A calculation method using a camera posture calculation target board 3 having a surface 2 has been known.

  That is, when calculating the camera posture using such a camera posture calculation target board 3, first, as shown in FIG. 22, the camera posture calculation target board 3 is placed on the mounting surface of the vehicle 5. It is installed with the pattern surface 2 facing upward (for example, a site surface in an automobile production factory or an automobile maintenance factory).

  Next, the pattern surface 2 of the camera posture calculation target board 3 installed on the mounting surface of the vehicle 5 is photographed from above by the camera 6 attached to the vehicle 5.

  Next, a feature point (for example, a center point of a circular pattern image) of a pattern (hereinafter referred to as a pattern image) in the captured image is extracted from the captured image of the pattern surface 2.

  Then, using the extracted feature points, the internal parameters of the camera 6 (focal length, distortion characteristics, image center, etc.) and the coordinate points of the known feature points in the three-dimensional world coordinate system, Calculate external parameters.

  As external parameters of the camera 6, a rotation matrix that defines the direction of the camera 6 in the three-dimensional world coordinate system and a translation vector that defines the position of the camera 6 in the three-dimensional world coordinate system are known. The external parameter indicates the attitude of the camera 6 with respect to the mounting surface of the vehicle.

  Then, after calculating the attitude of the camera 6 in this way, the mapping table is corrected based on the calculated attitude of the camera 6.

JP 2001-285681 A

  By the way, the camera posture calculation target board 3 as described above is located when the camera 6 is located directly above the camera posture calculation target board 3, that is, between the optical axis of the camera 6 and the pattern surface 2. When the formed angle is 90 °, a pattern image suitable for generating a good vehicle periphery monitoring image can be taken.

  However, as shown in FIG. 22, when the angle α [deg] between the optical axis OA of the camera 6 and the pattern surface 2 is other than 90 °, the captured image 7 of the pattern surface 2 by the camera 6 is as shown in FIG. As described above, the size of the pattern image 1 ′ is not uniform.

  Such a phenomenon is particularly noticeable when the camera 6 having a super wide angle lens such as a fisheye lens is used.

  In some cases, the feature point of the pattern image 1 ′ cannot be accurately extracted from the captured image 7.

  In particular, the pattern 1 arranged at a position far from the camera 6 on the pattern surface 2 is photographed as a smaller pattern image 1 ′ than the pattern 1 existing near the intersection between the optical axis OA of the camera 6 and the pattern surface 2 ( Image), and distortion with respect to the actual shape increases. It has been very difficult to accurately extract feature points from the pattern image 1 'having such a large distortion.

  As a result, there has conventionally been a problem that it is difficult to calculate the posture of the camera with high accuracy.

  Therefore, the present invention has been made in view of such problems, and can calculate the orientation of the camera with respect to the reference plane with high accuracy, and as a result, a good display image based on the captured image of the imaging target surface. It is an object of the present invention to provide a camera posture calculation target device capable of obtaining the above, a camera posture calculation method using the same, and an image display method.

In order to achieve the above-described object, a camera posture calculation target device according to the present invention is a camera attached at an attachment position above a reference plane so that the reference plane and the optical axis form an angle other than 90 °. A camera posture calculation target device used for calculating the posture of the reference device with respect to the reference surface, wherein the target device has a pattern surface on which a plurality of patterns are formed, and the target device main body is installed on the reference surface. The pattern surface is photographed by a camera, and a feature point is extracted from a pattern in the photographed image of the pattern surface so as to be used for calculation of the posture of the camera. A size of the plurality of patterns in the photographed image and a small interval between adjacent patterns in a predetermined direction. Kutomo one of which is formed can be obtained photographed image such that the uniform one, in installed state of the target device body onto the reference plane, so that the pattern surface is perpendicular to the optical axis of the camera The plurality of patterns are formed as a photographed image of the pattern surface in the installation state of the target device main body on the reference surface, and the size and predetermined direction of the plurality of patterns in the photographed image The shape of the plurality of patterns is uniform in size and / or period, so that a photographed image can be obtained in which at least one of the intervals between adjacent patterns is uniform. A plurality of positions in an image coordinate system, which is a pattern image, which is a two-dimensional coordinate system taken on the imaging surface of the camera. A set of virtual pattern images, which are pattern images that are the same as one pattern, is assumed within the field of view of the camera, and the coordinate points of the virtual pattern image in the image coordinate system include the pattern plane. a shape obtained by converting the coordinate point dimension coordinate system, yet, it is characterized in the shape der Rukoto in consideration of the influence of distortion due to the mounted lens to the camera.

  In the present application, the phrase that the pattern surface is orthogonal to the optical axis of the camera is not limited to the case where the pattern surface is strictly orthogonal to the optical axis of the camera, but includes the case where the pattern surface is approximately orthogonal (hereinafter referred to as approximately orthogonal). Shall be.

Furthermore, another camera posture calculation target device according to the present invention is such that the reference plane of the camera mounted at an attachment position above the reference plane so that the reference plane and the optical axis form an angle other than 90 °. A camera posture calculation target device used for calculation of the posture with respect to the pattern, the pattern device having a pattern surface on which a plurality of patterns are formed, and the pattern being formed by the camera in a state where the target device body is installed on the reference surface A surface is photographed, and a feature point is extracted from a pattern in the photographed image of the pattern surface, so that it is used for calculation of the posture of the camera. At least one of the size of a plurality of patterns and the interval between adjacent patterns in a predetermined direction is Is formed so as to be able to be obtained photographed image such that one, in installed state of the target device body onto the reference plane, the pattern surface is formed so as to be parallel to the reference plane, the A pattern in which a plurality of patterns are adjacent in the size and a predetermined direction of the plurality of patterns in the photographed image as a photographed image of the pattern surface in the installation state of the target device main body on the reference surface at least one of the spacing between each other is formed in a shape capable of obtaining a captured image such that the uniform one, the shape of the plurality of patterns, size and / or period was in a uniform pattern image The position in the image coordinate system, which is a two-dimensional coordinate system taken on the imaging surface of the camera, is the plurality of patterns in the captured image. A set of virtual pattern images that are the same pattern images is assumed within the field of view of the camera, and the coordinate points of the virtual pattern image in the image coordinate system are represented in a three-dimensional coordinate system including the pattern surface. a shape obtained by converting the coordinate point, yet, it is characterized in the shape der Rukoto in consideration of the influence of distortion due to the mounted lens to the camera.

  Furthermore, another camera posture calculation target device according to the present invention is characterized by comprising a plate-like body on which the pattern surface is formed.

  Furthermore, in another camera posture calculation target device according to the present invention, the pattern surface is arranged so that the pattern surface is orthogonal to the optical axis of the camera in the installation state of the target device body on the reference surface. It has the holding member hold | maintained in the state inclined with respect to the reference plane, It is characterized by the above-mentioned.

  Further, another camera posture calculation target device according to the present invention is characterized in that the camera is a camera provided with an ultra-wide angle lens.

  Furthermore, in another camera posture calculation target device according to the present invention, the camera is mounted at the predetermined mounting position in the vehicle, and the reference plane is a mounting surface of the vehicle. Yes.

Furthermore, the camera orientation calculation method according to the present invention provides an orientation relative to the reference plane of a camera attached at an attachment position above the reference plane so that the reference plane and the optical axis form an angle other than 90 °. A camera posture calculation method for calculating a camera posture calculation target device having a pattern surface on which a plurality of patterns are formed, and the camera in an installation state of the camera posture calculation target device on the reference surface as a photographic image of the pattern surface by at least one of the spacing between adjacent coordinated between patterns in size and a predetermined direction for said plurality of patterns in the captured image, the captured image such that the uniform one A camera posture calculation target device formed so as to be obtainable is installed on the reference plane, and the installed camera The pattern surface in the target device for force calculation is photographed by the camera, the feature point is extracted from the pattern in the photographed image of the pattern surface obtained by the photographing, and the posture of the camera is based on the extracted feature point When the camera posture calculation target device is installed on the reference plane, the pattern surface is formed so as to be orthogonal to the optical axis of the camera. The plurality of patterns are taken as a photographed image of the pattern surface in the installation state of the camera posture calculation target device on the reference surface, and the size of the plurality of patterns in the photographed image and a predetermined value Captured image in which at least one of the intervals between adjacent patterns in the direction is uniform The camera posture is calculated using a camera posture calculation target device formed in a shape capable of obtaining the shape, and the shape of the plurality of patterns is a pattern image having a uniform size and / or period. A set of virtual pattern images that are pattern images such that the positions in the image coordinate system, which is a two-dimensional coordinate system taken on the imaging surface of the camera, are the same as the plurality of patterns in the captured image, Assuming within the field of view of the camera, the coordinate point of the virtual pattern image in the image coordinate system is a shape obtained by converting the coordinate point of the two-dimensional coordinate system including the pattern surface, and the It is characterized in shape der Rukoto in consideration of the influence of distortion due to the mounted lens in the camera.

Further, according to another camera orientation calculation method of the present invention, an orientation of a camera attached to a reference position above the reference plane so that the reference plane and the optical axis form an angle other than 90 ° with respect to the reference plane. Is a camera posture calculation target device having a pattern surface on which a plurality of patterns are formed, and the camera posture calculation target device on the reference surface in the installed state As a photographed image of the pattern surface by the camera, a photographed image in which at least one of the size of the plurality of patterns in the photographed image and the interval between adjacent patterns in a predetermined direction is uniform. A camera posture calculation target device formed so as to be obtainable is installed on the reference plane, and the installed camera The pattern surface in the target device for force calculation is photographed by the camera, the feature point is extracted from the pattern in the photographed image of the pattern surface obtained by the photographing, and the posture of the camera is based on the extracted feature point When the camera posture calculation target device is installed on the reference surface as the camera posture calculation target device, the pattern surface is parallel to the reference surface. The plurality of patterns formed as the captured image of the pattern surface in the installation state of the target device for camera posture calculation on the reference surface is the size and the predetermined number of the plurality of patterns in the captured image. at least one of uniform primary and made such photographing field spacing between adjacent coordinated between patterns in the direction of A camera posture calculation target device formed in a shape capable of obtaining an image is used to calculate the camera posture, and the shape of the plurality of patterns is a pattern image having a uniform size and / or period. A set of virtual pattern images that are pattern images such that positions in an image coordinate system, which is a two-dimensional coordinate system taken on the imaging surface of the camera, are the same as the plurality of patterns in the captured image. Suppose within the field of view of the camera, and the shape obtained by converting the coordinate point of the virtual pattern image in the image coordinate system to the coordinate point of a three-dimensional coordinate system including the pattern surface, and It is characterized in shape der Rukoto in consideration of the influence of distortion due to the mounted lens to the camera.

  Furthermore, in another camera posture calculation method according to the present invention, a camera posture calculation target device including a plate-like body on which the pattern surface is formed is used as the camera posture calculation target device. It is characterized by calculating.

  Furthermore, in another camera posture calculation method according to the present invention, the pattern surface is an optical axis of the camera in the installation state of the camera posture calculation target device on the reference plane as the camera posture calculation target device. The camera posture is calculated using a camera posture calculation target device having a holding member that holds the pattern surface in a state inclined with respect to the reference surface so as to be orthogonal to the reference plane.

  Another camera posture calculation method according to the present invention is characterized in that a camera having an ultra-wide angle lens is used as the camera.

  Furthermore, another camera posture calculation method according to the present invention provides a camera posture calculation target device for calculating a posture of a camera attached to the predetermined attachment position in a vehicle with respect to a placement surface of the vehicle. It is characterized by using and calculating.

Furthermore, the image display method according to the present invention shoots an imaging target surface by a camera mounted at a predetermined mounting position on the vehicle so that the mounting surface of the vehicle and the optical axis form an angle other than 90 °. An image for generating a vehicle periphery monitoring image for monitoring the periphery of the vehicle by performing coordinate conversion on the captured image of the surface to be imaged, and displaying the generated vehicle periphery monitoring image on a display unit A display method, a camera posture calculation target device having a pattern surface on which a plurality of patterns are formed, and the camera in the installation state of the camera posture calculation target device on a placement surface of the vehicle As the captured image of the pattern surface, the size of the plurality of patterns in the captured image and adjacent patterns in a predetermined direction The camera pose at least one of the spacing between it, the camera orientation calculation target device to obtain a photographing image is formed so as to be able to be a uniform one was placed on a mounting surface of the vehicle, being this installed The pattern surface in the calculation target device is photographed by the camera, a feature point is extracted from a pattern in a photographed image of the pattern surface obtained by the photographing, and the posture of the camera is based on the extracted feature point If a deviation occurs between the calculated attitude of the camera and an appropriate attitude, the deviation relative to the captured image of the imaging target surface is generated when the vehicle periphery monitoring image is generated. there have row coordinate transformation as is eliminated, the shape of the plurality of patterns, a uniform pattern image size and / or period, taken on an imaging surface of the camera A set of virtual pattern images that are pattern images such that the positions in the image coordinate system, which is a two-dimensional coordinate system, are the same as the plurality of patterns in the captured image, are assumed within the field of view of the camera; A shape obtained by converting the coordinate point of the virtual pattern image in the image coordinate system into a coordinate point of a two-dimensional coordinate system or a three-dimensional coordinate system including the pattern surface, and mounted on the camera It is characterized by a shape that takes into account the influence of distortion by the lens .

According to the camera orientation calculation target device according to the present invention, the pattern image size and the feature points of pattern image by at least one evenly one spacing between adjacent coordinated pattern image together in a predetermined direction Since it can be extracted accurately, the posture of the camera with respect to the reference plane can be calculated with high accuracy, and as a result, a good display image can be obtained based on the captured image of the imaging target surface.

Further, according to another camera posture calculation target device according to the present invention, the shape of the pattern is adjacent in the size and the predetermined direction of the pattern image when photographing the pattern surface orthogonal to the optical axis of the camera. by at least one of the spacing between the pattern image is formed in a shape such that uniform one, since at least one of the spacing between the size and the pattern image between pattern image can be evenly one With a simple configuration, the posture of the camera with respect to the reference plane can be calculated with high accuracy.

Further, according to another camera posture calculation target device according to the present invention, the pattern shape is adjacent in the size of the pattern image and a predetermined direction when a pattern surface parallel to the reference surface is photographed. by at least one of the spacing between the pattern image is formed in a shape such that uniform one, since at least one of the spacing between the size and the pattern image between pattern image can be evenly one With a simple configuration, the posture of the camera with respect to the reference plane can be calculated with high accuracy.

Furthermore, according to another camera posture calculation target device according to the present invention, the plate-like camera posture calculation target device allows the size of the pattern image and the interval between adjacent pattern images in a predetermined direction. since at least one of can be evenly and foremost, it is possible to further simplify the configuration.

  Furthermore, according to another camera posture calculation target device according to the present invention, the size of the pattern image and the predetermined size can be determined by holding the pattern surface in an inclined state perpendicular to the optical axis of the camera by the holding member. Since at least one of the distances between the adjacent pattern images in the direction can be made uniform or substantially uniform, the camera posture with respect to the reference plane can be calculated with high accuracy with a simpler configuration.

  Further, according to another camera posture calculation target device according to the present invention, the size of a pattern image in a captured image of a pattern surface by a camera equipped with an ultra-wide-angle lens and a pattern image adjacent to each other in a predetermined direction. Since at least one of the intervals can be made uniform or substantially uniform, the attitude of the camera having the ultra-wide angle lens with respect to the reference surface can be calculated with high accuracy. A good display image can be obtained.

  Furthermore, according to another camera posture calculation target device according to the present invention, the size of the pattern image in the captured image of the pattern surface by the camera attached to the vehicle and the pattern images adjacent to each other in a predetermined direction. Since at least one of the intervals can be made uniform or substantially uniform, the posture of the camera with respect to the mounting surface of the vehicle can be calculated with high accuracy, and as a result, a good vehicle periphery based on the captured image of the imaging target surface A monitoring image can be obtained.

Furthermore, according to the camera orientation calculation method according to the present invention, feature points of pattern image by at least one evenly one spacing between the magnitude and next coordinated pattern images each other in the predetermined direction of the pattern image Can be accurately extracted, so that the posture of the camera with respect to the reference plane can be calculated with high accuracy. As a result, a good display image can be obtained based on the captured image of the imaging target surface.

Further, according to another camera orientation calculation method according to the present invention, the pattern shape is the size of the pattern image when the pattern surface orthogonal to the optical axis of the camera is photographed and the pattern image adjacent in the predetermined direction. by at least one of the spacing between each other to form a shape such that uniform one, since at least one of the spacing between the size and the pattern image between pattern image can be evenly one, simple With this method, the camera posture with respect to the reference plane can be calculated with high accuracy.

Furthermore, according to another camera posture calculation method according to the present invention, the pattern shape is the size of a pattern image when a pattern plane parallel to the reference plane is photographed and the pattern image adjacent in a predetermined direction. by at least one of the spacing between is shaped to have a uniform size, since at least one of the spacing between the size and the pattern image between pattern image can be evenly one By a simple method, the posture of the camera with respect to the reference plane can be calculated with high accuracy.

Furthermore, according to another camera orientation calculation method according to the present invention, at least a size of a pattern image and an interval between adjacent pattern images in a predetermined direction are detected by a plate-like camera orientation calculation target device. one of them can be evenly one, thereby, since it is possible to accurately extract feature points of pattern image, the configuration of the camera posture calculation target device can be simplified.

  Furthermore, according to another camera posture calculation method according to the present invention, the camera posture calculation target device that can hold the pattern surface in a tilted state orthogonal to the optical axis of the camera by the holding member is used. As a result, at least one of the size of the pattern image and the interval between adjacent pattern images in a predetermined direction can be made uniform or substantially uniform, and the feature points of the pattern image can be accurately extracted. In addition, the posture of the camera with respect to the reference plane can be calculated with high accuracy by a simpler method.

  In addition, according to another camera orientation calculation method according to the present invention, the size of the pattern image in the captured image of the pattern surface by the camera equipped with the super-wide-angle lens and the interval between adjacent pattern images in a predetermined direction At least one of them can be made uniform or substantially uniform, which makes it possible to accurately extract feature points of the pattern image, so that the attitude of the camera equipped with the ultra-wide-angle lens with respect to the reference plane can be calculated with high accuracy. As a result, a good display image can be obtained based on the photographed image of the photographing target surface.

  Furthermore, according to another camera orientation calculation method according to the present invention, the size of the pattern image in the captured image of the pattern surface by the camera attached to the vehicle and the interval between adjacent pattern images in a predetermined direction are determined. At least one of them can be made uniform or substantially uniform, which makes it possible to accurately extract feature points of the pattern image, so that the posture of the camera with respect to the reference plane can be calculated with high accuracy, and in turn, shooting A good vehicle periphery monitoring image can be obtained based on the captured image of the target surface.

  Furthermore, according to the image display method of the present invention, by making at least one of the size of the pattern image and the interval between the adjacent pattern images in a predetermined direction uniform or substantially uniform, It is possible to calculate the camera posture with respect to the reference plane after accurately extracting the feature points, and further generate a vehicle periphery monitoring image after correcting the mapping table based on the calculation result of the camera posture Therefore, the posture of the camera with respect to the reference plane can be calculated with high accuracy, and a good vehicle periphery monitoring image can be obtained based on the captured image of the imaging target surface.

(First embodiment of camera posture calculation target device and camera posture calculation method)
Hereinafter, a first embodiment of a camera posture calculation target device and a camera posture calculation method according to the present invention will be described with reference to FIGS. 1 to 9.

  Note that portions having the same or similar basic configuration as those in the related art will be described using the same reference numerals.

  FIG. 1 shows a camera posture calculation target device 10 according to this embodiment. The camera posture calculation target device 10 in the present embodiment is attached to a predetermined attachment position in the vehicle 5 in order to display the vehicle periphery monitoring image on the display in the vehicle, similarly to the camera posture calculation target board 3 described above. Before photographing a photographing target surface such as a road surface by the camera 6, the camera 6 is used for calculating the posture of the camera 6 with respect to the mounting surface of the vehicle 5 as a reference surface. The calculation of the posture of the camera 6 may be performed immediately after the camera 6 is attached to the vehicle 5 as in the prior art. Also, as the mounting surface of the vehicle 5 for calculating the attitude, various places such as the site surface in the automobile production factory, the site surface in the automobile maintenance factory, or the imaging target surface itself can be selected as in the past. it can.

  However, as will be described below, the camera posture calculation target device 10 according to the present embodiment is different from the conventional camera posture calculation target board 3 in that the posture of the camera 6 with respect to the mounting surface of the vehicle 5 is highly calculated. It is suitable for accuracy.

  That is, the camera posture calculation target device 10 in this embodiment has a pentahedral block 11 as a holding member, as shown in FIG. 1, and this block 11 is vertically above the bottom surface 11a. The outer surface of the inclined surface 11b is formed in a shape in which four sides of a rectangle are distorted inward. The block 11 may be formed of, for example, a resin material or the like, and may be hollow or solid.

  On the inclined surface 11b of the block 11, as shown in FIG. 1, a pattern surface 12 in which a plurality of patterns 14 are aligned vertically and horizontally is formed.

  The pattern surface 12 may be one in which the pattern 14 is directly drawn on the inclined surface 11b of the block 11 by printing or the like, or a sheet whose outer shape on which the pattern 14 is drawn coincides with the inclined surface 11b, A plate-like body may be fixed on the inclined surface 11b by adhesion or the like.

  Further, as shown in FIG. 2, the inclination angle α [deg] of the inclined surface 11 b of the block 11, that is, the inclination angle of the pattern surface 12 with respect to the mounting surface of the vehicle 5 is the optical axis of the camera 6 with respect to the vertical direction. It is the same as the angle α formed by OA (however, this α may be an approximate value based on a design value). In addition, when generating a top view image in which the vehicle and its periphery are looked down from above the vehicle as the vehicle periphery monitoring image, it is necessary to mount a plurality of cameras 6 on the vehicle. Of course, the value of α may be different for each camera 6. However, in this case, if α is different, the inclination angle of the pattern surface 12 is also different, so that it is necessary to use different camera orientation calculation target devices 10 for each camera 6 or the block 11 has a pattern surface. An adjustment mechanism (not shown) for adjusting the inclination angle of 12 is provided, and it is necessary to photograph the pattern surface 12 after adjusting (changing) the inclination angle for each camera 6 by this adjustment mechanism.

  Therefore, in the present embodiment, the pattern surface 12 is orthogonal (substantially orthogonal) to the optical axis OA of the camera 6 in the installation (placement) state of the camera posture calculation target device 10 on the placement surface of the vehicle 5. Can be included).

  In the present embodiment, the shape of the pattern 14 is such that, as shown in FIG. 3, the captured image 15 of the pattern surface 12 in a state where the camera posture calculation target device 10 is installed on the mounting surface of the vehicle 5. A photographed image in which the size of the pattern image 14 ′ in the image 15 and the interval between the adjacent pattern images 14 ′ in a predetermined direction (vertical direction, horizontal direction and diagonal direction in FIG. 3) are uniform. 15 can be obtained.

  More specifically, the shape of each pattern 14 is formed in consideration of the influence of distortion due to the ultra-wide-angle lens mounted on the camera 6.

  That is, among the patterns 14, the pattern 14 positioned near the optical axis OA of the camera 6 in the installed state of the camera posture calculation target device 10 on the placement surface of the vehicle 5, or near the center of the pattern surface 12. Since the arranged pattern 14 is less affected by distortion due to the super-wide-angle lens, the outer shape is formed in a perfect circle or a circle close thereto.

  On the other hand, among each pattern 14, the pattern 14 positioned far from the camera 6 in the installed state of the camera posture calculation target device 10 on the placement surface of the vehicle 5, and the pattern 14 is arranged on the peripheral side of the pattern surface 12. Since the pattern 14 is greatly affected by distortion, the outer shape of the pattern 14 is an ellipse or an ellipse extending radially from the center side to the peripheral side of the pattern surface 12.

  According to the camera posture calculation target device 10 in this embodiment, the feature points of each pattern image 14 'can be accurately extracted.

  The pattern surface 12 in such a camera posture calculation target device 10 can be designed as follows.

  That is, first, it is assumed that the internal parameters (cx, cy, f) of the camera 6 and the external parameters (rx, ry, rz, tex, ty, tz) indicating the posture of the camera 6 are known. However, the external parameter does not necessarily need to be known with a strict parameter value, and may be a rough one in vehicle design.

  Note that cx in the internal parameter is a center point (in other words, the camera 6) of an image taken by the camera 6 in a two-dimensional coordinate system (image coordinate system) taken on the imaging surface (image sensor surface, for example, CCD surface) of the camera 6. X-axis of the optical axis OA and the imaging surface). Similarly, cy in the internal parameter indicates the Y coordinate of the center point of the image captured by the camera 6 in the image coordinate system. Further, f in the internal parameter indicates the focal length of the camera 6.

  On the other hand, tx in the external parameter indicates the X coordinate of the position of the camera 6 in the three-dimensional world coordinate system, ty indicates the Y coordinate, and tz indicates the Z coordinate. In addition, rx in the external parameter is a direction around the X coordinate axis (direction of the optical axis) of the camera 6 in the three-dimensional world coordinate system, ry is a direction around the Y coordinate axis, and rz is a direction around the Z coordinate axis. Show.

  Next, as shown in FIG. 4, the pattern image has a uniform size suitable for feature point extraction (the period is uniform in FIG. 4), and the position in the image coordinate system is the same as the pattern image 14 ′. A set of pattern images (hereinafter referred to as a virtual pattern image 14 ″) is assumed within the field of view of the camera 6. The virtual pattern image 14 ″ is taken when the pattern 14 to be designed is photographed. This shows an ideal state of the pattern image 14 ′.

  Then, coordinate conversion shown in the following equations (1) to (14) is performed on the virtual pattern image 14 ″.

  Here, FIG. 5 shows one of the virtual pattern images 14 ″ together with the coordinate point (i, j) in the image coordinate system. The coordinate points of the virtual pattern image 14 ″ are The center point of the outer shape of the virtual pattern image 14 ″.

  As shown in FIG. 5, the intersection (cx, cy) between the optical axis OA of the camera 6 and the imaging surface represented by the internal parameters is different from the origin of the image coordinate system.

  The X coordinate difference ii and the Y coordinate difference jj between the coordinate point (cx, cy) and the coordinate point (i, j) of the virtual pattern image 14 ″ are obtained by the following equations (1) and (2), respectively. It is done.

ii = i-cx (1)
jj = j-cy (2)

  Next, as shown in FIG. 5, when a two-dimensional polar coordinate having a coordinate point (cx, cy) as an origin and a coordinate point defined by a radius rad and an angle φ [deg] is considered, the coordinate point (i, j) The angle φ is obtained as follows.

φ = tan ⁻¹ (| jj | / | ii |) (3)

  Further, the distance rad between the coordinate point (cx, cy) and the coordinate point (i, j) is obtained as in the following equation (4).

rad = (ii ² + jj ² ) ^1/2 (4)

  In this manner, the coordinate point (i, j) of the virtual pattern image 14 ″ in the image coordinate system is coordinate-converted to the coordinate point (rad, φ) in the two-dimensional polar coordinate system.

  Next, as shown in FIG. 6, there are U, V, and S coordinate axes orthogonal to each other, the S coordinate axis is the optical axis OA of the camera 6, and the tertiary is the intersection of the optical axis OA and the imaging surface as the origin. Taking the camera coordinate system which is the original Cartesian coordinate system, the left side of the equation (4) is expressed as the following equation (5).

  rad = f × θ (5)

  However, f in Formula (5) is the focal length (internal parameter) of the camera 6 mentioned above. Further, θ [deg] in the equation (5) is based on the optical axis OA (S coordinate axis) indicating the direction in which the pattern 14 corresponding to the virtual pattern image 14 ″ shown in FIG. 5 exists, as shown in FIG. However, the pattern 14 shown in Fig. 6 is designed from now on.

  The expression (5) is an expression in the case of an equidistant projection type fisheye lens, and the expression is different for a wide-angle lens and other projection type lenses, but the description is omitted.

  Next, from the equation (5), the angle θ is obtained as the following equation (6).

  θ = rad / f (6)

  Next, as shown in FIG. 6, when a sphere 16 having a unit radius centered on the origin of the camera coordinate system is taken on the camera coordinate system, the sphere 16 and the origin of the camera coordinate system are moved in the direction θ of the pattern 14. The coordinate point (u, v, s) of the intersection where the line segment L extending toward the intersection intersects is obtained as the determinant shown in the following (7).

  The coordinate point (u, v, s) represented by the equation (7) is regarded as a coordinate point in the camera coordinate system of the virtual pattern image 14 ″ (specifically, the center point of the outer shape of the virtual pattern image 14 ″). Can do.

  In this manner, the coordinate point (rad, φ) of the virtual pattern image 14 ″ in the two-dimensional polar coordinate system is coordinate-converted to the coordinate point (u, v, s) in the camera coordinate system.

  Next, as shown in FIG. 7, a three-dimensional world coordinate system having X, Y, and Z coordinate axes orthogonal to each other and having the XY plane as a mounting surface of the vehicle 5 is taken.

  The general formula for indicating the coordinate point (x, y, z) of the pattern 14 defined in the three-dimensional world coordinate system is three matrices using direction elements in the external parameters of the camera 6 (7 By using a matrix obtained by multiplying the coordinate point of the formula by a parameter λ indicating the distance from the intersection of the optical axis OA and the imaging surface to the pattern 14 and a matrix of position elements in the external parameters of the camera 6, (8) in the determinant.

  When the calculation is further advanced, the equation (8) becomes the determinant of the following (9).

  Here, as shown in FIG. 8, assuming that the pattern surface 17 is positioned on the XY plane in the three-dimensional world coordinate system, the Z coordinate of the pattern 18 on the pattern surface 17 is zero. Also, the value of λ can be obtained as the following equation (10) by substituting z = 0 into the equation (9).

λ = tz / {sin (ry) × u−cos (ry) × sin (rx) × v−cos (ry) × cos (rx) × s}
... (10)

  Then, the X coordinate xx ′ and the Y coordinate yy ′ of the pattern 18 on the pattern surface 17 are obtained by substituting the value of the equation (10) into the equation (9). However, it is assumed that one corner of the planar rectangular pattern surface 17 is located at the origin (0, 0, 0) of the three-dimensional world coordinate system.

  However, as described above, in the present embodiment, it is orthogonal (including substantially orthogonal) to the optical axis OA of the camera 6 and has an inclination (angle α [deg]) with respect to the mounting surface of the vehicle 5. Such a pattern surface 12 is designed.

  Therefore, as shown in FIG. 9, a pattern surface 12 having an inclination α in the Z-axis direction with respect to the XY plane in the three-dimensional world coordinate system is set. However, it is assumed that one corner of the pattern surface 12 is located at the origin (0, 0, 0) of the three-dimensional world coordinate system.

  Then, as a two-dimensional coordinate system including the pattern surface 12, as shown in FIG. 9, an X′Y coordinate composed of an X ′ coordinate axis extending in the direction of one side of the pattern surface 12 and a Y coordinate axis in the three-dimensional world coordinate system. Take a system.

  And the coordinate point (xx, yy) of the pattern 14 prescribed | regulated on this X'Y coordinate system is calculated | required by the following geometric calculations.

  Here, FIGS. 10A and 10B are shown for easy understanding of the calculation of the coordinate point (xx, yy) of the pattern 14, and FIG. 10A shows the three-dimensional world coordinate system. FIG. 10B shows a diagram viewed from the normal direction of the XZ plane, and FIG. 10B shows a diagram of the three-dimensional world coordinate system viewed from the normal direction of the YZ plane.

  As shown in FIG. 10 (a), the following relational expression (11) is established between xx and xx ′, and the following (12) is established between yy and yy ′. ) Is established. Note that β [deg] in the equation (11) is shown in FIG. 10A, and γ [deg] in the equation (12) is shown in FIG. 10B.

(Xx′−xx × cos (α)) / xx × sin (α)
= (Xx'-tx) / tz
= Tan β (11)

yy + xx × sin (α) × (yy′−ty) / tz
= Yy + xx × sin (α) × tan γ
= Yy + (yy'-yy)
= Yy '(12)

  Furthermore, xx is obtained from the following equation (13) from the equation (11), and yy is obtained from the following equation (14) from the equation (12).

xx = xx ′ × tz / {(xx′−tx) × sin (α) + tz × cos (α)}
... (13)

  yy = yy ′ − {xx × sin (α) × (yy′−ty)} / tz (14)

  In this way, after coordinate-transforming the coordinate point (u, v, s) of the virtual pattern image 14 ″ in the camera coordinate system to the coordinate point (xx ′, yy ′) in the three-dimensional world coordinate system, X′Y Coordinate conversion can be performed to a coordinate point (xx, yy) in the coordinate system.

  Thereby, since the coordinate point (xx, yy) of the pattern 14 can be defined, the pattern surface 12 in the camera posture calculation target device 10 of the present embodiment can be designed. In addition, about the coordinate of the height direction from the mounting surface of the vehicle 5 of the pattern 14 (Z coordinate in the three-dimensional world coordinate system), it is a pattern with respect to the coordinate point (xx, yy) and the mounting surface of the vehicle 5. It can be uniquely determined from the inclination α of the surface 12.

  In addition, for a specific distortion shape of the pattern 14 such as an ellipse or an ellipse, the distortion characteristics of the camera 6 corresponding to each coordinate point (xx, yy) are grasped in advance based on known internal parameters, What is necessary is just to design so that the shape back-calculated from the distortion characteristic may be assigned.

  Next, a camera posture calculation method using such a camera posture calculation target device 10 will be described.

  In this calculation method, the known parameters are the internal parameters (cx, cy, f) of the camera 6 and the coordinate points (xx, yy) of the pattern 14 corresponding to the pattern image 14 ′. However, in order that the coordinate point of the pattern 14 corresponding to the pattern image 14 ′ is known, the pattern image 14 ′ is used when designing the pattern 14 corresponding to the pattern image 14 ′ in the image coordinate system. It is necessary to take the same coordinate value as that of the virtual pattern image 14 ″, and for that purpose, the position of the pattern 14 must be matched with the design position (xx, yy).

  In the present embodiment, first, as shown in FIG. 2, the camera posture calculation target device 10 is installed on the placement surface of the vehicle 5. At this time, the installation position of the camera posture calculation target device 10 is adjusted so that the position of the pattern 14 in the camera posture calculation target device 10 matches the design position (xx, yy). The adjustment of the installation position at this time is assumed in advance, for example, such that the distance of the camera posture calculation target device 10 from the end of the vehicle 5 can match the position of the pattern 14 to the design position. It can be performed by setting the specified separation distance.

  Next, the pattern surface 12 of the camera posture calculation target device 10 is photographed by the camera 6.

  At this time, each pattern 14 on the pattern surface 12 is formed in a shape that takes into account the influence of distortion due to the super-wide-angle lens, so that the pattern image 14 'as shown in FIG. The captured image 15 of the pattern surface 12 with a uniform interval between adjacent pattern images 14 ′ can be acquired.

  Next, feature points of the pattern image 14 ′ in the captured image 15 are extracted by image recognition. Note that the feature point of the circular pattern image 14 ′ in this embodiment is a circular center point.

  At this time, since the pattern image 14 ′ has a uniform size, feature points can be accurately extracted.

  Next, the coordinates of the extracted feature points are regarded as the coordinates (i, j) of the center point of the virtual pattern image 14 ″, and the calculations of the expressions (1) to (7) are sequentially performed. Specifically, (1 Substituting the X coordinate of the extracted feature point into i in the formula (1), and substituting the Y coordinate of the extracted feature point into j in the formula (2), and performing calculations up to the formula (7). .

  In this calculation, known internal parameters (cx, cy, f) of the camera 6 are used.

  The coordinate point (u, v, s) of equation (7) obtained by such calculation corresponds to the coordinate value in the camera coordinate system of the feature point of the extracted pattern image 14 '.

  At this time, by using the equations (13) and (14), the coordinate point (xx, yy) of the pattern 14 corresponding to the feature point of the extracted pattern image 14 ′ is obtained along the optical axis OA. A coordinate point (xx ′, yy ′, 0) projected on the XY plane in the original world coordinate system is obtained.

Next, the coordinate point (u, v, s) obtained using the equation (7) and the coordinate point (xx ′, yy ′, 0) obtained using the equations (13) and (14) are expressed as ( 8) Assign to the left side of the equation.

  By performing such calculation for at least three pattern images 14 ′, a plurality of simultaneous equations are established, and by solving these simultaneous equations, the posture of the camera 6 with respect to the mounting surface of the vehicle 5 is determined. External parameters (rx, ry, rz, tex, ty, tz) can be calculated. In order to reduce the influence of measurement errors, it is preferable to extract as many feature points as possible scattered over a wide range and to calculate external parameters with high accuracy using a least square method or the like. In addition, since λ can be deleted in the process of calculating the external parameter, it is not necessary to consider a specific value of λ.

  As described above, according to the present embodiment, when calculating the posture of the camera 6 with respect to the mounting surface of the vehicle 5, the pattern image 12 having a uniform size is obtained by photographing the pattern surface 12 with the camera 6. Since the captured image 15 in which “is taken” can be obtained, the feature points of the pattern image 14 ′ can be accurately extracted, and the posture of the camera 6 with respect to the mounting surface of the vehicle 5 can be calculated with high accuracy. it can.

(Second Embodiment of Target Device for Camera Posture Calculation and Camera Posture Calculation Method)
Next, a second embodiment of the camera posture calculation target device and camera posture calculation method according to the present invention will be described with reference to FIGS.

  In the present embodiment as well, portions having the same or similar basic configuration as the conventional one will be described using the same reference numerals.

  FIG. 11 shows a camera posture calculation target board 20 as a camera posture calculation target device in the present embodiment. The camera posture calculation target board 20 is similar to the first embodiment in the vicinity of the vehicle. Before photographing a photographing target surface such as a road surface with a camera 6 attached at a predetermined attachment position in the vehicle 5 in order to display a monitoring image on a display inside the vehicle, the camera 6 with respect to the placement surface of the vehicle 5 as a reference surface. It is used to calculate the posture of

  Further, the camera orientation calculation target board 20 in this embodiment is also suitable for calculating the orientation of the camera 6 with respect to the mounting surface of the vehicle 5 with high accuracy, as in the first embodiment. .

  However, the camera orientation calculation target board 20 in the present embodiment can calculate the orientation of the camera 6 with high accuracy without giving the pattern surface 12 an inclination as in the first embodiment.

  That is, as shown in FIG. 11, the camera posture calculation target board 20 in the present embodiment is a pattern in which a plurality of patterns 22 are drawn on one surface of a flat substrate 21 made of a resin material or the like by printing or the like. It has a surface 23. The pattern surface 23 is arranged in parallel to the placement surface of the vehicle 5 in the installation state of the camera posture calculation target board 20 on the placement surface of the vehicle 5.

  Also in the present embodiment, the pattern 22 has a shape as a captured image 24 of the pattern surface 23 in a state in which the camera posture calculation target board 20 is installed on the mounting surface of the vehicle 5, as shown in FIG. Photographing in which the size of the pattern image 22 ′ in the photographed image 24 and the interval between adjacent pattern images 22 ′ in a predetermined direction (vertical direction, horizontal direction, and diagonal direction in FIG. 12) are uniform. The shape is such that the image 24 can be obtained.

  More specifically, the shape of each pattern 22 is formed in consideration of the influence of distortion due to the ultra-wide-angle lens mounted on the camera 6.

  That is, among the patterns 22, the pattern 22 positioned near the camera 6 in the installed state of the camera posture calculation target board 20 on the placement surface of the vehicle 5, and the pattern surface 23 are arranged near the center. Since the pattern 22 is less affected by distortion due to the ultra-wide-angle lens, the outer shape of the pattern 22 is a perfect circle or a circle close thereto.

  On the other hand, among the patterns 22, the pattern 22 positioned far from the camera 6 in the installed state of the camera posture calculation target board 20 on the mounting surface of the vehicle 5 and the pattern 22 are arranged on the peripheral side of the pattern surface 23. Since the pattern 22 is greatly affected by distortion, the outer shape of the pattern 22 is an ellipse or an ellipse extending radially from the center side to the peripheral side of the pattern surface 23.

  As a result, as shown in FIG. 12, the size of the pattern image 22 'can be made uniform, and the feature points of each pattern image 22' can be extracted accurately. In FIG. 12, the vertical and horizontal periods of the pattern image 22 'are also uniform.

  As shown in FIG. 13, such a pattern surface 23 assumes a virtual pattern image 22 ″ within the field of view of the camera 6 so that the position in the image coordinate system is the same as the pattern image 22 ′. ”Can be designed by performing calculations of equations (1) to (10) in the first embodiment.

  At this time, the pattern surface 17 on the same plane as the placement surface of the vehicle 5 shown in FIG. 8 is considered to be replaced with the pattern surface 23 designed in the present embodiment, and the coordinate points (xx ′) of the pattern 18 in FIG. , Yy ', 0) may be designed as the coordinate point (xx', yy ', 0) of the pattern 22 in this embodiment.

  At this time, for a specific distortion shape of the pattern 22 such as an ellipse or an ellipse, the distortion characteristics of the camera 6 corresponding to each coordinate point (xx ′, yy ′, 0) are based on known internal parameters. What is necessary is just to design in advance so as to assign a shape calculated backward from the distortion characteristics.

  Next, a camera posture calculation method using such a camera posture calculation target board 20 will be described.

  In this calculation method, the known parameters are the internal parameters (cx, cy, f) of the camera 6 and the coordinate points (xx ′, yy ′, 0) of the pattern 22 corresponding to the pattern image 22 ′. . However, as in the first embodiment, in order to know the coordinate points of the pattern 22 corresponding to the pattern image 22 ′, the pattern image 22 ′ corresponds to the pattern image 22 ′ in the image coordinate system. It is necessary to take the same coordinate value as that of the virtual pattern image 22 ″ used when the pattern 22 is designed. For this purpose, the position of the pattern 22 is adjusted to the design position (xx ′, yy ′, 0). It is a condition.

  In this embodiment, first, as shown in FIG. 14, a camera posture calculation target board 20 is installed on the mounting surface of the vehicle 5. At this time, the installation position of the camera posture calculation target board 20 is adjusted so that the position of the pattern 22 on the camera posture calculation target board 20 matches the design position (xx ′, yy ′, 0).

  Next, the pattern plane 23 of the camera posture calculation target board 20 is photographed by the camera 6.

  At this time, each pattern 22 on the pattern surface 23 is formed in a shape that takes into account the influence of distortion due to the super-wide-angle lens, so that the pattern surface 22 ′ as shown in FIG. 12 has a uniform size. 23 captured images 24 can be acquired.

  Next, feature points of the pattern image 22 'in the captured image 24 are extracted by image recognition. Note that the feature point of the circular pattern image 22 ′ in the present embodiment is a circular center point as in the first embodiment.

  At this time, since each pattern image 22 'has a uniform size, the feature points can be extracted accurately.

  Next, the coordinates of the extracted feature points are regarded as the coordinates (i, j) in the equation (1), and the coordinates (i, j) of the feature points and the known internal parameters (cx, cy, f) are Using these, the calculations of formulas (1) to (7) are sequentially performed.

  The coordinate point (u, v, s) of Equation (7) obtained by this calculation corresponds to the coordinate value in the camera coordinate system of the feature point of the extracted pattern image 22 '.

  Then, the coordinate point (u, v, s) obtained using the expression (7), the known coordinate point (xx ′, yy ′, 0) of the pattern 22 corresponding to the extracted pattern image 22 ′, and Is substituted into equation (8).

  By performing such a calculation for at least three pattern images 22 ′, a plurality of simultaneous equations are established, and by solving these simultaneous equations, the posture of the camera 6 with respect to the mounting surface of the vehicle 5 is determined. External parameters (rx, ry, rz, tex, ty, tz) can be calculated.

  As described above, also in the present embodiment, by capturing the pattern surface 23 with the camera 6, it is possible to obtain the captured image 24 in which the pattern image 22 ′ having a uniform size is captured, and the pattern image 22 ′. Since the feature point can be accurately extracted, the posture of the camera 6 with respect to the placement surface of the vehicle 5 can be calculated with high accuracy.

( Reference form of camera orientation calculation target device and camera orientation calculation method)
Hereinafter, reference embodiments of the present invention will be described with reference to FIGS. 15 to 17.

  Note that portions having the same or similar basic configuration as those in the related art will be described using the same reference numerals.

Figure 15 is a diagram showing a camera posture calculation target device 50 of Embodiment. Camera posture calculation target device 50 of Embodiment, similarly to the camera posture calculation target board 3 described above, attached to a predetermined attachment position in the vehicle 5 in order to display the vehicle surroundings monitoring image on the car display Before photographing a photographing target surface such as a road surface by the camera 6, the camera 6 is used for calculating the posture of the camera 6 with respect to the mounting surface of the vehicle 5 as a reference surface.

The camera posture calculation target device 50 of Embodiment also, unlike a conventional camera posture calculation target board 3, suitable for performing the calculation of the orientation of the camera 6 for the mounting surface of the vehicle 5 with high precision It is supposed to be.

That is, the camera posture calculation target device 50 of Embodiment has a block 51 as the holding member, the block 51 is perpendicular triangular prism shape having an inclined surface 51b which is inclined vertically upward relative to the bottom surface 51a Is formed. The block 51 may be formed of, for example, a resin material or the like, and may be hollow or solid.

  On the inclined surface 51b of the block 51, as shown in FIG. 15, there is formed a pattern surface 52 in which a plurality of patterns 54 having a circular outer shape are arranged vertically and horizontally. And the actual intervals between adjacent patterns 54 in a predetermined direction (vertical direction, horizontal direction and diagonal direction in FIG. 15) are uniform.

  The pattern surface 52 may be one in which the pattern 54 is directly drawn on the inclined surface 51b of the block 51 by printing, or the sheet or plate on which the pattern 54 is drawn is inclined by adhesion or the like. It may be fixed on the surface 51b.

  Further, as shown in FIG. 16, the inclination angle α [deg] of the inclined surface 51 b of the block 51, that is, the inclination angle of the pattern surface 52 with respect to the mounting surface of the vehicle 5 is the optical axis of the camera 6 with respect to the vertical direction. It is the same as the angle α formed by OA (however, this α may be an approximate value based on a design value). In addition, when generating a top view image in which the vehicle and its periphery are looked down from above the vehicle as the vehicle periphery monitoring image, it is necessary to mount a plurality of cameras 6 on the vehicle. Of course, the value of α may be different for each camera 6. However, in this case, if α is different, the inclination angle of the pattern surface 52 is also different, so that it is necessary to use different camera orientation calculation target devices 50 for each camera 6 or the block 51 has the pattern surface An adjustment mechanism (not shown) for adjusting the inclination angle of 52 is provided, and it is necessary to photograph the pattern surface 52 after adjusting (changing) the inclination angle for each camera 6 by this adjustment mechanism.

Therefore, in this reference embodiment, in the installation (placement) status of the camera posture calculation target device 50 onto the mounting surface of the vehicle 5, it is perpendicular to the pattern surface 52 with respect to the optical axis OA of the camera 6 It can be done.

  As a result, as shown in FIG. 17, the size of the pattern image 54 ′ in the captured image 55 of the pattern surface 52 by the camera 6 is approximately approximated to such an extent that the feature points can be accurately extracted from the pattern image 54 ′. Can be made substantially uniform.

  As a result, the feature points of each pattern image 54 ′ can be accurately extracted, so that the posture of the camera 6 with respect to the mounting surface of the vehicle 5 can be calculated with higher accuracy than in the past.

(Embodiment of image display method)
Hereinafter, an embodiment of an image display method according to the present invention will be described with reference to FIGS.

  FIG. 18 shows an image display device 30 as an example of a device for realizing the image display method according to the present embodiment. The image display device 30 is attached to the vehicle 5 at different positions. The cameras 6a, 6b, 6c and 6d are provided. One of the four cameras 6 a, 6 b, 6 c, 6 d is a front camera 6 a attached to the front portion (for example, an emblem portion) of the vehicle 5, and the front camera 6 a It is supposed to shoot forward. One of the other three cameras 6b, 6c, 6d is a back camera 6b attached to the rear portion (for example, a rear license garnish portion) of the vehicle 5, and the back camera 6b is the vehicle 5 The back of the camera is to be taken. One of the other two cameras 6c and 6d is a right side camera 6c attached to the right side portion (for example, the right door mirror portion) of the vehicle 5, and the right side camera 6c is the vehicle 5 The right side of is to be taken. The remaining one is a left side camera 6d attached to the left side of the vehicle 5 (for example, the left door mirror), and the left side camera 6d captures the left side of the vehicle 5. Yes.

  A camera image acquisition unit 33 is connected to each of the cameras 6a, 6b, 6c, and 6d, and the camera image acquisition unit 33 acquires captured images of the cameras 6a, 6b, 6c, and 6d. Yes.

  A camera posture calculation unit 31 is connected to the camera image acquisition unit 33. The camera posture calculation unit 31 extracts a feature point from a pattern image and a feature point extraction unit 31a. And a camera posture calculation unit 31b connected to the.

  This camera posture calculation unit 31 uses the camera point calculation method in the first embodiment or the second embodiment described above by the feature point extraction unit 31a and the camera posture calculation unit 31b. The postures of the cameras 6a, 6b, 6c, and 6d are individually calculated.

  In addition, a vehicle periphery monitoring image generation unit 32 is connected to the camera image acquisition unit 33, and the cameras 6 a, 6 b, 6 c, and 6 d are connected to the vehicle periphery monitoring image generation unit 32 from the camera image acquisition unit 33. Data of a photographed image of a photographing target surface such as a road surface is input.

  A mapping table storage unit 34 is connected to the vehicle periphery monitoring image generation unit 32, and a mapping table 35 as shown in FIG. 19 is stored in the mapping table storage unit 34, for example.

  In this mapping table 35, the correspondence relationship between the captured images of the cameras 6a, 6b, 6c, and 6d and a vehicle periphery monitoring image 38 (see FIG. 20) described later is described.

Specifically, as shown in FIG. 19, the mapping table 35 includes a coordinate point (Xc _i , Yc _j ) (i, j are arbitrary natural numbers in the image coordinate system. However, in the example of FIG. One pixel in the captured image of the camera 6a, 6b, 6c, 6d located at ≠ 5, j ≠ 2) is represented by a coordinate point (Xt _m , Yt _n ) (m, n) in the two-dimensional coordinate system of the display 36 described later. Is an arbitrary natural number, but in the example of FIG. 19, it is described that it is used to generate one pixel in a vehicle periphery monitoring image displayed with m ≠ 4 and n ≠ 3).

  It should be noted that the mapping table 35 describes that two or more pixels in the captured images of one camera 6a, 6b, 6c, 6d are used for generating one pixel in the vehicle periphery monitoring image 38. Good (see, for example, paragraphs 0041 and 0044 of Patent Document 1). Further, the mapping table 35 may be accompanied by coordinate conversion in consideration of correcting distortion of a captured image caused by using an ultra-wide angle lens as a lens for the cameras 6a, 6b, 6c, and 6d.

  By using the mapping table 35 stored in the mapping table storage unit 34, the vehicle periphery monitoring image generation unit 32 converts the image captured by the front camera 6a into a front monitoring image 41 for monitoring the front of the vehicle, and the back camera 6b. An image captured by the left side camera 6d, a captured image captured by the right side camera 6c, a captured image captured by the right side camera 6c, a captured image captured by the left side camera 6d, Are converted into left side monitoring images 44 for monitoring the left side.

  Then, the vehicle periphery monitoring image generation unit 32 combines the data of the monitoring images 41, 42, 43, and 44 and the data of the illustration image 39 indicating the own vehicle into one, thereby automatically A vehicle periphery monitoring image 38 whose viewpoint is changed so as to look down from above the vehicle is generated.

  A display 36 is connected to the vehicle periphery monitoring image generation unit 32, and a vehicle periphery monitoring image 38 generated by the vehicle periphery monitoring image generation unit 32 is displayed on the display 36.

  Furthermore, in this embodiment, the vehicle periphery monitoring image generation unit 32 includes a mapping table correction unit 37, and the mapping table correction unit 37 is configured by each camera 6 a, 6 b, 6 c, 6 d by the camera posture calculation unit 31. The calculation result of the posture is acquired.

Then, the mapping table correction unit 37 calculates the calculated postures of the cameras 6a, 6b, 6c, and 6d from the appropriate postures stored in advance in the mapping table correction unit 37 based on the obtained posture calculation results. It is determined whether or not there is a shift, and if there is a shift, the mapping table 35 stored in the mapping table storage unit 34 is corrected so that the shift is eliminated when the vehicle periphery monitoring image 38 is generated. It is supposed to be. Specifically, for example, the pixels of the image coordinate system used to generate the pixel of the coordinate point (Xt ₄ , Yt ₃ ) in the coordinate system of the display 36 shown in FIG. 19 are the coordinate points (Xc ₅ , Yc shown in FIG. 19). The description of the mapping table 35 is changed so that the pixel of ₂ ) becomes a pixel at a different coordinate point.

  Therefore, the vehicle periphery monitoring image generation unit 32 can generate a good vehicle periphery monitoring image 38 by correcting the mapping table 35 even when the posture of the camera 6 deviates from an appropriate posture due to the influence of an attachment error or the like. It is said that.

  An input operation unit 46 is connected to the vehicle periphery monitoring image generation unit 32, and the user operates the input operation unit 46 to display the vehicle periphery monitoring image 38 on the vehicle periphery monitoring image generation unit 32. Is to direct.

  The vehicle periphery monitoring image generation unit 32 is connected to a vehicle information acquisition unit 47 that acquires vehicle information. The vehicle periphery monitoring image generation unit 32 generates a vehicle periphery image 38 as necessary. Thus, the vehicle information (for example, the position of the shift lever, the vehicle speed, etc.) acquired by the vehicle information acquisition unit 47 is obtained and used to generate the vehicle periphery monitoring image 38. Note that the vehicle periphery monitoring image generation unit 32 may generate the vehicle periphery monitoring image 38 when the vehicle speed obtained from the vehicle information acquisition unit 47 is equal to or lower than a predetermined speed, for example.

  When displaying the vehicle periphery monitoring image 38 using such an image display device 30, first, as a premise, the postures of the cameras 6 a, 6 b, 6 c, 6 d with respect to the placement surface of the vehicle 5 are calculated.

  In other words, the camera posture calculation target device 10 shown in the first embodiment or the camera posture calculation target board 20 shown in the second embodiment is installed on the mounting surface of the vehicle 5, and then the cameras 6a and 6b. , 6c, 6d, the pattern surfaces 12, 23 are photographed.

  Data of the captured images of the pattern surfaces 12 and 23 are input from the cameras 6a, 6b, 6c, and 6d to the camera posture calculation unit 31 via the camera image acquisition unit 33.

  Next, the camera posture calculation unit 31 calculates the postures of the cameras 6 a, 6 b, 6 c, and 6 d and outputs the calculation results to the vehicle periphery monitoring image generation unit 32.

  Next, when the postures of the cameras 6a, 6b, 6c, and 6d are deviated from appropriate postures based on the calculation result of the camera posture calculation unit 31 by the mapping table correction unit 37 of the vehicle periphery monitoring image generation unit 32. The mapping table 35 stored in the mapping table storage unit 34 is corrected so that the deviation is eliminated when the vehicle periphery monitoring image 38 is generated (coordinate conversion and synthesis of the image).

  Thereafter, when the display of the vehicle periphery monitoring image 38 is instructed by the user's input operation using the input operation unit 46 on the imaging target surface such as the road surface, the imaging target surface is obtained by each camera 6a, 6b, 6c, 6d. The captured image data is input to the vehicle periphery monitoring image generation unit 32.

  Next, the vehicle periphery monitoring image generation unit 32 uses the conversion table 35 in the conversion table database 34 to convert the captured image data of the imaging target surface input from each camera 6 into the monitoring images 41, 42, 43,. The vehicle periphery monitoring image 38 is generated by converting the image data into 44 image data and combining the image data and the illustration image 39.

  Then, as shown in FIG. 20, the vehicle periphery monitoring image 38 generated by the vehicle periphery monitoring image generation unit 32 is displayed on the display 36.

  The vehicle periphery monitoring image 38 is generated by using a mapping table 35 corrected based on a highly accurate calculation result of the posture of the camera 6 using the camera posture calculation target device 10 or the camera posture calculation target board 20. As a result, the image quality is good.

  Therefore, according to the present embodiment, the posture of the camera 6 with respect to the mounting surface of the vehicle 5 can be calculated with high accuracy, and a good vehicle periphery monitoring image 38 can be obtained based on the captured image of the imaging target surface. it can.

  In addition, this invention is not limited to embodiment mentioned above, A various change is possible as needed.

  For example, the present invention may be applied to the calculation of the posture of the camera with respect to the reference plane for displaying an image other than the vehicle periphery monitoring image.

In addition, the pattern 54 in the reference form may be formed in a shape other than a circle, for example, a regular polygon such as a square, or a lattice line or two different rectangular shapes are orthogonal to each other. It may be formed in a rectangular shape of one color in a checkered pattern that is alternately arranged along the direction. However, the feature point of the pattern image for the regular polygonal pattern is the center point of the pattern image. Further, the feature points of the pattern image with respect to the grid line pattern are the grid points of the pattern image, that is, the intersection points of a plurality of grid pattern images intersecting each other. Furthermore, the feature points of the pattern image for the rectangular pattern of one color in the checkered pattern are the four corner points in this pattern image.

The perspective view which shows 1st Embodiment of the target apparatus for camera attitude | position calculation concerning this invention. 1 is a side view showing a first embodiment of a camera posture calculation target device and a camera posture calculation method according to the present invention. The figure which shows the picked-up image of a pattern surface in 1st Embodiment of the camera posture calculation target apparatus and camera posture calculation method which concern on this invention. The figure which shows a virtual pattern image in 1st Embodiment of the target apparatus for camera attitude | position calculation and camera attitude | position calculation method which concern on this invention. In the first embodiment of the camera posture calculation target device and camera posture calculation method according to the present invention, in order to explain the coordinate conversion of the virtual pattern image from the image coordinate system to the two-dimensional polar coordinate system in the process of designing the pattern surface Illustration used In the first embodiment of the camera posture calculation target device and camera posture calculation method according to the present invention, in order to explain the coordinate conversion of the virtual pattern image from the two-dimensional polar coordinate system to the camera coordinate system in the process of designing the pattern surface Illustration used In the first embodiment of the camera posture calculation target device and camera posture calculation method according to the present invention, in order to explain the coordinate conversion of the virtual pattern image from the camera coordinate system to the three-dimensional world coordinate system in the process of designing the pattern surface Explanatory drawing used for Explanatory drawing which shows the pattern surface assumed on XY plane in the three-dimensional world coordinate system in the process of the design of a pattern surface in 1st Embodiment of the target apparatus for camera attitude | position calculation and camera attitude calculation method concerning this invention In the first embodiment of the camera posture calculation target device and camera posture calculation method according to the present invention, as a final process of designing the pattern surface, the coordinate conversion of the virtual pattern image from the three-dimensional world coordinate system to the X′Y coordinate system is performed. Explanatory drawing used to explain (A) And (b) is the first embodiment of the camera posture calculation target device and camera posture calculation method according to the present invention, in which the coordinate conversion of the virtual pattern image from the three-dimensional world coordinate system to the X′Y coordinate system is performed. Explanatory drawing used with FIG. 9 for explanation The perspective view which shows 2nd Embodiment of the target apparatus for camera attitude | position calculation concerning this invention. The figure which shows the picked-up image of a pattern surface in 2nd Embodiment of the target apparatus for camera attitude | position calculation and camera attitude calculation method concerning this invention. The figure which shows the virtual pattern image in the process of the design of a pattern surface in 2nd Embodiment of the target apparatus for camera attitude | position calculation and camera attitude calculation method concerning this invention The figure which shows the installation state of the target board for camera attitude | position calculation on the mounting surface of a vehicle in 2nd Embodiment of the target apparatus for camera attitude | position calculation and camera attitude calculation method which concerns on this invention. The perspective view which shows the reference form of the target apparatus for camera attitude | position calculation which concerns on this invention The side view which shows the reference form of the target apparatus for camera attitude | position calculation which concerns on this invention, and the camera attitude | position calculation method The figure which shows the picked-up image of a pattern surface in the reference form of the camera posture calculation target apparatus and camera posture calculation method which concerns on this invention The block diagram which shows the image display apparatus used for the image display method in embodiment of the image display method which concerns on this invention Schematic of mapping table in the embodiment of the image display method according to the present invention. The figure which shows a vehicle periphery monitoring image in embodiment of the image display method which concerns on this invention. The perspective view which shows an example of the target board for camera posture calculation used conventionally The figure which shows the camera posture calculation method which has been adopted from the past The figure which shows an example of the picked-up image of the pattern surface in the conventional camera posture calculation target board

Explanation of symbols

6 Camera 10 Target device for camera posture calculation 12 Pattern surface 14 Pattern 15 Photographed image

Claims (13)

A camera posture calculation target device used for calculating the posture of a camera mounted at an attachment position above the reference surface so that the reference surface and the optical axis form an angle other than 90 ° with respect to the reference surface. And
A pattern surface on which a plurality of patterns are formed; the pattern surface is photographed by the camera in a state where the target apparatus main body is installed on the reference surface; and feature points are extracted from the pattern in the photographed image of the pattern surface To be used for calculating the posture of the camera,
As a photographic image of the pattern surface, at least one of the spacing between adjacent coordinated between patterns in size and a predetermined direction for said plurality of patterns in the captured image to obtain a captured image such that the uniform one Can be formed ,
In the installation state of the target device main body on the reference surface, the pattern surface is formed to be orthogonal to the optical axis of the camera,
The plurality of patterns are adjacent to each other in a size and a predetermined direction of the plurality of patterns in the photographed image as a photographed image of the pattern surface when the target apparatus main body is installed on the reference surface. Formed in a shape capable of obtaining a captured image in which at least one of the intervals between the patterns is uniform,
The shape of the plurality of patterns is
It is a pattern image having a uniform size and / or cycle, and the position in the image coordinate system, which is a two-dimensional coordinate system taken on the imaging surface of the camera, is the same as the plurality of patterns in the captured image. Assuming a set of virtual pattern images that are various pattern images within the field of view of the camera, the coordinate points of the virtual pattern image in the image coordinate system are converted into coordinate points of a two-dimensional coordinate system including the pattern surface to a shape obtained, yet, the camera posture calculation target and wherein the shape der Rukoto in consideration of the influence of distortion due to the mounted lens to the camera. A camera posture calculation target device used for calculating the posture of a camera mounted at an attachment position above the reference surface so that the reference surface and the optical axis form an angle other than 90 ° with respect to the reference surface. And
A pattern surface on which a plurality of patterns are formed; the pattern surface is photographed by the camera in a state where the target apparatus main body is installed on the reference surface; and feature points are extracted from the pattern in the photographed image of the pattern surface To be used for calculating the posture of the camera,
As a photographed image of the pattern surface, obtain a photographed image in which at least one of the size of the plurality of patterns in the photographed image and the interval between adjacent patterns in a predetermined direction is uniform. Can be formed,
In the installation state of the target device body on the reference surface, the pattern surface is formed to be parallel to the reference surface,
The plurality of patterns are adjacent to each other in a size and a predetermined direction of the plurality of patterns in the photographed image as a photographed image of the pattern surface when the target apparatus main body is installed on the reference surface. At least one of the distance between the patterns to each other is formed in a shape capable of obtaining a captured image such that the uniform one,
The shape of the plurality of patterns is
It is a pattern image having a uniform size and / or cycle, and the position in the image coordinate system, which is a two-dimensional coordinate system taken on the imaging surface of the camera, is the same as the plurality of patterns in the captured image. Assuming a set of virtual pattern images that are various pattern images within the field of view of the camera, the coordinate points of the virtual pattern image in the image coordinate system are converted into coordinate points of a three-dimensional coordinate system including the pattern surface to a shape obtained, yet, considering shape der Rukoto features and to Luke camera posture calculation target device the effects of the distortion due to the mounted lens to the camera. It consists of a plate-shaped object in which the said pattern surface was formed. The camera posture calculation target apparatus of Claim 2 characterized by these. A holding member that holds the pattern surface in an inclined state with respect to the reference surface so that the pattern surface is orthogonal to the optical axis of the camera in the installed state of the target device main body on the reference surface; The target apparatus for camera posture calculation according to claim 1 , wherein The camera posture calculation target device according to any one of claims 1 to 4 , wherein the camera is a camera including an ultra-wide-angle lens. The said camera is attached to the said predetermined attachment position in a vehicle, The said reference plane is used as the mounting surface of the said vehicle. The Claim 1 thru | or 5 characterized by the above-mentioned. Target device for camera posture calculation. A camera attitude calculation method for calculating an attitude relative to the reference plane of a camera mounted at an attachment position above the reference plane so that the reference plane and the optical axis form an angle other than 90 ° ,
A camera posture calculation target device having a pattern surface on which a plurality of patterns are formed, and as a captured image of the pattern surface by the camera in the installation state of the camera posture calculation target device on the reference surface, at least one of, can be obtained photographed image such that uniform one-formed camera pose spacing between adjacent coordinated between patterns in size and a predetermined direction for said plurality of patterns in the captured image Install the target device for calculation on the reference plane,
Photographing the pattern surface in the installed camera posture calculation target device with the camera,
Extract feature points from the pattern in the captured image of the pattern surface obtained by this shooting,
Based on the extracted feature points, calculate the posture of the camera ,
At that time,
As the camera posture calculation target device, in the installation state of the camera posture calculation target device on the reference plane, the pattern surface is formed to be orthogonal to the optical axis of the camera, and the plurality of patterns are As a captured image of the pattern surface in the installation state of the camera posture calculation target device on the reference surface, the size of the plurality of patterns in the captured image and the adjacent patterns in a predetermined direction Using a camera posture calculation target device formed in a shape capable of obtaining a captured image in which at least one of the intervals is uniform, and calculating the posture of the camera;
The shape of the plurality of patterns is
It is a pattern image having a uniform size and / or cycle, and the position in the image coordinate system, which is a two-dimensional coordinate system taken on the imaging surface of the camera, is the same as the plurality of patterns in the captured image. Assuming a set of virtual pattern images that are various pattern images within the field of view of the camera, the coordinate points of the virtual pattern image in the image coordinate system are converted into coordinate points of a two-dimensional coordinate system including the pattern surface and a shape obtained by, yet, the camera orientation calculation wherein the shape der Rukoto in consideration of the influence of distortion due to the mounted lens to the camera. A camera attitude calculation method for calculating an attitude relative to the reference plane of a camera mounted at an attachment position above the reference plane so that the reference plane and the optical axis form an angle other than 90 °,
A camera posture calculation target device having a pattern surface on which a plurality of patterns are formed, and as a captured image of the pattern surface by the camera in the installation state of the camera posture calculation target device on the reference surface, Camera posture calculation formed so as to obtain a photographed image in which at least one of the size of the plurality of patterns in the photographed image and the spacing between adjacent patterns in a predetermined direction is uniform. Set the target device on the reference plane,
Photographing the pattern surface in the installed camera posture calculation target device with the camera,
Extract feature points from the pattern in the captured image of the pattern surface obtained by this shooting,
Based on the extracted feature points, calculate the posture of the camera,
At that time,
As the camera posture calculation target device, in the installation state of the camera posture calculation target device on the reference surface, the pattern surface is formed to be parallel to the reference surface, and the plurality of patterns However, as a captured image of the pattern surface in the installation state of the camera posture calculation target device on the reference surface, adjacent patterns in the size and a predetermined direction of the plurality of patterns in the captured image using at least one camera orientation calculation target device formed in a shape capable of obtaining a captured image such that the uniform one spacing between, and calculating the orientation of the camera,
The shape of the plurality of patterns is
It is a pattern image having a uniform size and / or cycle, and the position in the image coordinate system, which is a two-dimensional coordinate system taken on the imaging surface of the camera, is the same as the plurality of patterns in the captured image. Assuming a set of virtual pattern images that are various pattern images within the field of view of the camera, the coordinate points of the virtual pattern image in the image coordinate system are converted into coordinate points of a three-dimensional coordinate system including the pattern surface to a shape obtained, yet, it mounted on the characteristic shape der Rukoto in consideration of the influence of distortion due to the lens were to Luke camera orientation calculation method to the camera. 9. The camera according to claim 8 , wherein the camera posture calculation target device is a camera posture calculation target device including a plate-like body on which the pattern surface is formed. Posture calculation method. As the camera posture calculation target device, the pattern surface is set with respect to the reference surface so that the pattern surface is orthogonal to the optical axis of the camera in the installation state of the camera posture calculation target device on the reference surface. The camera posture calculation method according to claim 7 , wherein the camera posture is calculated by using a camera posture calculation target device having a holding member that is held in an inclined state. As the camera, camera orientation calculation method according to any one of claims 7 to 10, characterized by using a camera with a super-wide-angle lens. The attitude to the mounting surface of the vehicle as the reference plane of the camera mounted at the predetermined mounting position in the vehicle, according to claim 7 or claims, characterized in that calculated using the camera orientation calculation target device camera posture calculation method according to any one of claim 11. The photographing target surface is photographed by a camera attached at a predetermined mounting position in the vehicle so that the mounting surface of the vehicle and the optical axis form an angle other than 90 °, and the photographing image of the photographing target surface is An image display method for generating a vehicle periphery monitoring image for monitoring the periphery of the vehicle by performing coordinate conversion, and displaying the generated vehicle periphery monitoring image on a display unit,
A camera posture calculation target device having a pattern surface on which a plurality of patterns are formed, and a photographed image of the pattern surface by the camera in the installation state of the camera posture calculation target device on the placement surface of the vehicle as, at least one of the spacing between adjacent coordinated between patterns in size and a predetermined direction for said plurality of patterns in the captured images are formed can be obtained photographed image such that the uniform one Installed the camera posture calculation target device on the mounting surface of the vehicle,
The pattern surface of the installed camera posture calculation target device is photographed by the camera,
Extract feature points from the pattern in the captured image of the pattern surface obtained by this shooting,
Based on the extracted feature points, calculate the posture of the camera,
When there is a deviation between the calculated camera posture and an appropriate posture, the deviation is eliminated with respect to the photographed image of the photographing target surface when the vehicle periphery monitoring image is generated. There row coordinate transformation, such as,
The shape of the plurality of patterns is
It is a pattern image having a uniform size and / or cycle, and the position in the image coordinate system, which is a two-dimensional coordinate system taken on the imaging surface of the camera, is the same as the plurality of patterns in the captured image. A set of virtual pattern images, which are various pattern images, is assumed within the field of view of the camera, and the coordinate points of the virtual pattern image in the image coordinate system are represented by a two-dimensional coordinate system or a three-dimensional coordinate system including the pattern surface. An image display method characterized in that the shape is obtained by converting to a coordinate point of the lens and is a shape that takes into account the influence of distortion by the lens mounted on the camera .

Images