JP2013024662A - Three-dimensional range measurement system, three-dimensional range measurement program and recording medium - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a three-dimensional range measurement system, etc. capable of measuring expansion of a three-dimensional recognition area of a driver of an automobile, etc.SOLUTION: A three-dimensional range measurement system 1 is constituted of: an exterior camera 20 having two wide-angle lens cameras which perform stereo measurement of exterior environment of a vehicle 10; an interior camera 30 which measures a visual line direction of a driver 12; and a PC 40 connected to the exterior camera 20 and the interior camera 30. An equidistance projection type fisheye lens camera is used for the exterior camera 20, and a twin-lens stereo method which does not perform conversion into a perspective projection model is devised. A coordinate system XwYwZw of the interior camera 30 is considered as a world coordinate system, and a coordinate system XcYcZc of the exterior camera 20 is matched to the coordinate system of the interior camera 30. Triangular patches are constituted for every three adjacent pixels on an image of exterior environment obtained from measurement of the exterior camera 20. Intersections between the triangular patches and a visual straight line of the driver obtained from measurement of the interior camera 30 are considered as three-dimensional coordinates of a fixation point of the driver 12.

Description

  The present invention relates to a three-dimensional range measurement system that measures a range of a three-dimensional space that is visible to a driver of a vehicle.

  In general, the human field of view is the range that can be seen without moving the eyes. Assuming that the angle in the direction of the point of interest (gaze point) is 0 °, the visual acuity is good at about 1.2 in the range of 13 ° left and right (view angle 26 °, central vision) from the gaze point. In the peripheral area (peripheral visual field), the visual acuity suddenly decreases to about 0.1. A range in which an object can be recognized in the peripheral visual field is called an effective visual field. In other words, the effective visual field is a part of the peripheral visual field that contributes to cognition. It is said that the effective visual field varies depending on age, sex and the like. The effective field of view is important when looking outside the vehicle in a moving state, such as a driver of an automobile or the like. In this case, it is said that the effective visual field varies depending on the driving condition, road condition, psychological condition, and individual differences in addition to age, sex, and the like.

  In general, in order to measure the effective visual field of a person, it is measured by detecting a target stimulus from various visual stimuli reflected on the screen within the visual field while having the subject gaze at a fixed viewpoint in the screen ( Non-patent document 1). As an effective visual field measurement under free eye movement, there is a limited visual field method (see Non-Patent Document 2), and its application to measurement during automobile driving (see Non-Patent Document 3) has also been proposed. For example, an in-vehicle stereo camera device such as faceLAB manufactured by Seeing Machines (registered trademark) is used to measure the eyeball rotation angle (pitch angle component of vertical rotation and yaw angle component of horizontal rotation) of the driver. It was broken.

  In the method for measuring the effective visual field described in Non-Patent Documents 1 and 2 described above, only the gaze direction in a plane such as a screen or on a cylinder is targeted. For this reason, the depth was not considered at all. Since the distance between the object to be watched by the driver and the vehicle varies and is not constant, it is necessary to measure the distance including the perspective. However, the method using the in-vehicle stereo camera device described above can measure the driver's line-of-sight direction (straight line) only two-dimensionally, and cannot measure the spread of the three-dimensional recognition area including the depth. There was a problem.

  Accordingly, an object of the present invention has been made to solve the above-described problem, and provides a three-dimensional range measurement system and the like that can measure the spread of a three-dimensional recognition area of a driver such as an automobile. It is in.

  A second object of the present invention is to provide a three-dimensional range measurement system and the like that can measure a temporal change of a three-dimensional effective visual field that changes depending on a mental factor such as a psychological state.

  The third object of the present invention is to determine whether or not the driver has watched an object (hazard) outside the vehicle, to issue a predetermined warning, and to provide driving assistance such as displaying an effective field of view. The object is to provide a three-dimensional range measurement system and the like that can be performed.

  A three-dimensional range measurement system of the present invention is a three-dimensional range measurement system that measures a range of a three-dimensional space that can be seen by a driver of a vehicle, and has an outside vehicle imager having two wide-angle lens cameras that measure the outside environment in stereo. An in-vehicle imager that measures the driver's line-of-sight direction, and an external imager and a computer connected to the in-vehicle imager, wherein the computer uses the coordinate system of the in-vehicle imager as a world coordinate system and the external vehicle The coordinate system of the image pickup device is matched with the coordinate system of the vehicle image pickup device, and the driver's line of sight obtained from the measurement of the vehicle image pickup device is adjacent to the image of the environment outside the vehicle obtained from the measurement of the vehicle image pickup device. The intersection of the triangular patch set for each of the three pixels is set as the three-dimensional coordinates of the driver's gazing point to measure the range of the three-dimensional space visible to the driver.

  Here, in the three-dimensional range measurement system of the present invention, the outside imager uses an equidistant projection left and right fisheye lens camera installed outside the vehicle as a wide-angle lens camera, and the inside imager is installed inside the vehicle. The rotation angle and position coordinates of the driver's eyes can be acquired using the sensor.

  Here, in the three-dimensional range measurement system according to the present invention, the computer includes a vehicle exterior image recording unit that records left and right exterior vehicle images captured by the left and right fisheye lens cameras captured using the vehicle exterior imager, and the vehicle interior imager. The line-of-sight information recording unit that records the rotation angle and position coordinates of the driver's eyes when watching a predetermined point in the world coordinate system and the coordinates of the predetermined point, acquired using the Corresponding point matching means for obtaining an epipolar line as a set of pixels satisfying a predetermined condition with respect to the left and right outside images recorded in the outside image recording unit and finding corresponding points in the left and right outside images by searching on the epipolar line And for the pixel for which the corresponding point is obtained by the corresponding point matching means, the position coordinates of the point in the three-dimensional space outside the vehicle projected onto the pixel are obtained. Both of the three-dimensional coordinate calculation means for obtaining the coordinates of the point in the three-dimensional space outside the vehicle projected onto the pixel by interpolating from the surrounding pixels of the pixel for which the corresponding point has not been obtained; Conversion parameter calculation for obtaining a conversion parameter when the coordinates of the point in the three-dimensional space outside the vehicle obtained by the coordinate calculation means are converted to the coordinates of the predetermined point in the world coordinate system recorded in the line-of-sight information recording unit. Pixel coordinate information recording by calculating coordinates in the world coordinate system for all the pixels on the left and right outside images recorded in the outside image recording unit using the conversion parameters obtained by the means and the conversion parameter calculating means A pixel coordinate calculation means for recording in the unit and a triangular pattern in a predetermined manner for each of the three adjacent pixels recorded in the pixel coordinate information recording unit. A triangular patch forming means for recording information relating to the triangular patch in a triangular patch information recording unit, and a driver's binocular when the predetermined point in the world coordinate system recorded in the line-of-sight information recording unit is watched A triangular patch having a point of intersection between the driver's line of sight obtained from the rotation angle and position coordinates and the coordinates of the predetermined point and the triangular patch recorded in the triangular patch information recording unit by a predetermined method, The intersection point may be a three-dimensional coordinate of the driver's gazing point, and the intersection point calculating unit may record the position coordinate of the gazing point in a gazing point position coordinate recording unit as time passes.

  Here, in the three-dimensional range measurement system of the present invention, the three-dimensional convex hull composed of the gaze point coordinates for a predetermined time recorded in the gaze point position coordinate recording unit is effectively obtained as the driver's effective visual space. Visual space calculation means can be further provided.

  Here, in the three-dimensional range measurement system of the present invention, an orthographic image on the road surface of the effective visual space calculated by the effective visual space calculating means is obtained, and the orthographic image is displayed on a display device connected to the computer. It is possible to further include effective visual space display means for displaying the image in a predetermined manner.

  Here, in the three-dimensional range measurement system of the present invention, hazard coordinate detection means for detecting hazard coordinates existing within the range of the three-dimensional space visible to the driver of the vehicle, and the hazard detected by the hazard coordinate detection means And a warning means for issuing a predetermined warning based on the coordinates of the gazing point and the gazing point coordinates for a predetermined time recorded in the gazing point position coordinate recording unit.

  A three-dimensional range measurement program of the present invention is a three-dimensional range measurement program for measuring a range of a three-dimensional space that can be seen by a driver of a vehicle, and is an outside-vehicle imager having two wide-angle lens cameras that measure the outside environment in stereo. And an in-vehicle imager that measures the driver's line-of-sight direction, and an external imager and a computer connected to the in-vehicle imager. The coordinate system of the vehicle exterior imager is aligned with the coordinate system of the vehicle interior imager, and the driver's line of sight obtained from the measurement of the vehicle interior imager and the image of the environment outside the vehicle obtained from the measurement of the vehicle exterior imager 3D for measuring the range of the three-dimensional space visible to the driver by using the intersection point with the triangular patch set for every three adjacent pixels as the three-dimensional coordinates of the driver's gazing point Is a circumference measurement program.

  Here, in the three-dimensional range measurement program of the present invention, the outside imager uses an equidistant projection left and right fisheye lens camera installed outside the vehicle as a wide-angle lens camera, and the inside imager is installed inside the vehicle. The rotation angle and position coordinates of the driver's eyes can be acquired using the sensor.

  Here, in the three-dimensional range measurement program according to the present invention, the computer includes a vehicle exterior image recording unit that records left and right exterior images captured by the left and right fisheye lens cameras imaged using the exterior vehicle imager, and the in-vehicle imager. A line-of-sight information recording unit that records the rotation angle and position coordinates of the driver's eyes when the user focuses on a predetermined point in the world coordinate system, and the coordinates of the predetermined point recorded in correspondence with each other. Use the computer to obtain an epipolar line as a set of pixels that satisfy a predetermined condition for the left and right outside images recorded in the outside image recording unit, and search the epipolar line to correspond points in the left and right outside images Corresponding point matching step for obtaining a pixel, and for a pixel for which a corresponding point is obtained in the corresponding point matching step, three-dimensional outside the vehicle projected on the pixel 3D to obtain the coordinates of the point in the three-dimensional space outside the vehicle projected on the pixel by interpolating from the surrounding pixels of the pixel for which the corresponding point is not obtained A coordinate calculation step, when the coordinates of a point in the three-dimensional space outside the vehicle obtained in the three-dimensional coordinate calculation step are Euclidean transformed into the coordinates of a predetermined point in the world coordinate system recorded in the line-of-sight information recording unit A conversion parameter calculation step for obtaining a conversion parameter, and using the conversion parameter obtained in the conversion parameter calculation step, coordinates in the world coordinate system are calculated for all the pixels on the left and right outside images recorded in the outside image recording unit. The pixel coordinate calculation step for recording in the pixel coordinate information recording unit, and the adjacent coordinates recorded in the pixel coordinate information recording unit A triangular patch forming step for forming a triangular patch by a predetermined method for every three pixels, and recording information about the triangular patch in the triangular patch information recording unit, a predetermined point in the world coordinate system recorded in the line-of-sight information recording unit The intersection of the driver's line of sight obtained from the rotation angle and position coordinates of the driver's eyes and the coordinates of the predetermined point and the triangle patch recorded in the triangle patch information recording unit A triangular patch is obtained by a predetermined method, the intersection is set as a three-dimensional coordinate of the driver's gazing point, and an intersection point calculating step is executed in which the position coordinate of the gazing point is recorded in the gazing point position coordinate recording unit over time. be able to.

  Here, in the three-dimensional range measurement program according to the present invention, the three-dimensional convex hull composed of the gaze point coordinates for a predetermined time recorded in the gaze point position coordinate recording unit is effectively obtained as the driver's effective visual space. A visual space calculation step may be further provided.

  Here, in the three-dimensional range measurement program of the present invention, an orthographic image on the road surface of the effective visual space calculated by the effective visual space calculation step is obtained, and the orthographic image is displayed on a display device connected to the computer. Can be further provided with an effective visual space display step of displaying the image in a predetermined manner.

  Here, in the three-dimensional range measurement program of the present invention, a hazard coordinate detection step for detecting hazard coordinates existing in the range of the three-dimensional space visible to the driver of the vehicle, and the hazard detected by the hazard coordinate detection step And a warning step for issuing a predetermined warning based on the coordinates of the gazing point and the gazing point coordinates for a predetermined time recorded in the gazing point position coordinate recording unit.

  The recording medium of the present invention is a computer-readable recording medium on which any of the three-dimensional range measurement programs of the present invention is recorded.

  According to the three-dimensional range measurement system and the like of the present invention, the driver's face image is captured by the in-vehicle camera, and the line-of-sight measurement process is performed to obtain the driver's eyeball position coordinates and the eyeball rotation angle. In parallel, a vehicle outside image is captured by the vehicle outside camera and stereo processing outside the vehicle is executed. Since the coordinate system of the in-vehicle camera is different from the coordinate system of the in-vehicle camera, the coordinate system XwYwZw of the in-vehicle camera is the world coordinate system, and the coordinate system XcYcZc of the out-of-vehicle camera is matched to the coordinate system of the in-vehicle camera 30. Since the two sets of fisheye stereo cameras (for in-car use and for outside the car) used in the present invention do not have a common shooting area, pay attention to the marker placed in a predetermined position while the driver is sitting on the vehicle seat in advance. And then devised a method for obtaining a calibration parameter that integrates the camera coordinate system from information on the driver's line-of-sight direction that can be obtained from the in-vehicle camera and the three-dimensional coordinates of the marker in the fish-eye camera coordinate system. A triangular patch is formed for every three adjacent pixels on the image of the environment outside the vehicle obtained from the measurement by the camera outside the vehicle. Next, the intersection of the driver's line of sight obtained from the measurement by the in-vehicle camera and the triangular patch is set as the three-dimensional coordinates of the driver's point of interest. The present invention has devised a binocular stereo method that does not convert to a perspective projection model using a fisheye lens camera of equidistant projection. Coordinates in the world coordinate system are calculated and recorded in the pixel coordinate information DB for all pixels on the left and right outside images recorded in the outside image DB. A triangular patch is formed by a predetermined method for every three adjacent pixels recorded in the pixel coordinate information DB, and information related to the triangular patch is recorded in the triangular patch information DB. The driver's line of sight obtained from the eyeball rotation angle and eyeball position coordinates of the driver's eyes when gazing at a predetermined point in the world coordinate system recorded in the line-of-sight information DB, and the coordinates of the predetermined point; A triangular patch having an intersection with the triangular patch recorded in the triangular patch information DB was obtained by a predetermined method, and the intersection was set as the three-dimensional coordinates of the driver's point of sight. As described above, according to the three-dimensional range measurement system and the like of the present invention, it is possible to provide a three-dimensional range measurement system and the like that can measure the spread of the three-dimensional recognition area of the driver of the vehicle. There is an effect.

  According to the three-dimensional range measurement system and the like of the present invention, by generating a three-dimensional convex hull from a time-series gaze point group, the region is made an effective visual space. The three-dimensional convex hull can be obtained by the sequential addition method. As a result, it is possible to provide a three-dimensional range measurement system and the like that can measure a temporal change in a three-dimensional effective visual field (effective visual space) that changes depending on mental factors such as the psychological state of the driver 12. There is an effect.

  According to the three-dimensional range measurement system and the like of the present invention, an orthographic image on the road surface of the effective visual space calculated by the effective visual space calculation unit is obtained, and the orthographic image is predetermined on a display device connected to the computer. Display in the method. As a result, it is possible to display the horizontal viewing angle, the vertical viewing angle, and the depth as images, and to express the size of the effective viewing space by color and display an oversight hazard.

  According to the three-dimensional range measurement system and the like of the present invention, by using the hazard coordinates detected by the hazard coordinate detection unit and the gaze point coordinates for a predetermined time recorded in the gaze point position coordinate DB, a variety of An accurate warning can be issued. For this reason, by mounting the three-dimensional range measurement system of the present invention on a vehicle, it is possible to provide a vehicle with higher safety and lower cost.

It is a figure which shows the outline | summary of the three-dimensional range measurement system 1 in Example 1 of this invention and another Example. It is a flowchart which shows the flow of a process of the three-dimensional range measurement program of this invention. It is a figure which shows the operating environment of the three-dimensional range measurement program of this invention. It is a figure which shows the projection model of the fish-eye stereo used with the binocular stereo method which comprises the three-dimensional range measurement program of this invention. 2 is a diagram showing an example in which fisheye lens cameras 25L and 25R are installed on the roof of a vehicle 10. FIG. It is a figure which shows the example of installation of the camera apparatus 30 in a vehicle. It is a figure which shows the positional relationship of the camera 30 inside a vehicle and the camera 20 outside a vehicle. It is a figure which shows the condition which measures the gaze direction of the driver | operator 12 with the gaze measurement camera 30. FIG. It is a figure for demonstrating derivation | leading-out of the intersection of the visual line S of the driver | operator 12, and a vehicle exterior triangle patch. It is a flowchart which shows the flow of a process of the binocular stereo method which comprises the three-dimensional range measurement program of this invention. It is a figure for demonstrating the effective visual space in Example 2 of this invention. It is a flowchart which shows the algorithm (refer nonpatent literature 7) which calculates | requires a three-dimensional convex hull by the sequential addition method. It is a figure which illustrates the mode of sequential addition. It is a figure which shows the example of a display of the effective visual space in Example 3 of this invention. It is a block diagram which shows the internal circuit 180 of the computer 10 which executes the program of this invention.

  Hereinafter, embodiments will be described in detail with reference to the drawings.

  FIG. 1 shows an outline of a three-dimensional range measurement system 1 in Embodiment 1 of the present invention and other embodiments. In FIG. 1, reference numeral 10 denotes a vehicle such as an automobile, 12 denotes a driver driving the vehicle 10, and 20 denotes an outside camera (an outside image pickup device, coordinates having two wide-angle lens cameras for stereo measurement of the outside environment of the vehicle 10. The traveling direction of the vehicle 10 is the Zc axis, the Xc axis and the Yc axis perpendicular to the Zc axis, and 30 is an in-vehicle camera (in-vehicle imager. The coordinates are the Zw axis and Zw axis for the driver direction). 40 is a computer such as a personal computer (PC) connected to the outside camera 20 and the inside camera 30. The three-dimensional range measurement system 1 measures the range of a three-dimensional space that can be seen from the driver 12 of the vehicle 10.

  As shown in FIG. 1, the wide-angle lens camera of the outside camera 20 installed on the roof of the vehicle 10 is preferably a fish-eye lens camera. Hereinafter, the camera of the outside camera 20 will be described as being a fisheye lens camera, but is not limited to the fisheye lens camera, and the distance between the object outside the vehicle and the vehicle 10 is long (30 m or more). In some cases, a wide-angle lens camera is preferable. The in-vehicle camera 20 shown in FIG. 1 acquires the rotation angle and position coordinates of both eyes (not shown) of the driver 12 using an image sensor such as a CCD installed in the vehicle.

  FIG. 2 is a flowchart showing the processing flow of the three-dimensional range measurement program of the present invention. In FIG. 2, the parts denoted by the same reference numerals as those in FIG. In FIG. 2, reference numeral 42 denotes a recording device that is connected to the PC 40 and records various data used by the three-dimensional range measurement program. FIG. 3 shows the operating environment of the three-dimensional range measurement program of the present invention. In FIG. 3, reference numeral 50 denotes a functional block indicating the function of a three-dimensional range measurement program executed on the PC 40, and 42 denotes the recording apparatus described with reference to FIG. Various functions in the functional block 50 are realized by software, but some may be realized by hardware. The operation of the three-dimensional range measurement program of the present invention will be described below with reference to the contents of FIGS.

  First, an outline of the three-dimensional range measurement program of the present invention will be described. As shown in FIG. 2, the face image of the driver 12 is captured by the in-vehicle camera 30 and the line-of-sight measurement process is performed, and the eyeball position coordinates and the eyeball rotation angle of the driver 12 are obtained (step S30). At the same time, a vehicle exterior image is taken by the vehicle exterior camera 20 and a vehicle exterior stereo process is executed (step S20). Here, as shown in FIG. 1, the in-vehicle camera 30 has a coordinate system XwYwZw, while the in-vehicle camera 20 has another coordinate system XcYcZc. Therefore, in order to integrate the information on the line-of-sight direction of the driver 12 and the information on the three-dimensional environment outside the vehicle, it is necessary to unify the coordinate systems of the left and right fisheye stereo cameras into the world coordinate system. In the three-dimensional range measurement program of the present invention, the coordinate system XwYwZw of the in-vehicle camera 30 is the world coordinate system, and the coordinate system XcYcZc of the in-vehicle camera 20 is set to the coordinate system of the in-vehicle camera 30 (step S22).

  Since the two sets of fish-eye stereo cameras (in-vehicle use and external-use use) used in the present invention do not have a common photographing area, camera calibration using a general calibration instrument is impossible. Therefore, the driver 12 pays attention to the marker placed in a predetermined position while sitting on the seat of the vehicle 10, and information on the driver's line-of-sight direction and the marker that can be obtained from the in-vehicle camera 30 (faceLAB) at that time Devised a method for obtaining calibration parameters that integrate the camera coordinate system from the three-dimensional coordinates in the fish-eye camera coordinate system. Details will be described later. As shown in FIG. 2, a triangular patch is formed for every three adjacent pixels on the image of the outside environment obtained from the measurement by the outside camera 20 (step S24). Next, the intersection of the driver's line of sight obtained from the measurement by the in-vehicle camera 30 and the triangular patch is set as the three-dimensional coordinates of the driver's gazing point (step S40). After calculating a three-dimensional convex hull (details will be described later) indicating the temporal change of the three-dimensional effective visual field (step S42), and recording effective visual space (details will be described later) information in the recording device 42 (step S44). The above-described processing is repeated.

  Next, the details of the three-dimensional range measurement program of the present invention will be described. The above-described vehicle exterior stereo process (step S20) is measurement by a fisheye stereo camera in the vehicle exterior environment. There have been some studies on converting images captured by a fisheye stereo camera into images of perspective projection models (see Non-Patent Documents 4 and 5), and converting fisheye model images into images of perspective projection models. The stereo measurement is performed in (Non-patent Document 6). However, the methods described in Non-Patent Documents 4 to 6 have a problem that the peripheral portion of the fisheye model image is lost by converting the image. Therefore, a binocular stereo method (mainly corresponding to steps S22 to S40) that does not convert to a perspective projection model using an equidistant projection type fisheye lens camera was devised.

  FIG. 4 shows a fish-eye stereo projection model used in the binocular stereo method constituting the three-dimensional range measurement program of the present invention. Hereinafter, “fisheye lens” is used to include “fisheye lens camera”. As shown in FIG. 4, FIG. 4A is an overhead view of the left fisheye lens 25L (center O, coordinates XYZ) and the right fisheye lens 25R (center O ′, coordinates X′Y′Z ′) of the camera 20 outside the vehicle. 4B is a plan view of the fisheye lenses 25L and 26R viewed from above, and FIG. 4C is a plan view of the fisheye lenses 25L and 26R viewed from below (in the camera). FIG. 4D is a diagram for explaining FIG. 4A and 4B, let Pw be an arbitrary point in the three-dimensional space, and let P and P ′ be points that are projected on the fisheye lenses 25L and 25R, respectively. θ and θ ′ represent zenith angles with respect to the left and right fisheye lenses 25L and 25R, respectively, and φ and φ ′ represent azimuth angles. The left and right fisheye lenses 25L and 25R, f and f ', are focal lengths and represent the radius of the spherical model. In FIG. 4C, the ul axis whose right direction is the positive direction is the horizontal image coordinate axis with the optical axis of the left image as the origin, and the vl axis whose downward direction is the positive direction is the optical axis of the left image as the origin. Is the vertical image coordinate axis. Similarly, in the right side of FIG. 4C, the ur axis whose right direction is the positive direction is a horizontal image coordinate axis whose origin is the optical axis of the right image, and the vr axis whose downward direction is the positive direction is the light of the right image. This is an image coordinate axis in the vertical direction with the axis as the origin.

  In FIG. 4C, the vl axis is downward in the image because of the arrangement in which the image data is stored in the three-dimensional range measurement program of the present invention. In general, images are arranged in the order of raster operation. If you move 1 from the upper left corner to the right in the horizontal direction, you go 1 from the left of the lower left corner to the right in the horizontal direction, and then move down one square. Data is held so that one row is from the left to the right in the horizontal direction. Therefore, the ul axis is often directed to the right and the vl axis is directed downward. Furthermore, if the origin of the coordinates is at the center of the image instead of the upper left corner of the image, the origin of the three-dimensional coordinates of the fisheye lens (such as XYZ above) and the ulvl of the image are on the same z-axis. To make it easier to remove, the origin is shifted accordingly.

  As described above, the Y-axis in FIG. 4B and the vl-axis in FIG. 4C have different axis directions. For this reason, the location of the azimuth angle φ also varies between the XY plane and the ulvl plane. The azimuth angle φ represents a positive rotation angle from the horizontal axis (ul axis, X axis) in the right-handed coordinate system. In the XY plane, as shown in FIG. 4B, the angle is taken from the positive position of the X axis toward the positive position of the Y axis. Also in the ulvl plane, as shown in FIG. 4C, the angle is from the positive position of the ul axis toward the positive position of the vl axis. Even in the ulvl plane, φ means a positive rotation angle with respect to the ul axis, so that φ shown in FIG. 4C is originally not enough to draw a curve by 180 degrees. In the normal azimuth angle, as shown in FIG. 4D, φl in the left figure is about 320 degrees and φr in the right figure is about 210 degrees. However, φ is obtained by arctan (vl / ul) as described later. Since the period of arctan is π (= 180 degrees), assuming that the angle from the third quadrant to the fourth quadrant of φl is α, tanφl = tan (π + α) = tanα. For this reason, in the third quadrant and the fourth quadrant, even if only the positive rotation angle from the negative part of the ul axis is used, the calculation result is the same value as the azimuth angle that is 180 degrees more. For this reason, the location of φ in FIG. 4C is written shorter by 180 degrees. The same is true for the right diagram in FIG.

  Based on the above preparation, the zenith angle and azimuth angle are obtained as shown in Equation 1.

  When the position vectors of the projection points P and P ′ by the left and right fisheye lens cameras 25L and 25R are p and p ′, respectively, when expressed using the zenith angles θ and θ ′ and the azimuth angles φ and φ ′, Expression 2 It becomes like this.

  The point Pw in the three-dimensional space is at a position obtained by multiplying the position vector of the left and right projection points by a constant, and the position vector Pw can be expressed as in Expression 3.

  However, kl and kr are arbitrary positive real numbers. In the present invention, since the directions and heights of the left and right fisheye lens cameras 25L and 25R are assumed to be parallel, the distance between the two cameras is assumed to be separated by tx in the X-axis direction, and the following condition (formula 4) is given.

  Therefore, the point Pw in the three-dimensional space is obtained as shown in Equation 5.

  When stereo matching is performed with a fisheye projection image, an epipolar line when matching corresponding points is a problem. In FIG. 4A, the plane connecting the center O-point Pw-center O 'is an epipolar plane, and the intersections P, P' between the epipolar plane and the fisheye lenses 25L, 25R become epipoles, and each epipole on the fisheye lenses 25L, 25R. A curve connecting P and P ′ is an epipolar line. That is, the epipolar line of the fisheye image is non-linear and the equation is different for each pixel. The epipolar line is obtained as a set of pixels satisfying the following conditional expression.

  The epipolar line is searched to perform corresponding point matching. Returning to FIG. 3, reference numeral 70 denotes an out-of-vehicle image database DB (outside image recording unit) that records left and right outside images taken by the left and right fisheye lens cameras 25 </ b> L and 25 </ b> R captured using the outside camera 20. The corresponding point matching unit 51 (corresponding point matching means) shown in the functional block 50 of FIG. 3 is epipolar as a set of pixels that satisfy Expression 6 (predetermined condition) for the left and right outside images recorded in the outside image DB 70. A line is obtained, and corresponding points in the left and right outside images are obtained by searching on the epipolar line.

  The three-dimensional coordinate Pw of the point projected onto the pixel (ul, vl) for which the corresponding point is obtained in the left and right fish-eye images by the above processing can be obtained using Equation 5. Where there is an edge or texture, three-dimensional coordinates can be obtained in this way. However, for pixels for which corresponding points cannot be obtained (for example, pixels that are indistinguishable from other pixels such as roads), the three-dimensional points projected for all pixels on the image are interpolated from the surrounding pixels. Calculate the coordinates. The three-dimensional coordinate calculation unit 52 (three-dimensional coordinate calculation means) shown in the functional block 50 of FIG. 3 is a three-dimensional space outside the vehicle that is projected onto the pixel for which the corresponding point is obtained by the corresponding point matching unit 51. The coordinates of the point in the three-dimensional space outside the vehicle projected on the pixel are obtained by interpolating from the surrounding pixels of the pixel for which the corresponding point has not been obtained.

  FIG. 5 shows an example in which fisheye lens cameras 25 </ b> L and 25 </ b> R are installed on the roof of the vehicle 10. As shown in FIG. 5, it is preferable that the outside camera 20 is installed on the roof of the vehicle 10 with two fisheye lens cameras divided into left and right. However, the installation position is not limited to the roof, and may be on the hood or on both side mirrors, and the installation position is not particularly specified. In this specification, FE185C057HA-1 manufactured by FUJIFILM (registered trademark) is used as a fisheye lens, but the fisheye lens is not limited to this fisheye lens.

  FIG. 6 shows an installation example of the in-vehicle camera 30. The installation position of the in-vehicle camera 20 is preferably installed on a dashboard, for example, but may be another position as long as it does not interfere with driving. The in-vehicle camera 20 is a low-cost so-called web camera connected to the computer 40 via a USB or the like, and a software called TrackEye (registered trademark) can be used as a driver monitoring system, for example. TrackEye (registered trademark) detects the face and eyes of the driver 12 and measures the driver's gazing point. In the above-described example, TrackEye (registered trademark) is exemplified as the in-vehicle camera 20, but it is preferable to use, for example, a FaceLab system having a higher number of frames per second.

  FIG. 7 shows the positional relationship between the in-vehicle camera 30 and the outside camera 20. In FIG. 7, the parts denoted by the same reference numerals as those in FIG. In FIG. 7, a point Pw (coordinates Cx, Cy, Cz) is a point in the three-dimensional space described with reference to FIG. 4 and is a gazing point that the driver 12 is gazing at.

  Next, the three-dimensional position measurement of the driver's gazing point Pw will be described. The three-dimensional position measurement of the gazing point Pw of the driver 12 is based on the unification of the exterior camera 20 coordinate system XcYcZc and the interior camera 30 coordinate system XwYwZw, and the derivation of the intersection of the driver's 12 line of sight and the exterior triangle patch (described later). It is made up. Each principle will be described below. In addition, a method for deriving a transformation matrix necessary in advance to integrate the coordinate systems of both cameras will be described.

Unification of the exterior camera 20 coordinate system XcYcZc and the interior camera 30 coordinate system XwYwZw.
Hereinafter, the outside camera 20 is called a fish-eye stereo camera 20, and the in-vehicle camera 30 is called a line-of-sight measurement camera 30. The fish-eye stereo camera 20 that captures the environment outside the vehicle and the line-of-sight measurement camera 30 (faceLAB camera) that performs the line-of-sight measurement have independent coordinate systems, so that the three-dimensional environment outside the vehicle and the information on the line-of-sight direction are integrated. Therefore, it is necessary to unify the coordinate system of each stereo camera 20 and 30 to the world coordinate system. In the present invention, the world coordinate system is used as the coordinate system of the visual line measurement camera 30, and the camera coordinate system of the fisheye stereo camera 20 is matched with the coordinate system of the visual line measurement camera 30.

  The relationship between the three-dimensional coordinates (Wx, Wy, Wz) in the coordinate system XwYwZw of the line-of-sight measurement camera 30 and the three-dimensional coordinates (Cx, Cy, Cz) in the coordinate system XcYcZc of the fisheye stereo camera 20 is as shown in Expression 7. Is represented by Euclidean transformation.

  After that, conversion parameters p1,..., P12 may be obtained. However, since the two sets of stereo cameras (for in-vehicle camera 30 and for in-vehicle camera 20) used in the present invention do not have a common shooting area, camera calibration using a general calibration instrument is impossible. Therefore, the driver 12 pays attention to a predetermined marker in a state where the driver 12 is seated in advance, and information on the driver's gaze direction that can be acquired from the gaze measuring camera 30 at that time and the marker in the fish-eye camera 20 coordinate system. A method for obtaining a calibration parameter for integrating a camera coordinate system from three-dimensional coordinates was devised as follows. That is, since the two stereo cameras 20 and 30 cannot shoot the calibration instrument in common, a role of connecting the information of both stereo cameras 20 and 30 through the line-of-sight data of the driver 12 (or the operator) is provided instead. . The marker was placed on the hood or windshield of the vehicle 10.

FIG. 8 shows a situation in which the line-of-sight direction of the driver 12 is measured by the line-of-sight measurement camera 30. In FIG. 8, the portions denoted by the same reference numerals as those in FIGS. 1 and 6 indicate the same elements, and thus the description thereof is omitted. In FIG. 8, symbol S is a line of sight of the driver 12, 12L and 12R schematically show the left and right eyeballs of the driver 12, Ψpitch is a pitch angle component of longitudinal rotation of the eyeball 12L and the like, and Ψyaw is a yaw angle component of lateral rotation of the eyeball 12L such a (Ψpitch, Ψyaw) constitute the eyeball rotation _{angle, (e x, e y,} e z) is the position coordinates such as the eyeball 12L.

As for the eyeball rotation angle and the position coordinates such as the eyeball 12L, data for both eyes are obtained by the line-of-sight measurement camera 30. In the present invention, the average value of each data is taken for these binocular data, and these are defined as the eyeball rotation angle and eyeball position coordinates. Eyeball rotation angle of the left eye 12L (Ψpitchl, Ψyawl), the eye position coordinates _{(e xl, e yl, e} zl) and the eyeball rotation angle of the right eye 12R (Ψpitchr, Ψyawr), the eye position coordinates (e _xr , e _yr , e _zr ), it can be expressed as in equations 8-12.

Based on the above preparation, a method for obtaining a calibration parameter for integrating the camera coordinate system will be described. Straight line representing the line of sight in the world coordinate system (view straight S) is eye position coordinates _{(e x, e y, e} z) and visual line-of-sight direction vector indicating the direction of the straight line _{S (n x, n y,} n z) from Eq It can be written as 13.

  In Equation 13, k is an arbitrary real number. The gaze direction vector can be written as shown in Equation 14 using the eyeball rotation angle.

  From Equations 13 and 14, the equation of the visual line S in the world coordinate system can be written as Equation 15. Hereinafter, the eyeball rotation angle (Ψpitch, Ψyaw) may be simply written as (Ψp, Ψy).

  By setting Equation 7 and Equation 15 to be equal, if data of at least three points (more points for improving accuracy) can be acquired, Equation 7 is solved and parameters p1,. . . , P12 can be obtained. Returning to FIG. 3, reference numeral 72 denotes the eyeball rotation angles and eyeballs of the binoculars 12 </ b> L and 12 </ b> R of the driver 12 when gazing at the point Pw (predetermined point) in the world coordinate system, acquired using the line-of-sight measurement camera 30. This is a line-of-sight information DB (line-of-sight information recording unit) in which position coordinates and coordinates of the point Pw are recorded in association with each other. The conversion parameter calculation unit 53 (conversion parameter calculation means) shown in the functional block 50 of FIG. 3 uses the Euclidean coordinates of the point Pw in the three-dimensional space outside the vehicle, which is obtained by the three-dimensional coordinate calculation unit 52 as described above. The conversion parameters p1,... Of Expression 7 when converting to the coordinates of the point Pw in the world coordinate system recorded in the line-of-sight information DB 72. . . , P12 can be obtained. Next, the obtained conversion parameters p1,. . . , P12 in Expression 7, the three-dimensional coordinates (Wx, Wy, Wz) in the world coordinates can be calculated for the coordinates (Cx, Cy, Cz) of all pixels on the image of the fisheye lens camera 20. The pixel coordinate calculation unit 54 (pixel coordinate calculation means) shown in the functional block 50 of FIG. 3 includes conversion parameters p1,. . . , P12, the coordinates in the world coordinate system are calculated and recorded in the pixel coordinate information DB 74 (pixel coordinate information recording unit) for all the pixels on the left and right outside images recorded in the outside image DB 70.

Derivation of intersection of driver's 12 line of sight S and exterior triangle patch.
When the three-dimensional coordinates can be calculated for all the pixels on the image of the fisheye lens camera 20, a pair is formed for each adjacent pixel to form a triangular patch. At this time, when the v coordinate of the image (u, v) is 0 or an even number, three points (u, v), (u + 1, v), (u, v + 1) are used, and when the v coordinate is an odd number (u , V), (u-1, v), and (u, v-1), a triangular patch is created (predetermined method). By this processing, the outside environment surrounding the vehicle 10 can be configured as a set of triangular patches. Returning to FIG. 3, the triangular patch forming unit 55 (triangular patch forming means) shown in the functional block 50 of FIG. 3 performs the triangular patch in the predetermined method described above for every three adjacent pixels recorded in the pixel coordinate information DB 74. And records information related to the triangular patch in the triangular patch information DB 76 (triangular patch information recording unit). The intersection point between the vehicle exterior triangle patch and the line of sight S in this world coordinate system is taken as the gaze point of the driver 12. Since the three-dimensional environment outside the vehicle surrounding the vehicle 10 is composed of a group of triangular patches, one of the triangular patches and the line of luck 12 visual line S have an intersection.

FIG. 9 is a diagram for explaining the derivation of the intersection point between the driver's sight line S and the outside triangle patch. In FIG. 9, p ₁ = (p _1x , p _1y , p _1z ), p ₂ = (p _2x , p _2y , p _2z ), p ₃ = (p _3x , p _3y , p _3z ) 3 3D coordinates of a point. The equation of the plane stretched by these three points is as shown in Equation 16.

  From Equation 15 and Equation 16, Equation 18 is obtained.

  However, the vector v is assumed to be Equation 19.

  here,

Then, from the Kramer official,

It becomes.

Therefore, it can be written as Equation 23.

Solving this, if the condition of the above equation 17 is satisfied, the triangular patch p ₁ p ₂ p ₃ has the line of sight S and the intersection point Pw, and if the condition of the equation 17 is not satisfied, the triangular patch p ₁ p ₂ is satisfied. It does not have the intersection and p _3. By substituting k at that time into Equation 15 when having an intersection, the three-dimensional coordinates of the gazing point Pw can be obtained. Returning to FIG. 3, the intersection calculation unit 56 (intersection calculation means) shown in the functional block 50 of FIG. 3 focuses on the point Pw (predetermined point) in the world coordinate system recorded in the line-of-sight information DB 72. The visual line S of the driver 12 obtained from the eyeball rotation angle and eyeball position coordinates of the twelve binoculars 12L and the like and the coordinates of a predetermined point, and the triangle patches p ₁ p ₂ p ₃ recorded in the triangle patch information DB 76 The triangular patch p ₁ p ₂ p ₃ having the intersection point Pw is obtained by the method described above (predetermined method) in the derivation of the intersection point of the driver's 12 line of sight S and the outside triangle patch p ₁ p ₂ p ₃ . The intersection point Pw is set as the three-dimensional coordinate of the driver's gazing point, and the position coordinate of the gazing point is recorded in the gazing point position coordinate DB 78 (gaze point position coordinate recording unit) with the passage of time.

In the above-described method, whether or not there is an intersection (parameters s, t) and the parameter k of the intersection coordinate are simultaneously determined by solving the equation 18 established from the coordinates of the three points and the line-of-sight direction vector S by Kramer's formula. Looking for. Alternatively, there is a method of obtaining an intersection coordinate after obtaining a plane equation, and then examining whether the intersection is in a triangular patch. Let the triangular patches p ₁ p ₂ p ₃ shown in FIG. 9 be triangular patches ABC. A = (a _x , a _y , a _z ), B = (b _x , b _y , b _z ) and C = (c _x , c _y , c _z ). The point Pw is assumed to be P (Wx, Wy, Wz) with the same coordinates. An equation of a plane including three points (A, B, C) in the space in the world coordinate system can be written as shown in Equation 24.

  In Expression 24, α, β, and γ are elements obtained from the outer product of the vector AB and the vector AC, and can be obtained as Expressions 25 to 27.

  When the visual line A of the driver 12 is expressed using the above-described eyeball rotation angle and eyeball position coordinates, it can be written as in Expression 28.

  From the simultaneous equations of Expression 24 and Expression 28, a linear expression (expression 29) for k can be derived.

  The intersection point P (Wx, Wy, Wz) between the plane ABC and the visual line S can be obtained by obtaining k in Expression 29 and substituting it into Expression 28. When the point P exists inside the plane ABC, it can be written as Equation 30.

  The intersection point P (Wx, Wy, Wz) that satisfies the condition of Expression 30 can be defined as the three-dimensional position coordinate of the gazing point.

  FIG. 10 is a flowchart showing the process flow of the binocular stereo method constituting the above-described three-dimensional range measurement program of the present invention. As shown in FIG. 10, first, an epipolar line is obtained as a set of pixels that satisfy the condition (predetermined condition) shown in Expression 6 for the left and right outside images recorded in the outside image DB 70, and the epipolar line is displayed on the epipolar line. By searching, corresponding points in the left and right outside images are obtained (step S50, corresponding point matching step). For the pixel for which the corresponding point is obtained in the corresponding point matching step (step S50), the position coordinates of the point in the three-dimensional space outside the vehicle projected onto the pixel are obtained, and for the pixel for which the corresponding point is not obtained, the pixel. Those skilled in the art obtain the coordinates of the point in the three-dimensional space outside the vehicle projected onto the pixel by performing interpolation from the neighboring pixels (step S52, three-dimensional coordinate calculation step). Conversion parameter p1 when the coordinates of the point in the three-dimensional space outside the vehicle obtained in the three-dimensional coordinate calculation step (step S52) are Euclidean transformed into the coordinates of the predetermined point in the world coordinate system recorded in the line-of-sight information DB 72 ,. . . , P12 (conversion parameter calculation step, step S54). The conversion parameters p1,... Obtained in the conversion parameter calculation step (step S54). . . , P12, the coordinates in the world coordinate system are calculated and recorded in the pixel coordinate information DB 74 for all the pixels on the left and right outside images recorded in the outside image DB 70 (step S56 or step S22 shown in FIG. 2). Pixel coordinate calculation step). A triangular patch is formed by a predetermined method for every three adjacent pixels recorded in the pixel coordinate information DB 74, and information relating to the triangular patch is recorded in the triangular patch information DB 76 (step S58 or step S24 shown in FIG. 2). Triangular patch composition step). The driver's 12 obtained from the eyeball rotation angle and eyeball position coordinates of the driver's 12 eyeballs 12L and the like when gazing at a predetermined point in the world coordinate system recorded in the line-of-sight information DB 72 and the coordinates of the predetermined point. A triangular patch having an intersection Pw between the line of sight S and the triangular patch recorded in the triangular patch information DB 76 is obtained by a predetermined method, the intersection is set as the three-dimensional coordinates of the driver's gazing point, and the position of the gazing point The coordinates are recorded in the gazing point position coordinate DB 78 with the passage of time (step S60 or step S40 shown in FIG. 2, intersection calculation step).

  As described above, according to the first embodiment of the present invention, the three-dimensional range measurement system 1 has the two outside cameras 20 that measure the environment outside the vehicle 10 in stereo (the coordinates are the traveling direction of the vehicle 10). Is the Zc axis, the Xc axis and the Yc axis perpendicular to the Zc axis), and the in-vehicle camera 30 that measures the direction of the driver's line of sight (the coordinates are the driver direction is the Zw axis and the Zw axis is perpendicular to the Zw axis, the Xw axis and the Yw axis) And an external camera 20 and a PC 40 connected to the in-vehicle camera 30. The wide-angle lens camera of the outside camera 20 is preferably a fisheye lens camera, but is not limited to this. The in-vehicle camera 20 acquires the eyeball rotation angle and eyeball position coordinates of the driver's 12 eyeballs 12L using an image sensor such as a CCD installed in the car.

  A face image of the driver 12 is picked up by the in-vehicle camera 30 and a line-of-sight measurement process is performed, and the eyeball position coordinates and the eyeball rotation angle of the driver 12 are obtained. In parallel, a vehicle outside image is captured by the vehicle outside camera 20 and the vehicle outside stereo processing is executed. Since the coordinate system of the in-vehicle camera 30 and the coordinate system of the outside camera 20 are different, in order to integrate the information of the driver's sight line direction and the information of the three-dimensional environment outside the vehicle, the left and right fisheye stereo cameras 30L It is necessary to unify coordinate systems such as the world coordinate system. In the three-dimensional range measurement program of the present invention, the coordinate system XwYwZw of the in-vehicle camera 30 is the world coordinate system, and the coordinate system XcYcZc of the in-vehicle camera 20 is matched to the coordinate system of the in-vehicle camera 30.

  Since the two sets of fish-eye stereo cameras (in-vehicle use and external-use use) used in the present invention do not have a common photographing area, camera calibration using a general calibration instrument is impossible. Therefore, the driver 12 pays attention to the marker placed in a predetermined position while sitting on the seat of the vehicle 10, and information on the driver's line-of-sight direction and the marker that can be obtained from the in-vehicle camera 30 (faceLAB) at that time Devised a method for obtaining calibration parameters that integrate the camera coordinate system from the three-dimensional coordinates in the fish-eye camera coordinate system. A triangular patch is formed for every three adjacent pixels on the image of the outside environment obtained from the measurement by the outside camera 20. Next, the intersection of the driver's line of sight obtained from the measurement by the in-vehicle camera 30 and the triangular patch is set as the three-dimensional coordinates of the driver's gazing point. After various information is recorded in the recording device 42, the above-described processing is repeated.

  The present invention has devised a binocular stereo method that does not convert to a perspective projection model using a fisheye lens camera of equidistant projection. First, an epipolar line is obtained as a set of pixels satisfying the condition shown in Expression 6 with respect to the left and right outside images recorded in the outside image DB 70, and corresponding points in the left and right outside images are obtained by searching on the epipolar line. . For the pixel for which the corresponding point is obtained, the position coordinates of the point in the three-dimensional space outside the vehicle projected onto the pixel are obtained, and for the pixel for which the corresponding point is not obtained, interpolation is performed from the surrounding pixels of the pixel. A person skilled in the art obtains the coordinates of a point in a three-dimensional space outside the vehicle projected onto the pixel. Conversion parameters p1,... When converting the coordinates of the obtained point in the three-dimensional space outside the vehicle into the coordinates of the predetermined point in the world coordinate system recorded in the line-of-sight information DB 72. . . , P12. The obtained conversion parameters p1,. . . , P12, the coordinates in the world coordinate system are calculated and recorded in the pixel coordinate information DB 74 for all the pixels on the left and right outside images recorded in the outside image DB 70. A triangular patch is formed by a predetermined method for every three adjacent pixels recorded in the pixel coordinate information DB 74, and information related to the triangular patch is recorded in the triangular patch information DB 76. The driver's 12 obtained from the eyeball rotation angle and eyeball position coordinates of the driver's 12 eyeballs 12L and the like when gazing at a predetermined point in the world coordinate system recorded in the line-of-sight information DB 72 and the coordinates of the predetermined point. A triangular patch having an intersection Pw between the line of sight S and the triangular patch recorded in the triangular patch information DB 76 is obtained by a predetermined method, the intersection is set as the three-dimensional coordinates of the driver's gazing point, and the position of the gazing point The coordinates are recorded in the gazing point position coordinate DB 78 with the passage of time.

  As described above, it is possible to provide a three-dimensional range measurement system or the like that can measure the spread of the three-dimensional recognition area of the driver 12 such as the vehicle 10.

  In the second embodiment, a method for generating a three-dimensional convex hull from the gazing point of the driver 12 using the gazing point position coordinate DB 78 obtained in the first embodiment will be described. A three-dimensional convex hull composed of a line-of-sight direction vector (gaze vector) between the past t frames with the eyeball 12L of the driver 12 as a base point is defined as an effective visual space. Since the vehicle is traveling even during the measurement of the effective visual space, the reference line S extending from the driver's eyeball position (gaze measurement camera coordinate system) is used as a reference, and the relative position of the three-dimensional position of the gazing point from there is determined. Build an effective visual space from changes. A three-dimensional convex hull is constructed from the position vector group of the gazing point, and this internal region is obtained as an effective visual space.

  FIG. 11 is a diagram for explaining an effective visual space according to the second embodiment of the present invention. In FIG. 11A, reference numeral 100 denotes a hazard (an object outside the vehicle that is particularly necessary for the driver 12 to pay attention), and 102 is an effective visual space. FIG. 11B shows the effective viewing section 102xz when projected onto the xz plane (XcZc plane in the coordinate system of the outside camera 20 in the first embodiment), and FIG. 11C shows the yz plane (in the first embodiment). An effective viewing section 102 yz when projected onto the YcZc plane in the coordinate system of the outside camera 20 is shown. As shown in FIG. 11A, it can be seen that the driver 12 recognizes the hazard 100 because the hazard 100 is included in the effective visual space 102 of the driver 12.

  The effective visual space calculation unit 57 (effective visual space calculation means) of the functional block 50 shown in FIG. 3 is composed of gaze point coordinates for the past t frames (for a predetermined time) recorded in the gaze point position coordinate DB 78. The three-dimensional convex hull is obtained as the driver's effective visual space. That is, by generating a three-dimensional convex hull from a time series of gazing point groups, that region is made an effective visual space. Details will be described below.

When the gazing point coordinates for t frames are X = {x ₁ , x ₂ ,..., x _t }, the three-dimensional convex hull formed from these is set as the effective viewing space of the driver 12. Specifically, in a set of gazing point coordinates,

All convex combinations that satisfy

Find the common part (convex hull) C of. The data structure of the convex hull considers the boundary of the three-dimensional convex polyhedron as a plane graph and holds it in a doubly linked list. FIG. 12 is a flowchart showing an algorithm for obtaining a three-dimensional convex hull by the sequential addition method (see Non-Patent Document 7). As shown in FIG. 12, a convex hull (tetrahedron) C (X ₄ ) is constituted by four points x ₁ , x ₂ , x ₃ , x ₄ (step S 70). m = 5 is set (step S72). The points _{x m} of the m-th addition to _{C (X m-1)} to create a C _{(X m)} (step S74). If x _m is inside C (X _m−1 ), x _m is a point inside the convex hull C (X _m ), so the process proceeds to step S80 without doing anything (step S76). Otherwise, the visible surface is deleted and a surface that can be composed of the side of the ground plane and x _m is added as a new facet (step S78). If m ≠ t in step S80, m is incremented by 1 and the process returns to step S74 (steps S80 and S82).

Determining the Visible Surface: {(x _i −x _j ) × (x _j −x _k )} · (x _i −x _m )> in the facets of the effective linked list in the order of x _i , x _j and x _k If 0, the facet is visible.
Determination of the horizontal plane: the neighborhood is a side of the horizontal plane if the invisible face against facet sharing deleted visible surface and sides x _m.

FIG. 13 illustrates the state of sequential addition. FIG. 13A shows a state where the _m-th point x _m is added to the convex hull C (X _m−1 ), and FIG. 13B shows that the _m-th point x _m is the convex hull C (X _m− was added to ₁₎ shows a state where the convex hull C _{(X m)} is generated.

  As described above, according to the second embodiment of the present invention, the three-dimensional convex hull composed of the gaze direction vector (gaze vector) between the past t frames with the eyeball 12L of the driver 12 as a base point is effective. It is defined as visual space. If the hazard 100 is included in the effective visual space 102 of the driver 12, it can be understood that the driver 12 recognizes the hazard 100. The effective visual space calculation unit 57 obtains a three-dimensional convex hull composed of gazing point coordinates for the past t frames (a predetermined time) recorded in the gazing point position coordinate DB 78 as the driver's effective visual space. That is, by generating a three-dimensional convex hull from a time series of gazing point groups, that region is made an effective visual space. The three-dimensional convex hull can be obtained by the sequential addition method. As described above, it is possible to provide a three-dimensional range measurement system and the like that can measure a temporal change in a three-dimensional effective visual field (effective visual space) that changes depending on mental factors such as the psychological state of the driver 12. .

  In the third embodiment, an example in which the effective visual space described in the second embodiment is displayed to the driver 12 will be described. The effective visual space display unit 58 (effective visual space display means) of the functional block 50 shown in FIG. 3 obtains an orthogonal projection image on the road surface of the effective visual space calculated by the effective visual space calculation unit 57, and the computer 40. The orthogonal projection image is displayed in a predetermined manner on a display device 84 (described later) connected to the.

  FIG. 14 shows a display example of the effective visual space according to the third embodiment of the present invention. The area of the effective viewing space 102 is represented by color (original drawing), and is displayed in green when a sufficiently wide effective viewing space is maintained (a wide and distant state in FIG. 14A, a wide view in FIG. 14B). When the effective visual space is narrow, a warning is displayed in red (narrow and close state in FIG. 14C, narrow and far state in FIG. 14D). If there is a hazard (such as a pedestrian in the direction of the vehicle and having a possibility of collision) outside the effective visual space, a pictogram (bicycle 100a in FIG. 14C, pedestrian 100b in FIG. 14D) ) Is displayed to alert the driver 12. However, the maximum depth of the effective visual space to be displayed depends on the stereo measurement accuracy outside the vehicle. When using a camera that can accurately measure far away, the effective viewing space that can be displayed also becomes large. The left and right extent of the effective visual space to be displayed also depends on the stereo measurement performance outside the vehicle. In the case of a fisheye camera that can accurately measure a wide field of view, the left and right spread that can be displayed is also widened.

  As described above, according to the third embodiment of the present invention, the orthogonal projection image on the road surface of the effective visual space calculated by the effective visual space calculation unit 57 is obtained, and the display device 84 (described later) connected to the computer 40 is obtained. ) Is displayed in a predetermined manner. As a result, the horizontal viewing angle, the vertical viewing angle, and the depth can be displayed as an image, and the size of the effective viewing space can be expressed by color and an oversight hazard can be displayed.

  In the fourth embodiment, various driving support mechanisms using the effective visual space described in the second embodiment will be described. The hazard coordinate detection unit 59 (hazard detection means) of the functional block 50 shown in FIG. 3 detects the coordinates of the hazard that exists in the range of the three-dimensional space that can be seen from the driver 12 of the vehicle 10, and the warning unit 59 (warning The means) issues a predetermined warning based on the hazard coordinates detected by the hazard coordinate detection unit 59 and the gaze point coordinates for a predetermined time recorded in the gaze point position coordinate DB 78.

  The hazard coordinate detection unit 59 detects a hazard outside the vehicle 10 from a vehicle outside image recorded in the vehicle outside image DB 70 by a predetermined moving body analysis method, specifies the coordinate system of the vehicle outside camera 20, and then the coordinates are described above. The coordinate system of the in-vehicle camera 30 is adjusted in the same manner as the method. As the predetermined moving body analysis method, an analysis method using an optical flow algorithm which is an existing technique is suitable. The inventors used the Halcon® machine vision library function (optical_flow_mg ()). For details, refer to the Halcon (registered trademark) reference manual (http: //www/mvtec.com/download/documentation/reference-9.0/hdevelop/optical_flow_mg.html).

  By using the hazard coordinates detected by the hazard coordinate detection unit 59 and the gaze point coordinates for a predetermined time recorded in the gaze point position coordinate DB 78, it is specified whether or not the driver 12 was watching the hazard. can do. In other words, it can be specified that the driver 12 has overlooked a certain hazard. Since the three-dimensional coordinates of the hazard can be obtained as described above, the distance between the vehicle 10 and the hazard can also be measured without using an expensive laser device. Therefore, it is possible to obtain the time until the collision with the hazard from the distance and the traveling speed of the vehicle 10. For this reason, a more accurate warning (audio or image display) can be issued. In addition, since it is possible to measure the line of sight of the driver 12, it is possible to detect a distraction in which the driver 12 is distracted by instruments, passengers, etc. and does not look forward.

  As described above, according to the fourth embodiment of the present invention, the hazard coordinates detected by the hazard coordinate detection unit 59 and the gaze point coordinates for a predetermined time recorded in the gaze point position coordinate DB 78 are used. Various accurate warnings can be issued. For this reason, by mounting the three-dimensional range measurement system of the present invention on the vehicle 10, it is possible to provide the vehicle 10 with higher safety and lower cost.

  FIG. 15 is a block diagram showing an internal circuit 80 of a PC (computer) 40 that executes the three-dimensional range measurement program of the present invention. As shown in FIG. 15, the CPU 81, ROM 82, RAM 83, image control unit 86, controller 87, input control unit 90, and external I / F unit 92 are connected to a bus 93. In FIG. 15, the above-described three-dimensional range measurement program of the present invention is recorded on a recording medium (including a removable recording medium) such as ROM 82, disk 88 (or 44), DVD or CD-ROM 89. On the disk 88 (or 44), the above-described various vehicle outside image DB 70, line-of-sight information DB 72, pixel coordinate information DB 74, triangle patch information DB 76, gazing point position coordinate DB 78, and the like can be recorded. The three-dimensional range measurement program is loaded into the RAM 83 from the ROM 82 via the bus 93 or from a recording medium such as the disk 88 or DVD or CD-ROM 89 via the controller 87 via the bus 93. The image control unit 86 sends image data of various screens displayed by the effective visual space display unit 58 or the warning unit 60 to the VRAM 85. A display device (display) 84 displays the above data sent from the VRAM 85. The VRAM 85 is an image memory having a capacity corresponding to the data capacity of one screen of the display device 84. The input operation unit 91 is an input device such as a mouse or a keyboard for inputting to the PC 40, and the input control unit 90 is connected to the input operation unit 91 and performs input control. The external I / F unit 92 has an interface function when connecting to the outside camera 20 and the inside camera 30 outside the PC 40.

  As described above, when the computer (CPU) 81 executes the three-dimensional range measurement program of the present invention, the object of the present invention can be achieved. As described above, the three-dimensional range measurement program can be supplied to the computer (CPU) 81 in the form of a recording medium such as a DVD or CD-ROM 89, and can be recorded on the DVD or CD-ROM 89 or the like on which the three-dimensional range measurement program is recorded. The medium similarly constitutes the present invention. As a recording medium on which the three-dimensional range measurement program is recorded, for example, a memory card, a memory stick, an optical disk, or the like can be used in addition to the recording medium described above.

  In the description of each embodiment described above, the prior camera calibration (step S1) in FIG. 2 has not been mentioned. Also in the present invention, camera calibration is performed in advance using an existing method, and internal parameters of the camera (which includes focal length, lens distortion, pixel size, etc.) are obtained in advance. On the premise that this calibration can be performed in advance, the above-described Equation 1 uses ul, vl, etc. normalized by the internal parameters of the camera (excluding effects such as lens distortion). The numerical values obtained by the camera calibration in advance are also used for the focal lengths f in Equations 1 and 2.

  As an application example of the present invention, it can be applied to an accident prevention tool. When we look at traffic accidents in Japan by violation of laws and regulations, it is said that there are about 30% of unconfirmed safety. Regarding the relationship between accidents and visual perception characteristics, it has been reported that there is a high correlation between the effective vision of middle-aged and elderly people and the accident rate. For this reason, using the 3D range measurement system 1 or the like of the present invention (particularly the function of the effective visual space display unit 58), it is necessary to overestimate by constantly grasping its own effective visual space according to traffic conditions. It is possible to keep a safe driving style in mind.

  1 3D range measurement system, 10 vehicle, 12 driver, 12L, 12R eyeball, 20, 25L, 25R exterior camera, 30, 30L, 30R in-vehicle camera, 40 PC, 42 recording device, 50 function block, 51 corresponding point matching Unit, 52 three-dimensional coordinate calculation unit, 53 conversion parameter calculation unit, 54 pixel coordinate calculation unit, 55 triangular patch configuration unit, 56 intersection calculation unit, 57 effective visual space calculation unit, 58 effective visual space display unit, 59 hazard coordinate detection Section, 60 warning section, 70 vehicle exterior image DB, 72 line-of-sight information DB, 74 pixel coordinate information DB, 76 triangle patch information DB, 78 gaze point position coordinate DB, 81 CPU, 82 ROM, 83 RAM, 84 display device, 85 VRAM , 86 Image control unit, 87 Control La, 88 disk, 89 CD-ROM, 80 an input control unit, 91 input operation unit, 92 external I / F unit, 93 a bus 100 hazard 102 effective visual space.

Tatsuya Nakakura, Takashi Irikura, “Relationship between fatigue and effective vision”, Journal of the Illuminating Engineering Society of Japan, Vol. 89, No. 11, 2005. McConkie George W. and RaynerKeith: “The span of the effective stimulus during a fixationsin reading”, Perceptionand Psychophisics, 17, pp.578-586,1975. Toshiaki Miura, “Visual Cognition in Behavioral Scenes”, Proceedings of the 48th Annual Meeting of the Psychological Society of Japan, S68-69, 1984. Yasuo Tachibana, Toshikatsu Kawai, Yosuke Kobayashi, Tetsuya Kurihara, Takayuki Kikuchi, "Conversion from Fisheye Lens Photo to Arbitrary Focus Lens Image", IEICE Technical Report. SIP, Signal Processing: IEICE technical report107, No.22, pp.25-30, 2007. Shota Kase, Ryota Okutsu, Naotsu Mitsumoto, Yohei Aragaki, Ryoko Shimomura, Kenji Terabayashi, and Kazunobu Umeda, "Overhead Image Generation by Identifying Internal and External Parameters of Multiple Fisheye Cameras", IEEJ Study Material. IP, Information Processing Society of Japan 2008, No.13, pp.43-48, 2008. Nagamori Chiki, Nishimoto Takeshi, Yamaguchi Junichi, "Fisheye Stereo Vision Sensor", Proceedings of the 12th Image Sensing Symposium, pp.137-140, 2006. K.L.Clarkson and P.W.Shor: “Applications of random sampling incomputational geometry, II”, Discrete Computer Geometry, No.4, pp.387-421, 1989.

Claims (13)

A three-dimensional range measurement system that measures a range of a three-dimensional space that can be seen by a driver of a vehicle, and that measures an outside image sensor having two wide-angle lens cameras that measure the outside environment in stereo and the direction of the driver's line of sight An in-vehicle imager, and an external imager and a computer connected to the in-vehicle imager, the computer comprising:
The coordinate system of the in-vehicle imager is the world coordinate system, the coordinate system of the in-vehicle imager is matched with the coordinate system of the in-vehicle imager, and the driver's line of sight obtained from the measurement of the in-vehicle imager and the out-of-vehicle imager The intersection of the triangle patch set for every three adjacent pixels on the image of the environment outside the vehicle obtained from the above measurement is set as the three-dimensional coordinates of the driver's point of sight, so that the three-dimensional space visible to the driver A three-dimensional range measurement system characterized by measuring a range.   2. The three-dimensional range measurement system according to claim 1, wherein the image sensor outside the vehicle uses an equidistant projection left and right fisheye lens camera installed outside the vehicle as a wide-angle lens camera, and the image sensor inside the vehicle is installed inside the vehicle. A three-dimensional range measurement system that acquires a rotation angle and position coordinates of both eyes of a driver using 3. The three-dimensional range measurement system according to claim 2, wherein the computer is
A vehicle exterior image recording unit that records left and right exterior images of the left and right fisheye lens cameras imaged using the exterior vehicle;
Line-of-sight information recorded using the in-vehicle imager in association with the rotation angle and position coordinates of the driver's eyes when watching a predetermined point in the world coordinate system and the coordinates of the predetermined point A recording section;
Corresponding point matching that obtains an epipolar line as a set of pixels that satisfy a predetermined condition with respect to the left and right outside images recorded in the outside image recording unit, and obtains corresponding points in the left and right outside images by searching on the epipolar line Means,
For the pixel for which the corresponding point is obtained by the corresponding point matching means, the position coordinates of the point in the three-dimensional space outside the vehicle projected onto the pixel are obtained, and for the pixel for which the corresponding point is not obtained, the peripheral pixels of the pixel Three-dimensional coordinate calculation means for obtaining coordinates of a point in a three-dimensional space outside the vehicle projected onto the pixel by performing interpolation from
A conversion parameter is obtained when the coordinates of the point in the three-dimensional space outside the vehicle obtained by the three-dimensional coordinate calculation means are converted into the coordinates of the predetermined point in the world coordinate system recorded in the line-of-sight information recording unit. Conversion parameter calculation means;
Using the conversion parameters obtained by the conversion parameter calculation means, the coordinates in the world coordinate system are calculated and recorded in the pixel coordinate information recording unit for all the pixels on the left and right outside images recorded in the outside image recording unit. Pixel coordinate calculating means for
A triangular patch forming unit configured to form a triangular patch in a predetermined manner for every three adjacent pixels recorded in the pixel coordinate information recording unit, and record information about the triangular patch in the triangular patch information recording unit;
The driver's line of sight obtained from the rotation angle and position coordinates of the driver's eyes when watching a predetermined point in the world coordinate system recorded in the line-of-sight information recording unit, and the coordinates of the predetermined point; A triangular patch having an intersection with the triangular patch recorded in the triangular patch information recording unit is obtained by a predetermined method, the intersection is set as a three-dimensional coordinate of the driver's gazing point, and the position coordinate of the gazing point is a passage of time. A three-dimensional range measurement system comprising: an intersection point calculation means for recording in the gazing point position coordinate recording unit.   4. The three-dimensional range measurement system according to claim 3, wherein an effective vision for obtaining a three-dimensional convex hull composed of gaze point coordinates for a predetermined time recorded in the gaze point position coordinate recording unit as an effective visual space for a driver. A three-dimensional range measurement system further comprising space calculation means.   5. The three-dimensional range measurement system according to claim 4, wherein an orthographic image on the road surface of the effective visual space calculated by the effective visual space calculating means is obtained, and the orthographic image is displayed on a display device connected to the computer. A three-dimensional range measurement system further comprising effective visual space display means for displaying in a predetermined manner. The three-dimensional range measurement system according to any one of claims 3 to 5,
Hazard coordinate detection means for detecting the coordinates of the hazard present in the range of the three-dimensional space visible to the driver of the vehicle;
Warning means for issuing a predetermined warning based on the hazard coordinates detected by the hazard coordinate detection means and the gazing point coordinates for a predetermined time recorded in the gazing point position coordinate recording unit. Characteristic 3D range measurement system. A three-dimensional range measurement program for measuring a range of a three-dimensional space that can be seen by a driver of a vehicle, and measuring an outside image pickup device having two wide-angle lens cameras for measuring the outside environment in stereo, and a driver's gaze direction An in-vehicle imager, and an external imager and a computer connected to the in-vehicle imager are used.
The coordinate system of the in-vehicle imager is the world coordinate system, the coordinate system of the in-vehicle imager is matched with the coordinate system of the in-vehicle imager, and the driver's line of sight obtained from the measurement of the in-vehicle imager and the out-of-vehicle imager The intersection of the triangle patch set for every three adjacent pixels on the image of the environment outside the vehicle obtained from the above measurement is set as the three-dimensional coordinates of the driver's point of sight, so that the three-dimensional space visible to the driver A three-dimensional range measurement program for executing range measurement.   8. The three-dimensional range measurement program according to claim 7, wherein the image sensor outside the vehicle uses an equidistant projection type left and right fisheye lens camera installed outside the vehicle as a wide-angle lens camera, and the image sensor inside the vehicle is installed inside the vehicle. A three-dimensional range measurement program for acquiring a rotation angle and position coordinates of both eyes of a driver using The three-dimensional range measurement program according to claim 8, wherein the computer includes:
A vehicle exterior image recording unit that records left and right exterior images of the left and right fisheye lens cameras imaged using the exterior vehicle;
Line-of-sight information recorded using the in-vehicle imager in association with the rotation angle and position coordinates of the driver's eyes when watching a predetermined point in the world coordinate system and the coordinates of the predetermined point Further using the recording unit, the computer,
Corresponding point matching that obtains an epipolar line as a set of pixels that satisfy a predetermined condition with respect to the left and right outside images recorded in the outside image recording unit, and obtains corresponding points in the left and right outside images by searching on the epipolar line Step,
For the pixel for which the corresponding point is obtained in the corresponding point matching step, the position coordinates of the point in the three-dimensional space outside the vehicle projected onto the pixel are obtained, and for the pixel for which the corresponding point is not obtained, the peripheral pixels of the pixel A three-dimensional coordinate calculation step for obtaining coordinates of a point in a three-dimensional space outside the vehicle projected onto the pixel by performing interpolation from
A conversion parameter for obtaining the coordinates of a predetermined point in the world coordinate system recorded in the line-of-sight information recording unit by Euclidean transformation of the coordinates of the point in the three-dimensional space outside the vehicle obtained in the three-dimensional coordinate calculation step is obtained. Conversion parameter calculation step,
Using the conversion parameters obtained in the conversion parameter calculation step, the coordinates in the world coordinate system are calculated and recorded in the pixel coordinate information recording unit for all the pixels on the left and right outside images recorded in the outside image recording unit. A pixel coordinate calculating step,
A triangular patch configuration step of configuring a triangular patch in a predetermined manner for every three adjacent pixels recorded in the pixel coordinate information recording unit, and recording information about the triangular patch in the triangular patch information recording unit,
The driver's line of sight obtained from the rotation angle and position coordinates of the driver's eyes when watching a predetermined point in the world coordinate system recorded in the line-of-sight information recording unit, and the coordinates of the predetermined point; A triangular patch having an intersection with the triangular patch recorded in the triangular patch information recording unit is obtained by a predetermined method, the intersection is set as a three-dimensional coordinate of the driver's gazing point, and the position coordinate of the gazing point is a passage of time. A three-dimensional range measurement program that executes an intersection calculation step that is recorded in the gazing point position coordinate recording unit.   10. The three-dimensional range measurement program according to claim 9, wherein an effective vision for obtaining a three-dimensional convex hull composed of gaze point coordinates for a predetermined time recorded in the gaze point position coordinate recording unit as an effective visual space for a driver. A three-dimensional range measurement program further comprising a space calculation step.   The three-dimensional range measurement program according to claim 10, wherein an orthographic image on the road surface of the effective visual space calculated by the effective visual space calculation step is obtained, and the orthographic image is displayed on a display device connected to the computer. A three-dimensional range measurement program further comprising an effective visual space display step for displaying in a predetermined method. In the three-dimensional range measurement program according to any one of claims 9 to 11,
A hazard coordinate detection step for detecting the coordinates of the hazard existing within the range of the three-dimensional space visible to the driver of the vehicle;
A warning step for issuing a predetermined warning based on the hazard coordinates detected by the hazard coordinate detection step and the gaze point coordinates for a predetermined time recorded in the gaze point position coordinate recording unit; Characteristic 3D range measurement program. A computer-readable recording medium on which the three-dimensional range measurement program according to any one of claims 7 to 12 is recorded.