JP6699523B2 - Eye tracking device and imaging system - Google Patents

Description

  The present invention relates to an eye tracking device that focuses an image on a driver's eye of a vehicle and performs imaging, and an imaging system including the eye tracking device.

  Patent Document 1 discloses an iris image pickup apparatus that picks up an image of an iris used for personal authentication of a security system. The iris image pickup device of Patent Document 1 has one wide-angle camera for picking up an upright subject, a rangefinder for irradiating infrared light to measure the distance to the subject, and an image of the iris of the subject. It is equipped with a telephoto camera. The iris imaging device of Patent Document 1 obtains the position of the subject's eye with respect to the telephoto camera from the position of the subject's eye in the image of the wide-angle camera and the subject distance measured by a rangefinder. Then, the pan position and tilt position of the telephoto camera are finely adjusted so that the obtained eye position is in the center of the captured image. In addition, drive control is performed so that the focus lens is focused on the subject distance measured by the distance meter.

JP, 2002-125142, A

  However, the iris image pickup device of Patent Document 1 is not supposed to be used in a vehicle, as it is supposed to be a standing subject. Therefore, no consideration is given to a mechanical error such as a deviation of the mechanism itself that performs panning, tilting, focusing, and the like of the telephoto camera, which is caused by a load such as vibration when using the vehicle. Therefore, when the vehicle is used, there is a problem that it is difficult to accurately focus on the eyes of the driver of the vehicle due to this mechanical error.

  The present invention has been made based on this situation, and an object of the present invention is to accurately focus on the eyes of the driver of the vehicle in an imaging system that focuses and images the eyes of the driver of the vehicle. It is an object of the present invention to provide an eye tracking device and an imaging system capable of facilitating eye movement matching.

  The above objective is achieved by a combination of features described in independent claims, and the subclaims define further advantageous embodiments of the invention. The reference numerals in parentheses in the claims indicate the correspondence with the specific means described in the embodiments described below as one aspect, and do not limit the technical scope of the present invention. ..

  In order to achieve the above object, the eye tracking device of the present invention is used in a vehicle, and a wide-angle camera (20) whose imaging direction is fixed so as to image a range including a face portion when a driver sits in the driver's seat of the vehicle. ), a wide-angle image acquisition unit (101) that sequentially acquires wide-angle images, and a pan-tilt mechanism (33) and a focus that have a fixed installation interval with respect to the wide-angle camera and have a narrower angle of view than the wide-angle camera. Based on the telephoto image acquisition unit (102) that sequentially acquires a telephoto image that is an image captured by the telephoto camera (30) having the mechanism (32), and the position of the driver's eye in the wide-angle image acquired by the wide-angle image acquisition unit. An adjusting unit (113) for operating the pan-tilt mechanism and the focus mechanism so as to magnify and image the driver's eye with the telephoto camera, the wide-angle image acquired by the wide-angle image acquisition unit, and the telephoto image acquired by the telephoto image acquisition unit. Of the pan-tilt mechanism based on the deviation between the parallax specifying unit (105) that sequentially specifies the parallax of the pan-tilt mechanism and the actual parallax specified by the parallax specifying unit when there is no mechanism error of the pan-tilt mechanism. A pan-tilt correction unit (111) that sequentially determines a correction amount of the pan-tilt mechanism according to the adjustment unit, and the adjustment unit, when operating the pan-tilt mechanism and the focus mechanism so as to magnify and image the driver's eye with the telephoto camera, The pan-tilt mechanism is operated according to the correction amount that is sequentially determined by the pan-tilt correction unit.

  According to this, the correction amount of the pan-tilt mechanism is calculated based on the difference between the parallax between the wide-angle image of the wide-angle camera and the telephoto image of the telephoto camera and the actual parallax when there is no mechanism error of the pan-tilt mechanism. The pan-tilt correction unit sequentially determines. Since the installation distance between the wide-angle camera and the telephoto camera is fixed, the parallax shift described above has a value corresponding to the mechanical error of the pan-tilt mechanism. Therefore, based on this deviation, it becomes possible to sequentially determine the correction amount of the pan-tilt mechanism according to the mechanical error of the pan-tilt mechanism. Further, the adjusting unit operates the pan-tilt mechanism according to the correction amount that is sequentially determined by the pan-tilt correcting unit when operating the pan-tilt mechanism and the focus mechanism so that the telephoto camera magnifies and images the driver's eyes. Therefore, even if a mechanical error of the pan-tilt mechanism of the telephoto camera occurs due to a load such as vibration during vehicle use, the deviation of the imaging direction due to this mechanical error is suppressed, and the telephoto camera magnifies the driver's eye for imaging. It is possible to make it easier. As a result, in the imaging system that focuses the image on the driver's eye of the own vehicle, it is possible to easily focus on the driver's eye of the own vehicle accurately.

  Further, in order to achieve the above object, the imaging system of the present invention is used in a vehicle, and a wide-angle camera whose imaging direction is fixed so as to image a range including a face portion when a driver sits in a driver's seat of the vehicle ( 20), a telephoto camera (30) whose position with respect to the wide-angle camera is fixed and whose angle of view is narrower than that of the wide-angle camera, and which has a pan-tilt mechanism (33) and a focus mechanism (32), and the above-mentioned eye tracking device. (10) is included.

  According to this, since the above-described eye tracking device is included, it is possible to facilitate accurate focusing on the eyes of the driver of the own vehicle in an imaging system that focuses on the eyes of the driver of the own vehicle. .

FIG. 1 is a diagram showing an example of a schematic configuration of an imaging system 1. It is a figure which shows an example of a schematic structure of the telephoto camera 30. 3 is a diagram showing an example of a schematic configuration of a control unit 10. FIG. It is a figure for explaining an example of pattern matching of a wide-angle image and a telephoto image. FIG. 6 is a diagram for describing an example in which an area that can be actually imaged by the telephoto camera 30 deviates from an area that is an object of imaging. 6 is a flowchart showing an example of the flow of eye tracking related processing in the control unit 10.

  A plurality of embodiments for disclosure will be described with reference to the drawings. Note that, for convenience of description, between a plurality of embodiments, parts having the same functions as the parts shown in the drawings used in the above description are denoted by the same reference numerals, and the description thereof may be omitted. is there. For the parts denoted by the same reference numerals, the description in other embodiments can be referred to.

(Embodiment 1)
<Schematic configuration of imaging system 1>
Hereinafter, the present embodiment will be described with reference to the drawings. The imaging system 1 shown in FIG. 1 is used in a vehicle such as an automobile, and includes a control unit 10, a wide-angle camera 20, a telephoto camera 30, a wide-angle illumination 40, and a telephoto illumination 50.

  The imaging system 1 is a so-called DSM (Driver Status Monitor), and is provided in a position where the wide-angle camera 20 and the telephoto camera 30 can image a range including the face portion of the driver seated in the driver's seat in the vehicle interior of the vehicle. Be done. As an example, the configuration may be such that it is installed on the steering column cover, the upper surface of the instrument panel, or the like, and the image is taken from the front side of the driver sitting in the driver's seat.

  The wide-angle camera 20 is a camera whose imaging direction is fixed so as to capture an image of a range including the face portion of the driver sitting in the driver's seat from the installation position of the imaging system 1. When the driver is seated in the driver's seat, the wide-angle camera 20 images the range including the face portion of the driver. As an example, the frame rate of the wide-angle camera 20 may be 30 Hz or the like. The wide-angle camera 20 outputs a captured image (hereinafter, a wide-angle image) captured to the control unit 10.

  The telephoto camera 30 is a camera having a narrower angle of view than the wide-angle camera 20, and is a camera for enlarging and imaging an eye of a driver seated in a driver's seat. As an example, the frame rate of the telephoto camera 30 may be 200 Hz or the like. The telephoto camera 30 outputs the picked-up image (hereinafter, telephoto image) to the control unit 10. Unlike the wide-angle camera 20 whose image pickup direction is fixed, the telephoto camera 30 can adjust the image pickup direction. Further, in the imaging system 1, the telephoto cameras 30 are arranged side by side at a fixed interval in the left-right direction of the vehicle with respect to the wide-angle camera 20.

  As shown in FIG. 2, the telephoto camera 30 includes a telephoto camera body 31, a focus mechanism 32, and a pan/tilt mechanism 33. The telephoto camera body 31 is a device body of the telephoto camera 30 having an image pickup element built therein. In the present embodiment, the telephoto camera body 31 is provided such that its optical axis is parallel to the arrangement direction of the wide-angle camera 20 and the telephoto camera 30, that is, the left-right direction of the vehicle.

  The focus mechanism 32 is a mechanism for changing the focus of the telephoto camera 30. The focus mechanism 32 changes the focus of the telephoto camera 30 by changing the distance between the lens forming the lens unit and the image pickup device of the telephoto camera body 31 using a motor, similarly to a known focus mechanism. It may be configured to. In the present embodiment, a case where a stepping motor is used as this motor will be described as an example.

  The pan/tilt mechanism 33 is a mechanism for changing the imaging direction of the telephoto camera 30 to the pan direction and the tilt direction. The pan/tilt mechanism 33 includes a mirror for causing the reflected light to enter the telephoto camera body 31, and a pan motor and a tilt motor for driving the mirror. The mirror is rotatable in a horizontal direction and a vertical direction when the imaging system 1 is provided on a horizontal plane. The pan motor is a motor for rotating the mirror in the horizontal direction. The tilting motor is a motor for rotating the mirror in the vertical direction. In the present embodiment, a case in which a stepping motor is used as the pan motor and the tilt motor will be described as an example. In addition, in the imaging system 1, the wide-angle camera 20 and the telephoto camera 30 are assumed to be assembled so that there is no parallax in the vertical direction when assembled.

  The telephoto camera body 31 outputs not only the information of the captured image, but also the information related to the focus by the focus mechanism (hereinafter, focus related information) to the control unit 10. For example, the focusing-related information includes the presence/absence of focusing and the amount of deviation from focusing. As an example, the telephoto camera 30 has an autofocus (AF) function, and the image pickup device of the telephoto camera body 31 may detect whether or not there is focus and the amount of deviation from the focus. If the AF function uses a known phase difference detection method, the amount of deviation from the focus may be detected by measuring the distance between two images generated by the separator lens with a line sensor. Further, when the AF function uses a known contrast detection method, it is possible to detect the amount of deviation from the focus based on the change in the contrast of the captured image obtained by driving the lens of the focus mechanism. Good.

  The wide-angle illumination 40 is provided so as to irradiate the imaging range of the wide-angle camera 20 with near-infrared light. As an example, a plurality of wide-angle illuminations 40 may be provided in the arrangement direction of the wide-angle camera 20 and the telephoto camera 30, as shown in FIG. The telephoto illumination 50 is provided so that near-infrared light is condensed and emitted to the imaging range according to the movement of the imaging range of the telecamera 30. As an example, the telephoto illumination 50 may be configured to collect light with a condenser lens. Further, the telescopic illumination 50 condenses the near-infrared light in the image capturing range according to the movement of the image capturing range of the telephoto camera 30 by the mechanism for tilting the irradiation direction of the telescopic illumination 50 in the pan direction and the tilt direction. Then, the irradiation may be performed. LEDs may be used as the wide-angle illumination 40 and the telephoto illumination 50. The wide-angle illumination 40 and the telephoto illumination 50 correspond to the illumination device in the claims. Note that the near-infrared light may be emitted only from the wide-angle illumination 40 among the wide-angle illumination 40 and the telephoto illumination 50 without providing the telephoto illumination 50.

  The control unit 10 includes a processor, a volatile memory, a non-volatile memory, an I/O, and a bus connecting these, and executes various processes by executing a control program stored in the non-volatile memory. The control unit 10 corresponds to the eye tracking device in the claims. For example, the control unit 10 sequentially adjusts the image capturing direction of the telephoto camera 30 to track the driver's eyes and sequentially capture images. The control unit 10 also detects the line of sight of the driver based on the captured image of the driver's eye captured by the telephoto camera 30. In addition, the control unit 10 adjusts the light emission intensity of the wide-angle illumination 40 and the telephoto illumination 50. Details of the control unit 10 will be described later.

<Schematic configuration of control unit 10>
Subsequently, a schematic configuration of the control unit 10 will be described with reference to FIG. As illustrated in FIG. 3, the control unit 10 includes a wide-angle image acquisition unit 101, a telephoto image acquisition unit 102, an image processing unit 103, a memory 104, a parallax identification unit 105, a line-of-sight detection unit 106, a focus-related information acquisition unit 107, and A fluctuation amount specifying unit 108, a distance estimating unit 109, an illumination control unit 110, a pan/tilt correcting unit 111, a movement related value specifying unit 112, an adjusting unit 113, and a focus correcting unit 114 are provided.

  The wide-angle image acquisition unit 101 sequentially acquires wide-angle images output from the wide-angle camera 20. The telephoto image acquisition unit 102 sequentially acquires the telephoto images sequentially output from the telephoto camera body 31 of the telephoto camera 30.

  The image processing unit 103 detects a feature point from the wide-angle image acquired by the wide-angle image acquisition unit 101 by a known image recognition process. The image processing unit 103 detects the position of the eye in the wide-angle image when the eye can be detected by a known image recognition process based on the feature points detected from the wide-angle image. The image processing unit 103 also detects a feature point from the telephoto image acquired by the telephoto image acquisition unit 102 by a known image recognition process. The image processing unit 103 detects the position of the eye in the telephoto image when the eye can be detected by a known image recognition process based on the feature points detected from the telephoto image. The image processing unit 103 sequentially stores the detected position of the feature point in the wide-angle image, the position of the eye in the wide-angle image, the position of the feature point in the telephoto image, and the position of the eye in the telephoto image in the memory 104. The memory 104 may be a volatile memory, for example. If the eye can be detected as the characteristic point, the eye position may be stored in the memory 104 as the position of the characteristic point. The characteristic points and the positions of the eyes may be represented by coordinates on the image such as pixel coordinates.

  When the eye can be detected from the telephoto image, the image processing unit 103 detects the position of the pupil and the corneal reflection from the eye in the telephoto image by a known image recognition process. Further, the image processing unit 103 may be configured to detect the size of the pupil in the telephoto image. When detecting the size of the pupil in the telescopic image, the image processing unit 103 may be configured to sequentially store the size of the pupil in the telescopic image to be sequentially detected in the memory 104. The size of the eye may be represented by the size on the image such as the number of pixels.

  Further, the image processing unit 103 performs pattern matching between the wide-angle image acquired by the wide-angle image acquisition unit 101 and the telephoto image acquired by the telephoto image acquisition unit 102. The pattern matching may be performed with the wide-angle image and the telephoto image captured at substantially the same timing. As an example, pattern matching may be performed between the latest wide-angle image acquired by the wide-angle image acquisition unit 101 and the latest telephoto image acquired by the telephoto image acquisition unit 102. The pattern matching may be performed after reducing the size of the telescopic image to match the common feature points that can be detected from both the wide-angle image and the telescopic image (see FIG. 4 ).

  The parallax specifying unit 105 sequentially specifies the parallax between the wide-angle image and the telephoto image based on the result of the pattern matching between the wide-angle image and the telephoto image that is sequentially performed by the image processing unit 103. As an example, the parallax identifying unit 105 sequentially identifies, for the pattern-matched wide-angle image and the telephoto image, the positional shift of the center of the telephoto image with respect to the center of the wide-angle image as parallax. For example, this positional deviation may be represented by coordinates with the center of the wide-angle image as the origin. The parallax identifying unit 105 sequentially stores the parallax to be identified in the memory 104. In the present embodiment, the parallax identifying unit 105 identifies the parallax between the images captured by the wide-angle camera 20 and the telephoto camera 30 having different frame rates at substantially the same timing.

  When the image processing unit 103 can detect the driver's pupil and corneal reflection from the telescopic image, the line-of-sight detection unit 106 uses a known corneal reflection method to determine the positional relationship between the detected pupil and corneal reflection in the vehicle interior. The line-of-sight direction of the driver with respect to the reference position is sequentially detected. The reference position may be, for example, the installation position of the imaging system 1 or the like. An eye movement such as a saccade is detected from the change in the line-of-sight direction sequentially detected by the line-of-sight detection unit 106. Then, based on this saccade, the driver's aimless state is determined. The detection of the saccade and/or the determination of the aimless state may be performed by the control unit 10 or may be performed by a device other than the control unit 10.

  The focus-related information acquisition unit 107 sequentially acquires the focus-related information described above that is sequentially output from the telephoto camera body 31 of the telephoto camera 30. The variation amount specifying unit 108 sequentially identifies the variation amount of the wide-angle image and the variation amount of the telephoto image from the variation of the position of the feature amount point of the wide-angle image and the position of the feature point of the telephoto image, which are sequentially stored in the memory 104. As an example, the variation specifying unit 108 calculates the amount of change between the previously stored position and the newly stored position every time the position of the feature point of the wide-angle image and the position of the feature point of the telephoto image are stored in the memory 104. It may be configured to calculate and specify the variation as the variation. When the eye position is stored in the memory 104 as the position of the feature point, the change amount of the eye position may be specified as the change amount.

  The distance estimation unit 109 sequentially estimates the distance from the telephoto camera 30 to the driver's eyes (hereinafter, target distance), and sequentially stores the distance in the memory 104. As an example, the distance from the mirror of the pan/tilt mechanism 33 to the driver's eye may be estimated. The distance estimation unit 109 estimates the distance to the target from the parallax of the stereo camera image based on the parallax between the wide-angle image and the telephoto image identified by the parallax identification unit 105, in the same manner as triangulation. The target distance is estimated by the principle. It is assumed that the values necessary for estimating the target distance other than the parallax, such as the installation distance between the wide-angle camera 20 and the telephoto camera 30, are stored in advance in the non-volatile memory.

  Further, when both the variation amount of the wide-angle image and the variation amount of the telephoto image identified by the variation amount identifying unit 108 are within the threshold range, the distance estimation unit 109 newly estimates the target distance. Instead, it is preferable to continuously use the previously estimated target distance as the estimation result. According to this, it becomes possible to reduce the load of the process of estimating the target distance. The threshold range here is a range that can be set arbitrarily, and may be a range in which the difference between the target distances estimated by the distance estimation unit 109 is within an error level, for example. The threshold range may be different between the wide-angle image and the telephoto image.

  Furthermore, the distance estimation unit 109 performs the following processing when the telephoto camera 30 is in focus by the adjustment of the focus mechanism 32 by the adjustment unit 113 described later according to the target distance estimated by the distance estimation unit 109 the previous time. It may be configured to perform. For example, the distance estimation unit 109 estimates the next target distance based on the target distance in which the telephoto camera 30 is focused, and the variation amount of the wide-angle image and the variation amount of the telephoto image identified by the variation amount identifying unit 108. It may be configured to. In this case, the next target distance may be estimated based on the correlation between the variation amount of the wide-angle image and the variation amount of the telephoto image and the variation amount of the target distance. This correlation may be stored in advance in a non-volatile memory, for example.

  The illumination control unit 110 controls the wide-angle illumination 40 and the telephoto illumination 50. For example, the illumination control unit 110 causes the wide-angle illumination 40 to irradiate the near-infrared light in synchronization with the image capturing timing of the wide-angle camera 20, and the near-infrared light 50 emits in accordance with the image capturing timing of the telescopic camera 30. It may be configured to irradiate infrared light. It should be noted that the illumination control unit 110 may be configured to constantly irradiate the near-infrared light from the wide-angle illumination 40 and the telephoto illumination 50 while the imaging system 1 is activated. Further, the illumination control unit 110 preferably reduces the emission intensity of the wide-angle illumination 40 and the telephoto illumination 50 as the target distance estimated by the distance estimation unit 109 becomes shorter. This makes it possible to further reduce the influence of near-infrared light on the eye.

  The pan-tilt correction unit 111 is based on the deviation between the parallax (hereinafter, referred to as reference parallax) between the wide-angle image and the telephoto image and the actual parallax specified by the parallax specifying unit 105 when there is no mechanical error of the pan-tilt mechanism 33. , The correction amount of the pan/tilt mechanism 33 is sequentially determined. For example, the mechanical error of the pan-tilt mechanism 33 may be caused by the step-out of the pan motor and the tilt motor due to the vibration of the vehicle. Further, the reference parallax is assumed to be stored in advance in the non-volatile memory. The deviation between the reference parallax and the actual parallax is a deviation according to the respective mechanism errors of the pan motor and the tilt motor of the pan/tilt mechanism 33. Therefore, by determining the correction amount of the pan-tilt mechanism 33 that eliminates the deviation between the reference parallax and the actual parallax, the correction amount according to the mechanical error of the pan-tilt mechanism 33 is determined. As an example, the non-volatile memory may be configured to store a correspondence relationship between a deviation between the reference parallax and the actual parallax and a correction amount of the pan motor and the tilt motor that eliminates the deviation. Then, the pan-tilt correction unit 111 may determine the correction amount of the pan-tilt mechanism 33 using this correspondence based on the deviation between the reference parallax and the actual parallax.

  Further, in the pan/tilt mechanism 33, it is preferable that the pan motor in which the deviation of the rotation center is likely to be large determines the correction amount by using the parallaxes stored in the memory 104 for a plurality of times. As an example, the correction amount may be determined based on the deviation between the reference parallax and the likely parallax calculated by the least squares method from the parallaxes stored in the memory 104 for a plurality of times. According to this, even when the error is large with only one parallax stored in the memory 104, it is possible to determine the correction amount with higher accuracy by using the parallax with the smaller error. When the memory 104 does not store a plurality of parallaxes, the correction amount may not be determined until the plurality of parallaxes are stored, or the correction amount may be determined using one parallax. Good. Note that the tilt motor may also be configured to determine the correction amount using the parallaxes stored in the memory 104 for a plurality of times.

  The movement-related value specifying unit 112 sequentially specifies the movement speed and movement range (hereinafter, movement-related value) of the driver's eye position from the eye position in the wide-angle image sequentially stored in the memory 104, and specifies the specified movement. The related values are sequentially stored in the memory 104. For example, the moving speed may be represented by the amount of change in the eye position per frame. Further, the movement range may be represented by coordinates in the wide-angle image.

  Based on the position of the eye in the wide-angle image stored in the memory 104, the adjustment unit 113 operates the pan-tilt mechanism 33 and the focus mechanism 32 so that the telephoto camera 30 magnifies and images the driver's eye, The imaging direction and focus of the telephoto camera 30 are adjusted. As a default, the adjustment unit 113 calculates the position of the left eye of the driver with respect to the telephoto camera 30 from the position of the left eye of the driver in the wide-angle image and the predetermined distance provisionally set as the target distance, and the center of the left eye is calculated. The pan-tilt mechanism 33 and the focus mechanism 32 are operated so that the optical axis is directed toward the front. The predetermined distance may be a configuration that uses a preset value in consideration of the rough distance between the driver and the telephoto camera 30 when the driver is seated in the driver's seat, and may be set to, for example, 500 mm.

  In the above-described default adjustment, when there is an error between the predetermined distance (see A in FIG. 5) and the actual target distance as shown in FIG. The area (see C in FIG. 5) that can be actually captured by the telephoto camera 30 is deviated from the area (see B in FIG. 5). The range surrounded by the broken line D in FIG. 5 indicates the imaging range of the wide-angle camera 20, and the range surrounded by the broken line E in FIG. 5 indicates the imaging range of the telephoto camera 30. Further, in the above-described default adjustment, even when there is a mechanical error in the pan/tilt mechanism 33 and the focus mechanism 32, the area that can be actually imaged by the telephoto camera 30 with respect to the area targeted for imaging is equal to this error. It will shift.

  In order to eliminate such a shift, the adjustment unit 113 performs the following processing. When the distance estimation unit 109 estimates the target distance, the adjustment unit 113 operates the focus mechanism 32 so that the closest target distance estimated by the distance estimation unit 109 is focused instead of the default distance. Note that the adjustment unit 113 may be configured to read the latest target distance from the memory 104 and use it. Further, when the pan/tilt correction unit 111 determines the correction amount of the pan/tilt mechanism 33, the adjustment unit 113 operates the pan/tilt mechanism 33 according to the latest correction amount determined by the pan/tilt correction unit 111. Further, when the focus correction unit 114, which will be described later, determines the correction amount of the focus mechanism 32, the adjustment unit 113 operates the focus mechanism 32 according to the latest correction amount determined by the focus correction unit 114.

  It should be noted that the presence or absence of focusing due to adjustment by the adjusting unit 113, the amount of deviation from focusing, and the like may be treated as the above-mentioned focusing related information. When focusing is not performed due to the adjustment by the adjusting unit 113, for example, an AF function may be used for focusing.

  When the image processing unit 103 can detect an eye from the telescopic image, the adjustment unit 113 calculates the amount of deviation between the center of the position of the pupil detected by the image processing unit 103 from the telescopic image and the center of the telescopic image. Then, the pan-tilt mechanism 33 is operated to eliminate the calculated shift amount, so that the center of the position of the pupil is adjusted to the center of the telephoto image. According to this, even when a mechanism error of the pan-tilt mechanism 33 newly occurs and a shift occurs in the imaging direction, the pupil is less likely to be dislocated from the telephoto image, and this shift is easily allowed. When the correction amount is determined by the pan/tilt correction unit 111, the pan/tilt mechanism 33 may be operated according to the correction amount so as to eliminate the calculated shift amount.

  Further, the adjusting unit 113 operates the pan-tilt mechanism 33 and the focus mechanism 32 when the movement-related value specified by the movement-related value specifying unit 112 is within the predetermined range, while the movement-related value is not within the predetermined range. The pan/tilt mechanism 33 and the focus mechanism 32 are not operated. The predetermined range mentioned here is a value that can be set arbitrarily, and is set so as to suppress excessive tracking of the eyes of the driver. As an example, a range in which the moving speed of the eye position is estimated to be the moving speed of a human may be set as a predetermined range. In addition, the range in which the driver is estimated to face the front may be set as the predetermined range as the range of movement of the eyes.

  Furthermore, it is preferable that the adjusting unit 113 is configured to switch the eyes of the driver to be enlarged and imaged by the telephoto camera 30 according to the face orientation of the driver in the wide-angle image acquired by the wide-angle image acquisition unit 101. .. When this configuration is adopted, the range in which it is estimated that the driver is facing the front is not set as the above-described predetermined range as the range of movement of the eyes.

  Note that the driver's face orientation may be configured to use the face orientation detected by the image processing unit 103 by a known image recognition process, or may be specified from the movement range of the driver's eye position identified by the movement-related value identification unit 112. It may be configured to use the face direction. As an example, the adjusting unit 113 may set the imaging target to the left eye of the driver when the face orientation of the driver is front and right, and may set the imaging target to the right eye of the driver when the face orientation is left. . As for the face direction, a state in which the face is tilted leftward by a predetermined value or more may be leftward, and a state in which the face is tilted rightward by a predetermined value or more may be rightward. The predetermined value referred to here is a value that can be set arbitrarily, and may be set, for example, so that a state of being greatly tilted to the left or right from the front is leftward or rightward.

  The focus correction unit 114 operates the pan-tilt mechanism 33 according to the correction amount determined by the pan-tilt correction unit 111, and operates the focus mechanism 32 according to the target distance estimated by the distance estimation unit 109. When is not focused, the correction amount of the focus mechanism 32 is determined. Specifically, the correction amount of the focus mechanism 32 that is estimated to be required to bring the telephoto camera 30 out of focus by the adjustment by the adjusting unit 113 into the in-focus state is determined.

  As an example, the correction amount of the focus mechanism 32 may be determined according to the amount of deviation from the focus in the focus related information described above. As another example, the correction of the focus mechanism 32 is performed according to the ratio of the size of the closest pupil in the telephoto image stored in the memory 104 to the size of the reference pupil when in focus. The amount may be determined. Note that another method may be used to determine the correction amount of the focus mechanism 32 that is estimated to be necessary for focusing the telephoto camera 30 that was not in focus due to the adjustment by the adjusting unit 113 described above.

<Eyeball Tracking-related Processing in Control Unit 10>
Next, with reference to the flowchart of FIG. 6, an example of the flow of processing (hereinafter, eye tracking-related processing) related to processing of enlarging and imaging the driver's eye in the control unit 10 while tracking the driver's eye. Will be described. The flowchart of FIG. 6 may be configured to start when the power of the imaging system 1 is turned on and end when the power of the imaging system 1 is turned off. In the flowchart of FIG. 6, it is assumed that the wide-angle camera 20 and the telephoto camera 30 sequentially capture images at their respective frame rates.

  First, in step S1, the image processing unit 103 detects the position of the eye in the wide-angle image acquired by the wide-angle image acquisition unit 101. In step S2, the adjustment unit 113 sets an imaging distance for the telephoto camera 30 to image the driver's eyes. By default, the above-described default distance is set, and when the distance estimation unit 109 has estimated the target distance, the estimated latest target distance is set.

  In step S3, the adjusting unit 113 calculates the position of the driver's eye with respect to the telephoto camera 30 from the position of the driver's target eye in the wide-angle image and the set imaging distance. Then, the pan/tilt mechanism 33 and the focus mechanism 32 are adjusted so that the optical axis is directed to the center of the eye.

  In step S4, the image processing unit 103 performs pattern matching between the wide-angle image acquired by the wide-angle image acquisition unit 101 after the adjustment in S3 and the telephoto image acquired by the telephoto image acquisition unit 102. Then, the parallax identifying unit 105 identifies the parallax between the wide-angle image and the telephoto image from the result of the pattern matching. In step S5, the distance estimation unit 109 estimates the target distance.

  In step S6, the line-of-sight detection unit 106 attempts to detect the line-of-sight direction of the driver. In step S7, if the line-of-sight detection unit 106 succeeds in detecting the driver's line-of-sight direction (YES in step S7), the process proceeds to step S11. On the other hand, if the detection of the driver's line-of-sight direction has failed (NO in S7), the process proceeds to step S8. An example of a case where the driver's line-of-sight direction detection fails is when the image processing unit 103 cannot detect the driver's pupil and corneal reflection from the telephoto image.

  In step S8, the pan-tilt correction unit 111 determines the correction amount of the pan-tilt mechanism 33 based on the deviation between the reference parallax and the actual parallax specified in S4. When it is necessary to determine the correction amount of the focus mechanism 32 in step S9 (YES in S9), the process proceeds to step S10. On the other hand, when it is not necessary to determine the correction amount of the focus mechanism 32 (NO in S9), the process returns to S1 and is repeated. When the correction amount of the focus mechanism 32 needs to be determined, the pan/tilt mechanism 33 is operated according to the correction amount determined by the pan/tilt correction unit 111, and the focus mechanism 32 is operated according to the target distance estimated by the distance estimation unit 109. This is the case where the telephoto camera 30 is out of focus due to the adjustment in the unit 113.

  In step S10, the focus correction unit 114 determines the correction amount of the focus mechanism 32, returns to S1, and repeats the processing. In step S11, which is a process performed when the driver's line-of-sight direction is successfully detected in step S7, the adjusting unit 113 adjusts the center of the pupil position to the center of the telephoto image, and returns to step S4 to repeat the process. .

  In the flowchart of FIG. 6, as described above, when the movement related value specified by the movement related value specifying unit 112 is not within the predetermined range, the pan/tilt mechanism 33 and the focus mechanism 32 are not operated and the adjustment is interrupted. It is preferable. According to this, it is possible to suppress the excessive tracking of the driver's eyes and shorten the return time until the tracking is repeated even when the tracking of the driver's eyes fails.

  In the flowchart of FIG. 6, when the pan-tilt mechanism 33 and the focus mechanism 32 both have a mechanical error, the pan-tilt mechanism 33 and the focus mechanism 32 are repeatedly processed to sequentially correct the mechanical errors. Both of the mechanism errors will be accurately corrected. Further, among the pan motor of the pan/tilt mechanism 33, the tilt motor of the pan/tilt mechanism 33, and the motor of the focus mechanism 32, the mechanical error is sequentially corrected on the basis of a motor having a small mechanical error such as out-of-step occurrence. It may be configured to do so. For example, the configuration may be such that correction is sequentially performed from those that are estimated in advance when the mechanical error increases.

<Summary of Embodiment 1>
Since the installation distance between the wide-angle camera 20 and the telephoto camera 30 is fixed, the deviation between the reference parallax and the actual parallax identified by the parallax identifying unit 105 has a value corresponding to the mechanical error of the pan-tilt mechanism 33. According to the configuration of the first embodiment, the pan-tilt correction unit 111 sequentially determines the correction amount of the pan-tilt mechanism 33 based on this deviation, and thus the pan-tilt mechanism correction amount is sequentially determined according to the mechanical error of the pan-tilt mechanism 33. It becomes possible to do. The adjusting unit 113 operates the pan-tilt mechanism 33 according to this correction amount when operating the pan-tilt mechanism and the focus mechanism so that the telephoto camera 30 magnifies and images the driver's eyes. Therefore, even if a mechanical error of the pan-tilt mechanism 33 of the telephoto camera 30 occurs due to a load such as vibration during use in a vehicle, the deviation of the imaging direction due to the mechanical error is suppressed and the telephoto camera 30 takes an image with the driver's eye. It becomes easier to match the directions. As a result, in an imaging system that focuses an image on the eyes of the driver of the own vehicle, it becomes easy to accurately focus on the eyes of the driver of the own vehicle.

  According to the configuration of the first embodiment, the correction amount of the focus mechanism 32 that is estimated to be necessary for focusing the telephoto camera 30 that has not been focused by the adjustment by the adjusting unit 113 is determined, and the focus is adjusted according to this correction amount. The mechanism 32 is operated. Therefore, even when a mechanical error of the focus mechanism 32 of the telephoto camera 30 occurs due to a load such as vibration during use on a vehicle, the focal point shift due to this mechanical error is suppressed, and the telephoto camera 30 focuses on the driver's eye. It will be easier to match. From the above, even if a mechanism error between the focus mechanism 32 and the pan-tilt mechanism 33 of the telephoto camera 30 occurs at the same time, the deviation of the imaging direction and the focus due to these mechanism errors is suppressed and the driver's eyes of the own vehicle see. It becomes possible to easily focus with high accuracy.

  In addition, according to the configuration of the first embodiment, since the target distance is sequentially estimated, the target distance can be continuously estimated for the driver whose position changes frequently. Further, according to the configuration of the first embodiment, the adjusting unit 113 operates the focus mechanism 32 according to the target distance. Further, the adjusting unit 113 operates the pan-tilt mechanism 33 so that the telephoto camera 30 magnifies and images the driver's eyes to sequentially adjust the imaging direction of the telephoto camera 30. Therefore, it becomes possible to track the eyes of the driver whose position changes frequently and to take an image with the telephoto camera 30.

  Further, according to the configuration of the first embodiment, the target distance is estimated based on the parallax between the wide-angle image of the wide-angle camera 20 and the telephoto image of the telephoto camera 30, so that the wide-angle camera 20 and the wide-angle camera 20 are used to estimate the target distance. It is not necessary to newly provide a distance measuring device other than the telephoto camera 30. Therefore, it is possible to further simplify the configuration of the imaging system 1.

(Embodiment 2)
In the above-described embodiment, the distance estimation unit 109 has shown the configuration in which the target distance is estimated based on the parallax between the wide-angle image of the wide-angle camera 20 and the telephoto image of the telephoto camera 30, but the present invention is not limited to this. For example, the distance estimation unit 109 may be configured to estimate the target distance from the distance between the image sensor of the telephoto camera 30 and the lens of the focus mechanism 32 when the telephoto camera 30 is focused by, for example, an AF function. Note that the distance estimation unit 109 may be configured to estimate the target distance based on the amount of deviation from the focus in the focus related information described above even if the telephoto camera 30 is out of focus.

  In addition, for example, a distance measuring device such as an infrared distance measuring sensor is provided in the imaging system 1 in addition to the wide-angle camera 20 and the telephoto camera 30, and the distance estimation unit 109 estimates the target distance using this distance measuring device. May be

(Embodiment 3)
In the above-described embodiment, the configuration in which the pan/tilt correction unit 111 determines the correction amount of the pan motor and the correction amount of the tilt motor independently has been described, but the present invention is not limited to this. For example, by storing the correlation between the correction amount of the tilt motor and the correction amount of the pan motor in the control unit 10 in advance, the correction amount of the tilt motor can be calculated based on this correlation. It may be configured to determine the correction amount of the motor. According to this, it becomes possible to more easily determine the correction amount of the pan motor.

(Embodiment 4)
In the above-described embodiment, the configuration in which the imaging system 1 is used in the automobile has been shown, but the configuration is not limited to this. The imaging system 1 can be used in various vehicles. For example, the imaging system 1 may be configured to be used in vehicles other than automobiles such as railway vehicles and motorbikes.

  It should be noted that the present invention is not limited to the above-described embodiments and modifications, and various modifications can be made within the scope of the claims, and the technical means disclosed in the different embodiments and modifications, respectively. Embodiments obtained by appropriately combining the above are also included in the technical scope of the present invention.

1 imaging system, 10 control unit (eye tracking device), 20 wide-angle camera, 30 telephoto camera, 31 telephoto camera body, 32 focus mechanism, 33 pan-tilt mechanism, 40
Wide-angle illumination (illumination device), 50 Telephoto illumination (illumination device), 101 Wide-angle image acquisition unit, 102 Telephoto image acquisition unit, 103 Image processing unit, 104 Memory, 105 Parallax identification unit, 106 Eye-gaze detection unit, 107 Focus Related information acquisition unit, 108 Variation amount specification unit, 109 Distance estimation unit, 110 Lighting control unit, 111 Pan/tilt correction unit, 112 Movement related value specification unit, 113 Adjustment unit, 114 Focus correction unit

Claims (11)

Used in vehicles,
A wide-angle image acquisition unit (101) that sequentially acquires a wide-angle image that is a captured image of a wide-angle camera (20) whose imaging direction is fixed so as to capture a range including a face portion when a driver is seated in the driver's seat of the vehicle; ,
A telescopic image, which is a captured image of a telescopic camera (30) having a fixed installation interval with respect to the wide-angle camera and a narrower angle of view than the wide-angle camera, and having a pan-tilt mechanism (33) and a focus mechanism (32). A telescopic image acquisition unit (102) that sequentially acquires,
Adjustment for operating the pan-tilt mechanism and the focus mechanism so that the telephoto camera magnifies and images the driver's eye based on the position of the driver's eye in the wide-angle image acquired by the wide-angle image acquisition unit Part (113),
A parallax specifying unit (105) that sequentially specifies the parallax between the wide-angle image acquired by the wide-angle image acquisition unit and the telephoto image acquired by the telephoto image acquisition unit;
A correction amount of the pan-tilt mechanism according to the mechanical error of the pan-tilt mechanism is calculated based on a deviation between the parallax when there is no mechanical error of the pan-tilt mechanism and the actual parallax specified by the parallax specifying unit. A pan-tilt correction unit (111) that sequentially determines,
The adjustment unit operates the pan-tilt mechanism according to a correction amount that is sequentially determined by the pan-tilt correction unit when operating the pan-tilt mechanism and the focus mechanism so that the telephoto camera magnifies and images the driver's eye. Eye tracking device. In claim 1,
A distance estimation unit (109) for sequentially estimating a target distance from the telephoto camera to the driver's eyes,
The adjusting unit operates the focus mechanism according to the target distance sequentially estimated by the distance estimating unit when operating the pan-tilt mechanism and the focus mechanism so as to magnify and image the driver's eye with the telephoto camera. Eye tracking device to operate. In claim 2,
The distance estimation unit is an eye tracking device that sequentially estimates a target distance from the telephoto camera to the driver's eye based on the parallax specified by the parallax specifying unit. In claim 2 or 3,
A variation amount specifying unit (108) for sequentially specifying the variation amount of the wide-angle image sequentially acquired by the wide-angle image acquisition unit and the variation amount of the telephoto image sequentially acquired by the telephoto image acquisition unit,
The distance estimation unit does not newly estimate the target distance when both the variation amount of the wide-angle image identified by the variation amount identifying unit and the variation amount of the telephoto image are within the threshold range. , An eye tracking device that continues to use the previously estimated target distance as an estimation result. In any one of Claims 2-4,
A variation amount specifying unit (108) for sequentially specifying a variation amount of the wide-angle image sequentially acquired by the wide-angle image acquisition unit and a variation amount of the telephoto image sequentially acquired by the telephoto image acquisition unit,
When the focus mechanism is operated according to the target distance estimated last time and the telephoto camera is in focus, the distance estimation unit specifies the target distance in which the telephoto camera is focused, and the variation amount specifying unit. An eye tracking device that estimates the next target distance based on a variation amount of a wide-angle image and a variation amount of the telephoto image. In any one of Claims 2-5,
When the pan-tilt mechanism is operated according to the correction amount determined by the pan-tilt correction unit and the focus mechanism is operated according to the target distance estimated by the distance estimation unit, but the telephoto camera is not in focus And a focus correction unit (114) for determining a correction amount of the focus mechanism estimated to bring the telephoto camera into focus,
The adjustment unit operates the focus mechanism according to the correction amount determined by the focus correction unit when operating the pan-tilt mechanism and the focus mechanism so that the telephoto camera magnifies and images the driver's eyes. Eye tracking device. In any one of Claims 2-6,
An illumination control unit (110) that controls an illumination device (40, 50) that emits light toward the driver,
The illumination control unit is an eye tracking device that reduces the light emission intensity of the illumination device as the target distance estimated by the distance estimation unit becomes shorter. In any one of Claims 1-7,
The pan-tilt correction unit, at least a correction amount according to a mechanical error in the pan direction of the pan-tilt mechanism, as the actual parallax specified by the parallax specifying unit, the parallax for a plurality of times sequentially specified by the parallax specifying unit. An eye tracking device that determines using. In any one of Claims 1-8,
A moving-related value specifying unit (112) that specifies a moving-related value that is at least one of a moving speed and a moving range of the driver's eye position from the wide-angle images sequentially acquired by the wide-angle image acquiring unit,
The adjustment unit is configured to perform the pan-tilt mechanism and the focus so that the telescopic camera magnifies and images the driver's eye only when the movement-related value specified by the movement-related value specifying unit is within a predetermined range. Eye tracking device that operates the mechanism. In any one of Claims 1-9,
The adjustment unit is an eye tracking device that switches the eyes of the driver to be enlarged and imaged by the telephoto camera according to the face orientation of the driver in the wide-angle image acquired by the wide-angle image acquisition unit. Used in vehicles,
A wide-angle camera (20) whose imaging direction is fixed so as to image a range including a face portion when a driver is seated in the driver's seat of the vehicle;
A telephoto camera (30) having a fixed position with respect to the wide-angle camera, a narrower angle of view than the wide-angle camera, and a pan-tilt mechanism (33) and a focus mechanism (32);
An imaging system comprising the eye tracking device (10) according to any one of claims 1 to 10.

Images