JP6930410B2 - Pupil detector and detection system - Google Patents

Description

The present disclosure relates to a pupil detection device for detecting the movement of the pupil and a detection system including the pupil detection device.

Patent Document 1 discloses an eye movement inspection device for estimating the eye movement and the head movement of a target person from a photographed image. The subject of the eye movement test device wears a predetermined contact lens. A retroreflective material is formed on the contact lens so as to form a predetermined reflection pattern. In the eye movement inspection device, the light source irradiates the subject with light having a predetermined wavelength, and the camera captures the image. Based on the captured image, the computer estimates the head movement of the subject for each axis in the world coordinate system, and estimates the eye movement of the subject in the head coordinate system for each axis of the head coordinate system. There is.

Patent Document 2 discloses an eye region detection program for the purpose of extracting the eye portion of a subject at high speed from image data obtained by photographing the subject using an image processing technique. The eye region detection program differentiates the density value in the original image including the subject's face in the vertical direction of the original image, integrates the differential values in the horizontal direction to generate a histogram, and generates the maximum value of the histogram or the maximum value of the histogram. The eye region is extracted from the original image using the maximum value.

Further, Patent Documents 3 and 4 also disclose an image processing technique for detecting an eye portion, a pupil portion, and the like from an image captured by a camera. Patent Document 5 discloses a state estimation device that estimates the state of a monitored person based on eye movements, with a vehicle driver or the like as the monitored person. Patent Document 6 discloses a system for estimating the attention of a monitored person in a state estimation device. Patent Document 7 proposes to use the measurement of fixation tremor when estimating the position that the user is paying attention to in the field of view of the user.

Japanese Unexamined Patent Publication No. 2016-87093 Japanese Patent No. 5276381 Japanese Patent No. 596458 Japanese Patent No. 5995217 Japanese Unexamined Patent Publication No. 2017-23519 Japanese Unexamined Patent Publication No. 2015-207163 Japanese Unexamined Patent Publication No. 2013-240469

Tobi Delbruck, "Frame-free dynamic digital vision", Proceedings of Intl. Symp. on Security-Life Electronics, Advanced Electronics for Quality Life and Society, Univ. of Tokyo, Mar. 6-7, 2008.

In the eye movement inspection device of Patent Document 1, it is necessary to attach a special contact lens made of a retroreflective material to the subject. In addition, when detecting the eye movement of the subject, it becomes necessary to first execute a process of estimating the head movement. Therefore, in the prior art, it is not easy to detect the movement of a specific pupil according to the eye movement to be detected in a situation where the head movement and the eye movement of the subject can occur at the same time. was there.

An object of the present disclosure is to provide a pupil detection device and a detection system capable of easily detecting the movement of a specific pupil.

The pupil detection device according to the present disclosure detects the movement of the pupil based on the imaging result of the imaging device that captures an image including the pupil. The pupil detection device includes an acquisition unit and a control unit. The acquisition unit acquires imaging data indicating an imaging result from the imaging apparatus. The control unit executes data processing based on the imaging data acquired by the acquisition unit. The control unit extracts information indicating the movement of the pupil based on the imaging data, performs time-frequency analysis on the extracted information, and detects the movement of the pupil. The time-frequency analysis is a numerical analysis that decomposes the waveform to be analyzed in the time direction for each frequency, and is executed by, for example, a Fourier transform.

The detection device according to the present disclosure includes an imaging device and a pupil detection device. The imaging device captures an image including the pupil and generates imaging data. The pupil detection device detects the movement of the pupil based on the imaging data.

According to the pupil detection device and detection system of the present disclosure, it is possible to easily detect the movement of a specific pupil.

The figure for demonstrating the application example of the detection system which concerns on this disclosure. The figure which illustrates the structure of the pupil detection device in the detection system A flowchart illustrating the operation of the pupil detection device according to the first embodiment. A flowchart illustrating the microsaccade detection process according to the first embodiment. The figure which illustrates the image processing in the microsaccade detection processing of Embodiment 1. The figure for demonstrating the time frequency analysis in the microsaccade detection process of Embodiment 1. The figure for demonstrating the image pickup apparatus which concerns on Embodiment 2. Schematic diagram of various images in the microsaccade detection process of the second embodiment A flowchart illustrating the microsaccade detection process according to the second embodiment. Scatter plot exemplifying the distribution of events in luminance change data when microsaccades occur The figure which illustrates the search method of the elliptical region in a luminance change image A graph showing the experimental results of a pupil model experiment relating to the pupil detection device of the second embodiment. Graph showing the frequency spectrum of the experimental result of FIG.

Hereinafter, embodiments of the pupil detection device and the detection system according to the present disclosure will be described with reference to the accompanying drawings. In each of the following embodiments, the same reference numerals are given to the same components.

(Application example)
An example to which the pupil detection device and the detection system according to the present disclosure can be applied will be described with reference to FIG. FIG. 1 is a diagram for explaining an application example of the detection system 1 according to the present disclosure.

FIG. 1 shows an application example of the detection system 1 for in-vehicle use. The detection system 1 detects, for example, the arousal level indicating the awakening state of the driver 3 who drives the vehicle 2, and notifies the driver 3 to prevent drowsy driving or involuntary driving according to the decrease in the arousal level. The detection system 1 may perform various controls for driving support of the vehicle 2 based on the detection result such as the arousal level.

As shown in FIG. 1, the detection system 1 includes an image pickup device 4 and a pupil detection device 5. The image pickup device 4 is arranged in the vehicle 2, for example, and captures an image of the vicinity of the eye 30 of the driver 3. The driver 3 is an example of a target person to be detected by the detection system 1.

The pupil detection device 5 is a device that detects the movement of the pupil according to the eye movement of the subject based on the image pickup result of the image pickup device 4. The pupil detection device 5 is connected to a notification unit 21 such as a voice output device provided in the vehicle 2, for example. The pupil detection device 5 may be connected to various ECUs (electronic control units) or the like of the vehicle 2.

It is thought that there is a physiological correlation between the frequency of occurrence of microsaccade, which is a type of eye movement, and the degree of alertness. For example, it is believed that the frequency of microsaccades increases when alertness decreases. Therefore, the detection system 1 of this application example detects the microsaccade by the eye 30 of the driver 3 by the pupil detection device 5 and uses it for detecting the alertness. Microsaccade is a high-speed eye movement that moves instantaneously in a fine eye movement classified as a fixed vision tremor.

In the vehicle 2, it is assumed that head movement and eye movement can occur at the same time, for example, microsaccade may occur when the driver 3 moves the head. Conventionally, under such a situation, each of the head movement and the eye movement has been measured, and a complicated process of analyzing the mutual correlation has been performed to detect the target eye movement. On the other hand, the detection system 1 is a pupil detection device 5 in order to enable detection of a target eye movement such as a microsaccade without the need to separately measure the head movement under the above circumstances. Information indicating the movement of the pupil is extracted and frequency analysis is performed in the time direction.

(Configuration example)
Hereinafter, each embodiment will be described as a configuration example of the detection system 1 and the pupil detection device 5.

(Embodiment 1)
In the first embodiment, a configuration example of the detection system 1 using the image pickup device 4 that captures an image for each frame will be described.

1. 1. Configuration The configuration of the imaging device 4 and the pupil detection device 5 of the detection system 1 according to the present embodiment will be described below.

1-1. About the image pickup device In the present embodiment, the image pickup device 4 is composed of a high-speed camera having a high-speed frame rate such as 500 fps. The frame rate of the image pickup apparatus 4 is set to be at least twice the peak frequency in the eye movement of the detection target in consideration of the detection target such as the microsaccade.

The image pickup device 4 of the present embodiment generates image data indicating a frame image, and outputs the generated image data to the pupil detection device 5. The image data generated by the image pickup apparatus 4 is an example of the image pickup data in the present embodiment.

The image pickup apparatus 4 may include an illumination light source for imaging. The illumination light source emits infrared light as illumination light, for example. The illumination light source is arranged so as to irradiate the target person such as the driver 3 with the illumination light. The illumination light source may be configured separately from the image pickup apparatus 4, and may be incorporated into the detection system 1 as appropriate.

1-2. About the pupil detection device The configuration of the pupil detection device 5 in the detection system 1 will be described with reference to FIG. FIG. 2 is a diagram illustrating the configuration of the pupil detection device 5. As shown in FIG. 2, the pupil detection device 5 includes an acquisition unit 51 and a control unit 52. Further, the pupil detection device 5 includes, for example, a storage unit 53 and an output unit 54.

The acquisition unit 51 is an input interface circuit for inputting data from the outside of the pupil detection device 5. The acquisition unit 51 is connected to the image pickup device 4 according to a predetermined communication standard such as USB. The acquisition unit 51 acquires imaging data or the like indicating the imaging result from the imaging device 4.

The control unit 52 includes a CPU, RAM, ROM, and the like, and controls each component according to information processing. For example, the control unit 52 executes data processing according to a predetermined program based on the data acquired by the acquisition unit 51. The predetermined program is a program for realizing various operations described later of the pupil detection device 5. The control unit 52 may hold various data such as imaging data from the imaging device 4 in an internal memory such as a RAM.

The storage unit 53 is, for example, an auxiliary storage device such as a hard disk drive or a solid state drive. The storage unit 53 stores a program executed by the control unit 52, various data, and the like.

The pupil detection device 5 may acquire the above program or the like from a portable storage medium. The storage medium stores the information of the program or the like by electrical, magnetic, optical, mechanical or chemical action so that the information of the program or the like recorded by the computer or other device or machine can be read. It is a medium.

The output unit 54 is an output interface circuit that outputs data from the pupil detection device 5 to the outside. The output unit 54 is connected to, for example, the notification unit 21 of the vehicle 2 (FIG. 1), various ECUs, and the like. In the pupil detection device 5, the output unit 54 and the acquisition unit 51 may be integrally configured so as to form an input / output interface.

2. Operation The operation of the detection system 1 and the pupil detection device 5 configured as described above will be described below.

2-1. Operation of the detection system The overall operation of the detection system 1 will be described with reference to FIG. FIG. 3 is a flowchart illustrating the operation of the pupil detection device 5 in the detection system 1.

Each process of the flowchart shown in FIG. 3 is executed by the control unit 52 of the pupil detection device 5. This flowchart is started in a state where the image pickup device 4 is capturing an image of the vicinity of the eye 30 of the driver 3 while the vehicle 2 shown in FIG. 1 is being driven.

In the flowchart of FIG. 3, the control unit 52 of the pupil detection device 5 executes the microsaccade detection process (S1). The microsaccade detection process is an example of data processing for detecting microsaccade in the movement of the pupil 6 based on the imaging result of the imaging device 4. Details of the microsaccade detection process will be described later.

Next, the frequency of occurrence of microsaccade is detected from the detection result of the microsaccade detection process (S2), and it is determined whether or not the current frequency of occurrence of microsaccade corresponds to the condition for lowering the alertness. (S2).

The determination in step S2 is performed by appropriately setting the conditions for lowering the arousal level according to the correlation between the assumed frequency of occurrence of microsaccades and the arousal level. For example, the control unit 52 determines whether or not the current frequency of occurrence of microsaccades is higher than a predetermined reference value, and whether or not the frequency of occurrence corresponds to the condition for lowering the arousal level (S2). ).

When the control unit 52 determines that the current frequency of occurrence of microsaccades does not correspond to the condition for lowering the arousal level (NO in S2), the control unit 52 re-executes the process of step S1 in a predetermined cycle. The cycle for repeating step S1 can be appropriately set, for example, 1 second.

On the other hand, when the control unit 52 determines that the current frequency of occurrence of microsaccades corresponds to the condition for reducing the arousal level (YES in S2), the control unit 52 generates notification information indicating that the arousal level has decreased and outputs the output unit. Notification information is output to the notification unit 21 and the like via 54 (FIG. 2) (S3).

The control unit 52 outputs the notification information (S3), and ends the process according to the flowchart of FIG.

According to the above processing, in the detection system 1, the pupil detection device 5 can detect the occurrence frequency of microsaccades by the target person such as the driver 3 and detect the decrease in alertness.

2-1-1. Microsaccade detection process The details of the microsaccade detection process in step S1 of FIG. 3 will be described with reference to FIGS. 4 to 6.

FIG. 4 is a flowchart illustrating the microsaccade detection process (S1 in FIG. 3) in the present embodiment. FIG. 5 is a diagram illustrating an image process (S12) in the microsaccade detection process of the present embodiment. FIG. 6 is a diagram for explaining the time frequency analysis (S13, S14) in the microsaccade detection process.

In the flowchart of FIG. 4, first, the control unit 52 of the pupil detection device 5 (FIG. 1) acquires image data for a predetermined period from the image pickup device 4 via the acquisition unit 51 (S11). The predetermined period in step S11 can be appropriately set in consideration of the number of frames required for frequency analysis. For example, the predetermined period is set to 1 second.

Next, the control unit 52 performs image processing on the image of each frame in the acquired image data to extract a temporal change (that is, time evolution) of the position of the pupil (S12). An example of the process of step S12 will be described with reference to FIG.

FIG. 5A illustrates a one-frame image 60 in the image data from the image pickup apparatus 4. In the example of FIG. 5A, the pupil 61 is included in the image 60. FIG. 5B illustrates the binarized image 62 of the image 60 of FIG. 5A. Hereinafter, as an XY coordinate system common to various images 60 and 62, the horizontal direction on the imaging surface of the imaging device 4 is defined as the “X direction” and the vertical direction is defined as the “Y direction”.

In step S12, the control unit 52 performs binarization processing on the brightness of each pixel in the image 60 of one frame as shown in FIG. 5A, for example. For example, the control unit 52 assigns a white gradation to a pixel having a brightness less than a predetermined threshold value in the original image 60 (FIG. 5A), and a black scale to a pixel having a brightness equal to or higher than the same threshold value. A key is assigned to generate a binarized image 62 (FIG. 5 (b)). The threshold value is set to a value larger than the brightness assumed as the pupil 61 in the image 60.

According to the above binarization process, as shown in FIG. 5B, the region 63 corresponding to the pupil 61 can be easily determined. The control unit 52 calculates the center position 64 of the region 63 determined by the binarization process as the pupil position of the pupil 61 at the time of the corresponding frame. In step S12, the control unit 52 performs the above processing on the image of each frame to generate time-series data of the pupil position. As a result, the temporal change of the pupil position is extracted. FIG. 6A exemplifies the temporal change of the pupil position. In the graph illustrated in FIG. 6A, the horizontal axis represents the time (ms) and the vertical axis represents the X coordinate of the pupil position.

Returning to FIG. 4, the control unit 52 frequency-decomposes the extracted temporal change of the pupil position in the time direction (S13). For example, the control unit 52 performs FFT (Fast Fourier Transform) on the generated time series data to calculate the frequency spectrum of the time series data. FIG. 6B is a graph showing the frequency spectrum for one second in FIG. 6A. In the graph of FIG. 6B, the horizontal axis represents the frequency (Hz) and the vertical axis represents the intensity of the frequency component.

Next, the control unit 52 analyzes the frequency spectrum as a result of the frequency decomposition shown in FIG. 6B, for example, and determines whether or not the movement of the pupil in a specific frequency band corresponding to the microsaccade has occurred. (S14). The specific frequency band is set in a range that includes the peak frequency of the microsaccade and does not include the frequency that is expected to occur remarkably in response to the head movement, for example, 220 to 250 Hz.

In step S14, the control unit 52 determines whether or not the frequency component included in the specific frequency band is equal to or higher than a predetermined threshold value in the frequency spectrum, for example. The threshold value is set in consideration of the intensity of the frequency component expected when microsaccade occurs. When the maximum value of the specific frequency band component is equal to or greater than the threshold value, the control unit 52 determines that the movement of the pupil in the specific frequency band, that is, microsaccade has occurred (YES in S14).

When the control unit 52 determines that the movement of the pupil in a specific frequency band has occurred (YES in S14), the control unit 52 records the occurrence time of the microsaccade in the storage unit 53 (S15). For example, the control unit 52 records the time when the image data was acquired in step S11 as the time when the microsaccade occurred.

On the other hand, when the control unit 52 determines that the movement of the pupil in a specific frequency band has not occurred (NO in S14), the control unit 52 ends the microsaccade detection process (S1 in FIG. 3) without performing the process in step S15. , Step S2.

According to the above processing, the microsaccade can be automatically detected by the time-frequency analysis based on the frequency spectrum of the pupil position in the image data from the image pickup apparatus 4. The time-frequency analysis in the microsaccade detection process of the present embodiment will be described with reference to FIGS. 6 (a) and 6 (b).

FIG. 6A shows an example in which microsaccade occurs at the timing indicated by the arrow A1 while the X coordinate of the position of the pupil gradually changes due to the head movement. The pupil detection device 5 of the present embodiment executes the microsaccade detection process every predetermined period (S11) to detect whether or not microsaccade has occurred during the period to be processed. For example, the control unit 52 of the pupil detection device 5 extracts the temporal change of the position of the pupil for 1 second in the example of FIG. 6A (S12), and as shown in FIG. 6B, the time direction. The frequency spectrum of is calculated (S13).

The frequency spectrum of the example of FIG. 6B includes a first peak value P1 corresponding to head movement and a second peak value P2 corresponding to microsaccades. With respect to the frequency spectrum of FIG. 6B, the control unit 52 of the pupil detection device 5 is, for example, 1 second to be processed based on the second peak value P2 which is the maximum within the range of the specific frequency band B1. It is detected that microsaccade has occurred inside (S14, S15). In this way, it is possible to detect whether or not microsaccade is occurring in a situation where head movements can occur at the same time without interfering with the head movements.

3. 3. Summary As described above, the pupil detection device 5 according to the present embodiment detects the movement of the pupil based on the image pickup result of the image pickup device 4 that captures the image including the pupil. The pupil detection device 5 includes an acquisition unit 51 and a control unit 52. The acquisition unit 51 acquires image data from the image pickup apparatus 4 as image pickup data indicating the image pickup result. The control unit 52 executes data processing based on the image data acquired by the acquisition unit 51. The control unit 52 extracts the time change of the pupil position as information indicating the movement of the pupil based on the image data (S12). The control unit 52 performs time-frequency analysis on the extracted information to detect the movement of the pupil (S13, S14).

According to the above-mentioned pupil detection device 5, by extracting information indicating the movement of the pupil and performing time-frequency analysis, for example, a specific pupil corresponding to an eye movement having a frequency band different from that of the head movement can be obtained. The movement can be easily detected.

In the present embodiment, the image pickup data of the image pickup apparatus 4 is image data indicating an image 60 for each frame. The control unit 52 extracts the temporal change of the position of the pupil 61 in the image 60 for each frame (S12). As a result, the time-frequency analysis of the movement of the pupil 61 can be performed within the frame rate range of the frame image 60 (FIG. 6) .

Further, in the present embodiment, the control unit 52 performs binarization processing on the brightness on the image 62 for each frame to determine the region 63 corresponding to the pupil 61 (see FIG. 5). The control unit 52 extracts the temporal change of the position 64 of the pupil 61 based on the region 63 determined for each frame (S12). Since the brightness of the pupil 61 is considered to be remarkably low in the image 62, the position of the pupil 61 can be accurately obtained by the binarization process.

Further, the detection system 1 according to the present embodiment includes an image pickup device 4 and a pupil detection device 5. The imaging device 4 captures an image including the pupil and generates imaging data. The pupil detection device 5 detects the movement of the pupil based on the imaging data. The detection system 1 can easily detect the movement of a specific pupil according to the microsaccade or the like by the pupil detection device 5, and can detect the arousal degree of the subject based on the detection result.

(Embodiment 2)
In the second embodiment, a configuration example of the detection system 1 using the differential detection type imaging device 4 for detecting the change in brightness will be described.

1. 1. Configuration In the detection system 1 according to the second embodiment, the configuration of the pupil detection device 5 is the same as that of the first embodiment. Hereinafter, the image pickup apparatus 4 according to the second embodiment will be described with reference to FIG. 7.

FIG. 7 is a diagram for explaining the image pickup apparatus 4 according to the present embodiment. FIG. 7 schematically shows the image pickup surface 4a in the image pickup apparatus 4 of the present embodiment and the output data for each pixel 40 on the image pickup surface 4a.

The detection system 1 of the present embodiment employs, as the image pickup device 4, a differential detection type camera device that detects the time change rate of the brightness for each pixel 40 on the image pickup surface 4a. As the configuration of the image pickup apparatus 4 of the present embodiment, for example, a known configuration known as a dynamic vision sensor (DVS) can be applied (see, for example, Non-Patent Document 1).

For example, the image pickup device 4 outputs an image pickup element having an image pickup surface 4a in which pixels 40 are arranged in a two-dimensional array, an optical lens that collects light on the image pickup surface 4a, and image pickup data (luminance change data D1). It is equipped with a data output circuit and the like. Each pixel 40 of the image pickup device includes, for example, a photodiode, a differentiator, a comparator, and the like. Each pixel 40 can be identified by the XY coordinates on the imaging surface 4a.

In the image pickup apparatus 4, an image is formed (imaged) on the image pickup surface 4a of the image pickup device by the light from the optical lens. The image pickup apparatus 4 of the present embodiment determines the threshold value of the time change rate of the brightness for each pixel 40 with respect to the image on the image pickup surface 4a, and detects an event in which the brightness of each pixel 40 changes. The event to be detected is an event in which a change in luminance occurs at the position of the pixel 40, and is specified by whether the change in luminance increases or decreases, the timing at which the luminance increases or decreases, and the XY coordinates of the pixel 40.

As shown in FIG. 7, the image pickup apparatus 4 generates luminance change data D1 indicating the detection result of each event. The luminance change data D1 illustrated in FIG. 7 includes the output data D11 when the luminance increases and the output data D12 when the luminance decreases. Of the luminance change data D1 illustrated in FIG. 7, the white squares mean the output data when the luminance increases. The black squares mean the output data when the brightness is reduced. The brightness change data D1 is an example of the image pickup data of the image pickup result by the image pickup apparatus 4 of the present embodiment.

In the image pickup apparatus 4, each pixel 40 detects an increase in luminance when the rate of change in brightness with time exceeds the upper limit threshold value, and detects a decrease in luminance when it falls below the lower threshold value. When each pixel 40 detects an increase or decrease in brightness, it immediately outputs a detection result at the detected timing. The pixel 40 whose brightness has not changed and the pixel 40 whose brightness with time change rate is between the upper limit threshold value and the lower limit threshold value do not particularly output the detection result. The image pickup apparatus 4 generates the luminance change data D1 by associating, for example, the XY coordinates of the pixel 40 that outputs the detection result, the timing of the detection result, and the information indicating the increase or decrease of the luminance. The image pickup apparatus 4 externally outputs the luminance change data D1 as time-series data in which the time width is set to a predetermined value, for example.

According to the above-mentioned imaging method of the imaging device 4, an operation cycle such as a frame cycle for synchronizing the detection results of all pixels does not occur. Therefore, the image pickup apparatus 4 of the present embodiment can detect a change in luminance in a time width significantly shorter than the frame period. The detectable time width is, for example, the reaction period in which the pixel 40 reacts to the change in brightness, for example, 100 microseconds.

2. Operation The operation of the detection system 1 according to the second embodiment will be described below.

2-1. Outline of Operation In the detection system 1 of the present embodiment, the pupil detection device 5 executes the microsaccade detection process (S1 in FIG. 3) by using the luminance change data D1 by the image pickup device 4. The outline of the microsaccade detection process in the present embodiment will be described with reference to FIG.

FIG. 8 shows a schematic diagram of various images in the microsaccade detection process of the present embodiment. 8 (a) and 8 (b) exemplify images including the pupil 6 before and after the occurrence of microsaccade. FIG. 8C exemplifies the luminance change image 7 corresponding to the image pickup result between FIGS. 8 (a) and 8 (b) by the image pickup apparatus 4 (FIG. 1).

The images illustrated in FIGS. 8 (a) and 8 (b) are formed on the image pickup surface 4a (FIG. 7) of the image pickup apparatus 4 before and after the occurrence of the microsaccade, respectively. The pupil 6 on the images of FIGS. 8 (a) and 8 (b) has been moved in the moving direction d1 by the microsaccade from the time of FIG. 8 (a) as shown in FIG. 8 (b). In the detection system 1 of the present embodiment, the image pickup apparatus 4 outputs a change in brightness for each pixel as brightness change data D1 at any time between FIGS. 8 (a) and 8 (b) (FIG. 7).

FIG. 8 (c) illustrates the luminance change image 7 corresponding to the luminance change data D1 obtained between FIGS. 8 (a) and 8 (b). Since the brightness of the region where the pupil 6 is located is significantly lower than the brightness of the region around the pupil 6, the movement of the pupil 6 such as a microsaccade causes a characteristic change in brightness in the brightness change image 7. it is conceivable that.

In the example of FIG. 8C, the brightness of the brightness change image 7 changes between the region 71 on the start end side and the region 72 on the end side in the moving direction d1 of the pupil 6. Specifically, in the region 71 on the starting end side, the pupil 6 was located at the time of FIG. 8 (a), but the pupil 6 is no longer located at the time of FIG. 8 (b), so that the brightness increases. .. Further, in the terminal region 72, the pupil 6 was not located at the time of FIG. 8 (a), but the pupil 6 was located at the time of FIG. 8 (b), so that the brightness decreased. do.

The microsaccade detection process of the present embodiment realizes automatic detection of microsaccades that may occur at any time by a simple process of counting changes in brightness as described above. Hereinafter, the details of the microsaccade detection process according to the present embodiment will be described.

2-2. Microsaccade detection process FIG. 9 is a flowchart illustrating the microsaccade detection process according to the second embodiment. For example, in step S1 of FIG. 3, the pupil detection device 5 of the present embodiment executes the microsaccade detection process according to the flowchart of FIG. 9 instead of the flowchart of FIG.

In the microsaccade detection process of the first embodiment, image data for a predetermined period including a plurality of frames was acquired (S11 in FIG. 4), and a temporal change in the pupil position was extracted (S12). In the microsaccade detection process of the present embodiment, the control unit 52 of the pupil detection device 5 acquires the brightness change data D1 for a predetermined period from the image pickup device 4 via the acquisition unit 51 (instead of the image data). S11A).

Based on the acquired luminance change data D1, the control unit 52 extracts the temporal change in the number of luminance changes instead of the temporal change in the pupil position (S12A). The brightness change number indicates the number of events in which the brightness of the same coordinates in the brightness change data D1 changes with time, that is, the number of events. For example, the control unit 52 divides the above-mentioned predetermined period into predetermined sample periods, and counts the number of luminance changes during each sample period in the acquired luminance change data D1 (S12A). The sample period is appropriately set to be 1/2 or less of the reciprocal of the frequency of the movement of the pupil to be detected.

With respect to the time change of the extracted luminance change number, the control unit 52 frequency-decomposes in the time direction as in the first embodiment (S13), and based on the result of the frequency decomposition, the movement of the pupil in a specific frequency band. Is determined (S14). The specific frequency is set near the peak frequency of the microsaccade speed, for example. As a result, the occurrence time of the microsaccade can be detected as in the first embodiment (S15).

2-2-1. About the method of searching the pupil region In step S12A in the above microsaccade detection process, the control unit 52 searches for the region corresponding to the pupil in the luminance change data D1, and the number of events having the coordinates in the searched region. May be counted as the number of changes in brightness. A method of searching the pupil region in the luminance change data D1 will be described with reference to FIGS. 10 to 11.

FIG. 10 is a scatter diagram illustrating the distribution of events in the luminance change data D1 when the microsaccade occurs. FIG. 10 shows the spatial distribution of the increase event E11 in which the brightness increases and the decrease event E12 in which the brightness decreases in the brightness change data D1 acquired in step S11A.

For example, the control unit 52 calculates the number of events E11 and E12 for each XY coordinate of each pixel in the acquired luminance change data D1 to generate the luminance change image 7. The luminance change image 7 may have two types of pixel values (for example, two colors) corresponding to the increase event E11 and the decrease event E12 in each pixel.

The distribution of FIG. 10 includes an elliptical region R1 corresponding to the microsaccade-moved pupil 6. In the elliptical region R1, the brightness increase event E11 and the brightness decrease event E12 are distributed in an elliptical shape. Therefore, the control unit 52 searches the elliptical region R1 due to the movement of the pupil 6 in the generated luminance change image 7.

FIG. 11 is a diagram illustrating a method of searching for the elliptical region R1 in the luminance change image 7. The control unit 52 scans the luminance change image 7 using the scanning region R10, for example, as shown in FIG. The scanning region R10 has a predetermined elliptical shape and is set at various scanning positions on the luminance change image 7. The elliptical shape of the scanning region R10 is set to the dimensions and shape assumed as the elliptical region R1 due to the movement of the pupil 6.

When scanning the luminance change image 7, the control unit 52 sequentially counts a predetermined evaluation value for the image located inside the scanning region R10. The evaluation value is counted by the number of pixels having at least one event E11, E12 in the image inside the scanning region R10, for example. The evaluation value may be counted by the number of brightness changes in the scanning region R10.

The elliptical region R1 is searched by comparing the evaluation values of the scanning region R10 at various scanning positions. For example, the control unit 52 detects the scanning region R10 at the scanning position where the evaluation value is the maximum value or the maximum value as the elliptical region R1.

Further, the search for the elliptic region R1 is not limited to scanning by the scanning region R10, and for example, pattern recognition may be applied, or a Hough transform using an elliptic curve equation or the like may be applied. Further, the data corresponding to the moving pupil 6 may be extracted not only by the image processing of the luminance change image 7 but also by the event processing of the luminance change data D1.

2-2-2. Confirmation experiment A confirmation experiment regarding the effect of the microsaccade detection process of the present embodiment will be described with reference to FIGS. 12 and 13.

FIG. 12 is a graph showing the experimental results of a pupil model experiment relating to the pupil detection device 5 of the second embodiment. In FIG. 12, time (μs) is shown, and the vertical axis shows the number of luminance changes.

In FIG. 12, a model experiment was conducted to simulate a state in which eye movements and head movements were superimposed. As a model of the pupil, a black round member having a diameter of 3 mm was used. While vibrating the model in response to eye movements, the model was moved at 100 m / s in response to head movements. The vibration of the model was set to a frequency of 100 Hz and an amplitude of 200 μm so as to correspond to a microsaccade, and a maximum speed of 20 mm / s was obtained.

The movement of the pupil model as described above was imaged by the imaging device 4, and the number of changes in brightness was counted by the pupil detecting device 5 (S15). The imaging conditions of the imaging device 4 were an optical magnification of 1.0 and a resolution of 304 × 240 pixels. The sample period (S12A) for counting the number of luminance changes was 0.5 milliseconds.

In the experimental results of FIG. 12, the brightness increase event E11 and the brightness decrease event E12 are, for example, 500 or more when the pupil model is within the angle of view of the image pickup apparatus 4. The number of changes in brightness increases as the object such as the pupil moves faster, and it is considered that there is a correlation with the speed of the object. Therefore, for example, if the pupil moves due to a head movement unrelated to the eye movement, it is considered that the number of changes in brightness increases. On the other hand, the pupil detection device 5 of the present embodiment frequency-decomposes the time change of the number of luminance changes and analyzes the high-frequency component that does not include the head movement (S13).

FIG. 13 is a graph showing the frequency spectrum of the experimental result of FIG. In FIG. 13, the frequency (Hz) is shown, and the vertical axis shows the intensity of the frequency component.

In FIG. 13, frequency analysis was performed on the number of luminance changes of 1024 samples for 512 milliseconds. In the frequency spectrum of FIG. 13, the first peak P11 and the second peak P12 separated from each other were confirmed. The first peak P11 is located near 0 Hz and is considered to correspond to a simulated locomotion of head movement. The second peak P12 is located near 100 Hz and is considered to correspond to a simulated vibrational motion of eye movement. Therefore, even when an actual microsaccade or the like occurs superimposed on the head movement, the temporal change in the number of luminance changes is frequency-decomposed to obtain the movement of the pupil in a specific frequency band according to the microsaccade. It was confirmed that it was detectable.

3. 3. Summary As described above, in the pupil detection device 5 according to the present embodiment, the image pickup data acquired from the image pickup device 4 is the brightness change data D1 indicating the timing of the event in which the brightness changes for each pixel. The control unit 52 extracts the temporal change of the number of events (that is, the number of brightness changes) of the event in which the brightness has changed based on the brightness change data D1 (S12A). The number of events correlates with the speed of movement of the pupil 6, and the temporal change in the number of events is an example of information indicating the movement of the pupil 6. According to the pupil detection device 5 of the present embodiment, information indicating the movement of the pupil 6 can be easily extracted by counting the number of events.

In the present embodiment, the control unit 52 frequency-decomposes the temporal change of the number of extracted events (S13) and detects the movement of the pupil in a specific frequency band (S14). As a result, high-resolution time-frequency analysis can be easily performed according to the change in the brightness in the brightness change data D1.

Further, in the present embodiment, the specific frequency band includes, for example, a frequency corresponding to the microsaccade of eye movement. By specifying the frequency band of the movement of the pupil 6 to be detected, the movement of the specific pupil 6 can be easily detected.

(Other embodiments)
In the first and second embodiments described above, an example in which the detection target by the pupil detection device 5 is a microsaccade has been described. The detection target by the pupil detection device 5 is not limited to the microsaccade, and may be other saccades, saccades, or blinks. In this case, a specific frequency band can be appropriately set according to the movement of the pupil to be detected.

Further, in each of the above embodiments, an example in which the detection system 1 detects the arousal level of the target person has been described, but the detection target of the detection system 1 is not limited to the arousal level, and depends on the detection result of the pupil detection device 5. It may be various indicators that can be detected. For example, the detection system 1 may detect the degree of collection or the degree of fatigue of the subject.

Further, in the above description, the application example of the detection system 1 to the in-vehicle use has been described, but the detection system 1 according to the present disclosure is not limited to the in-vehicle use. The detection system 1 according to the present disclosure can be applied to various environments in which the eyes of a subject can be imaged, and can be applied, for example, to monitor workers in a factory.

(Additional note)
As described above, various embodiments of the present disclosure have been described, but the present invention is not limited to the above-mentioned disclosure contents, and various modifications can be made within the scope of the present invention. Hereinafter, various aspects relating to the present disclosure will be added.

The first aspect according to the present disclosure is the pupil detection device (5) that detects the movement of the pupil based on the image pickup result of the image pickup device (4) that captures an image including the pupil. The pupil detection device includes an acquisition unit (51) and a control unit (52). The acquisition unit acquires imaging data indicating the imaging result from the imaging device. The control unit executes data processing based on the imaging data acquired by the acquisition unit. The control unit extracts information indicating the movement of the pupil based on the imaging data (S12, S12A). The control unit performs time-frequency analysis on the extracted information to detect the movement of the pupil (S13, S14).

In the second aspect, in the pupil detection device of the first aspect, the imaging data shows an image (60) for each frame. The information indicating the movement of the pupil indicates a temporal change in the position of the pupil. The control unit extracts the temporal change of the position of the pupil (61) in the image for each frame.

In the third aspect, in the pupil detection device of the second aspect, the control unit performs binarization processing on the brightness of the image for each frame to determine the region (63) corresponding to the pupil. .. The control unit extracts a temporal change in the position of the pupil based on the region determined for each frame (S12).

In the fourth aspect, in the pupil detection device of the first aspect, the imaging data (D1) indicates the timing at which the brightness changes for each pixel. The information indicating the movement of the pupil indicates a temporal change in the number of brightness changes, which is the number of timings at which the brightness changes. The control unit counts the number of brightness changes based on the imaging data and extracts the temporal change of the brightness change (S12A).

In the fifth aspect, in the pupil detection device of the fourth aspect, the control unit frequency-decomposes the temporal change of the extracted luminance change number (S13) and detects the movement of the pupil in a specific frequency band. (S14).

In the sixth aspect, in the pupil detection device of the fifth aspect, the specific frequency band includes a frequency corresponding to the microsaccade of eye movement.

A seventh aspect is a detection system (1) including an imaging device and a pupil detection device according to any one of the first to sixth aspects. The imaging device captures an image including the pupil and generates imaging data. The pupil detection device detects the movement of the pupil based on the imaging data.

1 Detection system 2 Vehicle 3 Driver 4 Imaging device 5 Pupil detection device 51 Acquisition unit 52 Control unit

Claims (4)

A pupil detection device that detects the movement of the pupil based on the imaging result of the image pickup device that captures an image including the pupil.
An acquisition unit that acquires imaging data indicating the imaging result from the imaging device, and
It is provided with a control unit that executes data processing based on the imaging data acquired by the acquisition unit.
The control unit
Based on the imaging data, information indicating the movement of the pupil is extracted.
Time-frequency analysis is performed on the extracted information to detect the movement of the pupil .
The imaging data indicates the timing at which the brightness changes for each pixel.
The information indicating the movement of the pupil indicates a temporal change in the number of brightness changes, which is the number of timings at which the brightness changes.
The control unit is a pupil detection device that counts the number of changes in luminance based on the imaging data and extracts the temporal change in the number of changes in luminance. The pupil detection device according to claim 1 , wherein the control unit frequency-decomposes the temporal change of the extracted luminance change number to detect the movement of the pupil in a specific frequency band. The pupil detection device according to claim 2 , wherein the specific frequency band includes a frequency corresponding to a microsaccade of eye movement. An imaging device that captures an image including the pupil and generates imaging data,
A detection system including the pupil detection device according to any one of claims 1 to 3 , which detects the movement of the pupil based on the imaging data.

Images