JP2022082280A - Image processing system and image processing method - Google Patents

Abstract

To accurately detect an object person's point of fixation on an observation object whose position and angle change.SOLUTION: An image processing system 1 comprises: an optical system 3 for object person detection for acquiring a face image necessary for detection of the angle and position of the line of sight of an object person Sp; an optical system 5 for object detection for acquiring an object image necessary for detection of the arrangement angle and position of an observation object So that is observed by the object person Sp; and a computer 9 that executes image processing on the face image and the object image. The computer 9 specifies the three-dimensional coordinates of a marker on the observation object So based on the object image, calculates the arrangement angle and position of the observation object So based on the three-dimensional coordinates of the marker; calculates the angle and position of the line of sight of the object person Sp based on the face image, and detects the object person Sp's point of fixation on a specific surface on the observation object So based on the angle and position of the line of sight and the arrangement angle and position of the observation object So.SELECTED DRAWING: Figure 1

Description

The present invention relates to an image processing system and an image processing method for processing an image.

In recent years, a face image processing device that processes a face image obtained by using a video camera has become widespread (see, for example, Patent Documents 1 and 2 below). For example, in such a face image processing device, by processing a face image, the three-dimensional coordinates of the pupil and the angle of the line of sight with respect to the straight line connecting the center of the pupil from the camera are obtained, and the line of sight is detected based on these. ..

Patent No. 4500992 Patent No. 4517049

In recent product manufacturing sites, product inspections are carried out by a target person such as an inspector holding the product in his hand and performing a visual inspection. Even in such a scene, the start point and direction of the line of sight of the subject can be detected by using the devices described in Patent Documents 1 and 2 described above. However, at the time of product inspection, the position and angle of the observation object such as the product may change constantly, and even in such a case, it is possible to detect the gazing point of the object on the observation object and output the detection information. It is desired. Such detection information can be utilized for improving the inspection technique of the inspector or improving the inspection efficiency of the inspector.

The present invention has been made in view of the above problems, and provides an image processing system and an image processing method capable of accurately detecting the gaze point of a subject with respect to an observation object whose position and angle change. The purpose is.

In order to solve the above problems, the image processing system according to one embodiment of the present invention includes a camera that captures the face of the subject, and acquires a face image necessary for detecting the angle and position of the line of sight of the subject. An optical system for detecting an object and an optical system for detecting an object for acquiring an object image necessary for detecting the arrangement angle and position of the object to be observed, including a camera for capturing the observation object of the object to be observed by the object. It includes a system and a calculation unit that executes image processing on the face image and the object image. The calculation unit identifies the three-dimensional position of the marker on the observation object based on the object image, and the marker. The placement angle and position of the observation object are calculated based on the three-dimensional position of, and the line-of-sight angle and position of the subject are calculated based on the face image. Based on the position and the position, the gaze point of the subject on a specific surface on the observation object is detected.

Alternatively, the image processing method according to another aspect of the present invention is an image processing method executed by an image processing system, which is necessary for capturing an image of the subject's face and detecting the angle and position of the subject's line of sight. A step of acquiring a face image, a step of acquiring an object image necessary for detecting the arrangement angle and position of the observation object by imaging the observation object of the object to be observed, and a face image and an object image. In the step of executing image processing, the three-dimensional position of the marker on the observation object is specified based on the object image, and the three-dimensional position of the marker is used as the basis. In addition, the arrangement angle and position of the observation object are calculated, the angle and position of the line of sight of the subject are calculated based on the face image, and the angle and position of the line of sight and the arrangement angle and position of the observation object are used as the basis. , Detects the gaze point of the subject on a specific surface of the object to be observed.

In the image processing system or image processing method of the above-described embodiment, the face image of the subject is acquired and the object image, which is an image of the observation object, is acquired. Then, the arrangement angle and position of the observation object are calculated using the three-dimensional position of the marker on the observation object detected from the object image, and the angle and position of the line of sight of the subject are calculated from the face image. Based on them, the gaze point of the subject on the observation object is detected. As a result, even when the position and angle of the observation object change due to a visual inspection of the observation object or the like, the gazing point of the subject with respect to the observation object can be accurately detected.

Here, the calculation unit specifies the three-dimensional positions of three or more markers on the observation object, and calculates the arrangement angle and position of the observation object based on the three-dimensional positions of the three or more markers. It may be that. In this case, by using the positions of three or more independent markers, the placement angle and position of the observation object can be calculated with higher accuracy, and the gazing point of the subject with respect to the observation object can be detected with higher accuracy. can.

Further, a support member that integrally supports the target person detection optical system and the target object detection optical system may be further provided. In such a configuration, it is possible to stabilize the relationship between the line of sight of the subject detected by using the optical system for detecting the subject and the arrangement of the observation object detected by using the optical system for detecting the object. , It is possible to accurately detect the gaze point of the subject with respect to the observation object.

Further, a flat plate member which is supported by the support member and serves as a reference when the subject grips the observation object may be further provided. In this case, since the position of the observation object with respect to the object detection optical system is stabilized, the arrangement angle and position of the observation object can be reliably detected. As a result, it is possible to more reliably detect the gaze point of the subject with respect to the observation object.

In addition, the calculation unit obtains the formula of a specific surface in the three-dimensional coordinate system based on the arrangement angle and position of the observation object, and calculates the gazing point based on the formula and the vector representing the line of sight. May be good. By doing so, since the arrangement of the specific surface on the observation object can be specified by the formula, the gazing point of the subject on the specific surface on the observation object can be calculated with high accuracy.

In addition, the calculation unit obtains an equation for each surface that divides a specific surface into a plurality of surfaces based on the arrangement angle and position of the observation object, and uses the equations of the plurality of surfaces to obtain a vector from the plurality of surfaces. A surface having an intersection may be searched for, and the intersection of the searched surface and the vector may be obtained as a gazing point. According to such a configuration, even when an observation object having a complicated shape is targeted, the arrangement of specific surfaces on the observation object can be specified by an equation for each surface divided into a plurality of areas. It is possible to calculate the gaze point of the subject on a specific surface of the observation object with high accuracy.

In addition, the calculation unit stores the positional relationship of a plurality of true markers in advance, narrows down the three-dimensional positions of the specified plurality of marker candidates to the three-dimensional positions of the plurality of true markers, and narrows down the plurality of positions. The placement angle and position of the observation object may be calculated based on the three-dimensional position of the true marker. In this case, when a plurality of markers including a fake marker are detected in the object image, the true marker can be narrowed down from those markers. As a result, the arrangement of the observation object can be detected without error, and the gazing point of the subject on a specific surface on the observation object can be detected without error.

In addition, the calculation unit identifies the three-dimensional position of the calibration optotype based on the image obtained in advance by the object detection optical system, and bases the face image on the basis of the three-dimensional position of the calibration optotype. A calibration process for calculating the angle and position of the line of sight may be performed in advance. In this case, the accuracy of detecting the gazing point of the subject can be improved, and the gazing point of the subject on a specific surface on the observation object can be detected with high accuracy.

According to the present invention, it is possible to accurately detect the gaze point of the subject with respect to the observation object whose position and angle change.

It is a side view which shows the image processing system which concerns on embodiment. FIG. 3 is an external view of an observation object So, which is a processing target of the image processing system of FIG. 1. It is a figure which shows the example of the aspect of the markers _{M A} , MB, and _MC attached to the observation object So. It is a figure which shows the mode of the optotype displayed on the observation object So. It is a figure which shows the hardware configuration of the computer which concerns on embodiment. It is a block diagram which shows the functional structure of the computer which concerns on embodiment. It is a flowchart which shows the procedure of the gaze point detection process by the image processing system which concerns on embodiment. It is a side view of the observation object So which is the processing target of the image processing system which concerns on embodiment.

Hereinafter, preferred embodiments of the image processing system according to the present invention will be described in detail with reference to the drawings. In the description of the drawings, the same or corresponding parts are designated by the same reference numerals, and duplicate description will be omitted.

First, with reference to FIG. 1, the overall configuration of the image processing system 1 according to the embodiment will be described. FIG. 1 is a side view of the image processing system 1. The image processing system 1 is a processing system that detects the trajectory of the inspector's gazing point on the surface of the product and records the detection result when the inspector visually inspects a plate-shaped product such as an electronic substrate. .. This image processing system 1 acquires a face image and an object image by imaging the subject Sp who is an inspector and the observation object So which is a product to be inspected, and targets the face image and the object image. A group of devices that perform image processing. The image processing system 1 includes an optical system for detecting an object, an optical system for detecting an object 5, a table 7, a computer (calculation unit) 9, and a support member 11. Hereinafter, each component of the image processing system 1 will be described.

The table 7 is a member having a structure in which the top plate portion 7a, which is a reference for gripping the observation object So when the subject Sp performs an inspection, is supported by the support column 7b. For example, the table 7 is used to support the forearm portion of the subject Sp when the subject Sp grips the observation object So to stabilize the arrangement angle and position of the observation object So, and the optical system 3 for detecting the subject. It is provided to cover and protect the. The top plate portion 7a is preferably made of a light-transmitting material such as glass in order to prevent the subject Sp from being photographed by the subject detection optical system 3. When the table 7 is provided mainly for the purpose of covering and protecting the target person detection optical system 3, the top plate portion 7a has a shape extended toward the target person Sp side, and the material thereof is glass or the like. It is composed of a light-transmitting material. On the other hand, when the table 7 is provided mainly for the purpose of stabilizing the arrangement angle and position of the observation object So, the top plate portion 7a is photographed by the subject detection optical system 3 as shown in FIG. The shape may be such that it does not become an obstacle, and the material may be made of an opaque material.

The target person detection optical system 3 is fixed to one end side (left side in FIG. 1) of the top plate portion 7a at the lower part of the top plate portion 7a of the table 7, and is located away from the one end side of the top plate portion 7a. Includes two cameras (stereo cameras) whose optical axes are adjusted to face Sp's face. Each camera is calibrated in advance after the subject detection optical system 3 is attached to the support member 11, and captures the face of the subject Sp to acquire and output a face image. In the present embodiment, the two cameras are NTSC CMOS (Complementary Metal Oxide Semiconductor) video cameras, which is one of the interlaced scan methods, and each camera has the pupil, corneal reflex, or nose of the subject Sp. In order to detect such feature points, a near-infrared light source having the configuration described in Patent No. 4500992 (for example, two types of light sources having emission wavelength bands of 850 nm and 940 nm) is attached. The subject detection optical system 3 turns on the near-infrared light source and images the face of the subject Sp in response to a command from the computer 9, and outputs the face image data to the computer 9. These facial images are used in the computer 9 to detect the angle and position of the line of sight of the subject Sp.

The object detection optical system 5 is fixed at a position away from the top plate portion 7a on the other end side (right side in FIG. 1) of the top plate portion 7a, and the optical axis is adjusted so as to face the upper portion of the top plate portion 7a. Includes two cameras (stereo cameras). Each camera is pre-calibrated after the object detection optical system 5 is attached to the support member 11, and the observation object So gripped by the object Sp is opposite to the surface facing the object Sp. The image is taken from the side, and the object image is acquired and output. In the present embodiment, the two cameras are NTSC CMOS video cameras, which is one of the interlaced scan methods, and each camera has a near-infrared light source (near-infrared light source) for detecting a marker on the observation object So. A light source with an emission wavelength band of 940 nm) is attached. The object detection optical system 5 turns on the near-infrared light source and images the observation object So in response to a command from the computer 9, and outputs the data of the object image to the computer 9. These object images are used in the computer 9 to detect the arrangement angle and position of the observation object So.

The computer 9 controls the object detection optical system 3 and the object detection optical system 5, and targets the face image and the object image acquired by the object detection optical system 3 and the object detection optical system 5. It is a data processing device that executes the image processing. The computer 9 may be built by a stationary or portable personal computer (PC), a workstation, or another type of computer. Alternatively, the computer 9 may be constructed by combining a plurality of arbitrary types of computers. When using multiple computers, these computers may be connected via a communication network such as the Internet or an intranet.

The support member 11 is a support member that is placed on a flat fixed surface such as a floor surface and integrally supports the table 7, the object detection optical system 3, and the object detection optical system 5. .. That is, the support member 11 is the subject detection optical so that the optical axis of the support member 11 faces diagonally to the upper left in the direction away from the top plate portion 7a at the lower end side (left side in FIG. 1) of the top plate portion 7a of the table 7. Supports system 3. At this time, it is preferable that the support member 11 supports the target person detection optical system 3 so that the optical axis of the camera of the target person detection optical system 3 does not pass through the top plate portion 7a. In addition, the support member 11 is positioned away from the other end side (right side in FIG. 1) of the top plate portion 7a of the table 7, and the optical axis for object detection is such that the optical axis of the support member 11 faces the upper portion of the top plate portion 7a. Supports system 5. Here, the support member 11 may be configured so that the height of the table 7 from the fixed surface can be adjusted according to the height of the subject Sp, and in that case, the height of the support member 11 may be adjusted according to the position of the face of the subject Sp. The tilt of the optical axis of the camera of the object detection optical system 3 may be adjustable, and the tilt of the optical axis of the camera of the object detection optical system 5 may be adjusted according to the height of the table 7. It may be configured to be possible.

When the image processing system 1 having the above configuration is used, the target person Sp is positioned so that the face is within the shooting range of the target person detection optical system 3. Then, the subject Sp is made to grip the observation target So in the vicinity of the upper surface of the top plate 7a with its face facing upward of the top plate 7a of the table 7, and a specific surface of the observation target So is held. The surface to be inspected is visually inspected.

FIG. 2 shows an example of the appearance of the observation object So, the part (a) shows the appearance of the observation object So as seen from the inspection target surface P1 _side , and the part (b) shows the observation object So. Is shown as seen from the surface P2 _on the opposite _side of the surface P1 to be inspected. The observation object So shown in FIG. 2 has a shape in which uneven surfaces are irregularly formed as a whole, and the inspection target surface P1 has an optotype M1 for calibration of the line _- of _- sight detection process of the subject Sp. It is attached. Further, the surface P ₂ on the opposite side of the surface P ₁ to be inspected is provided with three markers M _A , M _B , and _MC used as a reference for calculating the arrangement angle and the angle of the observation object So. ing. Here, as an example of the shape of the observation object So, a shape having an irregular uneven surface is given, but a flat plate shape having a flat surface may be used, or a surface shape such as a cylindrical shape may be used. It may be a shape having a curved surface that can be approximated by a mathematical formula.

As the markers _{M A} , MB, and _MC attached to the observation object So, a marker having a color conspicuous on the object image such as white is preferable, and a light emitting body such as an LED may be used as the marker. When is used, an LED having a low light emitting portion and a weak directivity is used so that the image appears in a shape that enhances the center accuracy in the object image. When a near-infrared camera is used as the camera of the optical system 5 for detecting an object, it is preferable to use a marker made of a retroreflective material, and a color camera is used as the camera of the optical system 5 for detecting an object. When used, a marker having a color that stands out against the background (for example, blue) may be used.

Part (a) and part (b) of FIG. 3 show examples of aspects of the markers M _A , M _B , and M _C. As described above, it is preferable that the markers _{M A} , MB, and _MC have an embodiment in which the inner region of a predetermined shape (for example, a rectangle) having a uniform color is surrounded by the outer region of another color. In such an embodiment, as shown in part (a) of FIG. 3, the inner region has a color that easily reflects infrared rays (for example, white), and the outer region has a color that easily absorbs infrared rays (for example, for example). It is preferable to have black), and as shown in part (b) of FIG. 3, the inner region has a color that easily absorbs infrared rays (for example, black), and the outer region has a color that easily reflects infrared rays (for example, black). For example, it is also preferable to have white).

Further, the calibration target M ₁ attached to the inspection target surface P ₁ of the observation target So may be a fixed size optotype, but the size may be once by using a small liquid crystal display or the like. However, the optotype M1 may be in _a form that changes repeatedly. By using the optotype M ₁ in such a form, it is possible to make the optotype easily attract the visual attention of the subject, so that the subject Sp can more accurately direct the line of sight to the optotype M ₁ . As a result, the calibration can be performed more accurately while the calibration reduces the burden on the subject Sp. FIG. 4 is a diagram showing an example of _an embodiment of the optotype M1 displayed on the observation object So. As described above, it is preferable that the optotype M1 is displayed only once or repeatedly in such a manner that the optotype M1 is dynamically changed from a large size to a small size.

FIG. 5 is a block diagram showing a general hardware configuration of the computer 9. The computer 9 includes a CPU (processor) 101 for executing an operating system, an application program, and the like, a main storage unit 102 composed of a ROM and a RAM, an auxiliary storage unit 103 composed of a hard disk, a flash memory, and the like, and a network. It includes a communication control unit 104 composed of a card or a wireless communication module, an input device 105 such as a keyboard and a mouse, and an output device 106 such as a display and a printer.

Each functional element of the computer 9, which will be described later, loads predetermined software on the CPU 101 or the main storage unit 102, and operates the communication control unit 104, the input device 105, the output device 106, and the like under the control of the CPU 101. This is realized by reading and writing data in the storage unit 102 or the auxiliary storage unit 103. The data and database required for processing are stored in the main storage unit 102 or the auxiliary storage unit 103.

As shown in FIG. 6, the computer 9 includes an image pickup control unit 21, an image acquisition unit 23, a posture calculation unit 25, a line-of-sight calculation unit 27, and a gaze point detection unit 29 as functional components. Hereinafter, the functions of each component of the computer 9 will be described.

The image pickup control unit 21 controls the image pickup by the target person detection optical system 3 and the object detection optical system 5 after the gaze point detection process for the target person Sp is started. At that time, as described in Patent No. 4500992, the image pickup control unit 21 has two types of emission wavelengths near red attached to the stereo camera according to the image pickup timing by the stereo camera of the subject detection optical system 3. Controls the lighting timing of the external light source. By controlling the image pickup control unit 21 in this way, in each camera constituting the stereo camera, as a one-frame face image obtained 30 images per second, an image of an odd field composed of odd-th horizontal pixel lines and an image of an odd field. , An even-field image composed of even-th horizontal pixel lines is obtained, and the odd-field image and the even-field image are a face image (bright pupil image) in which the pupil appears relatively bright and a relatively bright pupil. It is generated by alternately taking pictures at intervals of 1/60 second as a face image (dark pupil image) that appears dark. However, the frame rate for shooting with a stereo camera may be set as appropriate.

Further, the image pickup control unit 21 controls the image pickup of the object detection optical system 5 at the same time as the control of the image pickup of the object detection optical system 3. At that time, the image pickup control unit 21 controls the lighting timing of the near-infrared light source attached to the stereo camera according to the image pickup timing by the stereo camera of the object detection optical system 5. By such control of the image pickup control unit 21, one frame of an object image that can be simultaneously obtained at 30 images per second can be obtained in each camera constituting the stereo camera. Here, although the frame rate for shooting with the stereo camera can be appropriately set, it may be set to be the same as the frame rate for shooting the bright pupil image and the dark pupil image obtained by the subject detection optical system 3. preferable. When the subject Sp grips the observation object So on the table 7 when visually inspecting the observation object So, the surface P ₂ on the opposite side of the inspection target surface P ₁ of the observation object So (FIG. Since (part (b) of 2) faces the object detection optical system 5, the images of the markers _{M A} , MB, and _MC are reflected in the object image captured by the object detection optical system 5. Become. The image pickup control unit 21 is for detecting an object in order to prevent blurring of the images of the _{markers MA} , _MB , and MC and to accurately detect the positions of the _{markers MA} , _MB , and MC on the image. It is preferable to set the exposure time of each camera of the optical system 5 as short as possible (for example, 500 μsec). Further, when a light emitting body such as an LED whose lighting can be controlled by the computer 9 is used as the _{markers MA} , MB, and MC, the image pickup control unit 21 synchronizes with the lighting timing of the near _- infrared light source, and the marker _MA is used. , _MB , and MC are preferably _controlled to light up.

The image acquisition unit 23 transfers a face image and an object image in real time from the stereo camera of the object detection optical system 3 and the stereo camera of the object detection optical system 5 via a wired or wireless data transfer interface. Get it with. Then, the image acquisition unit 23 delivers the acquired face image to the line-of-sight calculation unit 27 each time, and delivers the acquired object image to the posture calculation unit 25 each time.

The posture calculation unit 25 calculates the arrangement angle and position (posture) of the observation target So based on the pair of object images simultaneously obtained by the stereo camera of the object detection optical system 5. The posture calculation function of the observation object So of the posture calculation unit 25 will be described below. The posture calculation unit 25 continuously executes the following posture calculation processing for each frame of the continuously obtained object image.

The posture calculation unit 25 detects the position of the marker on the image from each of the pair of object images. That is, after converting the object image into a binarized image using the threshold value of the pixel value and labeling the binarized image, the area and the vertical and horizontal directions that are typical of the reflection images of the _{markers MA} , _MB , and MC. By searching for the ratio image, a candidate reflection image of the marker is selected from the labels. At this time, the attitude calculation unit 25 can also select a candidate for the reflection image of the marker by geometric shape pattern matching corresponding to the shapes (for example, rectangles ₎ of the markers M _A , MB, and _MC , and is an observation target. Geometric shape pattern matching is performed so that the posture of the object So can be rotated and the shape of the reflected image can be changed.

Further, the attitude calculation unit 25 acquires the center coordinates on the image of the selected marker candidate with respect to the pair of object images, and stereos the center coordinates of the marker candidates on the pair of object images in a round-robin manner. By performing matching, the three-dimensional coordinates of the marker candidates in the world coordinate system _XW - _YW - _ZW (FIG. 1), which is a predetermined three-dimensional coordinate system, are specified. Further, when the specified marker candidate has four or more points, the posture calculation unit 25 has three true markers _{M A} , MB from the three-dimensional coordinates of the specified four or more marker candidates. , _MC 3D coordinates are narrowed down. At this time, the attitude calculation unit 25 stores in advance data indicating the positional relationship of the three _{markers MA} , _MB , and MC in a database such as the auxiliary storage unit 103, and the marker corresponding to the data. The candidate 3D coordinates are narrowed down as the 3D coordinates of the true marker. For example, the data of the vector V _A from the marker M _A to the marker M _B and the data of the vector V _B from the marker M _A to the marker M _C are recorded in advance, and the data correspond to these vectors V _A and V _B. The three-dimensional coordinates of the candidate markers to be used are selected as those of the true markers M _A , M _B , and M _C. This results in image noise due to reflection of near-infrared light or light from illumination provided in the inspection environment, and is a false marker _MX that was mistakenly recognized as a marker on the observation object So ((b) in FIG. 2). ) Can be excluded from the true markers.

In addition, the attitude calculation unit 25 uses the three _- dimensional coordinates of the narrowed _- down three markers _MA , MB, and MC as the attitude of the observation object So in the world coordinate system _XW - _YW - _ZW . , The position and direction of the object coordinate system X _h -Y _h -Z _h in the world coordinate system X _W -Y _W -Z _W are calculated. That is, the attitude calculation unit 25 obtains the midpoint of the three _- dimensional coordinates of the two markers M _A and MC as the coordinates Oh of the center of the object coordinate system X _h − Y _h − Z _h , and obtains the coordinates _Oh of the three points. The normal vector of the plane passing through the markers _{M A} , MB, and _MC is obtained as a direction vector defining the Z _h axis. Further, two vectors perpendicular to each other obtained by synthesizing the vector _VA from the marker M _A to the marker M _B and the vector V _B from the marker M _A to the marker _MC at a predetermined ratio are formed on the X _h axis and the Y. Obtained as a direction vector that defines the _h -axis. Then, the attitude calculation unit 25 determines the plane of the observation object So in the world coordinate system _XW - _YW - _ZW based on the normal vector of the plane and the direction vector defining the _Xh axis and the _Yh axis. The rotation angles α, β, and γ around the _YW axis, the _XW axis, and the ZW axis of the specified object coordinate system X _h - _{Y h} -Z _h are calculated, respectively. In this way, the posture calculation unit 25 calculates the center coordinates _Oh and the rotation angles α, β, and γ as parameters for specifying the posture of the observation object So.

Further, the attitude calculation unit 25 identifies the three-dimensional coordinates of the marker candidates described above, and at the time of the calibration process of the line-of-sight vector, the three _- dimensional of the _{optotype M1} in the world coordinate system XW- _YW -ZW. Specify the coordinates.

The line-of-sight calculation unit 27 targets each face image obtained by the stereo camera of the target person detection optical system 3, and detects the line-of-sight of the target person Sp in the same manner as the method described in Japanese Patent No. 4500992. To execute. At this time, the line-of-sight calculation unit 27 executes the line-of-sight detection process for each frame of the face image continuously obtained by the subject detection optical system 3.

Specifically, the line-of-sight calculation unit 27 performs stereo matching based on the position of the pupil detected on the face image to obtain the three-dimensional coordinates of the pupil position in the world coordinate system _XW - _YW - _ZW . calculate. At this time, the line-of-sight calculation unit 27 obtains a vector from the pinhole position toward the pupil based on the facial images of the two cameras constituting the stereo camera, and obtains the intersection of the two vectors to obtain the pupil position. Three-dimensional coordinates can be calculated. Further, the line-of-sight calculation unit 27 can also detect the latest points of the two vectors as the pupil positions by using the method described in International Publication WO2015 / 190204. In this case, the pupil position can be obtained even when the intersection of the two vectors does not exist due to an error.

Further, the line-of-sight calculation unit 27 uses the method described in Patent No. 4500992 based on the pupil position calculated based on the face image and the positions of the pupil and the corneal reflex on the face image detected based on the face image. Then, the line-of-sight vector of the subject Sp is calculated. Here, the line of sight is a straight line connecting the pupil and the gazing point, and the line of sight vector is data representing the position of the start point of the line of sight and the angle (direction) of the line of sight in the world coordinate system _XW - _YW - _ZW . be.

Further, the line-of-sight calculation unit 27 uses the three-dimensional coordinates of the calibration optotype M1 (part ( _a ) in FIG. 2) detected by the object image by using the method described in the international publication WO2012 / 020760. Then, a process of calibrating the coefficient used for calculating the line-of-sight vector may be performed in advance. However, this calibration process does not have to be executed at each time of the line-of-sight detection process, and may be executed at least once at every timing when the subject Sp changes and at every timing when the type of the observation object So changes.

For example, instead of the observation object So, a small liquid crystal display or the like having the same size as the observation object So (a display device having a configuration as shown in FIG. 8 to be described later) is displayed in the display mode shown in FIG. The subject Sp may hold the small liquid crystal display while M ₁ is displayed. That is, after the optotype M1 _first appears large on the small liquid crystal display and the subject Sp focuses on a small point at the center of the optotype M1, for a predetermined time (for example, about ₁ second). While gradually reducing the size of the optotype M ₁ and displaying it, pay attention to the center of the optotype M ₁ following the subject Sp. The line-of-sight calculation unit 27 uses the face image obtained in such a state to perform a process of calibrating the coefficient used for calculating the line-of-sight vector. After this calibration process, the line-of-sight calculation unit 27 executes the calculation process of the line-of-sight vector of the target person Sp in a state where the subject Sp is switched to the observation object So. _In this way, by executing the calibration process with the target person Sp paying attention to the optotype M1 displayed on the small display having the same size as the observation object So, the optotype is displayed at the time of the calibration process. _The distance from M1 to the face of the subject Sp can be set to be equal to the distance from the observation object So to the face, and the accuracy of the gaze point detection by the gaze point detection unit 29 can be improved.

The gazing point detection unit 29 uses the posture data of the observation object So calculated by the posture calculation unit 25 and the line-of-sight vector of the subject Sp calculated by the line-of-sight calculation unit to inspect the observation object So. _The gaze point of the subject Sp on the surface P1 is detected. Then, the gazing point detection unit 29 visually outputs the detected gazing point data to an output device 106 such as a display. Here, the gazing point data of the output target may be the gazing point coordinate data in the object coordinate system X _h − Y _h − Z _h . Further, the gaze point detection unit 29 may store the detected gaze point data in the database in the auxiliary storage unit 103.

Specifically, the gazing point detection unit 29 uses the position and direction of the object coordinate system X _h -Y _h -Z _h calculated by the attitude calculation unit 25 to obtain the line-of-sight vector in the object coordinate system X _h -Y. Convert to a vector in _h -Z _h . In this vector transformation, the line-of-sight vector is moved from the origin to the world coordinate system _XW - _YW - _ZW of the object coordinate system _Xh - _Yh - _Zh (for example, the center coordinate _Oh ) and the axis. It is realized by performing conversion using the surrounding rotation state (for example, rotation angles α, β, γ). Then, the gazing point detection unit 29 specifies the converted line-of _- sight vector and the shape and position of the inspection target surface P1 in the object coordinate system _Xh - _Yh - _Zh stored in advance in the auxiliary storage unit 103. By finding the intersection of the line _- of-sight vector and the inspection target surface P1 based on the equation for performing, the three-dimensional coordinates of the gazing _point on the inspection target surface P1 of the subject Sp are detected.

At this time, the gazing point detection unit 29 may detect the gazing point as follows. That is, the gazing point detection unit 29 is in the world coordinate system _XW - _YW - _ZW based on the position and direction of the object coordinate system _Xh - _{Yh - Zh} and the equation of the inspection target surface P1. _Calculate the formula for the surface to be inspected P1. The calculation of this equation is based on the movement state of the origin with respect to the world coordinate system _XW - _YW - _ZW of the object coordinate system _Xh - _Yh - _Zh (for example, the center coordinate _Oh ) and the rotation state around the axis. (For example, rotation angles α, β, γ) are used to perform coordinate transformation according to the moving state and the rotating state. Then, the gazing point detection unit 29 uses the equation of the inspection target surface P1 in the world coordinate system _XW - _YW - _ZW and the line _- of-sight vector of the subject Sp to be the intersection of the inspection target surface P1 and the line-of _- sight vector. By obtaining, the three-dimensional coordinates of the gazing _point on the inspection target surface P1 of the subject Sp are detected. For example, when the surface P1 to be inspected has _a cylindrical shape, the gazing point can be detected by using an equation obtained by transforming the equation representing the cylindrical shape into coordinates. Further, the three-dimensional coordinates of the gazing point in the world coordinate system _XW - _YW - _ZW are converted into the three-dimensional coordinates in the object coordinate system _Xh - _Yh - _Zh .

Here, in the case where the inspection target surface P ₁ has an irregular surface shape having an uneven surface, an equation approximated by a planar shape or a curved surface shape (for example, a spherical shape) for each surface divided into a mesh shape is stored in advance. It may be stored in the unit 103. In this case, the gazing point detection unit 29 obtains the equation of the inspection target surface P1 for _each surface divided into a mesh shape, searches for a surface having an intersection with the line-of-sight vector from the divided plurality of surfaces, and searches for the surface. Calculate using the intersection of the line-of-sight vector and the gaze point as the gaze point. At this time, the gazing point detection unit 29 calculates the distance between the coordinates of the tip and the plurality of faces while extending the line-of-sight vector in the direction from the viewpoint toward the tip at regular distance intervals, and the distance is equal to or less than the threshold value. You can search for the surface that became.

Next, the procedure of the gaze point detection process in the image processing system 1 described above will be described, and the image processing method according to the present embodiment will be described in detail. FIG. 7 is a flowchart showing the procedure of the gaze point detection process by the image processing system 1.

First, when the computer 9 receives an instruction input from the outside, the gaze point detection process is started, and the image pickup control unit 21 of the computer 9 controls the image pickup of the target person Sp by the target person detection optical system 3. And the control of the image pickup of the observation object So by the object detection optical system 5 is started (step S1). In response to this, the image acquisition unit 23 of the computer 9 continuously acquires a face image and an object image for each frame (step S2).

After that, the posture calculation unit 25 of the computer 9 executes the posture calculation process for the object image for each frame, and as data representing the posture of the observation target So, the object coordinate system _Xh -Y Data indicating the moving state and the rotating state of _h —Z _h are calculated (step S3). At the same time, the line-of-sight calculation unit 27 of the computer 9 executes the line-of-sight detection process on the face image for each frame, and calculates the line-of-sight vector of the target person Sp (step S4).

Next, in the world coordinate system _XW - _YW - _ZW , the gazing point detection unit 29 of the computer 9 uses data indicating the moving state and the rotating state of the object coordinate system _Xh - _Yh - _Zh . _The formula of the inspection target surface P1 of the observation target So is calculated (step S5). Then, the gazing point detection unit 29 obtains the intersection between the line-of-sight vector of the subject Sp and the inspection target surface P ₁ , so that the three-dimensional coordinates of the gazing point of the subject Sp on the inspection target surface P ₁ can be obtained. Detected (step 6). The above processes from steps S3 to S6 are repeated for each frame of the face image and the object image captured and acquired at the same timing in the object detection optical system 3 and the object detection optical system 5 (step). S7). Finally, the gazing point detection unit 29 outputs data indicating the three-dimensional coordinates of the gazing point continuously detected for each frame of the face image and the object image to the output device 106 (step S8). At this time, for example, an image in which a marker indicating the gazing point of the subject Sp is superimposed on the image of the observation object So may be output. Further, at this time, the data indicating the three-dimensional coordinates of the gazing point may be recorded in the database in the auxiliary storage unit 103 together with the data indicating the posture of the observation object So.

According to the image processing system 1 according to the present embodiment described above, the face image of the subject Sp is acquired and the object image which is the image of the observation object So is acquired. Then, the arrangement angle and position of the observation object So are calculated using the three _- dimensional positions of the three markers _MA , MB, and _MC on the observation object So detected from the object image, and the face image is obtained. The angle and position of the line of sight of the subject Sp are calculated from the above, and based on these, the position of the gazing _point of the subject Sp on the inspection target surface P1 of the observation target So is detected. This makes it possible to accurately detect the gazing point of the subject Sp with respect to the observation object So even when the position and angle of the observation object So change, such as when visually inspecting the observation object So.

Here, the image processing system 1 is further provided with a support member 11 that integrally supports the object detection optical system 3 and the object detection optical system 5. In such a configuration, the relationship between the line-of-sight vector of the subject Sp detected by using the subject detection optical system 3 and the arrangement of the observation object So detected by using the object detection optical system 5 is stable. It is possible to accurately detect the position of the gazing point of the subject Sp with respect to the observation object So.

Further, the image processing system 1 is further provided with a table 7 which is supported by the support member 11 and serves as a reference when the subject Sp grips the observation object So. In this case, since the position of the observation object So with respect to the object detection optical system 5 is stabilized, the arrangement angle and position of the observation object So can be reliably detected. As a result, the position of the gazing point of the subject Sp with respect to the observation object So can be detected more reliably.

Further, the computer 9 obtains _an equation of the inspection target surface P1 in the three-dimensional coordinate system based on the arrangement angle and position of the observation object So, and calculates the position of the gazing point based on this equation and the line-of-sight vector. is doing. By doing so, since the arrangement of the inspection target surface P1 of the observation target So can be specified by the formula, the position of the gazing _point of the subject Sp _on the inspection target surface P1 of the observation object So can be calculated with high accuracy. Can be done.

Further, the computer 9 obtains a formula for each of the planes P1 to be inspected divided into _a plurality of planes based on the arrangement angle and the position of the observation target surface So, and uses the formulas of the plurality of planes in the plurality of planes. A surface having an intersection with the line-of-sight vector is searched for, and the intersection of the searched surface and the line-of-sight vector is obtained as the position of the gazing point. According to such a configuration, even when the observation object So having a complicated shape is targeted, the arrangement of the inspection target surface P1 of the observation object So is specified by an equation for _each of the plurality of divided surfaces. Therefore, the position of the gazing _point of the subject Sp on the inspection target surface P1 of the observation target So can be calculated with high accuracy.

Further, the computer 9 stores the positional relationship of the three true markers _MA , MB, and _MC in advance, and among the three _- dimensional coordinates of the specified four or more marker candidates, the three true markers are true. The placement angle and position of the observation target So are calculated based on the _{3D coordinates} of the _{markers MA} , _MB , and _MC , and the true 3D coordinates of the narrowed down 3 points. is doing. In this case, when four or more markers including false markers are detected in the object image, the true markers M _A , M _B , and _MC can be narrowed down from those markers. As a result, the arrangement of the observation target So can be detected without error, and the position of the gazing _point of the subject Sp on the inspection target surface P1 of the observation target So can be detected without error.

The present invention is not limited to the above-described embodiment.

For example, on the inspection target surface P1 of the observation target So, _four or more markers for detecting the posture of the observation target So may be attached. At that time, the posture calculation unit 25 of the computer 9 specifies the three-dimensional coordinates of the four or more markers, and calculates the posture of the observation object So using those three-dimensional coordinates. For example, when using the three-dimensional coordinates of the four-point marker, the attitude calculation unit 25 obtains the center of gravity of the four-point marker as the coordinates _Oh of the center of the object coordinate system X _h − Y _h − Z _h . A plane formula is calculated from the three-dimensional coordinates of the four marker using the minimum square method, and the normal vector determined from the calculated plane formula is obtained as a direction vector defining the Z _h axis. Further, when the number of true markers is four, the posture calculation unit 25 selects the four true markers from the three-dimensional coordinates of the five or more marker candidates in the same manner as in the above-described embodiment. Narrow down the 3D coordinates.

Further, the form of the marker for detecting the posture of the observation object So attached on the inspection target surface P1 of the observation object So includes _a QR code (registered trademark) or a simplified code thereof. It may have a pattern that can be identified by image processing. Further, the plurality of markers attached to the inspection target surface P1 of the same observation target So may have different patterns from _each other. In this case, the posture calculation unit 25 of the computer 9 can detect the positions of the reflected images of the plurality of markers on the inspection target surface P1 in _a state in which each of them can be identified by simple image processing. As a result, it is possible to shorten the processing time of the process of narrowing down the three-dimensional coordinates of the true marker.

On the other hand, in _another modification, a marker for detecting the posture of the observation target So may be attached on the inspection target surface P1 of the observation target So. For example, a marker having the form shown in the part (a) of FIG. 3 and having different shapes of the corners at the four corners in the shape of the inner white rectangle may be attached at one point. As the marker of such a form, for example, a form in which the corners of the four corners are chipped differently is exemplified. The posture calculation unit 25 of the computer 9 can identify the positions of the four corners of one marker in the object image by the difference in their shapes. For example, it can be identified by rotating a template representing a shape and collating the template with an object image. As a result, the posture calculation unit 25 can detect the three-dimensional coordinates of the four corners of the marker at one point, and can calculate the posture of the observation target So by using the detection results.

Furthermore, without preparing a special marker and attaching it to the observation object So, three or more feature points or one characteristic shape portion originally existing in the observation object So can be used as a marker. May be good. In this case, the posture calculation unit 25 identifies the feature point or the characteristic shape portion as a marker by image processing for the object image.

Further, as the observation target So to be imaged by the image processing system 1, a flat plate-shaped display device such as a liquid crystal display, a tablet terminal, or a small game device as shown in FIG. 8 may be targeted. In this case, for example, it is possible to detect which part of the image displayed on the display the line of sight is directed to, and this may be reflected in the next processing content or image display depending on the detection result. _In this modification, the optotype M1 for calibration may be dynamically displayed on the inspection target surface P1, which is the display surface of the observation target So, at the time of imaging by the image processing system ₁ . By using the position of the optotype M ₁ on the face image, the coefficient used for calculating the line-of-sight vector can be calibrated in advance. However, in order to calibrate this coefficient accurately, it is necessary to have the subject Sp correctly align the line of sight toward the center of the _optotype M1. Therefore, in order to realize accurate calibration, it is preferable to display the optotype M1 so as to dynamically change from _a large size to a small size, as shown in FIG. By displaying in this way, the subject Sp can improve his / her concentration, and as _a result, the line of sight of both eyes can be easily aligned with the small area at the center of the optotype M1. As a result, accurate line-of-sight vector calibration can be realized.

Further, the target person detection optical system 3 or the object detection optical system 5 is not limited to being composed of a stereo camera, and may be composed of one camera. Alternatively, the target person detection optical system 3 and the target object detection optical system 5 are composed of one camera including a high-performance (high resolution, high speed, wide field of view, etc.) optical system, and are observed with the target person Sp. Corresponding to the positional relationship of the object So, the subject Sp and the observation object So may be configured to be capable of simultaneous imaging by one camera. According to the image processing system 1 including such an optical system, it is possible to simultaneously detect the posture of the observation object So and the line-of-sight direction of the subject Sp.

Further, the marker of the observation target of the image processing system 1 is not limited to being directly attached to the observation target So. For example, when the electronic substrate or the like is an observation object So, a jig for holding the jig may be prepared and a plurality of markers may be attached to the back surface of the jig.

Further, the marker to be observed in the image processing system 1 may be a marker given to the observation object So having a three-dimensional shape with four or more points. In this case, for example, even if one marker is hidden by the rotation operation during the visual inspection by the subject Sp, or one marker is removed from the field of view of the object detection optical system 5, the remaining three points are left. The three-dimensional position of the marker can be detected.

In addition to the elements shown in FIG. 1, the image processing system 1 may include an optical system for detecting an object, an optical system for detecting an object 5, and a housing (case) for accommodating a computer 9. .. In this case, the upper portion of the subject detection optical system 3 in the housing is composed of a member having light transmission at least in the wavelength band used for the light source such as glass. This protects (covers) the camera of the target person detection optical system 3 and enables the camera to capture the target person Sp.

Further, in the configuration shown in FIG. 1, the support member 11 integrally supports the table 7, the object detection optical system 3, and the object detection optical system 5, but the top plate portion 7a of the table 7 is integrally supported. Is shaped to extend to the object detection optical system 5 side, the object detection optical system 5 is supported by the top plate portion 7a, and another is placed at an appropriate position in the lower part of the top plate portion 7a on the target person Sp side. A table or a housing may be integrally provided, and the subject detection optical system 3 may be fixed to the table or the housing. That is, the role of the support member 11 may be played by the table 7 itself or a member integrally provided on the table 7.

Further, in the image processing system 1, the table 7 may be omitted.

1 ... Image processing system, 3 ... Optical system for object detection, 5 ... Optical system for object detection, 7 ... Table (flat plate member), 9 ... Computer (calculation unit), 11 ... Support member, _MA , _MB , MC ... Marker, _{P 1} ... Inspection target surface (specific surface), So ... Observation target, Sp ... Subject.

Claims (9)

An optical system for detecting a subject, including a camera that captures the face of the subject, and an optical system for detecting the subject, for acquiring a facial image necessary for detecting the angle and position of the line of sight of the subject.
An object detection optical system for acquiring an object image necessary for detecting the arrangement angle and position of the observation object, including a camera that captures the observation object of the observation object of the subject.
A calculation unit that executes image processing on the face image and the object image is provided.
The calculation unit
The three-dimensional position of the marker on the observation object is specified based on the object image, and the arrangement angle and position of the observation object are calculated based on the three-dimensional position of the marker.
The angle and position of the line of sight of the subject are calculated based on the face image, and the specific surface on the object to be observed is based on the angle and position of the line of sight and the arrangement angle and position of the object to be observed. Detects the gaze point of the subject in
Image processing system. The calculation unit identifies the three-dimensional positions of three or more markers on the observation object, and calculates the arrangement angle and position of the observation object based on the three-dimensional positions of the three or more markers. ,
The image processing system according to claim 1. Further, a support member for integrally supporting the object detection optical system and the object detection optical system is provided.
The image processing system according to claim 1 or 2. A flat plate member supported by the support member and used as a reference when the subject grips the observation object is further provided.
The image processing system according to claim 3. The calculation unit obtains an equation for the specific surface in the three-dimensional coordinate system based on the arrangement angle and position of the observation object, and calculates the gazing point based on the equation and the vector representing the line of sight. do,
The image processing system according to any one of claims 1 to 4. Based on the arrangement angle and position of the observation object, the calculation unit obtains an equation for each of the surfaces in which the specific surface is divided into a plurality of surfaces, and uses the equations of the plurality of the surfaces in the plurality of the surfaces. Searches for a surface having an intersection with the vector, and obtains the intersection of the searched surface with the vector as the gazing point.
The image processing system according to claim 5. The calculation unit stores the positional relationship of a plurality of true markers in advance, narrows down the three-dimensional positions of the specified plurality of marker candidates to the three-dimensional positions of the plurality of true markers, and narrows down the plurality of positions. The placement angle and position of the observation object are calculated based on the three-dimensional position of the true marker of.
The image processing system according to any one of claims 1 to 6. The calculation unit identifies the three-dimensional position of the calibration optotype based on the image obtained in advance by the object detection optical system, and creates the face image based on the three-dimensional position of the calibration optotype. Based on this, a calibration process for calculating the angle and position of the line of sight is executed in advance.
The image processing system according to any one of claims 1 to 6. An image processing method executed by an image processing system.
A step of taking an image of the subject's face and acquiring a facial image necessary for detecting the angle and position of the subject's line of sight.
A step of capturing an observation object of the observation target of the subject and acquiring an object image necessary for detecting the arrangement angle and position of the observation object, and
A step of executing image processing on the face image and the object image is provided.
In the step of executing the image processing,
The three-dimensional position of the marker on the observation object is specified based on the object image, and the arrangement angle and position of the observation object are calculated based on the three-dimensional position of the marker.
The angle and position of the line of sight of the subject are calculated based on the face image, and the angle and position of the line of sight and the arrangement angle and position of the object to be observed are used on a specific surface of the object to be observed. Detecting the gaze point of the subject,
Image processing method.

Images