WO2024034924A1 - Procédé et système de pointage utilisant une oculométrie basée sur une caméra stéréo - Google Patents
Procédé et système de pointage utilisant une oculométrie basée sur une caméra stéréo Download PDFInfo
- Publication number
- WO2024034924A1 WO2024034924A1 PCT/KR2023/010786 KR2023010786W WO2024034924A1 WO 2024034924 A1 WO2024034924 A1 WO 2024034924A1 KR 2023010786 W KR2023010786 W KR 2023010786W WO 2024034924 A1 WO2024034924 A1 WO 2024034924A1
- Authority
- WO
- WIPO (PCT)
- Prior art keywords
- user
- face
- stereo camera
- control device
- gaze
- Prior art date
Links
- 238000000034 method Methods 0.000 title claims abstract description 36
- 210000001747 pupil Anatomy 0.000 claims abstract description 40
- 230000008859 change Effects 0.000 claims abstract description 11
- 239000000284 extract Substances 0.000 claims abstract description 9
- 230000004424 eye movement Effects 0.000 claims description 2
- 210000001508 eye Anatomy 0.000 description 25
- 238000004364 calculation method Methods 0.000 description 20
- 238000001514 detection method Methods 0.000 description 19
- 230000006870 function Effects 0.000 description 15
- 238000010586 diagram Methods 0.000 description 12
- 238000005516 engineering process Methods 0.000 description 10
- 238000004891 communication Methods 0.000 description 8
- 230000008569 process Effects 0.000 description 6
- 238000012545 processing Methods 0.000 description 6
- 238000005070 sampling Methods 0.000 description 5
- 210000005252 bulbus oculi Anatomy 0.000 description 4
- 238000004422 calculation algorithm Methods 0.000 description 4
- 239000011159 matrix material Substances 0.000 description 4
- 230000001815 facial effect Effects 0.000 description 3
- 230000003287 optical effect Effects 0.000 description 3
- 238000012937 correction Methods 0.000 description 2
- 238000006073 displacement reaction Methods 0.000 description 2
- 230000004418 eye rotation Effects 0.000 description 2
- 230000014509 gene expression Effects 0.000 description 2
- 230000005484 gravity Effects 0.000 description 2
- 238000002372 labelling Methods 0.000 description 2
- 239000004065 semiconductor Substances 0.000 description 2
- 238000012546 transfer Methods 0.000 description 2
- 238000011426 transformation method Methods 0.000 description 2
- 238000012935 Averaging Methods 0.000 description 1
- 230000001133 acceleration Effects 0.000 description 1
- 230000003044 adaptive effect Effects 0.000 description 1
- 238000003491 array Methods 0.000 description 1
- 230000004397 blinking Effects 0.000 description 1
- 238000004590 computer program Methods 0.000 description 1
- 210000004087 cornea Anatomy 0.000 description 1
- 230000003111 delayed effect Effects 0.000 description 1
- 238000005401 electroluminescence Methods 0.000 description 1
- 210000004709 eyebrow Anatomy 0.000 description 1
- 210000000887 face Anatomy 0.000 description 1
- 238000001914 filtration Methods 0.000 description 1
- 230000003993 interaction Effects 0.000 description 1
- 239000004973 liquid crystal related substance Substances 0.000 description 1
- 238000013507 mapping Methods 0.000 description 1
- 239000000203 mixture Substances 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 230000000877 morphologic effect Effects 0.000 description 1
- 238000003672 processing method Methods 0.000 description 1
- 238000011160 research Methods 0.000 description 1
- 239000007787 solid Substances 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/10—Image acquisition
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/40—Extraction of image or video features
- G06V10/44—Local feature extraction by analysis of parts of the pattern, e.g. by detecting edges, contours, loops, corners, strokes or intersections; Connectivity analysis, e.g. of connected components
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V40/00—Recognition of biometric, human-related or animal-related patterns in image or video data
- G06V40/10—Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands
- G06V40/18—Eye characteristics, e.g. of the iris
Definitions
- the present invention relates to a pointing method, and more specifically, to a pointing method and system using stereo camera-based eye tracking.
- these gaze recognition technologies are being used in various fields. For example, in advertising, by showing advertisements to consumers and collecting information about where consumers look and when they look, it is possible to assess whether people look more at the product, the advertising model, or the company brand. You can. Additionally, in the marketing field, a large number of contents can be provided to people through devices, and user preference scores can be set high for contents that people are interested in and look at for a long time. And it is expected that this gaze recognition technology will be used in more and more fields.
- Patent Document 1 Domestic Patent Publication No. 10-1249263
- Patent Document 2 Domestic Patent Publication No. 10-2216020
- Patent Document 3 Domestic Patent Publication No. 10-2093198
- Patent Document 4 Domestic Patent Publication No. 10-2015-0032019
- Patent Document 5 Domestic Patent Publication No. 10-2011-0100987
- This specification was developed to solve the problems described above, and provides a pointing method and system using stereo camera-based eye tracking that allows the user to quickly and accurately move the mouse cursor on the screen without directly touching the screen. The purpose is to do this.
- a pointing system using eye tracking based on a stereo camera includes a display device divided into a plurality of areas according to the user's face angle; a plurality of lighting devices installed on the display device to radiate light to the user's face; a stereo camera that photographs the user's face while light is illuminated on the user's face by the plurality of lighting devices; and calculating the user's face angle based on the face image captured using the stereo camera, first moving the mouse cursor to an area corresponding to the user's face angle, and then calculating the face captured using the stereo camera.
- Control for extracting an eye image from an image and secondary moving the mouse cursor within an area corresponding to the user's face angle according to the movement of the gaze due to a change in the position of the reflected light reflected from the pupil included in the extracted eye image. Includes device.
- control device calculates face coordinate information from a face image captured using the stereo camera and calculates the user's face angle based on the coordinate difference between left and right face images.
- the control device extracts an eye image from the face image, detects a pupil from the extracted eye image, and detects reflected light reflected from the detected pupil to determine the location of the reflected light and the positions of the plurality of lighting devices. It is characterized by tracking the movement of the gaze with respect to the display device through a geometric transformation method corresponding to .
- control device detects the reflected light by applying a Kalman filter to every frame of the eye image.
- control device is characterized in that when the gaze stays at a specific location for more than a certain period of time, the control device clicks on the location.
- control device controls movement between pages by recognizing the user's hand gestures using the stereo camera.
- the control device switches screens within the same page when the user moves his hand left and right, moves to the previous page when the user moves his hand up, and moves to the next page when the user moves his hand down. It is characterized by ordering.
- control device samples the movement path along which the gaze has moved into a plurality of sections, calculates the movement speed for each section, extracts the direction of movement and the movement direction of the gaze for each section, and determines the change trend of the gaze. It is characterized by prediction in advance.
- a pointing method using eye tracking based on a stereo camera includes the steps of dividing a display device into a plurality of areas according to the user's face angle; A plurality of lighting devices installed on the display device irradiate light to the user's face; Using a stereo camera, photographing the user's face while light is irradiated on the user's face by the plurality of lighting devices; Calculating, by a control device, a face angle of the user based on a face image captured using the stereo camera; The control device first moving the mouse cursor to an area corresponding to the user's face angle; Extracting, by the control device, an eye image from a face image captured using the stereo camera; And a step where the control device secondarily moves the mouse cursor within an area corresponding to the user's face angle according to the movement of the gaze due to the change in the position of the reflected light reflected from the pupil included in the extracted eye image. do.
- the mouse cursor is first moved to an area corresponding to the user's face angle, and the user's gaze is moved according to the change in the position of the reflected light reflected from the pupil included in the user's eye image.
- FIG. 1 is a diagram showing the configuration of a pointing system using stereo camera-based eye tracking according to an embodiment of the present invention
- FIG. 2 is a block diagram showing an example of hardware capable of realizing the function of a pointing system according to an embodiment of the present invention
- FIG. 3 is a block diagram showing an example of the functions of the control device according to the embodiment of the present invention.
- Figure 4 is a diagram showing an example of feature detection using Adaboost
- FIG. 5 is a diagram for explaining a method of detecting a pupil area using a circular detection algorithm
- Figure 6 is a diagram showing the center of gravity of the black area
- Figure 7 is a diagram showing an example of finding reflected light using a Kalman filter for sequential images
- Figure 8 is a diagram showing a model for analyzing the difference between the amount of pupil movement due to eye rotation and the amount of movement of reflected light;
- Figure 9 is a diagram showing the correspondence between the distorted rectangle formed by reflected light and the rectangle formed by four lights;
- Figure 10 is a diagram showing a sampling table in which the movement path of the gaze is sampled in detail into a plurality of sections according to time, and
- 11 to 13 are flowcharts showing a pointing method using stereo camera-based eye tracking according to an embodiment of the present invention.
- first, second, etc. used in this specification may be used to describe various components, but the components should not be limited by the terms. The above terms are used only for the purpose of distinguishing one component from another. For example, a first component may be referred to as a second component, and similarly, the second component may also be referred to as a first component without departing from the scope of the present invention.
- Gaze position calculation unit 340 Control unit
- the pointing method of the present invention involves selecting a display device 10 divided into a plurality of areas according to the user's face angle with a plurality of menu items in relation to the control of an electronic device or control device. Use it as a menu board.
- the control device 30 photographs the part of the user's face illuminated by a plurality of lighting devices 21 with the stereo camera 20, and then moves the mouse to the area corresponding to the angle of the user's face through the image signal processing process. After first moving the cursor, tracking the gaze on the display device 10, and secondarily moving the mouse cursor within an area corresponding to the user's face angle according to the movement of the user's gaze, the user's gaze is moved for a certain period of time.
- the electronic device is controlled by a control signal (e.g., click processing) corresponding to the location.
- the control structure of electronic devices can be understood by a general level of technology.
- control device 30 can control movement between pages by recognizing the user's hand gestures using the stereo camera 20. Specifically, the control device 30 switches the screen within the same page when the user moves the hand left and right, moves to the previous page when the user moves the hand upward, and moves to the next page when the user moves the hand downward. Move to , and if the user shows his or her palm, the page movement is stopped.
- the pointing method of the present invention can be applied to a KIOSK screen installed in a store selling goods or products, or to a web shopping mall screen selling goods or products through a web page, and can be broadcast at home. It can be applied to a regular TV screen or satellite TV screen.
- the pointing method can be applied to all display devices, such as robot devices that display menus guiding relevant organizations, events, and services in public places such as exhibition halls, event halls, government agencies, local governments, banks, and department stores.
- Figure 2 is a block diagram showing an example of hardware capable of realizing the function of the pointing system according to the embodiment of the present invention.
- the functions of the pointing system can be realized using, for example, the hardware resources shown in FIG. 2.
- the functions of the pointing system are realized by controlling the hardware shown in FIG. 2 using a computer program.
- this hardware mainly includes a CPU 202, a read only memory (ROM) 204, a RAM 206, a host bus 208, and a bridge 210. Additionally, this hardware includes an external bus 212, an interface 214, an input unit 216, an output unit 218, a storage unit 220, a drive 222, a connection port 224, and a communication unit 226.
- ROM read only memory
- RAM random access memory
- bridge 210 mainly includes a bridge 210. Additionally, this hardware includes an external bus 212, an interface 214, an input unit 216, an output unit 218, a storage unit 220, a drive 222, a connection port 224, and a communication unit 226.
- the CPU 202 functions, for example, as an arithmetic processing unit or a control unit, based on various programs recorded in the ROM 204, RAM 206, storage unit 220, or removable recording medium 228. Controls all or part of the operation of each component.
- the ROM 204 is an example of a storage device that stores programs read by the CPU 202 and data used in calculations. In the RAM 206, for example, a program read by the CPU 202 and various parameters that change when executing the program are temporarily or permanently stored.
- a host bus 208 capable of high-speed data transfer.
- the host bus 208 is connected to the external bus 212, which has a relatively low data transfer speed, through a bridge 210, for example.
- a bridge 210 for example.
- the input unit 216 for example, a mouse, keyboard, touch panel, touch pad, buttons, switches, and levers are used.
- a remote controller capable of transmitting a control signal using infrared waves or other radio waves can be used.
- a display device such as a Cathode Ray Tube (CRT), Liquid Crystal Display (LCD), Plasma Display Panel (PDP), or Electro-Luminescence Display (ELD) may be used.
- an audio output device such as a speaker or headphone, or a printer may be used.
- the storage unit 220 is a device for storing various data.
- a magnetic storage device such as an HDD is used.
- a semiconductor storage device such as a solid state drive (SSD) or RAM disk, an optical storage device, or a magneto-optical storage device may be used.
- the drive 222 is a device that reads information recorded on the removable recording medium 228, which is a removable recording medium, or records information on the removable recording medium 228.
- the removable recording medium 228, for example, a magnetic disk, optical disk, magneto-optical disk, or semiconductor memory is used. Additionally, a program defining the operation of the pointing system may be stored in the removable recording medium 228.
- connection port 224 is for connecting an external connection device 230, such as a USB (Universal Serial Bus) port, IEEE 1394 port, SCSI (Small Computer System Interface), RS-232C port, or optical audio terminal. It's a port.
- an external connection device 230 such as a USB (Universal Serial Bus) port, IEEE 1394 port, SCSI (Small Computer System Interface), RS-232C port, or optical audio terminal. It's a port.
- the external connection device 230 for example, a printer or the like is used.
- the communication unit 226 is a communication device for connecting to the network 232.
- a communication circuit for a wired or wireless LAN for example, a communication circuit for a WUSB (Wireless USB), a communication circuit for a mobile phone network, etc. may be used.
- the network 232 is, for example, a network connected by wire or wireless.
- the hardware of the pointing system has been described.
- the hardware described above is an example, and modifications such as omitting some elements or adding new elements are possible.
- FIG. 3 is a block diagram showing an example of the functions of the control device 30 according to the embodiment of the present invention.
- the control device 30 may include a reception unit 310, a detection unit 320, a calculation unit 330, and a control unit 340.
- the calculation unit 330 may again include a face angle calculation unit 332 and a gaze position calculation unit 334.
- the function of the receiving unit 310 can be realized using the above-described communication unit 226, etc.
- the functions of the detection unit 320, the calculation unit 330, and the control unit 340 can be realized using the CPU 202 and the like described above.
- the receiver 310 acquires an image of the user using the stereo camera 20.
- the detection unit 320 first detects the face part from the image acquired through the reception unit 310.
- the detection unit 320 may use the Adaboost (Adaptive Boosting) method in the above process.
- Boosting is a method of finding a strong classifier with high accuracy by combining many weak classifiers.
- a representative boosting algorithm is Adaboost.
- Adaboost operates in real time by detecting faces with relatively high accuracy with simple and fast performance.
- Figure 4 shows an example of feature detection using Adaboost. As shown in Figure 4, the face is detected by using the characteristic that the eye area is darker than the nose and cheek areas, and then by using the fact that the eyes are darker than the bridge of the nose.
- the detection unit 320 After extracting the face area, the detection unit 320 detects the eye area using the Adaboost method. To extract the eye region, information trained through weak classifiers that reflect eye characteristics is used.
- the detector 320 detects the pupil area using the circular detection algorithm below. First, the detector 320 determines the initial pupil area through circular template matching, as shown in (a) of FIG. 5.
- the circular template matching method can be explained by Equation 1 below.
- I(x, y) represents the brightness of the image at the (x, y) location
- (x 0 , y 0 ) and r represent the center and radius of the circular template.
- the point where the difference in the sum of brightness between the two circular templates is maximized is determined as the pupil area (or iris area).
- the pupil may appear in the shape of an ellipse rather than a circle depending on the gaze position and camera shooting angle, the position determined by the circular template matching method cannot be said to be accurate. Therefore, a local binarization process is performed as shown in (b) of FIG. 5 based on the determined location.
- the threshold for binarization uses the method proposed by Gonzalez and the automatic threshold determination method proposed by Otzu. After local binarization, noise areas due to eyebrows or shadows may exist, and if reflected light is present inside the pupil area, it may appear in the form of a hole. To solve this, a component labeling method is performed on the binarized area to assign identities to adjacent areas, and then remove areas with different identities, leaving the widest area. Remove noise areas. Finally, a morphological closing operation is performed to fill the holed area. As a result, as shown in Figures 5(c) and 6, the center of gravity of the black area is obtained and determined as the final pupil center.
- reflected light generated by a plurality of lighting devices 21 installed on the display device 10 can be regarded as a plurality of lighting devices 21.
- the gaze position is calculated based on the relationship between the display device 10 and the center of the pupil. Therefore, it is important to accurately extract not only the pupil center but also the center of the reflected light.
- the detection unit 320 detects reflected light by performing a Kalman filter in every frame.
- the Kalman filter is a circular function that not only considers the observation process, but also considers the model in which the target variable changes. Therefore, it is widely used to track objects. Since location information of the reflected light is required to perform the Kalman filter, in the first frame of a system operating in real time, the centers of the four reflected lights are extracted using image processing methods (binarization, labeling, etc.) for the entire area. . The center of the extracted reflected light is used as information needed to find the corresponding reflected light in the next frame.
- the Kalman filter used in the embodiment of the present invention moves at a constant speed as shown in Equation 2 to find four reflected lights.
- the position of the reflected light in the next frame is predicted through a Kalman filter based on the position of the reflected light in the previous frame, only a certain area is searched around the predicted position. Therefore, it can be found faster than looking for reflected light in the entire area. Additionally, the location of the reflected light found is corrected to the exact location through a Kalman filter.
- Equation 2 x k and y k are the positions of reflected light in frame k, and x k' and y k' represent the velocities of x k and y k , respectively.
- the time interval of video input from the camera used in the embodiment of the present invention can be considered to be 66ms.
- Figure 7 shows an example of finding reflected light using a Kalman filter for sequential images.
- the display device 10 that is, the lighting 21 attached to the corner of the monitor
- the corresponding reflected light disappears from the image.
- the gaze position cannot be determined. Therefore, to solve this case, in the present invention, the position predicted by the Kalman filter for the previous frame is replaced with the position of the missing reflected light.
- the Kalman filter is a filter based on constant velocity or constant acceleration motion, there is a problem in that it is difficult to find when the speed of the object being tracked suddenly changes or changes direction.
- an existing method of using an iterative Kalman filter but this also has the disadvantage of taking a long time to process. Since the camera used in the embodiment of the present invention has an image input time interval of 66 ms per frame, if the Kalman filter is repeatedly performed on one image frame, the processing time is delayed, making it difficult to apply to the pointing system according to the embodiment of the present invention. .
- the present invention considers the gaze direction as the direction in which the pupil moves, and uses a method of re-searching by increasing the search area of reflected light according to the pupil's movement direction.
- the direction of the user's gaze movement can be known using the position of the pupil.
- the direction and size of movement can be determined based on the difference between the position of the pupil in the previous image and the position of the pupil in the current image. If the user suddenly gazes in a different direction and the center of the pupil moves significantly, the reflected light also changes position as the eyeball rotates.
- the reason for the difference in the amount of movement between the pupil and the reflected light is that the pupil rotates based on the rotation center of the eyeball, which has a radius of 13.5mm, and the reflected light from the lighting enters the center of the corneal sphere, which has a radius of 7.7mm. Therefore, because the distance from the monitor to the center of the pupil and reflected light is different, the amount of movement is also different.
- the z-distance between the monitor and the user must be known. Because the present invention is approached in a two-dimensional manner, it is difficult to determine the z distance. Therefore, in an embodiment of the present invention, the approximate amount of movement of the reflected light relative to the amount of pupil movement is obtained in advance through learning, defined as a LUT (Look Up Table), and then applied to the Kalman filtering step.
- LUT Look Up Table
- the number of pixels in the pupil area can be measured before detecting reflected light. If the number of pixels is less than 1/2 of the number of pixels in the pupil area in the previous frame, it is determined that the user has closed his or her eyes and the next process, detection of reflected light and calculation of gaze position, is not performed.
- the detection unit 320 can smooth the movement of the mouse cursor, which will be described later, by detecting reflected light using the Kalman filter algorithm.
- the detection unit 320 determines the initial hand area from the image captured using the stereo camera 20. For example, from an image containing a waving hand, an area with shaking can be detected by distinguishing it from an area without shaking, and this can be identified as the initial hand area. Additionally, in classifying the shaking area, the difference image method or Gaussian Mixture Model can be used to separate the shaking area into the hand area and the background. Additionally, the detector 320 may perform one of distortion correction, brightness adjustment, noise removal, and color correction on the image input from the stereo camera 20.
- the detection unit 320 confirms the hand area using the parallax of the stereo camera 20 with respect to the detected initial hand area. For example, using the parallax information of the initial image, the distance between the hand and the background can be calculated to check the closest point, and the point can be confirmed as the hand area.
- the detection unit 320 detects the hand shape for the determined hand area. More specifically, hand shape detection technology can be applied to detect the hand shape in the hand area. For example, various technologies can be applied, such as identifying the shape of the hand using border information of the hand area.
- the detection unit 320 recognizes hand movements according to changes in the detected hand shape. For example, technology that tracks the moving trajectory of the hand as it moves or the shape of the hand changes can be applied.
- the face angle calculation unit 332 calculates face coordinate information from the face image detected by the detection unit 320, and calculates the distance from the camera 20 to the face and the face angle (up and down) based on the coordinate difference between the left and right face images. left and right).
- the face angle calculation unit 332 knows the distance from the camera 20 to the face, it can also calculate the user's height and standing position using the Pythagorean theorem. Accordingly, the face angle calculation unit 332 can adjust the face angle according to the user's height and standing position.
- the gaze position calculation unit 334 calculates the distorted position formed by the reflected light (P 1 , P 2 , P 3 , P 4 ) as shown in FIG.
- the correspondence relationship between the square shape and the rectangle formed by the four lights (F 1 , F 2 , F 3 , F 4 ) can be defined, and the geometric transform method can be used for this purpose.
- Equation 3 Let the coordinates of the four reflected lights P 1 to P 4 be (x p1 , y p1 ), (x p2 , y p2 ), (x p3 , y p3 ), (x p4 , y p4 ), and the four lighting positions F If the coordinates of 1 to F 4 are (x f1 , y f1 ), (x f2 , y f2 ), (x f3 , y f3 ), (x f4 , y f4 ), the math below is based on the geometric transformation method. Equation 3 can be designed.
- Equation 3 is expressed in matrix form, it is as Equation 4 below.
- the mapping matrix T expressing the correspondence can be obtained by multiplying both sides by the inverse matrix of F.
- the gaze position (G x , G y ) within the rectangle formed by the four infrared lights can be calculated through Equation 5 below.
- the gaze position calculation unit 334 closely samples the movement path of the gaze moving on the plane of the display device 10 into a plurality of sections according to time, creates a sampling table, and calculates the movement path of the gaze moving on the plane of the display device 10. By referring to the sampling table created, you can predict future trends in eye movement in advance.
- the prediction can be more accurate as the number of sampling sections increases, and this can be set differently by the manager who manages the control device 30.
- the sampling table will be described in detail with reference to FIG. 10.
- the prediction of the moving speed is predicted based on the moving speed calculated for each sampled section according to Equation 6 below.
- Vb represents the moving speed for each section
- D(x, y) represents the moving distance (displacement amount) of the gaze in the x and y directions
- t represents the time the gaze moved for each section.
- F is a function for calculating the moving speed for each section, and calculates the moving speed of the gaze in the corresponding section by differentiating the moving time from the displacement amount.
- the gaze position calculation unit 334 compares the movement speed for each section, calculates an increase/decrease ratio with respect to the overall movement speed, and predicts the movement speed of the continuously moving gaze based on this. Meanwhile, the prediction of the movement speed of the gaze can also be made by adding up and averaging all the movement speeds calculated for each section.
- the gaze position calculation unit 334 can detect changes in the gaze direction for each section according to gaze tracking and predict trends in the gaze direction that will change thereafter.
- the gaze position calculation unit 334 can detect a vector for gaze movement for each section and predict a trend in the gaze movement path based on this.
- the control unit 340 moves the mouse cursor to the area corresponding to the user's face angle calculated by the face angle calculation unit 332. After moving the mouse cursor secondarily within the area corresponding to the user's face angle according to the movement of the gaze calculated by the gaze position calculation unit 334, if the gaze stays at a specific position for a certain period of time or more, Click on the location.
- control unit 340 can control movement between pages of the display device 10 by recognizing the user's hand gesture through the detection unit 320. Specifically, the control unit 340 switches the screen within the same page when the user moves his hand left or right, moves to the previous page when the user moves his hand up, and moves to the next page when the user moves his hand down. If the user shows his or her palm, the page movement is stopped.
- 11 to 13 are flowcharts showing a pointing method using stereo camera-based eye tracking according to an embodiment of the present invention.
- the pointing system provides voice guidance to the customer (user) to face the front the first time (S10).
- the pointing system can display a logo in the center of the screen along with an announcement and provide voice guidance saying, “Welcome. Please look at the logo on the screen.”
- the pointing system extracts 128 points of information from the face image captured using the stereo camera 20 (S12). That is, the pointing system extracts 128 points of information from each of the left and right face images.
- the pointing system calculates the depth of the face (DEPTH) based on the extracted information and calculates the distance from the stereo camera 20 to the face and the customer's height (S14).
- the pointing system identifies customers through facial recognition (S16).
- the pointing system implements a 3D image of the face (DEPTH) and can identify the customer's ID through facial recognition.
- the pointing system radiates infrared light to the customer's face (S18).
- the pointing system stores the positions of four infrared lights shining on the pupil (S20).
- service video information may represent a menu for customers to order and pay for food.
- the pointing system determines whether there is movement in the customer's face (S24), and when the customer moves his face up, down, left, and right, it calculates the face angle (up, down, left, and right) based on the coordinate difference between the left and right face images (S26). .
- the pointing system determines the location (area) the customer is looking at based on the face angle and initially moves the mouse cursor to the corresponding area (S28). In other words, the pointing system can initially move the mouse cursor to an area corresponding to the customer's face angle.
- the pointing system calculates the change in position of the four infrared lights shining on the pupil and moves the mouse cursor secondarily (finely) (S30). That is, the pointing system can track the gaze (pupil) of the display device 10 and secondarily move the mouse cursor within the corresponding area according to the movement of the customer's gaze.
- the pointing system determines whether the pupil stays at a specific location for more than a certain period of time (S32), and if the pupil stays at a specific location for more than a certain period of time, an electronic signal is sent by a control signal (e.g., click processing) corresponding to the location. Allow the device to be controlled (S34).
- a control signal e.g., click processing
- the pointing system can control movement between pages of the display device 10 by recognizing the movement of the customer's hand using the stereo camera 20. That is, after step S16, the pointing system determines whether there is movement of the customer's hand (S17), switches the screen within the same page when the customer moves his hand left or right (S19), and moves the customer's hand up. If the customer moves, it moves to the previous page (S21), and if the customer moves his hand down, it moves to the next page (S23). Although not shown in the drawing, the pointing system can stop page movement when the customer shows his/her palm.
- embodiments of the present invention may be implemented by hardware, firmware, software, or a combination thereof.
- the method according to embodiments of the present invention uses one or more Application Specific Integrated Circuits (ASICs), Digital Signal Processors (DSPs), Digital Signal Processing Devices (DSPDs), and Programmable Logic Devices (PLDs).
- ASICs Application Specific Integrated Circuits
- DSPs Digital Signal Processors
- DSPDs Digital Signal Processing Devices
- PLDs Programmable Logic Devices
- FPGAs Field Programmable Gate Arrays
- processors controllers, microcontrollers, and microprocessors.
- the method according to embodiments of the present invention may be implemented in the form of a module, procedure, or function that performs the functions or operations described above.
- Software code can be stored in a memory unit and run by a processor.
- the memory unit is located inside or outside the processor and can exchange data with the processor through various known means.
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Multimedia (AREA)
- General Engineering & Computer Science (AREA)
- Human Computer Interaction (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Health & Medical Sciences (AREA)
- General Health & Medical Sciences (AREA)
- Ophthalmology & Optometry (AREA)
- Position Input By Displaying (AREA)
Abstract
La présente invention concerne un procédé et un système de pointage utilisant une oculométrie basée sur une caméra stéréo, le système comprenant : un dispositif d'affichage qui est divisé en une pluralité de zones en fonction d'un angle de visage d'un utilisateur ; une pluralité de dispositifs d'éclairage qui sont installés sur le dispositif d'affichage de façon à projeter une lumière sur le visage de l'utilisateur ; une caméra stéréo qui capture le visage de l'utilisateur tandis que la lumière est projetée sur le visage de l'utilisateur au moyen de la pluralité de dispositifs d'éclairage ; et un dispositif de commande qui calcule l'angle de visage de l'utilisateur sur la base d'une image de visage capturée à l'aide de la caméra stéréo, déplace principalement un curseur de souris vers une zone correspondant à l'angle de visage de l'utilisateur, puis extrait une image d'œil à partir de l'image de visage capturée à l'aide de la caméra stéréo, et déplace secondairement le curseur de souris à l'intérieur d'une zone correspondant à l'angle de visage de l'utilisateur en fonction d'un mouvement de regard dû à un changement de la position de la lumière réfléchie réfléchie par la pupille incluse dans l'image d'œil extraite.
Applications Claiming Priority (2)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
KR1020220101396A KR102506037B1 (ko) | 2022-08-12 | 2022-08-12 | 스테레오 카메라 기반의 시선 추적을 이용한 포인팅 방법 및 시스템 |
KR10-2022-0101396 | 2022-08-12 |
Publications (1)
Publication Number | Publication Date |
---|---|
WO2024034924A1 true WO2024034924A1 (fr) | 2024-02-15 |
Family
ID=85510086
Family Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
PCT/KR2023/010786 WO2024034924A1 (fr) | 2022-08-12 | 2023-07-26 | Procédé et système de pointage utilisant une oculométrie basée sur une caméra stéréo |
Country Status (2)
Country | Link |
---|---|
KR (1) | KR102506037B1 (fr) |
WO (1) | WO2024034924A1 (fr) |
Families Citing this family (1)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
KR102506037B1 (ko) * | 2022-08-12 | 2023-03-06 | 주식회사 아이트 | 스테레오 카메라 기반의 시선 추적을 이용한 포인팅 방법 및 시스템 |
Citations (6)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
JP2005182247A (ja) * | 2003-12-17 | 2005-07-07 | National Univ Corp Shizuoka Univ | 瞳孔位置検出によるポインティング装置及び方法 |
KR20150032019A (ko) * | 2013-09-17 | 2015-03-25 | 한국전자통신연구원 | 시선 추적 기반의 사용자 인터페이스 방법 및 그 장치 |
KR101819164B1 (ko) * | 2016-07-29 | 2018-01-17 | 상명대학교산학협력단 | 시선 추적 장치 및 방법 |
KR20200043786A (ko) * | 2018-10-18 | 2020-04-28 | 주식회사 토비스 | 화면 제어 방법 및 표시 장치 |
KR20210073429A (ko) * | 2019-12-10 | 2021-06-18 | 한국전자기술연구원 | 착용형 ar 장치를 위한 시선 추적 및 손동작 인식 기반 통합 인터페이스 방법 및 시스템 |
KR102506037B1 (ko) * | 2022-08-12 | 2023-03-06 | 주식회사 아이트 | 스테레오 카메라 기반의 시선 추적을 이용한 포인팅 방법 및 시스템 |
Family Cites Families (5)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
KR20110100987A (ko) | 2010-03-05 | 2011-09-15 | 에스케이텔레콤 주식회사 | 시선 추적을 이용한 사용자 인터페이스 방법 및 이를 적용한 단말 |
KR101249263B1 (ko) | 2010-12-14 | 2013-04-09 | 한국기초과학지원연구원 | 시선 추적을 이용한 포인팅 방법 및 이를 적용한 엘리베이터와 탑승객 간 인터페이스 방법 |
KR102093198B1 (ko) | 2013-02-21 | 2020-03-25 | 삼성전자주식회사 | 시선 인식을 이용한 사용자 인터페이스 방법 및 장치 |
KR101855196B1 (ko) * | 2013-11-27 | 2018-06-08 | 선전 구딕스 테크놀로지 컴퍼니, 리미티드 | 아이 트래킹 및 사용자 반응 검출 |
KR102216020B1 (ko) | 2018-11-21 | 2021-02-17 | 네이버웹툰 유한회사 | 시선인식 인터페이스 제공방법 및 이를 위한 사용자 단말 |
-
2022
- 2022-08-12 KR KR1020220101396A patent/KR102506037B1/ko active IP Right Grant
-
2023
- 2023-07-26 WO PCT/KR2023/010786 patent/WO2024034924A1/fr unknown
Patent Citations (6)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
JP2005182247A (ja) * | 2003-12-17 | 2005-07-07 | National Univ Corp Shizuoka Univ | 瞳孔位置検出によるポインティング装置及び方法 |
KR20150032019A (ko) * | 2013-09-17 | 2015-03-25 | 한국전자통신연구원 | 시선 추적 기반의 사용자 인터페이스 방법 및 그 장치 |
KR101819164B1 (ko) * | 2016-07-29 | 2018-01-17 | 상명대학교산학협력단 | 시선 추적 장치 및 방법 |
KR20200043786A (ko) * | 2018-10-18 | 2020-04-28 | 주식회사 토비스 | 화면 제어 방법 및 표시 장치 |
KR20210073429A (ko) * | 2019-12-10 | 2021-06-18 | 한국전자기술연구원 | 착용형 ar 장치를 위한 시선 추적 및 손동작 인식 기반 통합 인터페이스 방법 및 시스템 |
KR102506037B1 (ko) * | 2022-08-12 | 2023-03-06 | 주식회사 아이트 | 스테레오 카메라 기반의 시선 추적을 이용한 포인팅 방법 및 시스템 |
Also Published As
Publication number | Publication date |
---|---|
KR102506037B1 (ko) | 2023-03-06 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
Letessier et al. | Visual tracking of bare fingers for interactive surfaces | |
Darrell et al. | Integrated person tracking using stereo, color, and pattern detection | |
JP4965653B2 (ja) | 視覚ディスプレイのためのバーチャルコントローラ | |
Crowley et al. | Vision for man machine interaction | |
Licsár et al. | User-adaptive hand gesture recognition system with interactive training | |
WO2024034924A1 (fr) | Procédé et système de pointage utilisant une oculométrie basée sur une caméra stéréo | |
KR20130004357A (ko) | 컴퓨팅 디바이스 인터페이스 | |
JP2001016606A (ja) | 動作認識システムおよび動作認識プログラムを記録した記録媒体 | |
Kawato et al. | Scale-adaptive face detection and tracking in real time with SSR filters and support vector machine | |
Iannizzotto et al. | Hand tracking for human-computer interaction with graylevel visualglove: turning back to the simple way | |
Lo et al. | Augmediated reality system based on 3D camera selfgesture sensing | |
WO2019004754A1 (fr) | Publicités à réalité augmentée sur des objets | |
Ueng et al. | Vision based multi-user human computer interaction | |
Yeh et al. | Vision-based virtual control mechanism via hand gesture recognition | |
WO2019199035A1 (fr) | Système et procédé de suivi du regard | |
Park et al. | A hand posture recognition system utilizing frequency difference of infrared light | |
CN115393965B (zh) | 一种基于红外遥感的光感装置智能控制系统 | |
Gobhinath et al. | Implementation of Real Time Static Hand Gestures Recognition for Sign Language | |
EP4407569A1 (fr) | Systèmes et procédés de suivi d'objets | |
WO2021261770A1 (fr) | Procédé de calcul de position de pupille d'utilisateur, et support d'enregistrement, sur lequel un programme d'exécution de procédé de calcul de position de pupille d'utilisateur est enregistré | |
WO2015056894A1 (fr) | Appareil de traitement d'image et son procédé de commande | |
Okada et al. | Recognition of typing motions on AR typing interface | |
Christ et al. | Vision-based Machine Learning Worker Assistance System for People with Disabilities on Low-Cost Hardware | |
El Sibai et al. | Vision-Based Approach for Real-Time Hand Detection and Gesture Recognition | |
KR20230043285A (ko) | 딥러닝을 이용한 손 움직임 추적방법 및 장치 |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
121 | Ep: the epo has been informed by wipo that ep was designated in this application |
Ref document number: 23852805 Country of ref document: EP Kind code of ref document: A1 |