WO2024034924A1 - Procédé et système de pointage utilisant une oculométrie basée sur une caméra stéréo - Google Patents

Procédé et système de pointage utilisant une oculométrie basée sur une caméra stéréo Download PDF

Info

Publication number
WO2024034924A1
WO2024034924A1 PCT/KR2023/010786 KR2023010786W WO2024034924A1 WO 2024034924 A1 WO2024034924 A1 WO 2024034924A1 KR 2023010786 W KR2023010786 W KR 2023010786W WO 2024034924 A1 WO2024034924 A1 WO 2024034924A1
Authority
WO
WIPO (PCT)
Prior art keywords
user
face
stereo camera
control device
gaze
Prior art date
Application number
PCT/KR2023/010786
Other languages
English (en)
Korean (ko)
Inventor
우광제
Original Assignee
주식회사 아이트
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by 주식회사 아이트 filed Critical 주식회사 아이트
Publication of WO2024034924A1 publication Critical patent/WO2024034924A1/fr

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/10—Image acquisition
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/40—Extraction of image or video features
    • G06V10/44—Local feature extraction by analysis of parts of the pattern, e.g. by detecting edges, contours, loops, corners, strokes or intersections; Connectivity analysis, e.g. of connected components
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V40/00—Recognition of biometric, human-related or animal-related patterns in image or video data
    • G06V40/10—Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands
    • G06V40/18—Eye characteristics, e.g. of the iris

Definitions

  • the present invention relates to a pointing method, and more specifically, to a pointing method and system using stereo camera-based eye tracking.
  • these gaze recognition technologies are being used in various fields. For example, in advertising, by showing advertisements to consumers and collecting information about where consumers look and when they look, it is possible to assess whether people look more at the product, the advertising model, or the company brand. You can. Additionally, in the marketing field, a large number of contents can be provided to people through devices, and user preference scores can be set high for contents that people are interested in and look at for a long time. And it is expected that this gaze recognition technology will be used in more and more fields.
  • Patent Document 1 Domestic Patent Publication No. 10-1249263
  • Patent Document 2 Domestic Patent Publication No. 10-2216020
  • Patent Document 3 Domestic Patent Publication No. 10-2093198
  • Patent Document 4 Domestic Patent Publication No. 10-2015-0032019
  • Patent Document 5 Domestic Patent Publication No. 10-2011-0100987
  • This specification was developed to solve the problems described above, and provides a pointing method and system using stereo camera-based eye tracking that allows the user to quickly and accurately move the mouse cursor on the screen without directly touching the screen. The purpose is to do this.
  • a pointing system using eye tracking based on a stereo camera includes a display device divided into a plurality of areas according to the user's face angle; a plurality of lighting devices installed on the display device to radiate light to the user's face; a stereo camera that photographs the user's face while light is illuminated on the user's face by the plurality of lighting devices; and calculating the user's face angle based on the face image captured using the stereo camera, first moving the mouse cursor to an area corresponding to the user's face angle, and then calculating the face captured using the stereo camera.
  • Control for extracting an eye image from an image and secondary moving the mouse cursor within an area corresponding to the user's face angle according to the movement of the gaze due to a change in the position of the reflected light reflected from the pupil included in the extracted eye image. Includes device.
  • control device calculates face coordinate information from a face image captured using the stereo camera and calculates the user's face angle based on the coordinate difference between left and right face images.
  • the control device extracts an eye image from the face image, detects a pupil from the extracted eye image, and detects reflected light reflected from the detected pupil to determine the location of the reflected light and the positions of the plurality of lighting devices. It is characterized by tracking the movement of the gaze with respect to the display device through a geometric transformation method corresponding to .
  • control device detects the reflected light by applying a Kalman filter to every frame of the eye image.
  • control device is characterized in that when the gaze stays at a specific location for more than a certain period of time, the control device clicks on the location.
  • control device controls movement between pages by recognizing the user's hand gestures using the stereo camera.
  • the control device switches screens within the same page when the user moves his hand left and right, moves to the previous page when the user moves his hand up, and moves to the next page when the user moves his hand down. It is characterized by ordering.
  • control device samples the movement path along which the gaze has moved into a plurality of sections, calculates the movement speed for each section, extracts the direction of movement and the movement direction of the gaze for each section, and determines the change trend of the gaze. It is characterized by prediction in advance.
  • a pointing method using eye tracking based on a stereo camera includes the steps of dividing a display device into a plurality of areas according to the user's face angle; A plurality of lighting devices installed on the display device irradiate light to the user's face; Using a stereo camera, photographing the user's face while light is irradiated on the user's face by the plurality of lighting devices; Calculating, by a control device, a face angle of the user based on a face image captured using the stereo camera; The control device first moving the mouse cursor to an area corresponding to the user's face angle; Extracting, by the control device, an eye image from a face image captured using the stereo camera; And a step where the control device secondarily moves the mouse cursor within an area corresponding to the user's face angle according to the movement of the gaze due to the change in the position of the reflected light reflected from the pupil included in the extracted eye image. do.
  • the mouse cursor is first moved to an area corresponding to the user's face angle, and the user's gaze is moved according to the change in the position of the reflected light reflected from the pupil included in the user's eye image.
  • FIG. 1 is a diagram showing the configuration of a pointing system using stereo camera-based eye tracking according to an embodiment of the present invention
  • FIG. 2 is a block diagram showing an example of hardware capable of realizing the function of a pointing system according to an embodiment of the present invention
  • FIG. 3 is a block diagram showing an example of the functions of the control device according to the embodiment of the present invention.
  • Figure 4 is a diagram showing an example of feature detection using Adaboost
  • FIG. 5 is a diagram for explaining a method of detecting a pupil area using a circular detection algorithm
  • Figure 6 is a diagram showing the center of gravity of the black area
  • Figure 7 is a diagram showing an example of finding reflected light using a Kalman filter for sequential images
  • Figure 8 is a diagram showing a model for analyzing the difference between the amount of pupil movement due to eye rotation and the amount of movement of reflected light;
  • Figure 9 is a diagram showing the correspondence between the distorted rectangle formed by reflected light and the rectangle formed by four lights;
  • Figure 10 is a diagram showing a sampling table in which the movement path of the gaze is sampled in detail into a plurality of sections according to time, and
  • 11 to 13 are flowcharts showing a pointing method using stereo camera-based eye tracking according to an embodiment of the present invention.
  • first, second, etc. used in this specification may be used to describe various components, but the components should not be limited by the terms. The above terms are used only for the purpose of distinguishing one component from another. For example, a first component may be referred to as a second component, and similarly, the second component may also be referred to as a first component without departing from the scope of the present invention.
  • Gaze position calculation unit 340 Control unit
  • the pointing method of the present invention involves selecting a display device 10 divided into a plurality of areas according to the user's face angle with a plurality of menu items in relation to the control of an electronic device or control device. Use it as a menu board.
  • the control device 30 photographs the part of the user's face illuminated by a plurality of lighting devices 21 with the stereo camera 20, and then moves the mouse to the area corresponding to the angle of the user's face through the image signal processing process. After first moving the cursor, tracking the gaze on the display device 10, and secondarily moving the mouse cursor within an area corresponding to the user's face angle according to the movement of the user's gaze, the user's gaze is moved for a certain period of time.
  • the electronic device is controlled by a control signal (e.g., click processing) corresponding to the location.
  • the control structure of electronic devices can be understood by a general level of technology.
  • control device 30 can control movement between pages by recognizing the user's hand gestures using the stereo camera 20. Specifically, the control device 30 switches the screen within the same page when the user moves the hand left and right, moves to the previous page when the user moves the hand upward, and moves to the next page when the user moves the hand downward. Move to , and if the user shows his or her palm, the page movement is stopped.
  • the pointing method of the present invention can be applied to a KIOSK screen installed in a store selling goods or products, or to a web shopping mall screen selling goods or products through a web page, and can be broadcast at home. It can be applied to a regular TV screen or satellite TV screen.
  • the pointing method can be applied to all display devices, such as robot devices that display menus guiding relevant organizations, events, and services in public places such as exhibition halls, event halls, government agencies, local governments, banks, and department stores.
  • Figure 2 is a block diagram showing an example of hardware capable of realizing the function of the pointing system according to the embodiment of the present invention.
  • the functions of the pointing system can be realized using, for example, the hardware resources shown in FIG. 2.
  • the functions of the pointing system are realized by controlling the hardware shown in FIG. 2 using a computer program.
  • this hardware mainly includes a CPU 202, a read only memory (ROM) 204, a RAM 206, a host bus 208, and a bridge 210. Additionally, this hardware includes an external bus 212, an interface 214, an input unit 216, an output unit 218, a storage unit 220, a drive 222, a connection port 224, and a communication unit 226.
  • ROM read only memory
  • RAM random access memory
  • bridge 210 mainly includes a bridge 210. Additionally, this hardware includes an external bus 212, an interface 214, an input unit 216, an output unit 218, a storage unit 220, a drive 222, a connection port 224, and a communication unit 226.
  • the CPU 202 functions, for example, as an arithmetic processing unit or a control unit, based on various programs recorded in the ROM 204, RAM 206, storage unit 220, or removable recording medium 228. Controls all or part of the operation of each component.
  • the ROM 204 is an example of a storage device that stores programs read by the CPU 202 and data used in calculations. In the RAM 206, for example, a program read by the CPU 202 and various parameters that change when executing the program are temporarily or permanently stored.
  • a host bus 208 capable of high-speed data transfer.
  • the host bus 208 is connected to the external bus 212, which has a relatively low data transfer speed, through a bridge 210, for example.
  • a bridge 210 for example.
  • the input unit 216 for example, a mouse, keyboard, touch panel, touch pad, buttons, switches, and levers are used.
  • a remote controller capable of transmitting a control signal using infrared waves or other radio waves can be used.
  • a display device such as a Cathode Ray Tube (CRT), Liquid Crystal Display (LCD), Plasma Display Panel (PDP), or Electro-Luminescence Display (ELD) may be used.
  • an audio output device such as a speaker or headphone, or a printer may be used.
  • the storage unit 220 is a device for storing various data.
  • a magnetic storage device such as an HDD is used.
  • a semiconductor storage device such as a solid state drive (SSD) or RAM disk, an optical storage device, or a magneto-optical storage device may be used.
  • the drive 222 is a device that reads information recorded on the removable recording medium 228, which is a removable recording medium, or records information on the removable recording medium 228.
  • the removable recording medium 228, for example, a magnetic disk, optical disk, magneto-optical disk, or semiconductor memory is used. Additionally, a program defining the operation of the pointing system may be stored in the removable recording medium 228.
  • connection port 224 is for connecting an external connection device 230, such as a USB (Universal Serial Bus) port, IEEE 1394 port, SCSI (Small Computer System Interface), RS-232C port, or optical audio terminal. It's a port.
  • an external connection device 230 such as a USB (Universal Serial Bus) port, IEEE 1394 port, SCSI (Small Computer System Interface), RS-232C port, or optical audio terminal. It's a port.
  • the external connection device 230 for example, a printer or the like is used.
  • the communication unit 226 is a communication device for connecting to the network 232.
  • a communication circuit for a wired or wireless LAN for example, a communication circuit for a WUSB (Wireless USB), a communication circuit for a mobile phone network, etc. may be used.
  • the network 232 is, for example, a network connected by wire or wireless.
  • the hardware of the pointing system has been described.
  • the hardware described above is an example, and modifications such as omitting some elements or adding new elements are possible.
  • FIG. 3 is a block diagram showing an example of the functions of the control device 30 according to the embodiment of the present invention.
  • the control device 30 may include a reception unit 310, a detection unit 320, a calculation unit 330, and a control unit 340.
  • the calculation unit 330 may again include a face angle calculation unit 332 and a gaze position calculation unit 334.
  • the function of the receiving unit 310 can be realized using the above-described communication unit 226, etc.
  • the functions of the detection unit 320, the calculation unit 330, and the control unit 340 can be realized using the CPU 202 and the like described above.
  • the receiver 310 acquires an image of the user using the stereo camera 20.
  • the detection unit 320 first detects the face part from the image acquired through the reception unit 310.
  • the detection unit 320 may use the Adaboost (Adaptive Boosting) method in the above process.
  • Boosting is a method of finding a strong classifier with high accuracy by combining many weak classifiers.
  • a representative boosting algorithm is Adaboost.
  • Adaboost operates in real time by detecting faces with relatively high accuracy with simple and fast performance.
  • Figure 4 shows an example of feature detection using Adaboost. As shown in Figure 4, the face is detected by using the characteristic that the eye area is darker than the nose and cheek areas, and then by using the fact that the eyes are darker than the bridge of the nose.
  • the detection unit 320 After extracting the face area, the detection unit 320 detects the eye area using the Adaboost method. To extract the eye region, information trained through weak classifiers that reflect eye characteristics is used.
  • the detector 320 detects the pupil area using the circular detection algorithm below. First, the detector 320 determines the initial pupil area through circular template matching, as shown in (a) of FIG. 5.
  • the circular template matching method can be explained by Equation 1 below.
  • I(x, y) represents the brightness of the image at the (x, y) location
  • (x 0 , y 0 ) and r represent the center and radius of the circular template.
  • the point where the difference in the sum of brightness between the two circular templates is maximized is determined as the pupil area (or iris area).
  • the pupil may appear in the shape of an ellipse rather than a circle depending on the gaze position and camera shooting angle, the position determined by the circular template matching method cannot be said to be accurate. Therefore, a local binarization process is performed as shown in (b) of FIG. 5 based on the determined location.
  • the threshold for binarization uses the method proposed by Gonzalez and the automatic threshold determination method proposed by Otzu. After local binarization, noise areas due to eyebrows or shadows may exist, and if reflected light is present inside the pupil area, it may appear in the form of a hole. To solve this, a component labeling method is performed on the binarized area to assign identities to adjacent areas, and then remove areas with different identities, leaving the widest area. Remove noise areas. Finally, a morphological closing operation is performed to fill the holed area. As a result, as shown in Figures 5(c) and 6, the center of gravity of the black area is obtained and determined as the final pupil center.
  • reflected light generated by a plurality of lighting devices 21 installed on the display device 10 can be regarded as a plurality of lighting devices 21.
  • the gaze position is calculated based on the relationship between the display device 10 and the center of the pupil. Therefore, it is important to accurately extract not only the pupil center but also the center of the reflected light.
  • the detection unit 320 detects reflected light by performing a Kalman filter in every frame.
  • the Kalman filter is a circular function that not only considers the observation process, but also considers the model in which the target variable changes. Therefore, it is widely used to track objects. Since location information of the reflected light is required to perform the Kalman filter, in the first frame of a system operating in real time, the centers of the four reflected lights are extracted using image processing methods (binarization, labeling, etc.) for the entire area. . The center of the extracted reflected light is used as information needed to find the corresponding reflected light in the next frame.
  • the Kalman filter used in the embodiment of the present invention moves at a constant speed as shown in Equation 2 to find four reflected lights.
  • the position of the reflected light in the next frame is predicted through a Kalman filter based on the position of the reflected light in the previous frame, only a certain area is searched around the predicted position. Therefore, it can be found faster than looking for reflected light in the entire area. Additionally, the location of the reflected light found is corrected to the exact location through a Kalman filter.
  • Equation 2 x k and y k are the positions of reflected light in frame k, and x k' and y k' represent the velocities of x k and y k , respectively.
  • the time interval of video input from the camera used in the embodiment of the present invention can be considered to be 66ms.
  • Figure 7 shows an example of finding reflected light using a Kalman filter for sequential images.
  • the display device 10 that is, the lighting 21 attached to the corner of the monitor
  • the corresponding reflected light disappears from the image.
  • the gaze position cannot be determined. Therefore, to solve this case, in the present invention, the position predicted by the Kalman filter for the previous frame is replaced with the position of the missing reflected light.
  • the Kalman filter is a filter based on constant velocity or constant acceleration motion, there is a problem in that it is difficult to find when the speed of the object being tracked suddenly changes or changes direction.
  • an existing method of using an iterative Kalman filter but this also has the disadvantage of taking a long time to process. Since the camera used in the embodiment of the present invention has an image input time interval of 66 ms per frame, if the Kalman filter is repeatedly performed on one image frame, the processing time is delayed, making it difficult to apply to the pointing system according to the embodiment of the present invention. .
  • the present invention considers the gaze direction as the direction in which the pupil moves, and uses a method of re-searching by increasing the search area of reflected light according to the pupil's movement direction.
  • the direction of the user's gaze movement can be known using the position of the pupil.
  • the direction and size of movement can be determined based on the difference between the position of the pupil in the previous image and the position of the pupil in the current image. If the user suddenly gazes in a different direction and the center of the pupil moves significantly, the reflected light also changes position as the eyeball rotates.
  • the reason for the difference in the amount of movement between the pupil and the reflected light is that the pupil rotates based on the rotation center of the eyeball, which has a radius of 13.5mm, and the reflected light from the lighting enters the center of the corneal sphere, which has a radius of 7.7mm. Therefore, because the distance from the monitor to the center of the pupil and reflected light is different, the amount of movement is also different.
  • the z-distance between the monitor and the user must be known. Because the present invention is approached in a two-dimensional manner, it is difficult to determine the z distance. Therefore, in an embodiment of the present invention, the approximate amount of movement of the reflected light relative to the amount of pupil movement is obtained in advance through learning, defined as a LUT (Look Up Table), and then applied to the Kalman filtering step.
  • LUT Look Up Table
  • the number of pixels in the pupil area can be measured before detecting reflected light. If the number of pixels is less than 1/2 of the number of pixels in the pupil area in the previous frame, it is determined that the user has closed his or her eyes and the next process, detection of reflected light and calculation of gaze position, is not performed.
  • the detection unit 320 can smooth the movement of the mouse cursor, which will be described later, by detecting reflected light using the Kalman filter algorithm.
  • the detection unit 320 determines the initial hand area from the image captured using the stereo camera 20. For example, from an image containing a waving hand, an area with shaking can be detected by distinguishing it from an area without shaking, and this can be identified as the initial hand area. Additionally, in classifying the shaking area, the difference image method or Gaussian Mixture Model can be used to separate the shaking area into the hand area and the background. Additionally, the detector 320 may perform one of distortion correction, brightness adjustment, noise removal, and color correction on the image input from the stereo camera 20.
  • the detection unit 320 confirms the hand area using the parallax of the stereo camera 20 with respect to the detected initial hand area. For example, using the parallax information of the initial image, the distance between the hand and the background can be calculated to check the closest point, and the point can be confirmed as the hand area.
  • the detection unit 320 detects the hand shape for the determined hand area. More specifically, hand shape detection technology can be applied to detect the hand shape in the hand area. For example, various technologies can be applied, such as identifying the shape of the hand using border information of the hand area.
  • the detection unit 320 recognizes hand movements according to changes in the detected hand shape. For example, technology that tracks the moving trajectory of the hand as it moves or the shape of the hand changes can be applied.
  • the face angle calculation unit 332 calculates face coordinate information from the face image detected by the detection unit 320, and calculates the distance from the camera 20 to the face and the face angle (up and down) based on the coordinate difference between the left and right face images. left and right).
  • the face angle calculation unit 332 knows the distance from the camera 20 to the face, it can also calculate the user's height and standing position using the Pythagorean theorem. Accordingly, the face angle calculation unit 332 can adjust the face angle according to the user's height and standing position.
  • the gaze position calculation unit 334 calculates the distorted position formed by the reflected light (P 1 , P 2 , P 3 , P 4 ) as shown in FIG.
  • the correspondence relationship between the square shape and the rectangle formed by the four lights (F 1 , F 2 , F 3 , F 4 ) can be defined, and the geometric transform method can be used for this purpose.
  • Equation 3 Let the coordinates of the four reflected lights P 1 to P 4 be (x p1 , y p1 ), (x p2 , y p2 ), (x p3 , y p3 ), (x p4 , y p4 ), and the four lighting positions F If the coordinates of 1 to F 4 are (x f1 , y f1 ), (x f2 , y f2 ), (x f3 , y f3 ), (x f4 , y f4 ), the math below is based on the geometric transformation method. Equation 3 can be designed.
  • Equation 3 is expressed in matrix form, it is as Equation 4 below.
  • the mapping matrix T expressing the correspondence can be obtained by multiplying both sides by the inverse matrix of F.
  • the gaze position (G x , G y ) within the rectangle formed by the four infrared lights can be calculated through Equation 5 below.
  • the gaze position calculation unit 334 closely samples the movement path of the gaze moving on the plane of the display device 10 into a plurality of sections according to time, creates a sampling table, and calculates the movement path of the gaze moving on the plane of the display device 10. By referring to the sampling table created, you can predict future trends in eye movement in advance.
  • the prediction can be more accurate as the number of sampling sections increases, and this can be set differently by the manager who manages the control device 30.
  • the sampling table will be described in detail with reference to FIG. 10.
  • the prediction of the moving speed is predicted based on the moving speed calculated for each sampled section according to Equation 6 below.
  • Vb represents the moving speed for each section
  • D(x, y) represents the moving distance (displacement amount) of the gaze in the x and y directions
  • t represents the time the gaze moved for each section.
  • F is a function for calculating the moving speed for each section, and calculates the moving speed of the gaze in the corresponding section by differentiating the moving time from the displacement amount.
  • the gaze position calculation unit 334 compares the movement speed for each section, calculates an increase/decrease ratio with respect to the overall movement speed, and predicts the movement speed of the continuously moving gaze based on this. Meanwhile, the prediction of the movement speed of the gaze can also be made by adding up and averaging all the movement speeds calculated for each section.
  • the gaze position calculation unit 334 can detect changes in the gaze direction for each section according to gaze tracking and predict trends in the gaze direction that will change thereafter.
  • the gaze position calculation unit 334 can detect a vector for gaze movement for each section and predict a trend in the gaze movement path based on this.
  • the control unit 340 moves the mouse cursor to the area corresponding to the user's face angle calculated by the face angle calculation unit 332. After moving the mouse cursor secondarily within the area corresponding to the user's face angle according to the movement of the gaze calculated by the gaze position calculation unit 334, if the gaze stays at a specific position for a certain period of time or more, Click on the location.
  • control unit 340 can control movement between pages of the display device 10 by recognizing the user's hand gesture through the detection unit 320. Specifically, the control unit 340 switches the screen within the same page when the user moves his hand left or right, moves to the previous page when the user moves his hand up, and moves to the next page when the user moves his hand down. If the user shows his or her palm, the page movement is stopped.
  • 11 to 13 are flowcharts showing a pointing method using stereo camera-based eye tracking according to an embodiment of the present invention.
  • the pointing system provides voice guidance to the customer (user) to face the front the first time (S10).
  • the pointing system can display a logo in the center of the screen along with an announcement and provide voice guidance saying, “Welcome. Please look at the logo on the screen.”
  • the pointing system extracts 128 points of information from the face image captured using the stereo camera 20 (S12). That is, the pointing system extracts 128 points of information from each of the left and right face images.
  • the pointing system calculates the depth of the face (DEPTH) based on the extracted information and calculates the distance from the stereo camera 20 to the face and the customer's height (S14).
  • the pointing system identifies customers through facial recognition (S16).
  • the pointing system implements a 3D image of the face (DEPTH) and can identify the customer's ID through facial recognition.
  • the pointing system radiates infrared light to the customer's face (S18).
  • the pointing system stores the positions of four infrared lights shining on the pupil (S20).
  • service video information may represent a menu for customers to order and pay for food.
  • the pointing system determines whether there is movement in the customer's face (S24), and when the customer moves his face up, down, left, and right, it calculates the face angle (up, down, left, and right) based on the coordinate difference between the left and right face images (S26). .
  • the pointing system determines the location (area) the customer is looking at based on the face angle and initially moves the mouse cursor to the corresponding area (S28). In other words, the pointing system can initially move the mouse cursor to an area corresponding to the customer's face angle.
  • the pointing system calculates the change in position of the four infrared lights shining on the pupil and moves the mouse cursor secondarily (finely) (S30). That is, the pointing system can track the gaze (pupil) of the display device 10 and secondarily move the mouse cursor within the corresponding area according to the movement of the customer's gaze.
  • the pointing system determines whether the pupil stays at a specific location for more than a certain period of time (S32), and if the pupil stays at a specific location for more than a certain period of time, an electronic signal is sent by a control signal (e.g., click processing) corresponding to the location. Allow the device to be controlled (S34).
  • a control signal e.g., click processing
  • the pointing system can control movement between pages of the display device 10 by recognizing the movement of the customer's hand using the stereo camera 20. That is, after step S16, the pointing system determines whether there is movement of the customer's hand (S17), switches the screen within the same page when the customer moves his hand left or right (S19), and moves the customer's hand up. If the customer moves, it moves to the previous page (S21), and if the customer moves his hand down, it moves to the next page (S23). Although not shown in the drawing, the pointing system can stop page movement when the customer shows his/her palm.
  • embodiments of the present invention may be implemented by hardware, firmware, software, or a combination thereof.
  • the method according to embodiments of the present invention uses one or more Application Specific Integrated Circuits (ASICs), Digital Signal Processors (DSPs), Digital Signal Processing Devices (DSPDs), and Programmable Logic Devices (PLDs).
  • ASICs Application Specific Integrated Circuits
  • DSPs Digital Signal Processors
  • DSPDs Digital Signal Processing Devices
  • PLDs Programmable Logic Devices
  • FPGAs Field Programmable Gate Arrays
  • processors controllers, microcontrollers, and microprocessors.
  • the method according to embodiments of the present invention may be implemented in the form of a module, procedure, or function that performs the functions or operations described above.
  • Software code can be stored in a memory unit and run by a processor.
  • the memory unit is located inside or outside the processor and can exchange data with the processor through various known means.

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Multimedia (AREA)
  • General Engineering & Computer Science (AREA)
  • Human Computer Interaction (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Health & Medical Sciences (AREA)
  • General Health & Medical Sciences (AREA)
  • Ophthalmology & Optometry (AREA)
  • Position Input By Displaying (AREA)

Abstract

La présente invention concerne un procédé et un système de pointage utilisant une oculométrie basée sur une caméra stéréo, le système comprenant : un dispositif d'affichage qui est divisé en une pluralité de zones en fonction d'un angle de visage d'un utilisateur ; une pluralité de dispositifs d'éclairage qui sont installés sur le dispositif d'affichage de façon à projeter une lumière sur le visage de l'utilisateur ; une caméra stéréo qui capture le visage de l'utilisateur tandis que la lumière est projetée sur le visage de l'utilisateur au moyen de la pluralité de dispositifs d'éclairage ; et un dispositif de commande qui calcule l'angle de visage de l'utilisateur sur la base d'une image de visage capturée à l'aide de la caméra stéréo, déplace principalement un curseur de souris vers une zone correspondant à l'angle de visage de l'utilisateur, puis extrait une image d'œil à partir de l'image de visage capturée à l'aide de la caméra stéréo, et déplace secondairement le curseur de souris à l'intérieur d'une zone correspondant à l'angle de visage de l'utilisateur en fonction d'un mouvement de regard dû à un changement de la position de la lumière réfléchie réfléchie par la pupille incluse dans l'image d'œil extraite.
PCT/KR2023/010786 2022-08-12 2023-07-26 Procédé et système de pointage utilisant une oculométrie basée sur une caméra stéréo WO2024034924A1 (fr)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
KR1020220101396A KR102506037B1 (ko) 2022-08-12 2022-08-12 스테레오 카메라 기반의 시선 추적을 이용한 포인팅 방법 및 시스템
KR10-2022-0101396 2022-08-12

Publications (1)

Publication Number Publication Date
WO2024034924A1 true WO2024034924A1 (fr) 2024-02-15

Family

ID=85510086

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/KR2023/010786 WO2024034924A1 (fr) 2022-08-12 2023-07-26 Procédé et système de pointage utilisant une oculométrie basée sur une caméra stéréo

Country Status (2)

Country Link
KR (1) KR102506037B1 (fr)
WO (1) WO2024034924A1 (fr)

Families Citing this family (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
KR102506037B1 (ko) * 2022-08-12 2023-03-06 주식회사 아이트 스테레오 카메라 기반의 시선 추적을 이용한 포인팅 방법 및 시스템

Citations (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP2005182247A (ja) * 2003-12-17 2005-07-07 National Univ Corp Shizuoka Univ 瞳孔位置検出によるポインティング装置及び方法
KR20150032019A (ko) * 2013-09-17 2015-03-25 한국전자통신연구원 시선 추적 기반의 사용자 인터페이스 방법 및 그 장치
KR101819164B1 (ko) * 2016-07-29 2018-01-17 상명대학교산학협력단 시선 추적 장치 및 방법
KR20200043786A (ko) * 2018-10-18 2020-04-28 주식회사 토비스 화면 제어 방법 및 표시 장치
KR20210073429A (ko) * 2019-12-10 2021-06-18 한국전자기술연구원 착용형 ar 장치를 위한 시선 추적 및 손동작 인식 기반 통합 인터페이스 방법 및 시스템
KR102506037B1 (ko) * 2022-08-12 2023-03-06 주식회사 아이트 스테레오 카메라 기반의 시선 추적을 이용한 포인팅 방법 및 시스템

Family Cites Families (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
KR20110100987A (ko) 2010-03-05 2011-09-15 에스케이텔레콤 주식회사 시선 추적을 이용한 사용자 인터페이스 방법 및 이를 적용한 단말
KR101249263B1 (ko) 2010-12-14 2013-04-09 한국기초과학지원연구원 시선 추적을 이용한 포인팅 방법 및 이를 적용한 엘리베이터와 탑승객 간 인터페이스 방법
KR102093198B1 (ko) 2013-02-21 2020-03-25 삼성전자주식회사 시선 인식을 이용한 사용자 인터페이스 방법 및 장치
KR101855196B1 (ko) * 2013-11-27 2018-06-08 선전 구딕스 테크놀로지 컴퍼니, 리미티드 아이 트래킹 및 사용자 반응 검출
KR102216020B1 (ko) 2018-11-21 2021-02-17 네이버웹툰 유한회사 시선인식 인터페이스 제공방법 및 이를 위한 사용자 단말

Patent Citations (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP2005182247A (ja) * 2003-12-17 2005-07-07 National Univ Corp Shizuoka Univ 瞳孔位置検出によるポインティング装置及び方法
KR20150032019A (ko) * 2013-09-17 2015-03-25 한국전자통신연구원 시선 추적 기반의 사용자 인터페이스 방법 및 그 장치
KR101819164B1 (ko) * 2016-07-29 2018-01-17 상명대학교산학협력단 시선 추적 장치 및 방법
KR20200043786A (ko) * 2018-10-18 2020-04-28 주식회사 토비스 화면 제어 방법 및 표시 장치
KR20210073429A (ko) * 2019-12-10 2021-06-18 한국전자기술연구원 착용형 ar 장치를 위한 시선 추적 및 손동작 인식 기반 통합 인터페이스 방법 및 시스템
KR102506037B1 (ko) * 2022-08-12 2023-03-06 주식회사 아이트 스테레오 카메라 기반의 시선 추적을 이용한 포인팅 방법 및 시스템

Also Published As

Publication number Publication date
KR102506037B1 (ko) 2023-03-06

Similar Documents

Publication Publication Date Title
Letessier et al. Visual tracking of bare fingers for interactive surfaces
Darrell et al. Integrated person tracking using stereo, color, and pattern detection
JP4965653B2 (ja) 視覚ディスプレイのためのバーチャルコントローラ
Crowley et al. Vision for man machine interaction
Licsár et al. User-adaptive hand gesture recognition system with interactive training
WO2024034924A1 (fr) Procédé et système de pointage utilisant une oculométrie basée sur une caméra stéréo
KR20130004357A (ko) 컴퓨팅 디바이스 인터페이스
JP2001016606A (ja) 動作認識システムおよび動作認識プログラムを記録した記録媒体
Kawato et al. Scale-adaptive face detection and tracking in real time with SSR filters and support vector machine
Iannizzotto et al. Hand tracking for human-computer interaction with graylevel visualglove: turning back to the simple way
Lo et al. Augmediated reality system based on 3D camera selfgesture sensing
WO2019004754A1 (fr) Publicités à réalité augmentée sur des objets
Ueng et al. Vision based multi-user human computer interaction
Yeh et al. Vision-based virtual control mechanism via hand gesture recognition
WO2019199035A1 (fr) Système et procédé de suivi du regard
Park et al. A hand posture recognition system utilizing frequency difference of infrared light
CN115393965B (zh) 一种基于红外遥感的光感装置智能控制系统
Gobhinath et al. Implementation of Real Time Static Hand Gestures Recognition for Sign Language
EP4407569A1 (fr) Systèmes et procédés de suivi d'objets
WO2021261770A1 (fr) Procédé de calcul de position de pupille d'utilisateur, et support d'enregistrement, sur lequel un programme d'exécution de procédé de calcul de position de pupille d'utilisateur est enregistré
WO2015056894A1 (fr) Appareil de traitement d'image et son procédé de commande
Okada et al. Recognition of typing motions on AR typing interface
Christ et al. Vision-based Machine Learning Worker Assistance System for People with Disabilities on Low-Cost Hardware
El Sibai et al. Vision-Based Approach for Real-Time Hand Detection and Gesture Recognition
KR20230043285A (ko) 딥러닝을 이용한 손 움직임 추적방법 및 장치

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 23852805

Country of ref document: EP

Kind code of ref document: A1