TW497065B - A method and its associated device to visually control the cursor of the Windows - Google Patents

Abstract

This invention introduces a method and its associated device to visually control the cursor of the Windows. The invention mainly uses an image-capturing device to acquire the moving information on the users' eyeballs relative to their facial appearances. This moving information serves as the foundation to control the relative position of the computer cursor. The computer users can operate on the movement of the cursor without resorting to a mouse. Therefore, it eliminates the inconvenience regarding the operation back and forth between the mouse and the keyboard.

Description

Description of the invention (1) [Scope of application of the invention] The present invention relates to a method and device for moving a computer cursor, and a mouthpiece, and a method for visually selecting a computer cursor position according to the movement of a user's eyeball. And device. [Background of the invention '] The basic input device of the early computers was the keyboard. No matter what ^ typing or instruction input is done on the keyboard, but using the keyboard to move ~ a small disadvantage, that is, the cursor position can not be as you want. The rapid movement was followed by the popularity of the Windows graphical interface system, which resulted in a mouse ^ M〇 = e) This pointing input device allows the user to quickly move the cursor (or pointer) by moving the mouse Add any ^ or option on the computer screen, so the input device such as mouse has become very important, and it is the basic equipment of the computer like the keyboard. Yes, while using the mouse as a pointing device is a very convenient thing, but the bone mouse is mainly designed to be used as the cursor pointing and simple command input, if you want to type or use hot keys (H. t Key) is not the use of the keyboard, so the user slowly finds that the hands must be moved between the keyboard and the mouse during the use of the computer, which is very inconvenient. In the current era of informationization, the window graphics interface has become the trend, so if you can quickly point the cursor to a specific position on the two sides, and on the other hand, you can concentrate on the keyboard operation, you must save operation Input device time, which greatly increases the efficiency of computer use. [Objective and Summary of the Invention] In view of this, the object of the present invention is to solve the problem that computer users sometimes use slippage.

Legislative and clarifying explanation (2) Simplified formula: “The inconvenience of using the keyboard to turn around” makes the computer cursor positioning method and installation: This: Provided by the head two t-a kind of window cursor visual point positioning identification, according to its = device The value of the positioning frame for the user's appearance is based on :: ;;:; the door will get the user's pupil and it will be recombined through a communication interface; the relative position change between the frames, and the computer cursor will be adjusted. The computer m electrical signal is transmitted to electricity, and the user does not need to shift the relative positional relationship between the teeth, so the computer is free from sliding back and forth. And advantages can be more clearly explained in detail below. The drawing of Yapei, detailed description [Exemplary Miaoming's consistent implementation of the illustration] Figures and pictures of the two installations, which are the basis of the eyes of the device of the present invention, so an imaging device is needed The k of m ^, is identified. This image capture ^; Γ: t is a digital camera 2. (Such as "Figure 1A"

Couple Device ^ CCD),] (Charge uses CCD 30 as the image, as shown in Figure 1B "). If it is to use the camera # 4, ~ image capture device, it must be installed in a spectacle frame 3 1 is convenient; -3 is worn, and its method of transmitting electrical signals is more wireless transmission using a digital camera 2 〇 As an image capture device, because of the 497065 V. Description of the Invention (3) The camera can be placed on a desk or computer 10 and then the camera is pointed at the user to capture the image, so it can be wired to the computer i 0 Pass the telegraph signal. Please refer to "Figure 2" is the main flowchart of the method of the present invention. The flow f is similar to the process of starting and ending a computer. First, start the computer, enter the Windows operating system (step 20 ", and then load the above). Image driver (digital camera 20 or CCD 30) to activate the device = related hardware equipment (step 202). In the first embodiment, since each digital camera 20 is provided by f The driver and application program interface (APPllcati〇n Program Interface, Αρι) of its products, so = users can directly use the driver and application program provided by the product to obtain image data through the digital camera 2 If you want to get an image, you must also write: soft == knowing type and driver. It should also be noted that, no matter which of the above bismuth examples, the interface program of the image capturing device must be set to resident. As is the case with the current mouse, its interface program is only able to dynamically capture image data and use it as a basis for moving the cursor position (Step 203). Finally, when the user no longer needs to use the window system: Close the window in the usual way (step 204), and leave the video system (step 205). Please refer to the "S3 diagram" and "4th diagram", which are the individual flowcharts of the first and second embodiments of the method of the present invention. During this process, the procedures are the same and small, because the image capture device used is different. The difference is that the first need to drive and set the aforementioned image capture device (step 301 or step M to make both software and hardware ready to capture the user's image data (step

Page 6 497065 V. Description of the invention (4) 302 or step 402) An embodiment t includes positioning of a human-shaped frame, and the embodiment includes positioning (step 4 03). The present invention may be performed first or an offset adjustment may be performed first. Mobile vision is to directly and dynamically follow step 305 or step 405, until the user follows, and / or the inner face shape, eye shape, and pupil position of the captured user image data. The first eye part is positioned (step 303). If 9 and the eye position and the pupil position are also expected to be V /, the second, although the position is completed, determine whether it is -month / restart (step 304 or step 4〇4) The next time you start the program (step 306 or step 4〇6), and then move the window without looking at the pupil offset according to the cursor on the second stack (step 305 or step 40). Cursor II). The above-mentioned motion detection process has been repeated. 4 It is necessary to shut down (step 3007 or step 407) and end. The specific implementation contents of the individual embodiments are given as follows: ·) First embodiment ^ (using a digital camera 20 as an image manipulation device) 1 · Please refer to "Figure 5 Α" to capture the captured image to the user to The stroke method determines its shape 'including the facial shape 51, the eye shape 5 2 and the pupil 5 2 1' and determines a first positioning frame 41 according to the facial shape 51. 2 · Please refer to "Figure 5B" to determine the position of the eye, and calculate the relative position coordinates (Xa, Ya) of the pupil 521 and the first positioning frame 41 for calculation by the program. 3 Please refer to "Figure 5C". When the eye is rotated, the relative position coordinates of the pupil 5 2 1 and the first positioning frame 41 will change (Xb, Yb), and the vector that moves sequentially changes, and As a map

497065 V. Description of the invention (5) --- Move to the position of the cursor in the window screen. #, ^ Gongli. The calculation content is as follows: ⑴ First set (Xa, Ya) as the center point of the window. If a window with a resolution of 80 0 X 60 0 is taken as an example, that is, the coordinate position on the window (400, 300). 2) The displacement of the pupil is represented by a vector as '= Ub-Xa, Yb-Ya)'; Yb)-(Xa, Ya) (3) ^ Ύ The calculated ratio relative to the X_Y coordinate axis of the window screen α, / 3 Substituting into the calculation: The cursor position after the movement = the center point seat # + the movement vector, and the mouse's movement vector = (αΔχ, if a window with a resolution of 80 0 χ 60 0 is taken as an example, the position of the cursor after the movement = / 400, 300) + (α Δχ ， 万 Λγ); (j) Move the cursor on the window screen according to the position of the difference. (II) First, embodiment (using CCD 30 as an image capturing device) 1. Second: For example, "Figure 6A", the captured image of the user is drawn out in a tracing manner. Including the position of the eye shape 5 2 and the pupil 5 2 1, and a second positioning frame 4 2 〇2 is determined according to the shape of the eye 5 2. ^ Refer to "6 β picture", determine the position of the eye, and Calculate the relative position coordinates (Xa, Ya) of the pupil 521 and the second positioning frame 42 for calculation by the program. 3. Please refer to "Figure 6C". When the eyes are turned, the pupil

4 ^ / 065

The relative position coordinates between 521 and the second positioning frame 42 will be changed (Xb, Yb), and the vector will be changed according to the successive movement vector, which will be used as the amount of position movement that is mapped to the cursor in the window screen. The calculation is as follows: (1) The first ib sets (Xa, Ya) as the center point of the window. If a 800 × 600 resolution window is taken as an example, (Xa, Ya) is equivalent to (400 , 300) Coordinate position; (2) The displacement of the pupil is represented by a vector △ M = (Δχ, ΔΥ) = (Xb, Yb)-(xa, ya) = (Xb — Xa, Yb — Ya); ( 3) Substitute the ratio α and / 3 calculated from the positioning adjustment relative to the window screen χ — γ coordinate axis into the calculation: the cursor position after the movement = the center point coordinate + the movement vector, and the mouse's movement vector = (αΑχ, / 3ΔΥ ), If a window with a resolution of 80 0 X 60 0 is taken as an example, the position of the cursor after the movement = (400, 300) + (α Δχ, cold δυ); (4) moves the window on the screen according to the calculated position cursor. The part of the positioning adjustment mentioned above is to calculate the relative relationship between the range of pupil movement up and down and the resolution of the window. Taking the second embodiment as an example, assuming that the second positioning frame defined above is composed of 100 × 100 pixels, then (Xa, Ya) = (50, 50), this point Corresponds to (Xa ,, Ya,) = (40 0, 3 0 0) on the window screen, and α & X-direction screen resolution / pupil maximum movement amount in X direction

497065 V. Description of the invention (7) = 8 0 0/1 0 0 = 8 (assuming that the distance in the X direction of the second positioning frame is exactly the maximum movement amount of the pupil X direction) / 3 = screen resolution in the Y direction / pupil in the Y direction Maximum amount of movement = 6 0 0/1 〇〇 = 6 (assuming that the distance in the y direction of the second positioning frame is exactly the maximum amount of movement in the y direction of the pupil) If the coordinates of the pupil after the movement (Xb, Yb) = (10, 60), Then (Δ ×, ΔΥ) = (Xb-Xa, Yb —Ya) = (10 -50, 6 0-5 0) = (-40, 10) So the cursor will move to (Xb ,, Yb,) (Xb ,, Yb ') = (40 0, 30 0) + (8 x -40, 6 x 10). The movement of the ball to the hole and the fixed ball is large. Of course, the eyeball mark moves. When the right button is pressed, the above button can be measured. The amount of movement including the speed between the frames and the speed of the cursor movement will be the same as the cursor input. The calculation of the formula and sound method uses "the opposite of the square as the fast, then the moving, the distance and the smaller. The sound recognition of the window command, 3 6 0) mode, the method of the quantity" and "position The relationship "size" is the same for the same party. If the former has the same function, it means that the auxiliary party calculates the coordinate position based on the coordinate position and calculates the size of the moving bit of the window's upstream index. "We can use it as the basis for" direction ". The eye consideration point 'that is, if the eye is upward' in the computer window, it will cause the eyeball to move slowly. Although it is converted to the computer window's vertex, you can add inputs such as left mouse button and command. You can use closed eyes or press the mouse or keyboard

Page 10

/ UOD V. Description of the invention (8) In terms of the data structure used in the present invention, in order to facilitate the memorization of the data, a matrix is used to record the image data obtained by the image capture device: _ as in Figure 7A In the example shown in the figure, the matrix can be found in the figure: the value of the prime can distinguish the polarized data (maximum or minimum), so we can simplify the matrix: let g (x, y) = 0 if f (x, y) ^ 128 g (x, y) = 1 If f (X, y) > 128 is replaced by the above, we can get a binary reserve η 'κ as in "Figure 7B" Reduce the memory space required to store data: 1 like $ has been reduced from Ibyte to lbit, so the speed of data reading and transmission is faster: the sample plot is calculated by the change amount of g (x, y), and corresponds to On the screen, i Chen Er is used as the basis for the cruise to move to a relative position. Among them, 1 ^ can be self-defining, usually the size of the input matrix of the eyeball = 撷 captured by the image capture device, in general, about W⑽pixel / hibiscus = technical aspects can be used to image Manipulate the obtained data and cooperate with 乂, Fuzzy Algorithm and neural network-like algorithms

(Netwc) rk Algorithm) constitutes a recognition system with fast parallel processing and two real offenses, making the automatic correction of the cursor position more effective. [Effect of the invention] The method disclosed by the betting invention is to use image recognition to do computer games. "J two position" can operate the cursor movement more easily, so the user can avoid the inconvenience of using the mouse to locate and use the mouse keyboard back and forth.

Page 11 497065 V. Description of the invention (9) Although the present invention has been disclosed above with several preferred embodiments, it is not intended to limit the present invention. Any person skilled in the art will not depart from the spirit and scope of the present invention. As some changes and retouching can be made, the scope of protection of the present invention shall be determined by the scope of the attached patent application.

Page 497065 Brief description of the drawings [Explanation of the drawings] Figure 1A shows the first embodiment of the device of the present invention; Figure 1B shows the second embodiment of the device of the present invention; Figure 2 shows Figure 3 shows the main flow chart of the method of the present invention; Figure 3 shows the flow chart of the first embodiment of the method of the present invention; Figure 4 shows the flow chart of the second embodiment of the method of the present invention; User appearance and positioning frame of the tracing edge captured by the image capturing device in the first embodiment; FIG. 5B shows the positioning and calculation of the relative position between the pupil and the positioning frame in the first embodiment; 5C Figure 'shows the change of the relative position between the pupil and the positioning frame in the first embodiment; Figure 6A shows the user's appearance captured by the image capture device in the second embodiment; Figure 6B Figure 6 illustrates the relative position between the pupil and the positioning frame in the second embodiment; Figure 6C 'illustrates the change in the relative position between the pupil and the positioning frame in the second embodiment; Figure 7A, An example of the data structure used in the present invention is shown; with FIG. 7B: “No. 7A The simplified binary matrix in the example in ". [Illustration of Symbols in the Schematic]

Page 13 497065

Page 14

Claims (1)

497065 6. Scope of patent application 1. A method for visually selecting a cursor of a window, which controls the movement of a computer cursor, and includes the use of eye movements to obtain the user's pupil relative to the information; and τ relative to the user's appearance The movement is adjusted according to the relative position of the pupil of the user 2 and the relative position and motion information between the cursor and the computer window, and the dynamic adjustment is made according to the method described in item (1) of the scope of patent application. Relative position. face. The shape, eye shape, and pupil are the same as described in item 2 of the scope of patent application & *, wherein the image manipulation device is a digital camera (Digital Camera). The window mentioned in the above item has a positioning frame. The machine is also based on the face shape. 5 'As the method of the scope of patent application No. 4 method, in which the user ’s pupil cursor is used to visually select the amount, as The user ’s pupil ’s relative movement? The change from the relative position. 6. The visual movement as described in item 5 of the application. Amount of change in position: Right: The relative positional relationship between the pupil and the pupil. Corresponds to Peregrine Falcon and computer window 7. As per the scope of patent application No. 5, arbitrarily click on the visual click The amount of change in the relative position between the user's pupil disk frame and the frame corresponds to the relative position relationship between the cursor and the computer window. According to the method of visually selecting a cursor on a window as described in item 2 of the scope of the patent application, the image capturing device may be a Charge Couple Device (CCD) mounted on a spectacle frame. According to the visual selection method of the cursor of the window described in item 8 of the scope of the patent application, wherein the shape of the user captured by the charge-coupled element is the shape of the eye and the position of the pupil. 〇 As shown in item 9 of the patent application, the cursor of the window is visually selected, and a positioning frame is determined based on the shape of the eye of the charge transfer element. 1. Such as the amount of change in the application method 2. Such as the method between the application method and the fixed window 3, such as between the method of the application method and the fixed window 4. The cursor vision of the window as described in item 10 of the patent scope of application 5 Click on the & relative position between the pupil of the user and the positioning frame as the relative movement information of the pupil of the user. " The visual point of the cursor of the window described in item 11 of the month "Liganwei" where the coordinate conversion method is used to change the relative position of the user ’s pupil t frame to the relative of the cursor to the computer Positional relationship. The visual point of the cursor of the window as described in item 11 of the Mingli area, which is a vector conversion method, corresponds to the change in the relative position of the pupil & +, corresponding to the relative position of the cursor and computer 2. relationship. Visual click on the cursor of the window described in item 1 of the patent scope 6. Method L for applying for a patent, which further includes a step of obtaining an input instruction. The cursor visual point of the window described in item i 4 of the scope of the patent application = method, wherein the method of inputting the instruction is to convert the information of the user's closed eyes to a specified operation instruction. σgen, 6:-A kind of visual cursor selection device for windows, which uses the eyeball as the basis for operating the cursor movement of the computer. It includes: Helps an image capture device to locate the appearance of the user and pay the user. The position of the pupil, and a reference frame is set, and the relative position information of the pupil and the positioning frame is output simultaneously; and a computer communicates with the image capture device through an interface, according to the relative position between the positioning frames And the relative position between the cursor and the computer window is dynamically adjusted. n 1 7. Cursor visual point-and-click device for window as described in item 16 of the scope of patent application, wherein the image capture device can be a digital camera. ', (Digital Camera). 18. The visual cursor selection device of the window described in item 17 of the tD month patent, wherein the user's appearance captured by the digital camera is a face appearance, an eye appearance, and a pupil position. 19 嚷 ί ϋ :: Cursor visual point k around the window described in item 18 /, the positioning frame is determined based on the shape of the face. 2〇 = Please =; The cursor point-clicking device around the window described in item 16, wherein the image capturing device can be a spectacle frame 497065 6. Charge Coupled Device (CCD) within the scope of patent application. 2 1. The cursor visual selection device for a window as described in item 20 of the scope of the patent application, wherein the shape of the user captured by the charge-coupled element is the shape of the eyes and the position of the pupil. 2 2. According to the application | patent scope of item 21 of the patent, the visual selection device of the cursor of the window, wherein the positioning frame is determined based on the shape of the eye. 2 3. The visual cursor selection device of the window according to item 16 of the scope of patent application, which further includes a step of obtaining an input instruction. 24. The cursor visual selection device for a window as described in item 23 of the scope of patent application, wherein the method of inputting the instruction is to convert the message of the user's eyes to a specified operation instruction. Page 18