CN101872237B - Method and system for pupil tracing as well as correction method and module for pupil tracing - Google Patents
Method and system for pupil tracing as well as correction method and module for pupil tracing Download PDFInfo
- Publication number
- CN101872237B CN101872237B CN2009101372119A CN200910137211A CN101872237B CN 101872237 B CN101872237 B CN 101872237B CN 2009101372119 A CN2009101372119 A CN 2009101372119A CN 200910137211 A CN200910137211 A CN 200910137211A CN 101872237 B CN101872237 B CN 101872237B
- Authority
- CN
- China
- Prior art keywords
- pupil
- angle
- display screen
- coordinate
- line
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Images
Abstract
The invention discloses a method and a system for pupil tracing as well as a correction method and a module for pupil tracing. The method for pupil tracing comprises the following steps: capturing an image of one user eye, wherein the image of one eye comprises a pupil image and two reflective point images; calculating an area parameter and an angle parameter based on the pupil and the two reflective points; marking the area parameter and the angle parameter on an angle-area plane graph to obtain original coordinates on the angle-area plane graph; and converting the original coordinates on the angle-area plane graph into display cursor coordinates by using a coordinates system conversion method. Since the eye moving range described by characteristic values (the area and the angle) of the invention is a rectangle, the conversion process of the coordinates is equivalent to the conversion of coordinates in two rectangles, and therefore, the error can be greatly reduced, and the pupil tracing accuracy can be improved.
Description
Technical field
The present invention relates to a kind of pupil tracing method, particularly relate to and a kind ofly use special coordinate and change preposition technology and improve the pupil tracing method of accuracy.
Background technology
The technological common computer that is applied to control of people's ocular pursuit (Eye tracking) can develop as the personage of gradually freezing the inconvenience of people or limbs and pass through through computer and extraneous accessory of linking up.This technology is meant follows the trail of moving of eyeball, obtains eyeball position coordinate or motion track, and according to this computer is produced certain preset steering order.Therefore this respect technology at first must detect moving of eyeball accurately; Then another emphasis is must convert computer exactly to produce the required data of steering order; For example eyeball position conversion correspondence is become the slider position on the computer display screen, otherwise will cause wrong steering order down.
Known person ocular pursuit technology generally can be divided into contact and contactless two big classes.Wherein, contact human eye tracer technique for example needs the user to wear the search coil method of the contact lenses of special adding inductive coil; For example need the electroculogram method of user again all around at two electrodes that are sticked.The wear-type (Head-mount) that need wear specific installation is then for example arranged at present and exempt from wear-type (Free-head) as for contactless human eye tracer technique.Aforementioned wear-type human eye tracer technique is located at video camera and light source shelf on user's head, utilizes the eyeball image of video camera acquisition to infer the sight line position, can get rid of user's head and move the error that causes; Exempting from wear-type human eye tracer technique then must overcome user's head and move the problem that causes error.
Intelligible, because contact human eye tracer technique has the foreign matter contact concerning the user, can bring strong discomfort; Contactless wear-type human eye tracer technique then is because helmet causes head to bear weight, lets the user feel heavy the user.The repulsion sense that above kind all causes the user to use, let us not go into the question now accuracy and cost problem are promoted difficult basically.
Exempt from wear-type human eye tracer technique aspect; Though mainly containing user's head moves the problem that causes error and need overcome; But owing to can not cause discomfort or bear added burden to the user; Industry develops and respectively releases different method for tracing towards this direction mostly, and is basic tool with the image analysing computer technology mostly.For instance, the U.S. the 7th, 197; The 165B2 patent; Disclose a kind of method of utilizing image data to carry out eyeball tracking, it utilizes the three-dimensional computer model of the head set up in advance, and seeks the zone that is defined as eyes and eyebrow in the three dimensions; These zones are projected the bidimensional image space, find out eyeball position by the pixel intensity value of analysis of two-dimensional image space then.This technology not only need be set up the three-dimensional computer model of user's head in advance and follow the trail of the operational method complicacy, must cooperate the processor of powerful calculation function just to have way to reduce the processing time, and cost is costliness very.
No matter which kind of human eye method for tracing; The developer focuses on the mobile aspect of detecting eyeball more; And the big multipotency of this respect reaches certain accuracy, but then carries no weight aspect the conversion of display screen coordinate in eyeball coordinate mapping (mapping), often takes place than multiple error in this section.In general; Can proofread and correct earlier when carrying out for the first time the coordinate conversion; Common bearing calibration is to make the user watch four corners of display screen or edge up and down; The eyeball coordinate that detects is as check point, and these check points are regarded as correct and are mapped to display screen, then calculates the position that projects display screen with interpolation method when the user watches all the other positions of display screen thereafter.Yet; Aforementioned corrected method not only check point is few; The coordinate of the most positions of whole display screen is all tried to achieve with interior inserting; A check point has deviation slightly in case cause wherein, then will produce large-scale error effect, and this mode that directly converts the eyeball coordinate into the display screen coordinate is not preferable by checking.
This shows, no matter above-mentioned existing which kind of human eye method for tracing on the structure and use of method and system, obviously still has inconvenience and defective, and demands urgently further improving.In order to solve the problem of above-mentioned existence, relevant manufacturer there's no one who doesn't or isn't seeks solution painstakingly, but does not see always that for a long time suitable design is developed completion, and this obviously is the problem that the anxious desire of relevant dealer solves.Therefore how to found a kind of new pupil tracing method and system, and be used for the bearing calibration and the module of pupil tracing, real one of the current important research and development problem that belongs to, also becoming the current industry utmost point needs improved target.
Summary of the invention
The objective of the invention is to, overcome the defective that existing human eye method for tracing exists, and a kind of new pupil tracing method and system are provided, technical matters to be solved be make it improve accuracy and cost low.
Another object of the present invention is to, a kind of bearing calibration of new structure is provided and proofreaies and correct module, be used for pupil tracing and can improve accuracy.
The object of the invention and solve its technical matters and adopt following technical scheme to realize.According to a kind of pupil tracing method that the present invention proposes, supply a user sight line is converted to the slider position on the display screen; Wherein this method comprises: an image capture step by this user's of image capture unit acquisition eyes image frame, comprises the image of a pupil and two reflective spots in this eyes image frame; One treatment step, by a processing unit according to this pupil and this two reflective spot obtain one be located in an angle-area plane original coordinate; And a coordinate switch process, change module by the coordinate that is connected with this processing unit and carry out, utilize the coordinate system conversion method to make the original coordinate on this angle-area plane convert a display screen vernier coordinate into.
The object of the invention and solve its technical matters and also can adopt following technical measures further to realize.
Aforesaid pupil tracing method, wherein this treatment step comprises an image analysing computer step, calculates an area parameter and an angle parameter by the image analysing computer module of this processing unit of being connected with this image capture unit according to this pupil and this two reflective spot; And a graphics process step, make this area parameters and this angle parameter be shown in this angle-area plane and obtain this original coordinate by the graphics process module of this processing unit of being connected with this image analysing computer module.
Aforesaid pupil tracing method, wherein in this image analysing computer step, with one pass the pupil center in this eyes image frame horizontal line, and this pupil and the line of a reflective spot wherein, the folded angle of two lines is this angle parameter.
Aforesaid pupil tracing method; Wherein in this image analysing computer step; With one pass the pupil center in this eyes image frame horizontal line, and this pupil and the line of a reflective spot wherein, the folded angle of two lines is first angle; This horizontal line and this pupil are second angle with the folded angle of line of another reflective spot wherein, and the mean value of first angle and this second angle is this angle parameter.
Aforesaid pupil tracing method in this image analysing computer step, is the summit with the pupil in this eyes image frame wherein, and this pupil and this two reflective spot be the angle of line formation respectively, is this angle parameter.
Aforesaid pupil tracing method, wherein in this image analysing computer step, this area parameters is meant this pupil and the formed leg-of-mutton area value of this two reflective spots line.
Aforesaid pupil tracing method; Wherein in this image analysing computer step; This area parameters is meant this pupil and the formed leg-of-mutton area value of this two reflective spots line divided by a normalization factor, and this normalization factor is that the square distance by this two reflective spot calculates.
Aforesaid pupil tracing method, wherein in this image analysing computer step, this area parameters is meant this pupil and the formed leg-of-mutton area value of this two reflective spots line divided by a normalization factor, this normalization factor equals the square distance of this two reflective spot divided by 2.
Aforesaid pupil tracing method, wherein this coordinate switch process is to adopt affine conversion method to carry out.
Aforesaid pupil tracing method; Wherein when using this method for the first time, should proofread and correct as follows: proofread and correct module by one and define 1-n check point or correction block in the display screen, and at least four four corners that are respectively adjacent to this display screen in these check points or the correction block; Capture 1-n eyes image frame by this image capture unit, these eyes image frames are that this user watches the 1-n check point of this display screen respectively attentively or proofread and correct block the corresponding acquisition of time institute; Obtain the original coordinate of 1-n on this angle-area plane according to pupil in this 1-n eyes image frame and reflective spot seriatim by this processing unit; Reach coordinate by this coordinate conversion module uses and the consistent coordinate system conversion method of this coordinate conversion module makes these original coordinates be mapped to the check point of this display screen or proofread and correct block.
Aforesaid pupil tracing method; Wherein in the step of proofreading and correct; This processing unit is with this 1-method; Pupil in its n eyes image frame calculates corresponding area parameters and angle parameter with reflective spot, is shown in seriatim and obtains the original coordinate of this 1-n on this angle-area plane.
Wherein in the step of proofreading and correct, the coordinate that this coordinate conversion module makes these original coordinates be mapped to the check point of this display screen or proofread and correct block is tried to achieve the coefficient in the coordinate system conversion formula to aforesaid pupil tracing method at last.
Aforesaid pupil tracing method, wherein 1-n proofreaies and correct block and is meant the array block by the i * j of this display screen five equilibrium in this display screen, so n=ij.
The object of the invention and solve its technical matters and also adopt following technical scheme to realize.A kind of pupil tracing system according to the present invention proposes supplies a user sight line is converted to the slider position on the display screen, and comprise one supply this user's of acquisition eyes image frame image capture unit; The image that wherein comprises a pupil and two reflective spots in this eyes image frame, and this system also comprises: a processing unit, according to this pupil and this two reflective spot obtain one be located in an angle-area plane original coordinate; And coordinate conversion module, be connected with this processing unit, utilize the coordinate system conversion method to make the original coordinate on this angle-area plane convert a display screen vernier coordinate into.
The object of the invention and solve its technical matters and also can adopt following technical measures further to realize.
Aforesaid pupil tracing system; Wherein this processing unit comprises the image analysing computer module that is connected with this image capture unit; It orients this pupil and this two reflective spot in this eyes image frame, and calculates an area parameter and an angle parameter according to this pupil and this two reflective spot; And the graphics process module that is connected with this image analysing computer module, make this area parameters and this angle parameter be shown in this angle-area plane and obtain this original coordinate.
Aforesaid pupil tracing system, wherein this system also comprises two light sources, in this user's eyes, forms this two reflective spot.
Aforesaid pupil tracing system, wherein this image analysing computer module is the horizontal line that passes the pupil center in this eyes image frame with, and this pupil and the line of a reflective spot wherein, the folded angle of two lines is this angle parameter.
Aforesaid pupil tracing system; Wherein this image analysing computer module is the horizontal line that passes the pupil center in this eyes image frame with; And this pupil and the line of a reflective spot wherein; The folded angle of two lines is first angle, and this horizontal line and this pupil are second angle with the folded angle of line of another reflective spot wherein, and the mean value of first angle and this second angle is this angle parameter.
Aforesaid pupil tracing system, wherein this image analysing computer module is to be the summit with the pupil in this eyes image frame, this pupil and this two reflective spot be the angle that forms of line respectively, is this angle parameter.
Aforesaid pupil tracing system, wherein the area parameters calculated of this image analysing computer module is meant this pupil and the formed leg-of-mutton area value of this two reflective spots line.
Aforesaid pupil tracing system; The area parameters calculated of this image analysing computer module wherein; Be meant this pupil and the formed leg-of-mutton area value of this two reflective spots line divided by a normalization factor, this normalization factor is that the square distance by this two reflective spot calculates.
Aforesaid pupil tracing system; The area parameters calculated of this image analysing computer module wherein; Be meant this pupil and the formed leg-of-mutton area value of this two reflective spots line divided by a normalization factor, this normalization factor equals the square distance of this two reflective spot divided by 2.
Aforesaid pupil tracing system, wherein this coordinate conversion module is to adopt affine conversion method to carry out the coordinate conversion.
Aforesaid pupil tracing system; Wherein this system also comprises a correction module; When using for the first time, should proofread and correct as follows: this correction module defines 1-n check point or correction block in the display screen in advance, and at least four four corners that are respectively adjacent to this display screen in these check points or the correction block; This image capture unit captures 1-n eyes image frame, and these eyes image frames are that this user watches the 1-n check point of this display screen respectively attentively or proofread and correct block the corresponding acquisition of time institute; This processing unit obtains the original coordinate of 1-n on this angle-area plane according to pupil in this 1-n eyes image frame and reflective spot seriatim; And this correction module use makes these original coordinates be mapped to the check point of this display screen or the coordinate of correction block with the consistent coordinate system conversion method of this coordinate conversion module.
Aforesaid pupil tracing system; Wherein this system also comprises an image analysing computer module; In this trimming process, calculate corresponding area parameters and angle parameter with the line of reflective spot, be shown in seriatim according to this area parameters and angle parameter by this graphics process module again and obtain the original coordinate of this 1-n on this angle-area plane with the pupil in this 1-n eyes image frame.
Aforesaid pupil tracing system, wherein this correction module makes the original coordinate of this 1-n be mapped to the check point of this display screen or proofreaies and correct the coordinate of block, tries to achieve the coefficient in the coordinate system conversion formula at last.
Aforesaid pupil tracing system, wherein 1-n proofreaies and correct block and is meant the array block by the i * j of this display screen five equilibrium in this display screen, so n=ij.
The object of the invention and solve its technical matters and also adopt following technical scheme to realize in addition.A kind of bearing calibration that is used for pupil tracing according to the present invention's proposition; Wherein the method includes the steps of: proofread and correct 1-n check point or correction block in module predefined one display screen, at least four four corners that are respectively adjacent to this display screen in these check points or the correction block by one; Capture 1-n eyes image frame by an image capture unit, these eyes image frames are that this user watches the 1-n check point of this display screen respectively attentively or proofread and correct block the corresponding acquisition of time institute; Obtain the original coordinate of 1-n on the angle-area plane according to pupil in this 1-n eyes image frame and reflective spot seriatim by a processing unit; And this correction module use makes these original coordinates be mapped to the check point of this display screen or the coordinate of correction block with the consistent coordinate system conversion method of coordinate conversion module.
The object of the invention and solve its technical matters and also can adopt following technical measures further to realize.
Aforesaid bearing calibration wherein is to calculate corresponding area parameters and angle parameter according to the pupil of this 1-n eyes image frame and the line of this two reflective spot seriatim by an image analysing computer module of this processing unit; Graphics process module by this processing unit is shown in this angle-area plane seriatim and obtains the original coordinate of this 1-n according to this area parameters and angle parameter again.
Aforesaid bearing calibration, wherein this correction module makes the original coordinate of this 1-n be mapped to the check point of this display screen or the coordinate of correction block, tries to achieve the coefficient in the coordinate system conversion formula at last.
Aforesaid bearing calibration, wherein 1-n proofreaies and correct block and is meant the array block by the i * j of this display screen five equilibrium in this display screen, so n=ij.
Aforesaid bearing calibration; Wherein capture the step of this 1-n eyes image frame; Must cooperate the 1-n check point that makes this display screen or proofread and correct block to be highlighted in regular turn and use and remind this user to watch attentively, when this user watches the check point that is highlighted attentively or proofreaies and correct block, carry out the acquisition of eyes image frame.
The object of the invention and solve its technical matters and adopt following technical scheme to realize in addition again.A kind of correction module that is used for pupil tracing according to the present invention's proposition; Wherein this module comprises: one highlights controller; Make the predefined 1-n check point of a display screen or proofread and correct block to be highlighted in regular turn and use and remind this user to watch attentively these check points or proofread and correct at least four four corners that are respectively adjacent to this display screen in the block; One capture controller highlights controller at this and makes wherein a check point or proofread and correct block when being highlighted, and an image capture unit is sent the controlling signal of pick-up image; And a counter, receiving the distribution range of the original coordinate of 1-n that obtains from a graphics process module, this distribution range is approximate rectangular, and makes these original coordinates be mapped to the coordinate of the correction block of this display screen.
The object of the invention and solve its technical matters and also can adopt following technical measures further to realize.
Aforesaid correction module, wherein this counter makes the original coordinate of this 1-n be mapped to the check point of this display screen or the coordinate of correction block, tries to achieve the coefficient in the coordinate system conversion formula at last.
Aforesaid correction module, wherein this 1-n proofreaies and correct block and is meant the array block by the i * j of this display screen five equilibrium, so n=ij.
The present invention compared with prior art has tangible advantage and beneficial effect.Can know that by above technical scheme major technique of the present invention thes contents are as follows:
Pupil tracing method of the present invention comprises an image capture step, an image analysing computer step, a graphics process step, and a coordinate switch process.Wherein, the image capture step is the eyes image frame that is captured a user by an image capture unit, comprises the image of a pupil and two reflective spots in this eyes image frame.The image analysing computer step is to calculate an area parameter and an angle parameter by an image analysing computer module according to this pupil and this two reflective spot.The graphics process step is undertaken by the graphics process module that is connected with this image analysing computer module, makes this area parameters and this angle parameter be shown in the planimetric map of one angle-area, obtain one be located in this angle-area plane original coordinate.Last coordinate switch process is undertaken by the coordinate conversion module that is connected with this graphics process module, utilizes the coordinate system conversion method to make the original coordinate on this angle-area plane convert a display screen vernier coordinate into.
Pupil tracing of the present invention system supplies a user sight line is converted to the slider position on the display screen, and comprises an image capture unit, an image analysing computer module, a graphics process module and coordinate conversion module.Image capture unit supplies this user's of acquisition eyes image frame, comprises the image of a pupil and two reflective spots in this eyes image frame.The image analysing computer module is connected with this image capture unit, orients this pupil and this two reflective spot certainly in this eyes image frame, and calculates an area parameter and an angle parameter according to this pupil and this two reflective spot; The graphics process module is connected with this image analysing computer module, makes this area parameters and this angle parameter be shown in the planimetric map of one angle-area, obtain one be located in this angle-area plane original coordinate.Coordinate conversion module and this graphics process module connect, and utilize the coordinate system conversion method to make the original coordinate on this angle-area plane convert a display screen vernier coordinate into.
The present invention is through experiment and mass data checking; When finding that the user watches the display screen difference; The distribution shape of parameter on angle-area plane that pupil and two reflective spots constitute in institute's pick-up image; Coincide the most with reply point distribution shape on the display screen-for example, the display screen profile is a rectangle, and the user watches each block of display screen or periphery or four resultant distribution shapes on angle-area plane in corner also very near rectangle; Directly pupil coordinate (watch the resulting pupil coordinate of display screen periphery, often constitute trapezoidal or non-equilateral quadrilateral) is mapped to the display screen coordinate compared to convention, can significantly reduces mapping error.
Pupil tracing method and system of the present invention should be proofreaied and correct earlier when using for the first time, and bearing calibration may further comprise the steps:
By 1-n check point or correction block in a correction module predefined one display screen, and at least four four corners that are respectively adjacent to this display screen in these check points or the correction block.Capture 1-n eyes image frame by an image capture unit, these eyes image frames are that this user watches the 1-n check point of this display screen respectively attentively or proofread and correct block the corresponding acquisition of time institute.Calculate corresponding area parameters and angle parameter according to the pupil in this 1-n eyes image frame with the line of reflective spot seriatim by an image analysing computer module.Seriatim the area parameters and the angle parameter of this this 1-n eyes image frame are shown in this angle-area plane by a graphics process module, obtain the original coordinate of 1-n.This correction module uses with the consistent coordinate system conversion method of coordinate conversion module and makes the original coordinate of this 1-n be mapped to the 1-n check point of this display screen or the coordinate of correction block, tries to achieve the coefficient in the coordinate system conversion formula at last.
The present invention is used for the correction module of pupil tracing, comprises that one highlights controller, a capture controller and a counter.Highlight controller and make the predefined 1-n check point of a display screen or proofread and correct block and highlighted in regular turn and use and remind this user to watch attentively, these check points or proofread and correct at least four four corners that are respectively adjacent to this display screen in the block.The capture controller highlights controller at this and makes wherein a check point or proofread and correct block when being highlighted, and an image capture unit is sent the controlling signal of pick-up image.Counter receives the distribution range of the original coordinate of 1-n that obtains from a graphics process module; This distribution range is approximate rectangular; And make these original coordinates be mapped to the coordinate of the correction block of this display screen, try to achieve the coefficient in the coordinate system conversion formula at last.
Beneficial effect of the present invention is: adopt angle parameter, the area parameters of the coordinate mapping pre-treatment of original creation-pupil is moved generation to be shown on the angle-area plane; Change with the original coordinate on this angle-area plane again; The transfer process of this moment; Be equivalent to two conversions between rectangle, can significantly reduce the error that convention is caused by non-rectangle torque shape, the accuracy that therefore improves pupil tracing.
In sum, the present invention has obvious improvement technically, and has tangible good effect, really is the new design of a novelty, progress, practicality.
Above-mentioned explanation only is the general introduction of technical scheme of the present invention; Understand technological means of the present invention in order can more to know; And can implement according to the content of instructions, and for let of the present invention above-mentioned with other purposes, feature and advantage can be more obviously understandable, below special act preferred embodiment; And conjunction with figs., specify as follows.
Description of drawings
Fig. 1 is a system schematic, and the preferred embodiment of pupil tracing of the present invention system is described;
Fig. 2 is a system block diagrams, and the detailed formation of present embodiment is described;
Fig. 3 is a process flow diagram, and the correcting process when present embodiment uses for the first time is described;
Fig. 4 is a process flow diagram, and the flow process when present embodiment generally uses is described;
Fig. 5 is the synoptic diagram of an eyes image frame;
Fig. 6 is the synoptic diagram of the correction block that defines in the display screen;
Fig. 7 is an angle-area plane, and individual original coordinate of the 1-n that is located on it and any point are described;
Fig. 8 is the synoptic diagram of changing between two coordinate systems.
Embodiment
Reach technological means and the effect that predetermined goal of the invention is taked for further setting forth the present invention; Below in conjunction with accompanying drawing and preferred embodiment; To the pupil tracing method and system that proposes according to the present invention, and be used for the bearing calibration and the module of pupil tracing, specify as after.
See also Fig. 1 and Fig. 2; Pupil tracing of the present invention system 100 is to be converted to the slider position on the display screen 10 in order to the sight line with a user; Its preferred embodiment comprises processing unit 40, the coordinate conversion module 6 that an image capture unit 2, two light sources 3, include an image analysing computer module 4 and a graphics process module 5, and one proofreaies and correct module 7.Aforementioned processing unit 40, coordinate conversion module 6 and correction module 7 are at present embodiment; Be to produce after being stored in the computer-readable program fetch of a memory body when host computer 1 CPU (figure do not show) that is connected with this display screen 10 reads and carry out one; But not as limit; Also can be in this host computer, to realize with integrated circuit or firmware mode; Perhaps utilize the electronic installation (figure do not show) that is connected with this host computer 1 to read and executive routine after produce, and operation result is sent to this host computer 1 with the slider position on the control display screen 10.
Please cooperate and consult Fig. 5, this two light source 3 can form two reflective spots 31,32 in user's eyes.Image capture unit 2 supplies acquisition user's eyes image frame 20, comprises the image of a pupil 30 and this two reflective spot 31,32 in this eyes image frame 20.In order to help image analysing computer module 4 to orient pupil 30 and reflective spot 31,32 images, can shine this user with infrared ray light projector (figure does not show) earlier, again by image capture unit 2 these eyes image frames 20 of acquisition.
Image analysing computer module 4 is connected with image capture unit 2; It comprises a steady arm 41, is that eyes image frame 20 is carried out binary conversion treatment (thresholding claims the gray scale graduation again) at present embodiment; Make the part of gray scale in the image, be adjusted into black greater than preset gray-scale value; Gray scale is adjusted into white less than the part of preset gray-scale value in the image, the clear by this pupil 30 of finding out; Likewise, by the preset gray-scale value of adjustment, can find out this two reflective spot 31,32.Image analysing computer module 4 also comprises an area parameter calculator 42 and an angle parameter counter 43, calculates an area parameter and an angle parameter according to this pupil 30 with this two reflective spot 31,32 respectively.
The method of area parameters counter 42 reference area parameters of the present invention has two: one of which is a hypothesis pupil 30 and 31,32 3 of reflective spots conllinear not, calculates 3 formed leg-of-mutton area values of line.Its two, be to further consider user's the relative image capture unit 2 of head and error correction problem under the situation that moves forward and backward; Since when user's head than near image capture unit 2 time; The distance of this two reflective spot 31,32 is bigger in the eyes image frame 20; Along with user's head away from image capture unit 2; These two reflective spots, 31,32 distances diminish, and that is to say that mobile the causing of user's head increases another parameter when calculating the triangle area of pupil 30 and reflective spot 31,32 formations, therefore should this parameter be prevented; Present embodiment is promptly considered this error correction problem; The method that the actual calculating area parameter adopts is; To pupil 30 and reflective spot 31,32 formed triangle areas divided by a normalization factor, this normalization factor equal this two reflective spot 31,32 distance square divided by 2.
The method that angle parameter counter 43 of the present invention calculates angle parameter has three, selects fixed wherein a kind of pupil tracing effect accurately that all can obtain during calculation.One of which, with one be horizontally through pupil 30 centers horizontal line L, and the pupil 30 and the line of a reflective spot 31 wherein, the folded angle of two lines is this angle parameter.They are two years old; Make horizontal line L and the pupil 30 and the line of a reflective spot 31 wherein; The folded angle of two lines is first angle [alpha] 1; This horizontal line L and pupil 30 and the folded angle of line of another reflective spot 32 wherein be second angle [alpha], 2, the first angle [alpha] 1 with the mean value of this second angle [alpha] 2 be explanation below this angle parameter-present embodiment and graphic be to carry out with this account form.Its three, be the summit with pupil 30, pupil 30 and this two reflective spot 31,32 be the angle (being equivalent to α 2-α 1) that forms of lines respectively, is this angle parameter.
Coordinate conversion module 6 is connected with graphics process module 5, utilize the coordinate system conversion method make original coordinate on this angle-area plane (x, y) be mapped to display screen 10 the vernier coordinate (u, v).At present embodiment, coordinate conversion module 6 is to adopt affine conversion (affine transformation) method to carry out the coordinate conversion.
When using for the first time, need make the user watch earlier proofreading and correct in the module 7 predefined display screens 10 the 1-n check point or proofread and correct block to proofread and correct, obtain the correction coordinate on the angle-area plane by this.Present embodiment is to proofread and correct block with 1-18 shown in Figure 6 to illustrate, and 1-18 proofreaies and correct block and is meant 3 * 6 array block by display screen 10 five equilibriums among the n=18 just, and display screen 10.But not as limit, as long as at least four four corners that are respectively adjacent to display screen 10 in these check points or the correction block.
Consult Fig. 2, Fig. 3 and Fig. 6 simultaneously, the correcting mode when using for the first time is following, and the mainly leading correction module of proofreading and correct 7, comprises that one highlights controller 71, a capture controller 72, and a counter 73, and its effect branch is set forth in hereinafter.
Step S11-highlights controller 71 and makes k proofread and correct block to be highlighted in regular turn and use and remind this user to watch attentively, wherein k=1-18.Capture controller 72 is highlighting controller 71 when making wherein one to proofread and correct block and highlighted, and image capture unit 2 is sent the controlling signal of pick-up image.
2 acquisitions of step S12-image capture unit obtain k eyes image frame 20 (Fig. 5).
Step S13-image analysing computer module 4 calculates area parameters and angle parameter according to pupil in this k eyes image frame 20 30 and reflective spot 31,32.
Step S14-graphics process module 5 is shown in angle-area plane with area parameters and angle parameter.
The abovementioned steps S11-step S14 execution of pulling over is finished up to k=n, then can on angle-area plane, obtain the original coordinate of 1-n (with reference to figure 7), and the original coordinate of this 1-n is the aforementioned corrected coordinate.
Then; In step S15; The distribution range that counter 73 receives from all original coordinates of graphics process module 5, this distribution range is approximate rectangular, and uses affine transformation approach and the coordinate that makes these original coordinates be mapped to the correction block of this display screen 10 as shown in Figure 8; Can try to achieve the coefficient in the coordinate system conversion formula of affine transformation approach, use for the coordinate conversion module 6 of follow-up this pupil tracing system 100.
After calibrated, as long as the user watches arbitrary position on the display screen 10 attentively, this pupil tracing system 100 promptly carries out flow process as shown in Figure 4, and explanation is as follows:
2 acquisitions of step S21-image capture unit obtain an eyes image frame 20.Need to prove that this image capture unit 2 is continuous captures, so the eyes image frame in the said acquisition of this step road is the image frame in the sequential.
Step S22-image analysing computer module 4 calculates area parameters and angle parameter according to pupil in this eyes image frame 20 30 and reflective spot 31,32.
Step S23-graphics process module 5 is shown in angle-area plane with area parameters and angle parameter, obtains the original coordinate on the angle-area plane.
Step S24-coordinate conversion module 6 utilizes the coordinate system conversion method to make the original coordinate on this angle-area plane be mapped to display screen vernier coordinate.
Because empirical evidence is found; When the user watches the display screen difference; The distribution shape of parameter on angle-area plane that pupil and two reflective spots constitute in institute's pick-up image; Coincide the most with reply point distribution shape on the display screen-for example, the display screen profile is a rectangle, and the user watches each block of display screen or periphery or four resultant distribution shapes on angle-area plane in corner also very near rectangle; Therefore the present invention is shown in angle parameter, the area parameters that pupil moves generation on the angle-area plane; Change with the original coordinate on this angle-area plane again; The transfer process of this moment is equivalent to two conversions between rectangle, can significantly reduce mapping error; Therefore the accuracy that improves pupil tracing can be reached the object of the invention really.
The above only is preferred embodiment of the present invention, is not the present invention is done any pro forma restriction; Though the present invention discloses as above with preferred embodiment; Yet be not in order to limiting the present invention, anyly be familiar with the professional and technical personnel, in not breaking away from technical scheme scope of the present invention; When the technology contents of above-mentioned announcement capable of using is made a little change or is modified to the equivalent embodiment of equivalent variations; In every case be not break away from technical scheme content of the present invention, to any simple modification, equivalent variations and modification that above embodiment did, all still belong in the scope of technical scheme of the present invention according to technical spirit of the present invention.
Claims (16)
1. a pupil tracing method supplies a user sight line is converted to the slider position on the display screen; It is characterized in that this method comprises:
One image capture step by this user's of image capture unit acquisition eyes image frame, comprises the image of a pupil and two reflective spots in this eyes image frame;
One treatment step; Image analysing computer module by a processing unit that is connected with this image capture unit is confirmed the position of pupil in the eyes image frame and two reflective spots; And calculate an area parameter and an angle parameter according to this pupil and two reflective spots, make this area parameters and this angle parameter respectively as the variable of the longitudinal axis of angle-area plane and transverse axis and obtain the corresponding original coordinate that is seated this angle-area plane by the graphics process module of this processing unit of connecting with this image analysing computer module; And
One coordinate switch process is undertaken by the coordinate conversion module that is connected with this processing unit, utilizes affine conversion method to make the original coordinate on this angle-area plane convert a display screen vernier coordinate into;
Wherein, said area parameters is: this pupil and the formed leg-of-mutton area value of this two reflective spots line; Or this pupil and the formed leg-of-mutton area value of this two reflective spots line be divided by a normalization factor, this normalization factor be the square distance by this two reflective spot calculate or the square distance that equals this two reflective spot divided by 2;
Wherein, said angle parameter is: pass horizontal line and this pupil of the pupil center in this eyes image frame and folded angle between the line of a reflective spot wherein with one; Or with one pass the pupil center in this eyes image frame horizontal line; And this pupil and the line of a reflective spot wherein; The folded angle of two lines is first angle; This horizontal line and this pupil are second angle with the folded angle of line of another reflective spot wherein, and the mean value of first angle and this second angle is angle parameter; Or be the summit with the pupil in this eyes image frame, this pupil and this two reflective spot be the angle that forms of line respectively.
2. pupil tracing method as claimed in claim 1 is characterized in that when using this method for the first time, should proofreading and correct as follows:
Proofread and correct module by one and define 1-n check point or correction block in the display screen, and at least four four corners that are respectively adjacent to this display screen in these check points or the correction block;
Capture 1-n eyes image frame by this image capture unit, these eyes image frames are that this user watches the 1-n check point of this display screen respectively attentively or proofread and correct block the corresponding acquisition of time institute;
Obtain the original coordinate of 1-n on this angle-area plane according to pupil in this 1-n eyes image frame and reflective spot seriatim by this processing unit; And
Use the coordinate that makes these original coordinates be mapped to the check point of this display screen or proofread and correct block with the consistent coordinate system conversion method of this coordinate conversion module by this coordinate conversion module.
3. pupil tracing method as claimed in claim 2; It is characterized in that in the step of proofreading and correct; This processing unit is to calculate corresponding area parameters and angle parameter with the pupil in this 1-n eyes image frame with reflective spot, is shown in seriatim and obtains the original coordinate of this 1-n on this angle-area plane.
4. pupil tracing method as claimed in claim 2; It is characterized in that in the step of proofreading and correct; The coordinate that this coordinate conversion module makes these original coordinates be mapped to the check point of this display screen or proofread and correct block is tried to achieve the coefficient in the coordinate system conversion formula at last.
5. pupil tracing method as claimed in claim 2 is characterized in that 1-n correction block is meant the array block by the i * j of this display screen five equilibrium in this display screen, so n=ij.
6. a pupil tracing system supplies a user sight line is converted to the slider position on the display screen, and comprises the image capture unit of this user of confession acquisition eyes image frame; It is characterized in that comprising in this eyes image frame the image of a pupil and two reflective spots, and this system comprises also:
One processing unit; It comprises the graphics process module that the image analysing computer module and that is connected with this image capture unit is connected with this image analysing computer module; This image analysing computer module is confirmed the position of pupil in the eyes image frame and two reflective spots; And calculating an area parameter and an angle parameter according to this pupil and two reflective spots, this graphics process module makes this area parameters and this angle parameter respectively as the variable of the longitudinal axis of angle-area plane and transverse axis and obtain the corresponding original coordinate that is seated this angle-area plane; And
One coordinate conversion module is connected with this processing unit, utilizes affine conversion method to make the original coordinate on this angle-area plane convert a display screen vernier coordinate into;
Wherein, said area parameters is: this pupil and the formed leg-of-mutton area value of this two reflective spots line; Or this pupil and the formed leg-of-mutton area value of this two reflective spots line be divided by a normalization factor, this normalization factor be the square distance by this two reflective spot calculate or the square distance that equals this two reflective spot divided by 2;
Wherein, said angle parameter is: pass horizontal line and this pupil of the pupil center in this eyes image frame and folded angle between the line of a reflective spot wherein with one; Or with one pass the pupil center in this eyes image frame horizontal line; And this pupil and the line of a reflective spot wherein; The folded angle of two lines is first angle; This horizontal line and this pupil are second angle with the folded angle of line of another reflective spot wherein, and the mean value of first angle and this second angle is angle parameter; Or be the summit with the pupil in this eyes image frame, this pupil and this two reflective spot be the angle that forms of line respectively.
7. pupil tracing as claimed in claim 6 system is characterized in that this system also comprises two light sources, in this user's eyes, forms this two reflective spot.
8. like claim 6 or 7 described pupil tracing systems, it is characterized in that this system also comprises one and proofreaies and correct module, when using for the first time, should proofread and correct as follows:
This correction module defines 1-n check point or correction block in the display screen in advance, and at least four four corners that are respectively adjacent to this display screen in these check points or the correction block;
This image capture unit captures l-n eyes image frame, and these eyes image frames are that this user watches the l-n check point of this display screen respectively attentively or proofread and correct block the corresponding acquisition of time institute;
This processing unit obtains the original coordinate of 1-n on this angle-area plane according to pupil in this 1-n eyes image frame and reflective spot seriatim; And
This correction module uses with the consistent coordinate system conversion method of this coordinate conversion module and makes these original coordinates be mapped to the check point of this display screen or the coordinate of correction block.
9. pupil tracing as claimed in claim 8 system; It is characterized in that; Said image analysing computer module calculates corresponding area parameters and angle parameter with the pupil in this 1-n eyes image frame with the line of reflective spot in this trimming process, be shown in seriatim according to this area parameters and angle parameter by this graphics process module and obtain the original coordinate of this 1-n on this angle-area plane.
10. pupil tracing as claimed in claim 8 system is characterized in that this correction module makes the original coordinate of this 1-n be mapped to the check point of this display screen or proofreaies and correct the coordinate of block, tries to achieve the coefficient in the coordinate system conversion formula at last.
11. pupil tracing as claimed in claim 8 system is characterized in that 1-n proofreaies and correct block in this display screen to be meant the array block by the i * j of this display screen five equilibrium, so n=ij.
12. a bearing calibration that is used for pupil tracing is characterized in that the method includes the steps of:
By 1-n check point or correction block in a correction module predefined one display screen, and at least four four corners that are respectively adjacent to this display screen in these check points or the correction block;
Capture 1-n eyes image frame by an image capture unit, these eyes image frames person of being to use watches the 1-n check point of this display screen respectively attentively or proofreaies and correct block the corresponding acquisition of time institute;
Image analysing computer module by a processing unit calculates corresponding area parameters and angle parameter according to the pupil in this 1-n eyes image frame with the line of two reflective spots seriatim, by a graphics process module of this processing unit this area parameters and angle parameter is shown in seriatim and obtains the original coordinate of 1-n on the angle-area plane; And
This correction module uses with the consistent affine conversion method of coordinate conversion module and makes these original coordinates be mapped to the check point of this display screen or the coordinate of correction block, tries to achieve the coefficient in the coordinate system conversion formula of affine conversion method;
Wherein, said area parameters is: this pupil and the formed leg-of-mutton area value of this two reflective spots line; Or this pupil and the formed leg-of-mutton area value of this two reflective spots line be divided by a normalization factor, this normalization factor be the square distance by this two reflective spot calculate or the square distance that equals this two reflective spot divided by 2;
Wherein, said angle parameter is: pass horizontal line and this pupil of the pupil center in this eyes image frame and folded angle between the line of a reflective spot wherein with one; Or with one pass the pupil center in this eyes image frame horizontal line; And this pupil and the line of a reflective spot wherein; The folded angle of two lines is first angle; This horizontal line and this pupil are second angle with the folded angle of line of another reflective spot wherein, and the mean value of first angle and this second angle is angle parameter; Or be the summit with the pupil in this eyes image frame, this pupil and this two reflective spot be the angle that forms of line respectively.
13. bearing calibration as claimed in claim 12 is characterized in that 1-n correction block is meant the array block by the i * j of this display screen five equilibrium in this display screen, so n=ij.
14. bearing calibration as claimed in claim 12; It is characterized in that capturing the step of this 1-n eyes image frame; Must cooperate the 1-n check point that makes this display screen or proofread and correct block to be highlighted in regular turn and use and remind this user to watch attentively, when this user watches the check point that is highlighted attentively or proofreaies and correct block, carry out the acquisition of eyes image frame.
15. a correction module that is used for pupil tracing is characterized in that this module comprises:
One highlights controller, makes the predefined 1-n check point of a display screen or proofread and correct block to be highlighted in regular turn and use and remind the user to watch attentively these check points or proofread and correct at least four four corners that are respectively adjacent to this display screen in the block;
One capture controller; Highlighting controller at this makes wherein a check point or proofreaies and correct block when being highlighted; One image capture unit is sent the controlling signal of pick-up image; This image capture unit acquisition obtains 1-n eyes image frame; One image analysing computer module of one processing unit calculates corresponding area parameters and angle parameter according to the pupil in this 1-n eyes image frame with the line of two reflective spots, by a graphics process module of this processing unit this area parameters and angle parameter is shown in seriatim and obtains the original coordinate of 1-n on the angle-area plane; And
One counter; Receive the distribution range of the original coordinate of this 1-n; This distribution range is approximate rectangular, and adopts affine conversion method to make these original coordinates be mapped to the coordinate of the correction block of this display screen, tries to achieve the coefficient in the coordinate system conversion formula of affine conversion method;
Wherein, said area parameters is: this pupil and the formed leg-of-mutton area value of this two reflective spots line; Or this pupil and the formed leg-of-mutton area value of this two reflective spots line be divided by a normalization factor, this normalization factor be the square distance by this two reflective spot calculate or the square distance that equals this two reflective spot divided by 2;
Wherein, said angle parameter is: pass horizontal line and this pupil of the pupil center in this eyes image frame and folded angle between the line of a reflective spot wherein with one; Or with one pass the pupil center in this eyes image frame horizontal line; And this pupil and the line of a reflective spot wherein; The folded angle of two lines is first angle; This horizontal line and this pupil are second angle with the folded angle of line of another reflective spot wherein, and the mean value of first angle and this second angle is angle parameter; Or be the summit with the pupil in this eyes image frame, this pupil and this two reflective spot be the angle that forms of line respectively.
16. correction module as claimed in claim 15 is characterized in that this 1-n proofreaies and correct block and is meant the array block by the i * j of this display screen five equilibrium, so n=ij.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN2009101372119A CN101872237B (en) | 2009-04-23 | 2009-04-23 | Method and system for pupil tracing as well as correction method and module for pupil tracing |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN2009101372119A CN101872237B (en) | 2009-04-23 | 2009-04-23 | Method and system for pupil tracing as well as correction method and module for pupil tracing |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN101872237A CN101872237A (en) | 2010-10-27 |
| CN101872237B true CN101872237B (en) | 2012-04-18 |
Family
ID=42997140
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN2009101372119A Active CN101872237B (en) | 2009-04-23 | 2009-04-23 | Method and system for pupil tracing as well as correction method and module for pupil tracing |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN101872237B (en) |
Families Citing this family (21)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| JP5689850B2 (en) * | 2012-05-25 | 2015-03-25 | 株式会社ソニー・コンピュータエンタテインメント | Video analysis apparatus, video analysis method, and gaze point display system |
| US9189095B2 (en) * | 2013-06-06 | 2015-11-17 | Microsoft Technology Licensing, Llc | Calibrating eye tracking system by touch input |
| CN103356163B (en) * | 2013-07-08 | 2016-03-30 | 东北电力大学 | Based on fixation point measuring device and the method thereof of video image and artificial neural network |
| CN103455147B (en) * | 2013-09-10 | 2016-08-31 | 惠州学院 | A kind of cursor control method |
| TWI498769B (en) * | 2013-11-18 | 2015-09-01 | Quanta Comp Inc | Head mounted display apparatus and login method thereof |
| US20150302585A1 (en) * | 2014-04-22 | 2015-10-22 | Lenovo (Singapore) Pte. Ltd. | Automatic gaze calibration |
| CN106659380B (en) * | 2014-06-12 | 2019-03-12 | Sr实验室有限公司 | Calibrate the device and method of eye tracker and the eyes control device including calibrator (-ter) unit |
| TWI524215B (en) * | 2014-10-15 | 2016-03-01 | 由田新技股份有限公司 | Network authentication method and system based on eye tracking |
| TWI617948B (en) * | 2015-07-24 | 2018-03-11 | 由田新技股份有限公司 | Module, method and computer readable medium for eye-tracking correction |
| CN105929954B (en) * | 2016-04-19 | 2019-10-18 | 京东方科技集团股份有限公司 | It is a kind of control cursor method and device, display equipment |
| DE102016210288A1 (en) * | 2016-06-10 | 2017-12-14 | Volkswagen Aktiengesellschaft | Eyetracker unit operating device and method for calibrating an eyetracker unit of an operating device |
| CN106339087B (en) * | 2016-08-29 | 2019-01-29 | 上海青研科技有限公司 | A kind of eyeball tracking method and device thereof based on multidimensional coordinate |
| CN107122046A (en) * | 2017-04-18 | 2017-09-01 | 湘潭大学 | A kind of wearable amyotrophic lateral sclerosis servicing unit and its implementation based on eye control technology |
| TWI646466B (en) * | 2017-08-09 | 2019-01-01 | 宏碁股份有限公司 | Vision range mapping method and related eyeball tracking device and system |
| CN107609516B (en) * | 2017-09-13 | 2019-10-08 | 重庆爱威视科技有限公司 | Adaptive eye movement method for tracing |
| US11690509B2 (en) * | 2017-12-13 | 2023-07-04 | Medical Diagnostech Pty Ltd. | System and method for obtaining a pupil response profile |
| TWI674518B (en) * | 2018-11-28 | 2019-10-11 | 國立臺灣大學 | Calibration method of eye-tracking and device thereof |
| US11615549B2 (en) | 2018-12-06 | 2023-03-28 | Htc Corporation | Image processing system and image processing method |
| CN111399633B (en) * | 2019-01-03 | 2023-03-31 | 见臻科技股份有限公司 | Correction method for eyeball tracking application |
| CN110568930B (en) * | 2019-09-10 | 2022-05-17 | Oppo广东移动通信有限公司 | Method for calibrating fixation point and related equipment |
| CN110780742B (en) * | 2019-10-31 | 2021-11-02 | Oppo广东移动通信有限公司 | Eyeball tracking processing method and related device |
Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US4836670A (en) * | 1987-08-19 | 1989-06-06 | Center For Innovative Technology | Eye movement detector |
| CN1889016A (en) * | 2006-07-25 | 2007-01-03 | 周辰 | Eye-to-computer cursor automatic positioning controlling method and system |
| CN101336089A (en) * | 2006-01-26 | 2008-12-31 | 诺基亚公司 | Eye tracker equipment |
-
2009
- 2009-04-23 CN CN2009101372119A patent/CN101872237B/en active Active
Patent Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US4836670A (en) * | 1987-08-19 | 1989-06-06 | Center For Innovative Technology | Eye movement detector |
| CN101336089A (en) * | 2006-01-26 | 2008-12-31 | 诺基亚公司 | Eye tracker equipment |
| CN1889016A (en) * | 2006-07-25 | 2007-01-03 | 周辰 | Eye-to-computer cursor automatic positioning controlling method and system |
Also Published As
| Publication number | Publication date |
|---|---|
| CN101872237A (en) | 2010-10-27 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN101872237B (en) | Method and system for pupil tracing as well as correction method and module for pupil tracing | |
| TWI398796B (en) | Pupil tracking methods and systems, and correction methods and correction modules for pupil tracking | |
| CN107609516B (en) | Adaptive eye movement method for tracing | |
| CN105138965B (en) | A kind of near-to-eye sight tracing and its system | |
| CN103210421B (en) | Article detection device and object detecting method | |
| CN104978012B (en) | One kind points to exchange method, apparatus and system | |
| CN102509303B (en) | Binarization image registration method based on improved structural similarity | |
| CN106650682B (en) | Face tracking method and device | |
| US11715231B2 (en) | Head pose estimation from local eye region | |
| US20170374331A1 (en) | Auto keystone correction and auto focus adjustment | |
| CN107831900B (en) | human-computer interaction method and system of eye-controlled mouse | |
| CN105957041B (en) | A kind of wide-angle lens infrared image distortion correction method | |
| CN102473282B (en) | External light glare assessment device, line of sight detection device and external light glare assessment method | |
| CN103217108B (en) | A kind of spectacle frame geometric parameter detection method | |
| CN104133548A (en) | Method and device for determining viewpoint area and controlling screen luminance | |
| CN103761519A (en) | Non-contact sight-line tracking method based on self-adaptive calibration | |
| CN103886107A (en) | Robot locating and map building system based on ceiling image information | |
| JPH0798214A (en) | Method and device for three dimensional position and attitude recognition method based on sense of sight | |
| CN103870814A (en) | Non-contact real-time eye movement identification method based on intelligent camera | |
| US20200271447A1 (en) | Stereo Camera | |
| US11080822B2 (en) | Method, system and recording medium for building environment map | |
| US9741171B2 (en) | Image processing device and image processing method | |
| CN112651279A (en) | 3D face recognition method and system based on short-distance application | |
| JP2017173455A (en) | Information processing device, information processing method, and program | |
| WO2023103145A1 (en) | Head pose truth value acquisition method, apparatus and device, and storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| C06 | Publication | ||
| PB01 | Publication | ||
| C10 | Entry into substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| C14 | Grant of patent or utility model | ||
| GR01 | Patent grant |