CN114136209A - Eyeball position positioning circuit, method, substrate and virtual reality wearable device - Google Patents
Eyeball position positioning circuit, method, substrate and virtual reality wearable device Download PDFInfo
- Publication number
- CN114136209A CN114136209A CN202111420155.7A CN202111420155A CN114136209A CN 114136209 A CN114136209 A CN 114136209A CN 202111420155 A CN202111420155 A CN 202111420155A CN 114136209 A CN114136209 A CN 114136209A
- Authority
- CN
- China
- Prior art keywords
- module
- comparator
- photosensitive
- jth
- mos tube
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
- 210000005252 bulbus oculi Anatomy 0.000 title claims abstract description 36
- 238000000034 method Methods 0.000 title claims abstract description 27
- 239000000758 substrate Substances 0.000 title claims abstract description 12
- 210000001508 eye Anatomy 0.000 claims description 29
- 230000001502 supplementing effect Effects 0.000 claims description 6
- 238000012546 transfer Methods 0.000 claims description 6
- 238000010586 diagram Methods 0.000 description 13
- 210000001747 pupil Anatomy 0.000 description 8
- 238000013461 design Methods 0.000 description 5
- 238000007781 pre-processing Methods 0.000 description 5
- 230000008569 process Effects 0.000 description 5
- 230000009471 action Effects 0.000 description 4
- 238000005070 sampling Methods 0.000 description 4
- 230000002093 peripheral effect Effects 0.000 description 3
- 238000012545 processing Methods 0.000 description 3
- 238000003491 array Methods 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 238000006243 chemical reaction Methods 0.000 description 1
- 238000005286 illumination Methods 0.000 description 1
- 230000006872 improvement Effects 0.000 description 1
- 230000001678 irradiating effect Effects 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
- 239000002699 waste material Substances 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G01—MEASURING; TESTING
- G01B—MEASURING LENGTH, THICKNESS OR SIMILAR LINEAR DIMENSIONS; MEASURING ANGLES; MEASURING AREAS; MEASURING IRREGULARITIES OF SURFACES OR CONTOURS
- G01B11/00—Measuring arrangements characterised by the use of optical techniques
- G01B11/002—Measuring arrangements characterised by the use of optical techniques for measuring two or more coordinates
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
- G06F3/011—Arrangements for interaction with the human body, e.g. for user immersion in virtual reality
- G06F3/013—Eye tracking input arrangements
Landscapes
- Engineering & Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Human Computer Interaction (AREA)
- Photometry And Measurement Of Optical Pulse Characteristics (AREA)
- Length Measuring Devices By Optical Means (AREA)
Abstract
The embodiment of the invention provides an eyeball position positioning circuit, a eyeball position positioning method, a substrate and virtual reality wearing equipment, wherein the eyeball position positioning circuit comprises a plurality of photosensitive modules, a plurality of comparator modules and a plurality of switch modules; each comparator module is connected with two adjacent photosensitive modules, and each switch module is connected with two adjacent comparator modules; the photosensitive module is used for collecting the light intensity at the designated position, converting the light intensity into an electric signal and outputting the electric signal to the comparator module connected with the photosensitive module; the comparator module is used for comparing the magnitude of each electric signal received by the comparator module and outputting a corresponding level signal to the switch module connected with the comparator module according to the comparison result; and the switch module is used for outputting the position information of the specified position of the light sensing module with the minimum light intensity according to the level signal received by the switch module. The scheme of this application need not the camera and carries out image acquisition, has improved the report rate, has reduced the consumption, can be quick carry out the location of eyeball position.
Description
Technical Field
The invention relates to the technical field of intelligent wearable equipment, in particular to a circuit and a method for locating eyeball positions, a substrate and virtual reality wearable equipment.
Background
At present, in a VR (Virtual Reality) product, an eyeball position positioning scheme mainly performs image sampling through a camera chip, emits infrared light through an infrared emission tube, reflects the infrared light after irradiating human eyes, and acquires an image through a camera module, as shown in fig. 1. Analog signals acquired by multiple channels of the camera are converted into digital matrixes through an analog-to-digital converter (ADC) to acquire image information, an image preprocessing process is completed, then a Micro Controller Unit (MCU) performs digital positioning algorithm processing through pupil identification, an eyeball position coordinate is calculated, and the eyeball position coordinate information is reported, wherein the specific flow is as shown in fig. 2.
Because the sampling channels of the cameras are multiple, the sampling period is long, and the processing time of image preprocessing and position identification algorithms is long, the point reporting rate is low, and the existing conventional product has the point reporting rate of 60 HZ. The eyeball position location only needs to locate the position coordinates of the pupil, the pupil area needing to be identified is small, the pupil shape is regular, the whole image is collected by the camera, a lot of useless data can be collected, and the resource waste can be caused by preprocessing and calculating the whole image. When a single position needs to be identified and reported, only the difference between the pupil point and the peripheral point needs to be identified, high-precision ADC conversion is not needed, and meanwhile, the circuit area and the power consumption of the ADC circuit can be increased.
Therefore, how to provide an eyeball position positioning circuit with higher report rate, smaller circuit area, lower power consumption and faster operation method is an important problem to be solved.
Disclosure of Invention
An embodiment of the invention provides an eyeball position positioning circuit, an eyeball position positioning method, a substrate and a virtual reality wearing device, so as to solve at least one of the problems. The specific technical scheme is as follows:
in a first aspect, an embodiment of the present application provides an eyeball position locating circuit, including:
the device comprises a plurality of photosensitive modules, a plurality of comparator modules and a plurality of switch modules; each comparator module is connected with two adjacent photosensitive modules, and each switch module is connected with two adjacent comparator modules;
the photosensitive module is used for collecting the light intensity at the designated position, converting the light intensity into an electric signal and outputting the electric signal to the comparator module connected with the photosensitive module;
the comparator module is used for comparing the magnitude of each electric signal received by the comparator module and outputting a corresponding level signal to the switch module connected with the comparator module according to the comparison result;
and the switch module is used for outputting the position information of the specified position of the light sensing module with the minimum light intensity according to the level signal received by the switch module.
In one possible embodiment, the plurality of photosensitive modules includes: the 1 st photosensitive module, the 2 nd photosensitive module, … … and the Nth photosensitive module; the plurality of comparator modules includes: a 1 st comparator module, a 2 nd comparator module, … …, an N-1 th comparator module; the plurality of switch modules comprise a 1 st switch module, a 2 nd switch module, … … and an N-2 nd switch module, wherein N is an integer greater than 3;
the ith comparator module is connected with the ith photosensitive module and the (i + 1) th photosensitive module; the jth switch module is connected with the jth comparator module and the jth +1 comparator module; the ith photosensitive module is adjacent to the (i + 1) th photosensitive module, the jth comparator module is adjacent to the (j + 1) th comparator module, and i is {1,2, … …, N-1}, and j is {1,2, … …, N-2 }.
In a possible implementation manner, the ith comparator module is specifically configured to compare magnitudes of the electrical signal of the ith photosensitive module and the electrical signal of the (i + 1) th photosensitive module, and output a first level signal when the electrical signal of the ith photosensitive module is not less than the electrical signal of the (i + 1) th photosensitive module; under the condition that the electrical signal of the ith photosensitive module is smaller than the electrical signal of the (i + 1) th photosensitive module, outputting a second level signal, wherein the polarity of the first level signal is opposite to that of the second level signal;
the jth switch module comprises a jth NOT gate submodule, a jth double-switch control submodule and a jth storage submodule; the jth storage submodule stores position information of a specified position of a jth + 1-th photosensitive module;
after the polarity of the level signal output by the jth comparator module is changed by the jth NOT gate submodule, the level signal is transmitted to a first switch input end of the jth double-switch control submodule; the level signal output by the jth comparator module is transmitted to a second switch input end of the jth double-switch control submodule; and under the condition that the first switch input end and the second switch input end of the jth double-switch control submodule input second level signals, the jth double-switch control submodule outputs the position information stored in the jth storage submodule.
In one possible implementation, the circuit further includes: and the position coordinate determination module is used for acquiring the position information and outputting the position coordinate according to the corresponding relation between the position information and the position coordinate.
In a possible implementation manner, the photosensitive module includes an outer frame with a slit in the middle and an infrared sensor, where the infrared sensor includes a photodiode, and the photodiode is disposed in the slit of the outer frame.
In one possible embodiment, the infrared sensor further includes:
the MOS transistor comprises a first MOS transistor, a second MOS transistor, a third MOS transistor and a fourth MOS transistor;
the grid electrode of the first MOS tube is connected with a reset voltage end, and the first end of the first MOS tube is connected with a first preset voltage end;
the second end of the first MOS tube is respectively connected with the first end of the second MOS tube and the grid electrode of the third MOS tube;
the grid electrode of the second MOS tube is connected with a first driving signal end, the second end of the second MOS tube is connected with the second end of the photodiode, and the first end of the photodiode is connected with a second preset voltage end;
the first end of the third MOS tube is connected with the first preset voltage end, and the second end of the third MOS tube is connected with the second end of the fourth MOS tube;
the grid electrode of the fourth MOS tube is connected with the second driving signal end, and the first end of the fourth MOS tube is used for outputting the collected light intensity signal of the infrared light.
In a possible implementation manner, in a reset phase, the reset voltage terminal, the second driving signal terminal receive a high-level signal, the first driving signal terminal receives a low-level signal, the first MOS transistor, the fourth MOS transistor are turned on, the second MOS transistor is turned off, and a gate voltage of the third MOS transistor is a voltage of the first preset voltage terminal;
in an exposure stage, the reset voltage end, the first driving signal end receive a low level signal, the second driving signal end receives a high level signal, the first MOS tube, the second MOS tube and the fourth MOS tube are closed, and the photodiode collects a light intensity signal of infrared light and stores charges;
in a transfer stage, the reset voltage end receives a low level signal, the first driving signal end and the second driving signal end receive a high level signal, the first MOS tube is closed, the second MOS tube and the fourth MOS tube are opened, and the photodiode transfers the stored charges to the third MOS tube;
in the output stage, the reset voltage end, the first driving signal end receive low level signals, the second driving signal end receive high level signals, the first MOS tube and the second MOS tube are closed, the fourth MOS tube is opened, the stored charges are output through the fourth MOS tube, and then light intensity signals of infrared light collected by the photodiode are output.
In one possible implementation, the circuit further includes: and the infrared light supplementing module is used for transmitting infrared light.
In a second aspect, an embodiment of the present application provides an eyeball position locating method, which is applied to the circuit described in any one of the above first aspects, and the method includes:
acquiring intensity signals of human eye reflected light at each designated position;
comparing the intensity signals of the reflected light of the human eyes to obtain a comparison result;
and outputting the position information at the designated position with the minimum intensity signal according to the comparison result.
In a third aspect, an application embodiment provides an eyeball position positioning substrate, comprising:
the eye position positioning circuit of any one of the first aspect.
In a fourth aspect, an application embodiment provides a virtual reality wearable device, including: the eyeball position positioning substrate according to the third aspect.
The embodiment of the invention has the following beneficial effects:
the eyeball position positioning circuit comprises a plurality of photosensitive modules, a plurality of comparator modules and a plurality of switch modules; each comparator module is connected with two adjacent photosensitive modules, and each switch module is connected with two adjacent comparator modules; the photosensitive module is used for collecting the light intensity at the designated position, converting the light intensity into an electric signal and outputting the electric signal to the comparator module connected with the photosensitive module; the comparator module is used for comparing the magnitude of each electric signal received by the comparator module and outputting a corresponding level signal to the switch module connected with the comparator module according to the comparison result; and the switch module is used for outputting the position information of the specified position of the light sensing module with the minimum light intensity according to the level signal received by the switch module. Compared with the prior art, the scheme of the application does not need a camera to acquire images, reduces the time for image acquisition and preprocessing, improves the report rate, does not need an ADC (analog to digital converter) to convert analog signals, can identify the designated position of the light intensity minimum light sensing module only through the comparator, reduces the circuit area and reduces the power consumption, does not need an MCU (micro control unit) to calculate the position information through an image positioning algorithm, outputs the position information of the designated position of the light intensity minimum light sensing module through the switch module, and can quickly position the eyeball position.
Of course, not all of the advantages described above need to be achieved at the same time in the practice of any one product or method of the invention.
Drawings
In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the embodiments or the prior art will be briefly described below, it is obvious that the drawings in the following description are only some embodiments of the present invention, and it is obvious for those skilled in the art that other embodiments can be obtained by referring to these drawings.
FIG. 1 is a first schematic diagram of an eyeball position location scheme in the related art;
FIG. 2 is a second schematic diagram of an eyeball position positioning scheme in the related art;
FIG. 3 is a first schematic diagram of an eye position circuit according to an embodiment of the present disclosure;
FIG. 4 is a second schematic diagram of an eye position circuit according to an embodiment of the present disclosure;
FIG. 5 is a third schematic diagram of an eye position circuit according to an embodiment of the present disclosure;
FIG. 6 is a fourth schematic diagram of an eye position circuit according to an embodiment of the present disclosure;
FIG. 7 is a fifth schematic diagram of an eye position circuit according to an embodiment of the present disclosure;
FIG. 8 is a sixth schematic diagram of an eye position circuit according to an embodiment of the present application;
FIG. 9 is a seventh schematic diagram of an eye position circuit according to an embodiment of the present disclosure;
FIG. 10a is a diagram illustrating an eighth exemplary embodiment of an eye position circuit;
FIG. 10b is a ninth exemplary diagram of an eye position circuit according to an embodiment of the present disclosure;
FIG. 11 is a tenth exemplary schematic diagram of an eye position circuit according to an embodiment of the present disclosure;
FIG. 12 is an eleventh exemplary schematic diagram of an eye position circuit according to an embodiment of the disclosure;
fig. 13 is a schematic view illustrating an eyeball position locating method according to an embodiment of the present application.
Detailed Description
The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments. All other embodiments, which can be derived from the embodiments given herein by one of ordinary skill in the art, are within the scope of the invention.
In order to provide an eyeball position positioning scheme with a higher report rate, a smaller circuit area, lower power consumption and a faster operation method, an embodiment of the present application provides an eyeball position positioning circuit, see fig. 3, which includes:
a plurality of photosensitive modules 31, a plurality of comparator modules 32, a plurality of switch modules 33; each comparator module is connected with two adjacent photosensitive modules, and each switch module is connected with two adjacent comparator modules;
the photosensitive module 31 is used for collecting the light intensity at the designated position, converting the light intensity into an electric signal and outputting the electric signal to a comparator module connected with the photosensitive module;
the comparator module 32 is configured to compare the magnitude of each received electrical signal, and output a corresponding level signal to the switch module connected to the comparator module according to the comparison result;
the switch module 33 is configured to output the position information of the designated position of the light sensing module with the minimum light intensity according to the received level signal.
The number of the photosensitive modules 31, the comparator modules 32 and the switch modules 33 is not limited, and the specific number of the photosensitive modules, the comparator modules and the switch modules is determined according to the actual design requirement. Each comparator module is connected with two adjacent photosensitive modules, so as to compare light intensity signals collected by each two adjacent photosensitive modules. Each switch module is connected with two adjacent comparator modules, and is used for judging the received electric signals of the adjacent comparators and outputting corresponding position information according to the judgment result.
The photosensitive module is placed on the periphery of a display area in a VR product and used for collecting infrared light reflected by human eyes, and electric signals with different amplitudes can be output according to the intensity of the received infrared light, for example, when the photosensitive module does not receive the infrared light, the output current is less than 0.1uA, and when the infrared light intensity received by the photosensitive module is 600lux (lux of illumination), the output current is 20 uA.
The plurality of photosensitive modules are placed at different positions according to actual design, and each photosensitive module is used for indicating position information of the position.
The comparator module can adopt an operational amplifier as a comparator, and a forward input end and a reverse input end of the comparator module are respectively connected with two adjacent photosensitive modules at positions and used for comparing the magnitude of the electric signals received by the forward input end and the reverse input end.
Since the pupil of the human eye is black, the intensity of the infrared light reflected by the human eye is the smallest as can be seen from fig. 4. In the process of rotating human eyes, the light sensing module captures infrared light reflected by pupils, the light sensing module with the minimum output light intensity is compared through the plurality of comparator modules, and the switch module outputs position information of the designated position of the light sensing module with the minimum light intensity at the moment.
Compared with the method for receiving infrared light by using the camera in the prior art, the method has the advantages that the number of the photosensitive modules is reduced, the array arrangement mode of whole rows and lines is not adopted, the image acquisition by the camera is not needed, the time for image acquisition and preprocessing is reduced, the report rate is improved, the analog signals are not needed to be converted by the ADC, the appointed position of the photosensitive module with the minimum light intensity can be identified only by the comparator, the circuit area is reduced, the power consumption is reduced, the MCU is not needed to calculate the position information by an image positioning algorithm, the position information of the appointed position of the photosensitive module with the minimum light intensity is output by the switch module, and the eyeball position can be quickly positioned.
In one possible embodiment, the plurality of photosensitive modules includes: the 1 st photosensitive module, the 2 nd photosensitive module, … … and the Nth photosensitive module; the plurality of comparator modules includes: a 1 st comparator module, a 2 nd comparator module, … …, an N-1 th comparator module; the plurality of switch modules comprise a 1 st switch module, a 2 nd switch module, … … and an N-2 nd switch module, wherein N is an integer greater than 3;
the ith comparator module is connected with the ith photosensitive module and the (i + 1) th photosensitive module; the jth switch module is connected with the jth comparator module and the jth +1 comparator module; the ith photosensitive module is adjacent to the (i + 1) th photosensitive module, the jth comparator module is adjacent to the (j + 1) th comparator module, and i is {1,2, … …, N-1}, and j is {1,2, … …, N-2 }.
In one example, referring to fig. 5, the 1 st comparator module is configured to compare infrared intensity signals collected by the 1 st photosensitive module and the 2 nd photosensitive module, the 2 nd comparator module is configured to compare infrared intensity signals collected by the 2 nd photosensitive module and the 3 rd photosensitive module in sequence, the N-1 st comparator module is configured to compare infrared intensity signals collected by the N-1 st photosensitive module and the N rd photosensitive module, the photosensitive module with the minimum light intensity among the plurality of photosensitive modules can be selected through one-to-one comparison, and then the comparator outputs position information corresponding to the designated position of the photosensitive module.
In the embodiment of the application, the comparator module is connected with the adjacent photosensitive modules, and the switch module is connected with the adjacent comparators, so that the position information of the specified position of the photosensitive module with the minimum light intensity is determined, the conventional ADC sampling design is not needed, and the circuit area and the power consumption are reduced.
In a possible embodiment, the ith comparator module is specifically configured to compare magnitudes of the electrical signal of the ith photosensitive module and the electrical signal of the (i + 1) th photosensitive module, and output a first level signal when the electrical signal of the ith photosensitive module is not less than the electrical signal of the (i + 1) th photosensitive module; under the condition that the electrical signal of the ith photosensitive module is smaller than the electrical signal of the (i + 1) th photosensitive module, outputting a second level signal, wherein the polarity of the first level signal is opposite to that of the second level signal;
the jth switch module comprises a jth NOT gate submodule, a jth double-switch control submodule and a jth storage submodule; the jth storage submodule stores position information of a specified position of a jth + 1-th photosensitive module;
after the polarity of the level signal output by the jth comparator module is changed by the jth NOT gate submodule, the level signal is transmitted to a first switch input end of the jth double-switch control submodule; the level signal output by the jth comparator module is transmitted to a second switch input end of the jth double-switch control submodule; and under the condition that the first switch input end and the second switch input end of the jth double-switch control submodule input second level signals, the jth double-switch control submodule outputs the position information stored in the jth storage submodule.
A plurality of adjacent photosensitive modules simultaneously collect infrared light intensity signals reflected by human eyes, the adjacent photosensitive modules output the collected signals to two input ends of the same comparator, and the comparator module outputs a first level signal or a second level signal after comparing the magnitude of the two signals.
The first level signal and the second level signal are logic level signals 0 or 1, and the first level signal and the second level signal have opposite polarities, namely, when the first level signal is 1 or 0, the second level signal is 0 or 1. In one example, referring to fig. 6, in the case of a > B < C, the comparator module outputs 0 when the comparator module inputs a and B and 1 when the comparator module inputs B and C.
Referring to fig. 7, the switch module includes a not-gate sub-module, a dual-switch control sub-module, and a storage sub-module. The not-gate submodule is configured to perform polarity inversion on a logic level output by the comparator, for example, a logic level signal output by the comparator module is 1, and after passing through the not-gate submodule, the output logic level signal becomes 0. The storage submodule is used for storing the position information at the designated position after the digital circuit is coded.
And determining a logic operation unit of the rear-stage switch module according to the connection mode of the photosensitive modules and the comparator modules. In one example, referring to fig. 8, in a case where a > B < C, when a and B are input to the comparator module, the comparator module compares the size of a-B and outputs 0, the comparator module changes to 1 after passing through the nor sub-module, and transmits the 1 to the first switch input end of the dual-switch control sub-module, and when B and C are input to the comparator module, the comparator module compares the size of B-C and outputs 1, and transmits the 1 to the second switch input end of the dual-switch control sub-module, and when the dual-switch control sub-module determines that both the two input ends are 1, the B position information is output.
In the embodiment of the present application, the logic operation unit of the switch module behind two adjacent comparator modules corresponds to unique position information, and only after the comparator module compares the photosensitive module with the minimum light intensity, the switch module outputs the position information corresponding to the photosensitive module with the minimum light intensity. The position information can be determined without a complex positioning algorithm carried out by the MCU. The MCU removing design is simpler and faster.
In one possible embodiment, the circuit further comprises: and the position coordinate determination module is used for acquiring the position information and outputting the position coordinate according to the corresponding relation between the position information and the position coordinate.
And determining the corresponding relation between the position information and the position coordinates through a digital circuit, and storing the corresponding relation in a position coordinate determination module.
In one example, referring to fig. 9, the position coordinate module stores a corresponding relationship between the position information and the position coordinates, and outputs the corresponding position coordinates according to the position coordinate information output by the switch module.
In the embodiment of the application, the method for determining the position coordinates through the corresponding relation between the position information and the position coordinates does not need complex digital positioning algorithm for processing, and the operation speed is higher compared with the prior art.
In a possible embodiment, the photosensitive module includes an outer frame with a slit in the middle and an infrared sensor, the infrared sensor includes a photodiode, and the photodiode is disposed in the slit of the outer frame.
A photodiode is a photoelectric sensing device that converts an optical signal into an electrical signal. The photodiode operates under the action of reverse voltage, when no light is applied, the reverse current is very weak, called dark current, and when light is applied, the reverse current is rapidly increased to dozens of microamperes, called photocurrent. The greater the intensity of the light, the greater the reverse current.
The outer frame with the slit in the middle is used for reducing the influence of adjacent photodiodes on infrared ray intensity receiving. The photosensitive modules have different arrays, in one example, see fig. 10(a) and 10(b), where D1 and D2 are adjacent photodiodes, which are evaluated at a minimum collimation angle, such that the photodiodes are disposed within the slots of the housing and cover the maximum area of the slots, ensuring better reception of the infrared light signal. It is understood that the examples of the present application show only two array patterns, but are not limited to these two array patterns.
In this application embodiment, be provided with the frame of slit through the centre, can reduce adjacent photodiode to the influence that infrared light intensity received, improve the precision that infrared light detected, and then promote eyeball position location's accuracy.
In a possible embodiment, referring to fig. 11, the infrared sensor further comprises:
a first MOS transistor T1, a second MOS transistor T2, a third MOS transistor T3 and a fourth MOS transistor T4;
the gate of the first MOS transistor T1 is connected to a reset voltage terminal RST, and the first end of the first MOS transistor T1 is connected to a first preset voltage terminal VDD;
the second end of the first MOS transistor T1 is connected to the first end of the second MOS transistor T2 and the gate of the third MOS transistor T3, respectively;
the gate of the second MOS transistor T2 is connected to a first driving signal terminal TG, the second terminal of the second MOS transistor T2 is connected to the second terminal of the photodiode, and the first terminal of the photodiode is connected to a second preset voltage terminal VSS;
a first end of the third MOS transistor T3 is connected to the first preset voltage terminal VDD, and a second end of the third MOS transistor T3 is connected to a second end of the fourth MOS transistor T4;
the gate of the fourth MOS transistor T4 is connected to a second driving signal terminal SEL, and the first terminal of the fourth MOS transistor T4 is configured to output a light intensity signal of the collected infrared light.
The first preset voltage and the second preset voltage are both voltage values which can be set according to actual needs. For example, the first preset voltage VDD may be set to 5V or 3V, etc., and the second preset voltage VDD may be set to 0V.
Any MOS tube in the infrared sensor can be an N-type MOS tube or a P-type MOS tube, and can be selected according to actual conditions; the first end of the MOS tube is a source electrode or a drain electrode, and the second end of the MOS tube is a drain electrode or a source electrode corresponding to the first end.
In the embodiment of the application, the photocurrent of the light emitting diode can be amplified by the connection mode of the four MOS tubes so as to improve the signal intensity of the collected infrared light.
In a possible embodiment, referring to fig. 11 and 12, in the reset phase T1, the reset voltage terminal RST, the second driving signal terminal SEL receive a high level signal, the first driving signal terminal TG receive a low level signal, the first MOS transistor T1, the fourth MOS transistor T4 are turned on, the second MOS transistor T2 is turned off, and the gate voltage of the third MOS transistor T3 is the voltage of the first preset voltage terminal;
in an exposure stage T2, the reset voltage terminal RST, the first driving signal terminal TG receive a low level signal, the second driving signal terminal SEL receives a high level signal, the first MOS transistor T1, the second MOS transistor T2 are turned off, the fourth MOS transistor T4 is turned on, and the photodiode collects a light intensity signal of infrared light and stores charges;
in a transfer stage, the reset voltage terminal RST receives a low level signal, the first driving signal terminal TG and the second driving signal terminal SEL receive a high level signal, the first MOS transistor T1 is turned off, the second MOS transistor T2 and the fourth MOS transistor T4 are turned on, and the photodiode transfers the stored charges to the third MOS transistor T3;
in the output stage, the reset voltage end RST, the first driving signal end TG receive a low level signal, the second driving signal end SEL receives a high level signal, the first MOS transistor T1 and the second MOS transistor T2 are turned off, the fourth MOS transistor T4 is turned on, the stored charges are output through the fourth MOS transistor T4, and then the light intensity signal of the infrared light collected by the photodiode is output.
The high level signal is a voltage signal provided for turning on the MOS transistor, and a specific voltage value is determined by a selected MOS transistor type, for example, the high level signal may be 1.2V or 1.8V. The low level signal is used to turn off the voltage signal provided by the MOS transistor, which is usually 0V.
In the embodiment of the application, each MOS transistor receives the high-low level timing sequence, and is used for outputting the signal received by the photodiode.
In one possible embodiment, the circuit further comprises: and the infrared light supplementing module is used for transmitting infrared light.
The infrared light supplementing module is used for emitting infrared light, and the number and the placement position of the infrared light supplementing module are determined according to actual design requirements.
In the embodiment of the application, the infrared light supplementing module emits infrared light, and the light sensing module receives infrared light signals, so that signal sources are provided for determining the positions of eyeballs.
The embodiment of the present application further provides an eyeball position locating method, see fig. 13, which is applied to the above circuit, and the method includes:
s11, acquiring intensity signals of human eye reflected light at each designated position;
s12, comparing the intensity signals of the reflected light of the human eyes to obtain a comparison result;
and S13, outputting the position information of the designated position with the minimum intensity signal according to the comparison result.
In one example, the above-mentioned eyeball position locating method is applied to the circuit shown in fig. 9, the 1 st photo sensing module, the 2 nd photo sensing module, … …, and the N th photo sensing module collect infrared light intensity signals reflected by human eyes at each designated position, and output the collected intensity signals to the 1 st comparator module, the 2 nd comparator module, … …, and the N-1 th comparator module, wherein the i th comparator module is connected to the i th photo sensing module and the i +1 th photo sensing module, the i th photo sensing module is adjacent to the i +1 th photo sensing module, i is {1,2, … …, N-1}, the intensity signals output by the plurality of photo sensing modules are compared one by the plurality of comparator modules, and output the comparison result to the 1 st switch module, the 2 nd switch module, … …, and the N-2 th switch module, wherein the j th switch module is connected to the j th comparator module and the j +1 th comparator module, the j comparator module is adjacent to the j +1 comparator module, j is {1,2, … …, N-2}, the light sensing module with the minimum light intensity is selected through comparison, the switch module outputs corresponding position information according to the position information stored in the sub-module, and the position coordinate determination module outputs the position coordinate through the corresponding relation between the position information and the position coordinate.
In the embodiment of the application, the position of the photosensitive module corresponding to the pupil in the human eye is identified through the comparator, so that the point reporting rate can be improved to more than 1000HZ by the method for outputting the corresponding position information.
The embodiment of the present application further provides an eyeball position locating substrate, including: the eyeball position positioning circuit.
In one example, the eyeball position positioning substrate further includes a peripheral circuit, and the peripheral circuit is used for providing voltages at the ends, such as VDD, VSS, RST, TG, SEL, and the like.
The embodiment of the application further provides a virtual reality wearing device, including: the eyeball position positioning substrate.
It is noted that, herein, relational terms such as first and second, and the like may be used solely to distinguish one entity or action from another entity or action without necessarily requiring or implying any actual such relationship or order between such entities or actions. Also, the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising an … …" does not exclude the presence of other identical elements in a process, method, article, or apparatus that comprises the element.
The embodiments in the present specification are described in a related manner, each embodiment focuses on differences from other embodiments, and the same and similar parts in the embodiments are referred to each other.
The above description is only for the preferred embodiment of the present invention, and is not intended to limit the scope of the present invention. Any modification, equivalent replacement, or improvement made within the spirit and principle of the present invention shall fall within the protection scope of the present invention.
Claims (11)
1. An eye position locator circuit, the circuit comprising:
the device comprises a plurality of photosensitive modules, a plurality of comparator modules and a plurality of switch modules; each comparator module is connected with two adjacent photosensitive modules, and each switch module is connected with two adjacent comparator modules;
the photosensitive module is used for collecting the light intensity at the designated position, converting the light intensity into an electric signal and outputting the electric signal to the comparator module connected with the photosensitive module;
the comparator module is used for comparing the magnitude of each electric signal received by the comparator module and outputting a corresponding level signal to the switch module connected with the comparator module according to the comparison result;
and the switch module is used for outputting the position information of the specified position of the light sensing module with the minimum light intensity according to the level signal received by the switch module.
2. The circuit of claim 1, wherein the plurality of photosensitive modules comprises: the 1 st photosensitive module, the 2 nd photosensitive module, … … and the Nth photosensitive module; the plurality of comparator modules includes: a 1 st comparator module, a 2 nd comparator module, … …, an N-1 th comparator module; the plurality of switch modules comprise a 1 st switch module, a 2 nd switch module, … … and an N-2 nd switch module, wherein N is an integer greater than 3;
the ith comparator module is connected with the ith photosensitive module and the (i + 1) th photosensitive module; the jth switch module is connected with the jth comparator module and the jth +1 comparator module; the ith photosensitive module is adjacent to the (i + 1) th photosensitive module, the jth comparator module is adjacent to the (j + 1) th comparator module, and i is {1,2, … …, N-1}, and j is {1,2, … …, N-2 }.
3. The circuit according to claim 2, wherein the ith comparator module is specifically configured to compare magnitudes of the electrical signal of the ith photo-sensing module and the electrical signal of the (i + 1) th photo-sensing module, and output a first level signal when the electrical signal of the ith photo-sensing module is not less than the electrical signal of the (i + 1) th photo-sensing module; under the condition that the electrical signal of the ith photosensitive module is smaller than the electrical signal of the (i + 1) th photosensitive module, outputting a second level signal, wherein the polarity of the first level signal is opposite to that of the second level signal;
the jth switch module comprises a jth NOT gate submodule, a jth double-switch control submodule and a jth storage submodule; the jth storage submodule stores position information of a specified position of a jth + 1-th photosensitive module;
after the polarity of the level signal output by the jth comparator module is changed by the jth NOT gate submodule, the level signal is transmitted to a first switch input end of the jth double-switch control submodule; the level signal output by the jth comparator module is transmitted to a second switch input end of the jth double-switch control submodule; and under the condition that the first switch input end and the second switch input end of the jth double-switch control submodule input second level signals, the jth double-switch control submodule outputs the position information stored in the jth storage submodule.
4. The circuit of claim 1, further comprising: and the position coordinate determination module is used for acquiring the position information and outputting the position coordinate according to the corresponding relation between the position information and the position coordinate.
5. The circuit of claim 1, wherein the photosensitive module comprises a frame with a slit in the middle and an infrared sensor, the infrared sensor comprises a photodiode, and the photodiode is disposed in the slit of the frame.
6. The circuit of claim 5, wherein the infrared sensor further comprises:
the MOS transistor comprises a first MOS transistor, a second MOS transistor, a third MOS transistor and a fourth MOS transistor;
the grid electrode of the first MOS tube is connected with a reset voltage end, and the first end of the first MOS tube is connected with a first preset voltage end;
the second end of the first MOS tube is respectively connected with the first end of the second MOS tube and the grid electrode of the third MOS tube;
the grid electrode of the second MOS tube is connected with a first driving signal end, the second end of the second MOS tube is connected with the second end of the photodiode, and the first end of the photodiode is connected with a second preset voltage end;
the first end of the third MOS tube is connected with the first preset voltage end, and the second end of the third MOS tube is connected with the second end of the fourth MOS tube;
the grid electrode of the fourth MOS tube is connected with the second driving signal end, and the first end of the fourth MOS tube is used for outputting the collected light intensity signal of the infrared light.
7. The circuit of claim 6,
in a reset stage, the reset voltage end, the second driving signal end receive a high level signal, the first driving signal end receives a low level signal, the first MOS transistor, the fourth MOS transistor are turned on, the second MOS transistor is turned off, and the gate voltage of the third MOS transistor is the voltage of the first preset voltage end;
in an exposure stage, the reset voltage end, the first driving signal end receive a low level signal, the second driving signal end receives a high level signal, the first MOS tube, the second MOS tube and the fourth MOS tube are closed, and the photodiode collects a light intensity signal of infrared light and stores charges;
in a transfer stage, the reset voltage end receives a low level signal, the first driving signal end and the second driving signal end receive a high level signal, the first MOS tube is closed, the second MOS tube and the fourth MOS tube are opened, and the photodiode transfers the stored charges to the third MOS tube;
in the output stage, the reset voltage end, the first driving signal end receive low level signals, the second driving signal end receive high level signals, the first MOS tube and the second MOS tube are closed, the fourth MOS tube is opened, the stored charges are output through the fourth MOS tube, and then light intensity signals of infrared light collected by the photodiode are output.
8. The circuit of claim 1, further comprising: and the infrared light supplementing module is used for transmitting infrared light.
9. An eye position locating method, applied to the circuit of any one of claims 1 to 8, comprising:
acquiring intensity signals of human eye reflected light at each designated position;
comparing the intensity signals of the reflected light of the human eyes to obtain a comparison result;
and outputting the position information at the designated position with the minimum intensity signal according to the comparison result.
10. An eyeball position positioning substrate, comprising:
the eye position locating circuit according to any one of claims 1-8.
11. A virtual reality wearing equipment, comprising: the eyeball position positioning substrate according to claim 10.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202111420155.7A CN114136209B (en) | 2021-11-24 | 2021-11-24 | Eyeball position positioning circuit, eyeball position positioning method, substrate and virtual reality wearable device |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202111420155.7A CN114136209B (en) | 2021-11-24 | 2021-11-24 | Eyeball position positioning circuit, eyeball position positioning method, substrate and virtual reality wearable device |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN114136209A true CN114136209A (en) | 2022-03-04 |
| CN114136209B CN114136209B (en) | 2023-11-24 |
Family
ID=80388704
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202111420155.7A Active CN114136209B (en) | 2021-11-24 | 2021-11-24 | Eyeball position positioning circuit, eyeball position positioning method, substrate and virtual reality wearable device |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN114136209B (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2024113371A1 (en) * | 2022-12-02 | 2024-06-06 | 京东方科技集团股份有限公司 | Photosensitive system for determining positions of human eyes, and display apparatus and method |
Citations (25)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN2255047Y (en) * | 1996-06-06 | 1997-05-28 | 北京自动化技术研究院 | Device for indicating the position of a moving object |
| JPH10289094A (en) * | 1997-04-16 | 1998-10-27 | Sony Corp | Maximum value selecting circuit |
| JPH1185467A (en) * | 1997-09-12 | 1999-03-30 | Sony Corp | Selection method and selection circuit |
| US6120461A (en) * | 1999-08-09 | 2000-09-19 | The United States Of America As Represented By The Secretary Of The Army | Apparatus for tracking the human eye with a retinal scanning display, and method thereof |
| US20090225278A1 (en) * | 2008-03-05 | 2009-09-10 | Coretronic Corporation | Eyeball locating method and system |
| CN103604968A (en) * | 2013-12-10 | 2014-02-26 | 南京国睿安泰信科技股份有限公司 | System for eliminating burrs erroneously generated by peak detection |
| US20140138544A1 (en) * | 2012-09-04 | 2014-05-22 | Innovega Inc. | Eye tracking system and related methods |
| US8824779B1 (en) * | 2011-12-20 | 2014-09-02 | Christopher Charles Smyth | Apparatus and method for determining eye gaze from stereo-optic views |
| CN203811941U (en) * | 2014-04-23 | 2014-09-03 | 无锡芯盟科技有限公司 | Micro-display chip with optical sensitization function |
| TW201519103A (en) * | 2013-11-14 | 2015-05-16 | Pixart Imaging Inc | Eye detecting device and methodes of detecting pupil and identifying iris |
| CN104834381A (en) * | 2015-05-15 | 2015-08-12 | 中国科学院深圳先进技术研究院 | Wearable device for sight focus positioning and sight focus positioning method |
| CN105138996A (en) * | 2015-09-01 | 2015-12-09 | 北京上古视觉科技有限公司 | Iris identification system with living body detecting function |
| CN205181314U (en) * | 2015-03-14 | 2016-04-27 | 中国科学院苏州生物医学工程技术研究所 | Portable pair of mesh pupil detection device |
| US20160270656A1 (en) * | 2015-03-16 | 2016-09-22 | Magic Leap, Inc. | Methods and systems for diagnosing and treating health ailments |
| US20170115483A1 (en) * | 2015-10-23 | 2017-04-27 | Thalmic Labs Inc. | Systems, devices, and methods for laser eye tracking |
| CN106874895A (en) * | 2017-03-31 | 2017-06-20 | 北京七鑫易维信息技术有限公司 | A kind of Eye-controlling focus device and head-mounted display apparatus |
| CN107515466A (en) * | 2017-08-14 | 2017-12-26 | 华为技术有限公司 | A kind of eyeball tracking system and eyeball tracking method |
| CN107997737A (en) * | 2017-10-31 | 2018-05-08 | 上海美沃精密仪器股份有限公司 | Ocular imaging system, method and device |
| US20180249906A1 (en) * | 2017-03-01 | 2018-09-06 | The Johns Hopkins University | Fast x-y axis bright pupil tracker |
| CN108702162A (en) * | 2015-12-07 | 2018-10-23 | Macom连接解决有限公司 | High speed Gabi selection for Viterbi decoder |
| US20180314325A1 (en) * | 2017-04-28 | 2018-11-01 | Microsoft Technology Licensing, Llc | Eye tracking using scanned beam and multiple detectors |
| CN109143581A (en) * | 2018-08-28 | 2019-01-04 | 北京七鑫易维信息技术有限公司 | A kind of head-mounted display apparatus and its eyeball tracking method |
| CN110703904A (en) * | 2019-08-26 | 2020-01-17 | 深圳疆程技术有限公司 | Augmented virtual reality projection method and system based on sight tracking |
| CN110738952A (en) * | 2019-10-31 | 2020-01-31 | 的卢技术有限公司 | HUD display method and system combining human eye recognition, tracking and ray recognition |
| WO2021218315A1 (en) * | 2020-04-26 | 2021-11-04 | Oppo广东移动通信有限公司 | Image sensing circuit, image sensor, and terminal device |
-
2021
- 2021-11-24 CN CN202111420155.7A patent/CN114136209B/en active Active
Patent Citations (26)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN2255047Y (en) * | 1996-06-06 | 1997-05-28 | 北京自动化技术研究院 | Device for indicating the position of a moving object |
| JPH10289094A (en) * | 1997-04-16 | 1998-10-27 | Sony Corp | Maximum value selecting circuit |
| JPH1185467A (en) * | 1997-09-12 | 1999-03-30 | Sony Corp | Selection method and selection circuit |
| US6120461A (en) * | 1999-08-09 | 2000-09-19 | The United States Of America As Represented By The Secretary Of The Army | Apparatus for tracking the human eye with a retinal scanning display, and method thereof |
| US20090225278A1 (en) * | 2008-03-05 | 2009-09-10 | Coretronic Corporation | Eyeball locating method and system |
| US8824779B1 (en) * | 2011-12-20 | 2014-09-02 | Christopher Charles Smyth | Apparatus and method for determining eye gaze from stereo-optic views |
| US20140138544A1 (en) * | 2012-09-04 | 2014-05-22 | Innovega Inc. | Eye tracking system and related methods |
| TW201519103A (en) * | 2013-11-14 | 2015-05-16 | Pixart Imaging Inc | Eye detecting device and methodes of detecting pupil and identifying iris |
| CN103604968A (en) * | 2013-12-10 | 2014-02-26 | 南京国睿安泰信科技股份有限公司 | System for eliminating burrs erroneously generated by peak detection |
| CN203811941U (en) * | 2014-04-23 | 2014-09-03 | 无锡芯盟科技有限公司 | Micro-display chip with optical sensitization function |
| CN205181314U (en) * | 2015-03-14 | 2016-04-27 | 中国科学院苏州生物医学工程技术研究所 | Portable pair of mesh pupil detection device |
| US20160270656A1 (en) * | 2015-03-16 | 2016-09-22 | Magic Leap, Inc. | Methods and systems for diagnosing and treating health ailments |
| CN104834381A (en) * | 2015-05-15 | 2015-08-12 | 中国科学院深圳先进技术研究院 | Wearable device for sight focus positioning and sight focus positioning method |
| CN105138996A (en) * | 2015-09-01 | 2015-12-09 | 北京上古视觉科技有限公司 | Iris identification system with living body detecting function |
| US20180129041A1 (en) * | 2015-10-23 | 2018-05-10 | Thalmic Labs Inc. | Systems, devices, and methods for laser eye tracking |
| US20170115483A1 (en) * | 2015-10-23 | 2017-04-27 | Thalmic Labs Inc. | Systems, devices, and methods for laser eye tracking |
| CN108702162A (en) * | 2015-12-07 | 2018-10-23 | Macom连接解决有限公司 | High speed Gabi selection for Viterbi decoder |
| US20180249906A1 (en) * | 2017-03-01 | 2018-09-06 | The Johns Hopkins University | Fast x-y axis bright pupil tracker |
| CN106874895A (en) * | 2017-03-31 | 2017-06-20 | 北京七鑫易维信息技术有限公司 | A kind of Eye-controlling focus device and head-mounted display apparatus |
| US20180314325A1 (en) * | 2017-04-28 | 2018-11-01 | Microsoft Technology Licensing, Llc | Eye tracking using scanned beam and multiple detectors |
| CN107515466A (en) * | 2017-08-14 | 2017-12-26 | 华为技术有限公司 | A kind of eyeball tracking system and eyeball tracking method |
| CN107997737A (en) * | 2017-10-31 | 2018-05-08 | 上海美沃精密仪器股份有限公司 | Ocular imaging system, method and device |
| CN109143581A (en) * | 2018-08-28 | 2019-01-04 | 北京七鑫易维信息技术有限公司 | A kind of head-mounted display apparatus and its eyeball tracking method |
| CN110703904A (en) * | 2019-08-26 | 2020-01-17 | 深圳疆程技术有限公司 | Augmented virtual reality projection method and system based on sight tracking |
| CN110738952A (en) * | 2019-10-31 | 2020-01-31 | 的卢技术有限公司 | HUD display method and system combining human eye recognition, tracking and ray recognition |
| WO2021218315A1 (en) * | 2020-04-26 | 2021-11-04 | Oppo广东移动通信有限公司 | Image sensing circuit, image sensor, and terminal device |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2024113371A1 (en) * | 2022-12-02 | 2024-06-06 | 京东方科技集团股份有限公司 | Photosensitive system for determining positions of human eyes, and display apparatus and method |
Also Published As
| Publication number | Publication date |
|---|---|
| CN114136209B (en) | 2023-11-24 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US10212380B2 (en) | Pixel acquisition circuit, image sensor and image acquisition system | |
| CN108200362B (en) | Bionical retina imaging circuit and sub-circuit based on space contrast degree | |
| US9635302B2 (en) | Active pixel sensor device and operating method of the same | |
| CN109002218B (en) | Display panel, driving method thereof and display device | |
| US9380237B2 (en) | Semiconductor integrated circuit, current control method, ad converter, solid-state imaging device, and electronic system | |
| EP3354012B1 (en) | Imaging system unit cell and methods for dynamic range imaging | |
| CN108168695B (en) | Light detection unit and method, light detection circuit and method, and display device | |
| CN108304803A (en) | Optical detection circuit, light detection method and display device | |
| US20200175245A1 (en) | Fingerprint Detection Device and Fingerprint Detection Method | |
| CN114136209B (en) | Eyeball position positioning circuit, eyeball position positioning method, substrate and virtual reality wearable device | |
| CN111428697A (en) | Optical sensor circuit, detector, imaging system, sensor, and display panel | |
| JP7344667B2 (en) | Imaging device, its control method, and imaging device | |
| US20200221048A1 (en) | Circuit of detecting light, image sensor and electronic device using the same and method of detecting light based on the same | |
| US11232278B2 (en) | Fingerprint identification and detection circuit, method and display device | |
| US4360833A (en) | Light sensitive semi-conductor element and arrangement | |
| Brajovic et al. | Temporal photoreception for adaptive dynamic range image sensing and encoding | |
| EP3376754A1 (en) | Pixel acquisition circuit, optical flow sensor, and image acquisition system | |
| US11523073B2 (en) | Image sensor with noise cancellation feature and electronic device with same | |
| US20230139066A1 (en) | Image sensing apparatus | |
| JPH0437630B2 (en) | ||
| CN109348150B (en) | Pixel circuit for realizing CMOS active pixel flexible image sensor based on organic thin film phototransistor | |
| Yin et al. | An information sensor with in-pixel-processing for geriatric nursing | |
| CN111476164A (en) | Biological characteristic image acquisition device and method and intelligent equipment | |
| US12020501B2 (en) | Sensing circuit, detection control method, fingerprint identification module and display apparatus | |
| US11953373B1 (en) | Sensor device and sensing method |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |