CN109753153B - Haptic interaction device and method for 360-degree suspended light field three-dimensional display system - Google Patents
Haptic interaction device and method for 360-degree suspended light field three-dimensional display system Download PDFInfo
- Publication number
- CN109753153B CN109753153B CN201811601655.9A CN201811601655A CN109753153B CN 109753153 B CN109753153 B CN 109753153B CN 201811601655 A CN201811601655 A CN 201811601655A CN 109753153 B CN109753153 B CN 109753153B
- Authority
- CN
- China
- Prior art keywords
- finger
- coordinates
- model
- leap motion
- bluetooth
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Images
Landscapes
- User Interface Of Digital Computer (AREA)
Abstract
The invention discloses a touch interaction device and a touch interaction method for a 360-degree suspended light field three-dimensional display system, which belong to the technical field of computer vision and comprise a Leap Motion, a finger sleeve and a processor, wherein the Leap Motion is arranged on one side of a high-speed rotating screen and is used for acquiring hand position information; the treater is connected with the Leap Motion communication, acquires hand position information, and the treater is equipped with bluetooth emission module, sends out the instruction to the dactylotheca through bluetooth emission module, and the vibration of control dactylotheca is equipped with bluetooth receiving module in the dactylotheca. The interaction of the finger and the model force touch is realized by reading the finger information, converting coordinates, judging a model, transmitting Bluetooth, driving hardware and the like. When the finger and the boundary of the display model are contacted with each other within a certain range, the finger sleeve can provide a vibration force for the finger, and the effect of force-touch interaction is achieved. That is, when the hand touches the display model, the finger has a vibrating tactile sensation. Thereby creating a more realistic sensory experience for the user.
Description
Technical Field
The invention relates to the technical field of computer vision, in particular to a touch interaction device and method for a 360-degree suspended light field three-dimensional display system.
Background
With the development of computer science and projection technology, three-dimensional display technology has been rapidly developed in recent years. The real three-dimensional image enables an observer to see display in a space, the surrounding of a scene has correct vision and a shielding relation, and the naked-eye three-dimensional display becomes a popular research.
By reconstructing the light field of the three-dimensional scene, large-range high-quality three-dimensional color image display with correct shielding relation can be realized. The light field distribution of the three-dimensional scene is reconstructed by simulating the light emitting mode of the real object, so that different observers around the display system can see correct three-dimensional images. Scanning the light field display scene can bring a strong sense of suspension to an observer, and the observer is not physically isolated from the virtual scene, so that the observer tends to touch the three-dimensional display scene with hands, and scientists provide a suspension scene gesture interaction method based on the somatosensory detection device.
The most common somatosensory detection devices in the market at present comprise Microsoft Kinect and Leap Motion, wherein the Kinect can track 6 persons at most and has 25 skeleton nodes, so that the method is suitable for detection of large actions; the Leap Motion can simultaneously track all joints of the two hands of the person, so that the Leap Motion is more suitable for detecting small movements of fingers.
The Leap Motion is a sensor designed for recognizing gesture Motion, and has a small size, and the recognition frame rate is determined according to the performance of the computer, and the general frame rate ranges from 20 to 290 frames/s. It uses three infrared LEDs as infrared light emitters by using two infrared cameras as sensors. And transmitting the obtained gray level image and the corresponding depth information to a computer for calculation so as to convert the three-dimensional gesture information.
In the prior art, distributed tactile stimulation consistent with the three-dimensional shape information of the virtual object and the contact posture information of the finger and the virtual object is generated and fed back to an operator, so that the tactile sensation generated when the hand of a human contacts the virtual objects with different shapes at different postures is simulated, and good tactile interaction experience is provided for a user, for example, a method for reproducing the three-dimensional shape of the virtual object based on a fingerstall type device is disclosed in Chinese patent document with publication number CN107831892A, but the technology is applied to a virtual three-dimensional scene, so that the user cannot generate real sensory experience.
Disclosure of Invention
The invention aims to provide a touch interaction device for a 360-degree suspended light field three-dimensional display system, which is light and portable, has good coordinate judgment robustness, strong universality, obvious interaction effect and obvious touch feeling, and can be used for man-machine interaction of various light field three-dimensional display systems, so that a user can generate real sensory experience.
Another object of the present invention is to provide a haptic interaction method for a 360 ° floating light field three-dimensional display system, which is implemented based on the above apparatus.
In order to achieve the purpose, the haptic interaction device for the 360-degree suspended light field three-dimensional display system comprises a Leap Motion, a finger sleeve and a processor, wherein the Leap Motion is arranged on one side of a high-speed rotating screen and is used for collecting hand position information; the treater is connected with the Leap Motion communication, acquires hand position information, and the treater is equipped with bluetooth emission module, sends out the instruction to the dactylotheca through bluetooth emission module, and the vibration of control dactylotheca is equipped with bluetooth receiving module in the dactylotheca.
According to the technical scheme, the interaction of the finger and the model force touch is realized by the aid of the device through steps of reading finger information, coordinate conversion, model judgment, Bluetooth transmission, driving hardware and the like. When the finger and the boundary of the display model are contacted with each other within a certain range, the finger sleeve can provide a vibration force for the finger, and the effect of force-touch interaction is achieved. That is, when the hand touches the display model, the finger has a vibrating tactile sensation. Thereby creating a more realistic sensory experience for the user.
Preferably, an electric motor is provided on the finger cot to realize vibration of the finger cot.
Preferably, the electric motor comprises a dc motor and an eccentric wheel connected to an output of the dc motor. When the finger sleeve is in a vibration state, the control circuit is switched on, the direct current motor is started, and the eccentric wheel is driven to rotate at a high speed, so that vibration is generated.
Preferably, the bluetooth transmitting module comprises a usb to ttl component, a bluetooth transmitting end and a bluetooth receiving end.
Preferably, a power supply module is arranged in the finger sleeve. The power supply module may be a secondary lithium battery.
In order to achieve the above another object, the present invention provides a haptic interaction method for a 360 ° floating light field three-dimensional display system, comprising the steps of:
1) establishing a world coordinate system, manufacturing a 3D model to be placed on a high-speed rotating screen, and performing pose estimation on the Leap Motion by using the manufactured printing model, wherein the coordinates of the 3D model in the Leap Motion and the coordinates of the 3D model in the world coordinate are respectively as follows:
finding a transformation parameter by an iterative closest point method: rotation matrix R, and translational vector
And obtaining a conversion relation between the Leap Motion and the world coordinates:
2) calibrating the 3D model by using a binocular camera at a specific position in a world coordinate system, wherein the coordinates of each point on the 3D model in the world coordinate system are as follows:
respectively obtaining internal and external reference matrixes of the binocular camera by a Zhangyingyou calibration method and a PnP algorithm, and respectively taking the mean value of the internal and external reference matrixes as the projection matrixes P of the two cameras1，P2；
The coordinates of each point in the three-dimensional scene in world coordinates are P ', the plane coordinates of P' in two camera views are x1，x2From this it follows:
x1＝P1P″，x2＝P2P″
obtaining coordinates of each point in three-dimensional scene by linear triangle method
3) A transformation R' is found by an iterative closest point method,
obtaining a coordinate conversion formula of the Leap Motion and the 3D model:
4) the finger with the fingerstall is deeply inserted into the detectable area, the finger position information is detected through the Leap Motion, and the finger position information is converted into world coordinates;
5) and comparing and judging the obtained finger coordinates with the model coordinates, and sending an instruction to the finger sleeve to start the vibration mode of the finger sleeve when the finger coordinates are consistent with the 3D model coordinates.
Preferably, in step 1), the method for obtaining the transformation relationship between the Leap Motion and the world coordinates includes:
and (3) placing and fixing the Leap Motion at one side of a high-speed rotating screen, measuring the position and pose of the Leap Motion by four groups of coordinates, averaging to obtain coordinate parameters, and obtaining a coordinate conversion relation.
Preferably, the 3D model is made by 3D printing.
Preferably, the binocular camera is an industrial camera, the effective pixels are 130 ten thousand, and when the resolution is 1280 × 720, the frame rate is 34 FPS.
Compared with the prior art, the invention has the beneficial effects that:
according to the fingerstall type force and touch interaction device, when a finger is in contact with the surface of the three-dimensional display model of the suspended light field, the Leap Motion identifies the position of the finger, transmits the position to the processor to be compared with the coordinates of the model, and transmits a braking signal to the electric motor module through Bluetooth transmission, so that fingerstall type force and touch interaction is realized. The whole device is light and portable, good in coordinate judgment robustness, strong in universality, obvious in interaction effect and touch, and can be used for a human-computer interaction module of various light field three-dimensional display systems.
Drawings
FIG. 1 is a schematic diagram of a haptic interaction device for a 360 suspended light field three-dimensional display system according to an embodiment of the present invention;
FIG. 2 is a schematic diagram of the internal structure of a haptic interaction device for a 360-degree suspended light field three-dimensional display system according to an embodiment of the present invention;
FIG. 3 is a schematic workflow diagram of a haptic interaction device for a 360-degree suspended light field three-dimensional display system according to an embodiment of the present invention;
FIG. 4 is a schematic diagram of coordinate transformation measurement of a haptic interaction device for a 360-degree suspended light field three-dimensional display system according to an embodiment of the present invention.
Detailed Description
In order to make the objects, technical solutions and advantages of the present invention more apparent, the present invention will be further described with reference to the following embodiments and accompanying drawings.
Examples
Referring to fig. 1 to 4, the haptic interaction device for a 360 ° floating light field three-dimensional display system of the present invention comprises:
the Leap Motion1 is arranged on one side of the high-speed rotating screen and is used for collecting hand position information;
the fingerstall 2 is worn on the finger of a user and has a vibration function, and a Bluetooth receiving module is arranged in the fingerstall;
a processor 3 for processing information and transmitting signals; the processor 3 is connected with the Leap Motion1 in a communication way and acquires hand position information. Be equipped with bluetooth emission module in the treater 3, bluetooth emission module includes that usb changes ttl subassembly, bluetooth sending end and bluetooth receiving end. The Bluetooth transmitting module sends an instruction to the fingerstall 2 to control the fingerstall 2 to vibrate.
Be equipped with electric motor and power module on dactylotheca 2, electric motor includes piezoceramics actuator and vibrating motor, realizes the vibration of dactylotheca. The power supply module is composed of two 3.7V lithium batteries and supplies power for the electric motor and the Bluetooth receiving module.
A three-dimensional model is projected by a high-speed projector and a rotating directional diffuser screen 001. The reflective directional scattering screen 001 is connected with the servo motor and rotates at a high speed in a horizontal plane, the high-speed projector is positioned right above the rotation center of the screen, and the optical axis of the projector is coaxial with the rotation center of the directional scattering screen 001 so as to ensure that a projected image is not distorted. The light field reconstruction technique is used to render 600 images updated by the projector every second into a three-dimensional scene 002 for the surrounding observers to watch.
Since the optimal range of Leap Motion interaction is 20 to 60 centimeters, in order to expand the fusion area, the Leap Motion is obliquely arranged, the inclination angle is theta, the Leap Motion recognition effect is the best when the theta is measured by experiments at about 45 degrees, the vertical distance and the horizontal distance from the central area of the three-dimensional scene 002 are respectively delta H and delta L, and the position relationship is shown in fig. 4.
When the pose estimation is carried out on the Leap Motion, a 3D printing model 003 which is 210mm long, 110cm wide and 110cm high is used as a tool.
The hand information collection control used in this embodiment may be determined according to system requirements, and is not limited to LeapMotion, and the scope of the present invention should not be limited thereby.
Based on the sleeve type force haptic interaction device, the haptic interaction method for the 360 ° floating light field three-dimensional display system of the embodiment includes the following steps:
(1) adjusting the ambient brightness around the display system to be darker;
(2) placing and fixing the Leap Motion1, adjusting the angle of the Leap Motion1 to enable the fusion area to be the largest, wherein the inclination angle is theta, and the vertical distance and the horizontal distance from the central area of the three-dimensional scene 002 are respectively delta H and delta L;
(3) placing the 3D printing model 003, measuring four groups of delta H, delta L and theta by using the four groups of coordinates, and averaging to obtain a final parameter to obtain a conversion relation between the Leap Motion1 and the world center coordinate, as shown in FIG. 4;
and performing pose estimation on the Leap Motion by using the manufactured 3D printing model, wherein the coordinates of each point on the 3D model in the Leap Motion and the world coordinates are respectively as follows:
finding transformation parameters by an iterative closest point method: rotation matrix R, and translational vector
And obtaining a conversion relation between the Leap Motion and the world coordinates:
(4) verifying the coordinate conversion relation, measuring the coordinates of 18 points, and keeping the final error between the converted coordinates and the actual position coordinates within 6 mm;
(5) placing a binocular camera 004, and calibrating internal references of the binocular camera 004 respectively by using a standard checkerboard;
(6) respectively imaging the left camera and the right camera at the same 3D printing model 003 position (namely a world coordinate system), and calculating external parameters of the binocular camera 004 relative to the world coordinate;
(7) projecting and displaying the 3D printing model 003 with the same size, respectively shooting by using two cameras, obtaining a three-dimensional reconstruction result according to the internal and external reference matrixes, and completing coordinate conversion of a world coordinate system and a 3D printing model 003 coordinate system;
calibrating the 3D model by using a binocular camera at a specific position in a world coordinate system, wherein the coordinates of each point on the 3D model in the world coordinate system are as follows:
respectively obtaining internal and external reference matrixes of the binocular camera by a Zhangyingyou calibration method and a PnP algorithm, and respectively taking the mean value of the internal and external reference matrixes as the projection matrixes P of the two cameras1，P2；
The coordinates of each point in the three-dimensional scene in world coordinates are P ', the plane coordinates of P' in two camera views are x1，x2From this it follows:
x1＝P1P″，x2＝P2P″
obtaining coordinates of each point in three-dimensional scene by linear triangle method
(8) Obtaining a conversion relation between the Leap Motion1 and a 3D printing model 003 coordinate system;
a transformation R' is found by an iterative closest point method,
obtaining a coordinate conversion formula of the Leap Motion and the 3D printing model:
(9) the Bluetooth sending end is connected with the processor 3, and the Bluetooth receiving end is connected with the finger touch generator (namely a vibration motor);
(10) as shown in the block diagram in fig. 2, the hand with the fingerstall vibration device is inserted into the detectable area, the LeapMotion1 detects the coordinates of the finger in real time, and compares the coordinates with the 002 coordinates of the three-dimensional scene after coordinate conversion, if the finger is sensed to touch the 002 surface of the three-dimensional scene in a certain range, a signal is sent to the bluetooth sending end through serial port communication, and the bluetooth receiving end receives the signal and then brakes the electric motor through serial port communication.
After all the above processing and operations are completed, the interaction experiencer at the corresponding position successfully performs fingerstall type force touch interaction with the three-dimensional scene 002. Repeating step (10) to perform an all-around interactive experience for the three-dimensional scene 002.
The above fingerstall-type force tactile interaction device is not limited to the above embodiments and display systems. The method can be suitable for human-computer interaction systems and three-dimensional display systems in various large-angle scene ranges.
While the invention has been further described herein by way of illustration and example, it is to be understood that the invention is not limited to the embodiments and examples described above, and that the foregoing description is intended to be illustrative and not limiting, and that various changes and modifications may be made by one skilled in the art without departing from the scope and spirit of the invention as defined by the appended claims.
Claims (8)
1. A touch interaction method for a 360-degree suspended light field three-dimensional display system is characterized by being realized based on a touch interaction device, wherein the touch interaction device comprises a Leap Motion, a finger sleeve and a processor, the Leap Motion is arranged on one side of a high-speed rotating screen and used for collecting hand position information, the finger sleeve has a vibration function, and the processor is used for processing information and transmitting signals; the processor is in communication connection with the Leap Motion to acquire hand position information, the processor is provided with a Bluetooth transmitting module, the Bluetooth transmitting module sends an instruction to the finger sleeve to control the finger sleeve to vibrate, and a Bluetooth receiving module is arranged in the finger sleeve;
the haptic interaction method comprises the following steps:
1) establishing a world coordinate system, manufacturing a 3D model to be placed on a high-speed rotating screen, and estimating the pose of the Leap Motion by using the manufactured 3D printing model, wherein the coordinates of each point on the 3D model in the Leap Motion and the world coordinates are respectively as follows:
finding transformation parameters by an iterative closest point method: rotation matrix R, and translational vector
And obtaining a conversion relation between the Leap Motion and the world coordinates:
2) calibrating the 3D model by using a binocular camera at a specific position in a world coordinate system, wherein the coordinates of each point on the 3D model in the world coordinate system are as follows:
respectively obtaining internal and external reference matrixes of the binocular camera by a Zhangyingyou calibration method and a PnP algorithm, and respectively taking the mean value of the internal and external reference matrixes as the projection matrixes P of the two cameras1，P2；
The coordinates of each point in the three-dimensional scene in world coordinates are P ', the plane coordinates of P' in two camera views are x1，x2From this it follows:
x1＝P1P″，x2＝P2P″
obtaining coordinates of each point in three-dimensional scene by linear triangle method
3) A transformation R' is found by an iterative closest point method,
obtaining a coordinate conversion formula of the Leap Motion and the 3D model:
4) the finger with the fingerstall is deeply inserted into the detectable area, the finger position information is detected through the Leap Motion, and the finger position information is converted into world coordinates;
5) and comparing and judging the obtained finger coordinates with the model coordinates, and sending an instruction to the finger sleeve to start the vibration mode of the finger sleeve when the finger coordinates are consistent with the 3D model coordinates.
2. A haptic interaction method as recited in claim 1 wherein: the electric motor is arranged on the finger stall to realize the vibration of the finger stall.
3. A haptic interaction method as recited in claim 2 wherein: the electric motor comprises a direct current motor and an eccentric wheel connected with the output end of the direct current motor.
4. A haptic interaction method as recited in claim 1 wherein: the Bluetooth transmitting module comprises a usb to ttl component, a Bluetooth transmitting end and a Bluetooth receiving end.
5. A haptic interaction method as recited in claim 1 wherein: a power supply module is arranged in the finger sleeve.
6. A haptic interaction method as claimed in claim 1, wherein in step 1), the transformation relationship between Leap Motion and world coordinates is obtained by:
and (3) placing and fixing the Leap Motion at one side of a high-speed rotating screen, measuring the position and pose of the Leap Motion by four groups of coordinates, measuring for many times, taking an average number to obtain coordinate parameters, and obtaining a coordinate conversion relation.
7. A haptic interaction method as recited in claim 1 wherein said 3D model is made by means of 3D printing.
8. A haptic interaction method as recited in claim 1 wherein said binocular camera is an industrial camera with 130 thousand active pixels and a resolution of 1280 x 720, and wherein said frame rate is 34 FPS.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201811601655.9A CN109753153B (en) | 2018-12-26 | 2018-12-26 | Haptic interaction device and method for 360-degree suspended light field three-dimensional display system |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201811601655.9A CN109753153B (en) | 2018-12-26 | 2018-12-26 | Haptic interaction device and method for 360-degree suspended light field three-dimensional display system |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN109753153A CN109753153A (en) | 2019-05-14 |
| CN109753153B true CN109753153B (en) | 2020-05-22 |
Family
ID=66404200
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201811601655.9A Active CN109753153B (en) | 2018-12-26 | 2018-12-26 | Haptic interaction device and method for 360-degree suspended light field three-dimensional display system |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN109753153B (en) |
Families Citing this family (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110390686A (en) * | 2019-07-24 | 2019-10-29 | 张天 | Naked eye 3D display method and system |
| CN110865704B (en) * | 2019-10-21 | 2021-04-27 | 浙江大学 | Gesture interaction device and method for 360-degree suspended light field three-dimensional display system |
| CN111240474B (en) * | 2020-01-03 | 2023-01-20 | 云南电网有限责任公司电力科学研究院 | Building block type touch interaction system |
| CN113223344B (en) * | 2021-05-25 | 2022-08-23 | 湖南汽车工程职业学院 | Big data-based professional teaching display system for art design |
Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102183873A (en) * | 2011-04-27 | 2011-09-14 | 浙江大学 | Suspended three-dimensional 360-degree field space display device based on high-speed projector |
| CN104298065A (en) * | 2014-05-07 | 2015-01-21 | 浙江大学 | 360-degree three-dimensional display device and method based on splicing of multiple high-speed projectors |
| CN108664126A (en) * | 2018-05-16 | 2018-10-16 | 上海大学 | Deformable hand captures exchange method under a kind of reality environment |
| CN208077116U (en) * | 2018-02-09 | 2018-11-09 | 北京航天长征科技信息研究所 | A kind of line holographic projections interactive system |
Family Cites Families (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20180184948A1 (en) * | 2016-12-30 | 2018-07-05 | Mindmaze Holding Sa | System, method and apparatus for diagnosis and therapy of neuromuscular or neurological deficits |
-
2018
- 2018-12-26 CN CN201811601655.9A patent/CN109753153B/en active Active
Patent Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102183873A (en) * | 2011-04-27 | 2011-09-14 | 浙江大学 | Suspended three-dimensional 360-degree field space display device based on high-speed projector |
| CN104298065A (en) * | 2014-05-07 | 2015-01-21 | 浙江大学 | 360-degree three-dimensional display device and method based on splicing of multiple high-speed projectors |
| CN208077116U (en) * | 2018-02-09 | 2018-11-09 | 北京航天长征科技信息研究所 | A kind of line holographic projections interactive system |
| CN108664126A (en) * | 2018-05-16 | 2018-10-16 | 上海大学 | Deformable hand captures exchange method under a kind of reality environment |
Also Published As
| Publication number | Publication date |
|---|---|
| CN109753153A (en) | 2019-05-14 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN109753153B (en) | Haptic interaction device and method for 360-degree suspended light field three-dimensional display system | |
| WO2015180497A1 (en) | Motion collection and feedback method and system based on stereoscopic vision | |
| US9229540B2 (en) | Deriving input from six degrees of freedom interfaces | |
| US20160098095A1 (en) | Deriving Input from Six Degrees of Freedom Interfaces | |
| KR101518727B1 (en) | A stereoscopic interaction system and stereoscopic interaction method | |
| CN102426486B (en) | Stereo interaction method and operated apparatus | |
| US20120135803A1 (en) | Game device utilizing stereoscopic display, method of providing game, recording medium storing game program, and game system | |
| CN109804220A (en) | System and method for tracking the fortune dynamic posture on head and eyes | |
| EP2932358A1 (en) | Direct interaction system for mixed reality environments | |
| CN110865704B (en) | Gesture interaction device and method for 360-degree suspended light field three-dimensional display system | |
| WO2019171557A1 (en) | Image display system | |
| WO2022012194A1 (en) | Interaction method and apparatus, display device, and storage medium | |
| CN102508562B (en) | Three-dimensional interaction system | |
| US20160232708A1 (en) | Intuitive interaction apparatus and method | |
| KR20220125362A (en) | Position Tracking System for Head-Worn Display Systems Including Angle Sensing Detectors | |
| Hu et al. | StereoPilot: A wearable target location system for blind and visually impaired using spatial audio rendering | |
| US20190369807A1 (en) | Information processing device, information processing method, and program | |
| WO2017061890A1 (en) | Wireless full body motion control sensor | |
| CN102508561B (en) | Operating rod | |
| US10296098B2 (en) | Input/output device, input/output program, and input/output method | |
| JP2016541042A (en) | Method and system for providing position or motion information for controlling at least one function of a vehicle | |
| CN109426336A (en) | A kind of virtual reality auxiliary type selecting equipment | |
| WO2021029256A1 (en) | Information processing device, information processing method, and program | |
| CN102508563B (en) | Stereo interactive method and operated device | |
| US11550397B1 (en) | Systems and methods for simulating a sensation of expending effort in a virtual environment |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |