CN114666485A - Lens module automatic focusing control system and control method based on MTF - Google Patents
Lens module automatic focusing control system and control method based on MTF Download PDFInfo
- Publication number
- CN114666485A CN114666485A CN202210438341.1A CN202210438341A CN114666485A CN 114666485 A CN114666485 A CN 114666485A CN 202210438341 A CN202210438341 A CN 202210438341A CN 114666485 A CN114666485 A CN 114666485A
- Authority
- CN
- China
- Prior art keywords
- mtf
- camera
- lens module
- automatic focusing
- light source
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000000034 method Methods 0.000 title claims abstract description 48
- 238000003384 imaging method Methods 0.000 claims abstract description 18
- 230000008569 process Effects 0.000 claims abstract description 16
- 238000003491 array Methods 0.000 claims abstract description 7
- 238000006073 displacement reaction Methods 0.000 claims abstract description 5
- 238000004590 computer program Methods 0.000 claims description 12
- 238000004422 calculation algorithm Methods 0.000 claims description 11
- 238000012546 transfer Methods 0.000 abstract description 4
- 230000006870 function Effects 0.000 description 7
- 238000004364 calculation method Methods 0.000 description 4
- 230000008859 change Effects 0.000 description 3
- 238000012360 testing method Methods 0.000 description 3
- 230000000007 visual effect Effects 0.000 description 3
- 230000000694 effects Effects 0.000 description 2
- 230000006872 improvement Effects 0.000 description 2
- 241001156002 Anthonomus pomorum Species 0.000 description 1
- 238000006243 chemical reaction Methods 0.000 description 1
- 125000004122 cyclic group Chemical group 0.000 description 1
- 238000013461 design Methods 0.000 description 1
- 238000010586 diagram Methods 0.000 description 1
- 238000005516 engineering process Methods 0.000 description 1
- 238000004519 manufacturing process Methods 0.000 description 1
- 230000002093 peripheral effect Effects 0.000 description 1
- 238000013441 quality evaluation Methods 0.000 description 1
- 238000005070 sampling Methods 0.000 description 1
Images
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
- H04N23/50—Constructional details
- H04N23/55—Optical parts specially adapted for electronic image sensors; Mounting thereof
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
- H04N23/60—Control of cameras or camera modules
- H04N23/67—Focus control based on electronic image sensor signals
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
- H04N23/60—Control of cameras or camera modules
- H04N23/67—Focus control based on electronic image sensor signals
- H04N23/675—Focus control based on electronic image sensor signals comprising setting of focusing regions
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
- H04N23/95—Computational photography systems, e.g. light-field imaging systems
- H04N23/958—Computational photography systems, e.g. light-field imaging systems for extended depth of field imaging
- H04N23/959—Computational photography systems, e.g. light-field imaging systems for extended depth of field imaging by adjusting depth of field during image capture, e.g. maximising or setting range based on scene characteristics
Abstract
The invention discloses an automatic focusing control system and method of a lens module based on MTF (modulation transfer function), wherein the system comprises an imaging module, an actuator and an automatic focusing controller, the imaging module comprises a guide rail, a camera, a lens module and a target light source which are coaxially and sequentially arranged on the guide rail, the lens module and the target light source are fixed, the actuator controls the camera to move so as to adjust the distance between the camera and the lens module, the camera collects a plurality of target light source images under different displacements through the lens module and sends the images to the automatic focusing controller, the automatic focusing controller processes the target light source images to obtain MTF arrays corresponding to the target light source images under specific frequency, and the actuator controls the camera to move to the position corresponding to the maximum MTF value, so that automatic focusing control is realized. The invention realizes full-automatic focusing control, can realize configurability according to different lenses, and provides an efficient and accurate focusing method for the lens module and the camera for operators.
Description
Technical Field
The present invention relates to a lens module control technology, and more particularly, to an MTF-based automatic focusing control system and method for a lens module.
Background
With the continuous improvement of the industrial automation level, the traditional manual focusing mode and the low-quality automatic focusing mode can not meet the requirements of production and application. For a long time, workers can only consume a large amount of time in the repeated definition measuring process, so that the problems in the calibration and adjustment of a real lens imaging system are solved practically, the working efficiency and the operation quality of adjustment personnel are improved, the requirements of enterprises for continuous improvement are met, and the requirements of focusing software systems are increased continuously.
In the quality evaluation of the non-reference image, the definition of the image is an important index for measuring the quality of the image, and the image can better correspond to the subjective feeling of people, and the image is not high in definition and shows the blurring of the image. Several commonly used algorithms with representative definition, such as variance function, Brenner gradient function, Tenengrad gradient function, Laplacian gradient function. Modulation Transfer Function (MTF) is a scientific method for analyzing the image of a lens.
Disclosure of Invention
The purpose of the invention is as follows: one objective of the present invention is to provide an MTF-based automatic focus control system for a lens module.
Another objective of the present invention is to provide a method for controlling the lens module auto-focusing based on MTF, in which the imaging quality is measured by the MTF value under a specific frequency, when the imaging module is at different focusing positions of the lens, the imaging MTF value changes, and the larger the value, the better the visual quality, and the focus control and MTF calculation are realized by software, so as to achieve the method for controlling the auto-focusing.
The technical scheme is as follows: the invention relates to an MTF-based lens module automatic focusing control system which comprises an imaging module, an actuator and an automatic focusing controller, wherein the imaging module comprises a guide rail, a camera, a lens module and a target light source which are coaxially and sequentially arranged on the guide rail, the lens module and the target light source are fixed, the actuator controls the camera to move so as to adjust the distance between the camera and the lens module, the camera collects a plurality of target light source images under different displacements through the lens module and sends the target light source images to the automatic focusing controller, the automatic focusing controller processes the target light source images to obtain MTF arrays corresponding to the target light source images under specific frequency, the maximum MTF value is obtained, a control command is sent to the actuator, and the actuator controls the camera to move to the position corresponding to the maximum MTF value so as to realize automatic focusing control.
Optionally, the automatic focusing controller includes an MTF algorithm module, a display module, and an automatic focusing control module, where the MTF algorithm module processes the target light source image to obtain an MTF value of the image, and stores the MTF value in the MTF array, the automatic focusing control module controls the camera to move to a position corresponding to a maximum MTF value in the MTF array through the actuator, and the automatic focusing control module sends the automatic focusing process information to the display module to be displayed.
Optionally, the MTF algorithm module calculates an MTF value of the image by using a knife edge method.
Optionally, the target light source is a checkerboard panel.
Optionally, the actuator includes a linear motor and a motor controller, the auto-focus controller controls the motor to drive the camera to move along the guide rail through the motor controller, and the moving range of the camera covers the focus of the lens module.
The invention relates to a lens module automatic focusing control method based on MTF, which comprises the following steps:
s1, setting the moving range of the camera to be 0-M, the moving step length to be S and the MTF specific frequency to be F;
s2, resetting the camera to a point 0, and initializing an MTF array;
s3, controlling the camera to move by a step S, acquiring a target light source image at the moment, and selecting an interested area;
s4, calculating an MTF value corresponding to the region of interest at a specific frequency F, and assigning the MTF value to an MTF array;
s5, judging whether the current position of the camera is larger than M, if not, returning to the step S3; if so, stopping circulation, and calculating the maximum value in the MTF array and the array index N corresponding to the maximum value;
and S6, controlling the camera to move to the position of S x N, wherein the camera is at the best focus of the lens module, and realizing automatic focusing control.
Further, MTF values were calculated by the edge method.
Furthermore, the focus of the lens module is within the moving range of the camera, and the moving step S is smaller than the focal depth of the lens module.
An apparatus of the present invention includes a memory and a processor, wherein:
a memory for storing a computer program capable of running on the processor;
and the processor is used for executing the steps of the MTF-based lens module automatic focusing control method when the computer program is run.
A storage medium of the present invention stores thereon a computer program, and the computer program is executed by at least one processor to implement the steps of the above-mentioned MTF-based lens module auto-focus control method.
Has the advantages that: compared with the prior art, the automatic focusing control system based on the MTF controls the moving part through software, so that the imaging system is controlled to move in the front and back range of the focal point of the lens to shoot a plurality of groups of images with the precision smaller than the focal depth, the MTF value under the specific frequency is calculated to obtain the MTF arrays corresponding to the plurality of groups of images, and the maximum value in the MTF arrays is analyzed, so that the imaging system is controlled in the focal depth range.
Drawings
FIG. 1 is a schematic diagram of the control system connections of the present invention;
FIG. 2 is a schematic flow chart of calculating MTF value by edge method;
FIG. 3 is a plot of MTF for a particular frequency;
FIG. 4 is a control camera selecting a region of interest (ROI) image;
FIG. 5 is an image obtained in actual testing;
FIG. 6 is a focusing curve of the corresponding MTF array for the image of FIG. 5.
Detailed Description
The present invention will be described in detail with reference to specific examples.
The invention relates to an automatic focusing control system of a lens module based on MTF, which comprises an imaging module, an actuator and an automatic focusing controller, wherein the imaging module comprises a camera, a lens module, a guide rail and a target light source, the camera, the lens module and the target light source are coaxially and sequentially arranged on the guide rail, the distance between the lens module and the target light source is fixed, the distance between the camera and the lens module is adjustable, the camera collects target light source images through the lens module and sends the target light source images to the automatic focusing controller, the actuator controls the camera to move and collects a plurality of target light source images under different displacements, the automatic focusing controller processes the target light source images to obtain a plurality of MTF values of the target light source images under specific frequency, the maximum MTF value is obtained, a control instruction is sent to the actuator, and the actuator controls the camera to move to a position corresponding to the maximum MTF value, automatic focusing control is realized.
The automatic focusing controller is integrated in a PC (personal computer), the PC comprises computer peripherals such as a display and the like and is connected with a camera and an actuator, the actuator comprises a linear motor and a motor controller, a target light source adopts a checkerboard panel, as shown in figure 1, the camera, a lens module and the checkerboard panel are sequentially arranged on a guide rail, the lens module and the checkerboard panel are fixed, and the camera can move on the guide rail so as to adjust the distance between the camera and the lens module and realize automatic focusing control; the checkerboard panel provides a checkerboard target light source, the checkerboard target is projected into the camera through the lens module, the camera uploads the acquired target image to the PC, and an automatic focusing controller in the PC processes the target image to obtain an MTF value under a specific frequency; the linear motor is used as a carrier of the camera, the camera moves along the guide rail through the motor controller, the camera, the lens module and the checkerboard panel are ensured to be on the same axis, a target image is collected once when the camera moves once and is uploaded to the PC, and the cyclic collection of the target image is stopped until the camera exceeds a displacement range; the automatic focusing controller processes all target images to obtain an MTF array, takes the maximum value in the MTF array and an array index corresponding to the maximum value, sends a control instruction to the motor controller, controls the motor to drive the camera to move to a position corresponding to the maximum MTF value, and at the moment, the camera completes automatic focusing on the best focus of the lens module.
And the automatic focusing controller is used for acquiring the target images acquired by the camera, calculating the MTF value of each target image, and controlling the linear motor to drive the camera to realize the automatic focusing process and the realization of the MTF algorithm. The automatic focusing controller comprises an MTF algorithm module, a display module and an automatic focusing control module, wherein the MTF algorithm module processes a target light source image to obtain an MTF value of the image and stores the MTF value into an MTF array, the automatic focusing control module controls the camera to move to a position corresponding to the maximum MTF value in the MTF array through the actuator, and meanwhile, the automatic focusing control module sends automatic focusing process information to the display module to be displayed.
In the embodiment of the invention, the display module is a UI interface, can input automatic focusing parameters, controls focusing by one key, draws a focusing process curve, outputs a focusing result and displays the focusing result in a visual way through the interface.
The mtf (modulation Transfer function) modulation Transfer function is a comprehensive representation of the camera imaging contrast and resolution. Three luminophores are commonly used in the MTF test: the design in the embodiment of the invention is realized by adopting a knife edge method, and a checkerboard panel is taken as a target light source calculation method as an example, and a region of interest (ROI), namely a knife edge image, is selected as shown in figure 4. As shown in fig. 2, the edge method comprises the following calculation steps:
(1) and (3) performing super sampling through the edge image to obtain a more exquisite black-white conversion straight line (ESF).
(2) The rate of change of the Line (LSF) is derived from this line.
(3) The change rate is subjected to fourier transform fft (dft) to obtain MTF values at each frequency.
In the embodiment of the invention, the MTF algorithm module adopts a blade edge method to calculate the MTF value under the specific frequency, obtains MTF arrays corresponding to a plurality of groups of target images, and analyzes the maximum value in the MTF arrays, thereby transmitting a control instruction to the motor controller, and driving the camera within the focal depth range by controlling the linear motor.
The autofocus control module selects an MTF value at a specific frequency, and the specific frequency needs to be set according to a value of the actual lens module, for example, a frequency value corresponding to MTF-60, as shown in fig. 3, a frequency 29 corresponding to MTF60, and the specific frequency can be set to 29.
The invention relates to a lens module automatic focusing control method based on MTF, which comprises the following steps:
s1, setting the moving range of the camera to be 0-M, the moving step length to be S and the MTF specific frequency to be F;
the focal point of the lens module is in the moving range of the camera, and the moving step length S is smaller than the focal depth of the lens module;
s2, resetting the camera to a point 0, and initializing an MTF array;
s3, controlling the moving step length S of the camera, acquiring the target light source image at the moment, and selecting an interested area, namely a blade edge image, as shown in FIG. 4;
s4, calculating an MTF value corresponding to the region of interest at a specific frequency F, and assigning the MTF value to an MTF array;
s5, judging whether the current position of the camera is larger than M, if not, returning to the step S3; if so, stopping circulation, and calculating the maximum value in the MTF array and the array index N corresponding to the maximum value;
and S6, controlling the camera to move to the position of S × N, and realizing automatic focusing control when the camera is at the optimal focus of the lens module.
In the practical test of the embodiment of the invention, the obtained images are shown in fig. 5 and 26 images in total, the MTF value of each image is calculated to obtain an MTF array, and the focusing process curve of the MTF array is further obtained and is shown in fig. 6, so that the MTF value corresponding to the image img14 is 98.3304 as the maximum value, the imaging quality is the best, and the best focusing position is obtained.
The method of the invention measures the imaging quality through the MTF value under specific frequency, when the industrial camera is at different focusing positions of the lens, the MTF value calculated by the imaging image can change, the larger the value is, the better the visual quality is, and the method realizes image acquisition, focusing control and MTF calculation through software, thereby achieving the method of automatic focusing control.
An apparatus device of the present invention includes a memory and a processor, wherein:
a memory for storing a computer program capable of running on the processor;
and the processor is used for executing the steps of the MTF-based lens module automatic focusing control method when the computer program is run, and achieving the technical effect consistent with the method.
The storage medium of the present invention stores a computer program, and the computer program, when executed by at least one processor, implements the steps of the MTF-based lens module auto-focus control method, and achieves technical effects consistent with the above-mentioned methods.
Claims (10)
1. The system is characterized by comprising an imaging module, an actuator and an automatic focusing controller, wherein the imaging module comprises a guide rail, a camera, a lens module and a target light source which are coaxially and sequentially arranged on the guide rail, the lens module and the target light source are fixed, the actuator controls the camera to move so as to adjust the distance between the camera and the lens module, the camera collects a plurality of target light source images under different displacements through the lens module and sends the target light source images to the automatic focusing controller, the automatic focusing controller processes the target light source images to obtain MTF arrays corresponding to the target light source images under specific frequency, the maximum MTF value is obtained, a control command is sent to the actuator, and the actuator controls the camera to move to the position corresponding to the maximum MTF value, so that automatic focusing control is realized.
2. The system of claim 1, wherein the auto-focus controller comprises an MTF algorithm module, a display module, and an auto-focus control module, wherein the MTF algorithm module processes the target light source image to obtain an MTF value of the image, and stores the MTF value in the MTF array, the auto-focus control module controls the camera to move to a position corresponding to a maximum MTF value in the MTF array through the actuator, and the auto-focus control module sends the auto-focus process information to the display module for display.
3. The system of claim 2, wherein the MTF algorithm module calculates the MTF value of the image by edge method.
4. The system of claim 1, wherein the target light source is a checkerboard panel.
5. The system of claim 1, wherein the actuator comprises a linear motor and a motor controller, the auto-focus controller controls the motor to move the camera along the rail via the motor controller, and the range of movement of the camera covers the focus of the lens module.
6. A lens module automatic focusing control method based on MTF is characterized by comprising the following steps:
s1, setting the moving range of the camera to be 0-M, the moving step length to be S and the MTF specific frequency to be F;
s2, resetting the camera to a point 0, and initializing an MTF array;
s3, controlling the camera to move by a step length S, acquiring the target light source image at the moment, and selecting an interested area;
s4, calculating an MTF value corresponding to the region of interest at a specific frequency F, and assigning the MTF value to an MTF array;
s5, judging whether the current position of the camera is larger than M, if not, returning to the step S3; if so, stopping circulation, and calculating the maximum value in the MTF array and the array index N corresponding to the maximum value;
and S6, controlling the camera to move to the position of S × N, and realizing automatic focusing control when the camera is at the optimal focus of the lens module.
7. The method of claim 6, wherein the MTF value is calculated by a knife edge method.
8. The method of claim 6, wherein the focus of the lens module is within the range of the camera movement and the movement step S is smaller than the lens module focal depth.
9. An apparatus, comprising a memory and a processor, wherein:
a memory for storing a computer program capable of running on the processor;
a processor for executing the steps of the MTF-based lens module autofocus control method according to any of claims 6 to 8 when running the computer program.
10. A storage medium having a computer program stored thereon, wherein the computer program when executed by at least one processor implements the steps of the MTF-based lens module autofocus control method of any of claims 6-8.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210438341.1A CN114666485A (en) | 2022-04-25 | 2022-04-25 | Lens module automatic focusing control system and control method based on MTF |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210438341.1A CN114666485A (en) | 2022-04-25 | 2022-04-25 | Lens module automatic focusing control system and control method based on MTF |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN114666485A true CN114666485A (en) | 2022-06-24 |
Family
ID=82038099
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202210438341.1A Pending CN114666485A (en) | 2022-04-25 | 2022-04-25 | Lens module automatic focusing control system and control method based on MTF |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN114666485A (en) |
Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| JP2007212620A (en) * | 2006-02-08 | 2007-08-23 | Matsushita Electric Ind Co Ltd | Method for manufacturing lens system, and device and method for evaluating lens system |
| CN103278934A (en) * | 2013-05-31 | 2013-09-04 | 中国科学院西安光学精密机械研究所 | Focal plane butt-joint device and method for satellite-borne optical remote sensor |
| CN103424978A (en) * | 2012-05-15 | 2013-12-04 | 鸿富锦精密工业（深圳）有限公司 | System and method for testing camera module |
| CN107421722A (en) * | 2017-07-01 | 2017-12-01 | 南京理工大学 | Based on CCD camera image intensifier modulation transfer function test method and system |
| CN107860773A (en) * | 2017-11-06 | 2018-03-30 | 凌云光技术集团有限责任公司 | Automatic optical detecting system and its bearing calibration for PCB |
| CN109862346A (en) * | 2019-01-11 | 2019-06-07 | 歌尔股份有限公司 | Test method of focusing and equipment |
| CN212206548U (en) * | 2020-03-31 | 2020-12-22 | 歌尔光学科技有限公司 | Optical lens MTF testing device |
-
2022
- 2022-04-25 CN CN202210438341.1A patent/CN114666485A/en active Pending
Patent Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| JP2007212620A (en) * | 2006-02-08 | 2007-08-23 | Matsushita Electric Ind Co Ltd | Method for manufacturing lens system, and device and method for evaluating lens system |
| CN103424978A (en) * | 2012-05-15 | 2013-12-04 | 鸿富锦精密工业（深圳）有限公司 | System and method for testing camera module |
| CN103278934A (en) * | 2013-05-31 | 2013-09-04 | 中国科学院西安光学精密机械研究所 | Focal plane butt-joint device and method for satellite-borne optical remote sensor |
| CN107421722A (en) * | 2017-07-01 | 2017-12-01 | 南京理工大学 | Based on CCD camera image intensifier modulation transfer function test method and system |
| CN107860773A (en) * | 2017-11-06 | 2018-03-30 | 凌云光技术集团有限责任公司 | Automatic optical detecting system and its bearing calibration for PCB |
| CN109862346A (en) * | 2019-01-11 | 2019-06-07 | 歌尔股份有限公司 | Test method of focusing and equipment |
| CN212206548U (en) * | 2020-03-31 | 2020-12-22 | 歌尔光学科技有限公司 | Optical lens MTF testing device |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US10101572B2 (en) | Variable focal length lens system with multi-level extended depth of field image processing | |
| EP3374817B1 (en) | Autofocus system for a computational microscope | |
| US7769219B2 (en) | Method for assessing image focus quality | |
| CN108152869B (en) | Small step focusing method suitable for bionic vision rapid focusing | |
| US9830694B2 (en) | Multi-level image focus using a tunable lens in a machine vision inspection system | |
| CN109451244B (en) | Automatic focusing method and system based on liquid lens | |
| JP6469368B2 (en) | Machine vision inspection system and method for performing high speed focused height measurement operation | |
| CN109521547B (en) | Variable-step-length automatic focusing method and system | |
| CN100548022C (en) | Estimate the method and apparatus of position in focus | |
| CN102692347A (en) | Camera automatic regulating image acquisition device and method for fatigue crack propagation test | |
| CN109085113B (en) | Automatic focusing method and device for cervical exfoliated cell detection device | |
| CN104181685B (en) | Based on microscopical digital slices autofocus and its method | |
| CN111948784B (en) | Iterative optimization automatic focusing method based on hill climbing method | |
| US10534164B2 (en) | Digital microscope and focusing method thereof | |
| US11119382B2 (en) | Tunable acoustic gradient lens system with amplitude adjustment corresponding to z-height as indicated by calibration data | |
| CN113923358A (en) | Online automatic focusing method and system in flying shooting mode | |
| CN106873142B (en) | High-quality image acquisition device and method of tubercle bacillus detector | |
| JP7319903B2 (en) | High-speed TAG lens-assisted 3D measurement and extended depth-of-field imaging | |
| CN102261986A (en) | Zoom-stereo microscope detection device and method | |
| US20220046180A1 (en) | Deep Learning Model for Auto-Focusing Microscope Systems | |
| CN105227802A (en) | A kind of self-adaptation control method towards best quality image rapid scanning and system | |
| US20210190999A1 (en) | Tunable acoustic gradient lens system utilizing amplitude adjustments for acquiring images focused at different z-heights | |
| CN114666485A (en) | Lens module automatic focusing control system and control method based on MTF | |
| CN114815211B (en) | Microscope automatic focusing method based on image processing | |
| CN112839168B (en) | Method for automatically adjusting camera imaging resolution in AOI detection system |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |