CN115775281A - Global calibration method for non-target four-eye stereoscopic vision system - Google Patents
Global calibration method for non-target four-eye stereoscopic vision system Download PDFInfo
- Publication number
- CN115775281A CN115775281A CN202211617024.2A CN202211617024A CN115775281A CN 115775281 A CN115775281 A CN 115775281A CN 202211617024 A CN202211617024 A CN 202211617024A CN 115775281 A CN115775281 A CN 115775281A
- Authority
- CN
- China
- Prior art keywords
- cone
- vision system
- stereoscopic vision
- target
- binocular
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000000034 method Methods 0.000 title claims abstract description 28
- 238000013519 translation Methods 0.000 claims abstract description 7
- 230000009466 transformation Effects 0.000 claims abstract description 6
- 239000011159 matrix material Substances 0.000 claims description 10
- 238000005259 measurement Methods 0.000 claims description 8
- 238000012545 processing Methods 0.000 claims description 4
- 241000287196 Asthenes Species 0.000 claims description 3
- 238000005457 optimization Methods 0.000 claims description 3
- 238000007781 pre-processing Methods 0.000 claims description 3
- 230000000007 visual effect Effects 0.000 claims description 2
- 238000013461 design Methods 0.000 description 1
- 238000010586 diagram Methods 0.000 description 1
- 239000003550 marker Substances 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
Images
Abstract
The invention relates to a global calibration method of a non-target four-eye stereoscopic vision system, which comprises the following steps: acquiring three-dimensional coordinates of mark points on one-dimensional targets in the fields of view of two binocular stereoscopic vision systems, and acquiring an angle interval when a specific target image is shot; respectively carrying out cone fitting on the mark points acquired by the two binocular stereoscopic vision systems, and calculating the axis direction of a cone and the vertex coordinates of the cone; calculating the angle deviation of the two binocular stereoscopic vision systems in the direction perpendicular to the axis of the cone by using the angle interval obtained by the angle sensor; and converting the three-dimensional coordinates of the mark points in the second binocular stereoscopic vision system into the coordinate system of the first binocular stereoscopic vision system through rotation and translation transformation according to the relationship that the two conical surfaces physically belong to the same conical surface, and finishing the global calibration. The invention does not need a public view field between stereoscopic vision systems, and has the advantages of convenient operation, high calibration speed, high calibration precision and stronger practicability.
Description
Technical Field
The invention relates to a global calibration method for a non-target four-eye stereoscopic vision system.
Background
When a video camera is used for performing stereoscopic vision calibration, the range of a field of view is often limited, and no or only a small overlapped field of view exists between cameras, and particularly for high-precision and large-field industrial measurement, a measurement system with a larger field of view is generally required to be constructed by a plurality of cameras, and how to perform high-precision global calibration on the cameras with non-overlapped fields of view becomes the key of a multi-camera measurement system.
Currently, the adopted global calibration method includes:
(1) The spatial position of each camera is directly obtained by means of three-dimensional measuring equipment such as a high-precision theodolite or a laser tracker, so that all camera coordinate systems are unified under the same coordinate system. The method can achieve higher precision, but the operation process is very complicated, and the method has great limitation on an industrial measurement field with smaller available space;
(2) Establishing a corresponding refraction model by utilizing optical characteristics, such as light refraction, and simultaneously covering the same area of a calibration target; or indirectly observe the public target through reflection of a mirror surface, but the calibration precision can be reduced along with the increase of the distance between the cameras;
(3) Reconstructing a large plane calibration object, acquiring local characteristic information in each small target, calculating a transformation matrix among cameras according to the relative position relation among the small targets, wherein the multi-scene information of the targets needs to be acquired in the solving process, and the calibration result is not stable enough;
(4) The position relation between the two cameras is obtained by tracking a moving target and calculating the time difference between the target exiting the field of view of the camera and entering the field of view of the other camera, but more scene information is needed during calibration, which is often difficult to obtain in industrial measurement, and the precision still needs to be improved.
Disclosure of Invention
In view of this, it is necessary to provide a global calibration method for a non-target four-eye stereo vision system.
The invention provides a global calibration method for a non-target four-eye stereoscopic vision system, which comprises the following steps: the method comprises the following steps of S1, respectively obtaining three-dimensional coordinates of mark points on one-dimensional targets in the visual fields of two binocular stereoscopic vision systems, and obtaining an angle interval when a specific target image is shot by using an angle sensor; s2, respectively carrying out cone fitting on the mark points acquired by the two binocular stereoscopic vision systems according to the characteristics of a cone formed by the rotation motion of the one-dimensional target, and calculating the axis direction of the cone and the vertex coordinates of the cone; s3, calculating the angle deviation of the two binocular stereoscopic vision systems in the direction vertical to the axis of the cone by using the angle interval obtained by the angle sensor; and S4, converting the three-dimensional coordinates of the mark points in the second binocular stereoscopic vision system into the coordinate system of the first binocular stereoscopic vision system through rotation and translation transformation according to the relationship that the two conical surfaces physically belong to the same conical surface, and finishing overall calibration.
Specifically, the step S1 includes:
s11, mounting an angle sensor and determining a measurement view field;
s12, designing a mark point;
s13, calibrating the high-speed cameras, and determining stereoscopic vision calibration parameters among each group of high-speed cameras;
s14, collecting blade rotation motion images at intervals;
step S15, processing the paddle rotation motion image, extracting circle center pixel coordinates, and calculating the three-dimensional coordinates of the mark points in the field of view of each subsystem of the four-eye stereoscopic vision system by using the stereoscopic vision calibration parameters obtained in the step S13;
and S16, preprocessing the three-dimensional information of the mark points.
Specifically, the step S12 includes:
the marking points are round marking points insensitive to noise, and the size and the distance of the marking points are as small as possible on the premise that the positioning accuracy can be met.
Specifically, the step S16 includes:
firstly, sorting the coordinate values of the same marking point at the same position in the order from small to large, removing the first 20 percent and the second 20 percent, and taking the average value of the residual 60 percent of data as the coordinate value of the marking point.
Specifically, the step S2 includes:
s21, selecting 9 uniformly distributed mark points in the first binocular stereo vision system, roughly fitting a cone to solve 9 parameters of a cone equation, and solving the cone axis of the first binocular stereo vision system
And the vertex O of the cone c1 The initial solution of (a);
s22, through a Levenberg-Marquardt optimization algorithm, all mark points in the first binocular stereo vision system are brought into a target function, the sum of distances from the points to a fitting conical surface is made to be minimum, and an optimized conical axis is obtained
And the vertex O of the cone 1 ；
Step S23, cone fitting is carried out on the mark points collected by the second binocular stereo vision system, and cone axes are obtained according to the step S21 and the step S22 in the same way
And the vertex O of the cone 2 。
Specifically, step S21 includes:
setting the initial solution of the cone axis
Initial solution of the cone vertex O c1 (x 1 ,y 1 ,z 1 ) The equation is expressed as:
let the conic generatrix equation be:
and (2) combining the formula (1) and the formula (2) to obtain a cone surface equation:
F 1 (x,y,z)＝0#(3)。
specifically, the step S3 includes:
the angular interval is recorded as
The azimuth angle of the ith phase in the coordinate system of the first binocular stereo vision system is recorded as theta i And the azimuth angle of the j th phase in the coordinate system of the second binocular stereo vision system is recorded as theta j The angular deviation of the two binocular stereo vision systems in the direction of the vertical cone axis is recorded as
Then
Specifically, the step S4 includes:
step S41, the
Rotate to and
parallel, its rotation matrix denoted R 1 ；
Step S42, adding O 2 Is translated to and O 1 Overlapping, and recording a translation matrix as T;
step S43, rotating the coordinate system of the second binocular stereo vision system around the conical axis of the first binocular stereo vision system
Its rotation matrix is denoted as R 2 ；
Step S44, combining the steps S41-S43, the three-dimensional homogeneous coordinate of any mark point in the second binocular stereo vision system
Change to the first twoHomogeneous coordinates of the coordinate system of the body vision system are marked as P ij ′：P ij ′＝R 2 [R 1 ,T]P ij 。
The invention acquires the track image of the one-dimensional target in the rotating motion, obtains the three-dimensional coordinates of the marked points on the target by using the stereoscopic vision calibration technology and the image processing technology, respectively calculates the axis direction and the vertex coordinates of the cone under two binocular stereoscopic vision systems by using the common conicity property of the one-dimensional target in the rotating motion, and unifies the four-eye stereoscopic vision systems by the rotating and translating transformation of the coordinate system, thereby completing the global calibration. The invention does not need a public view field between stereoscopic vision systems, has convenient operation, high calibration speed and high calibration precision and has stronger practicability.
Drawings
FIG. 1 is a flowchart of a global calibration method for a non-target four-eye stereoscopic vision system according to the present invention;
FIG. 2 is a schematic diagram of a marker provided in an embodiment of the present invention;
fig. 3 is a schematic view of the rotation of the blade according to the embodiment of the present invention.
Detailed Description
The present invention will be described in further detail with reference to the accompanying drawings and specific embodiments.
Please refer to fig. 1, which is a flowchart illustrating a global calibration method for a target-free four-eye stereo vision system according to a preferred embodiment of the present invention.
Step S1, three-dimensional coordinates of mark points on one-dimensional targets in the fields of view of two binocular stereoscopic vision systems are respectively obtained, and an angle sensor is used for obtaining an angle interval when a specific target image is shot. The method specifically comprises the following steps:
step S11, mounting an angle sensor, determining a measurement view field: according to the length of the paddle, a binocular stereoscopic vision system is arranged on a bearing platform below the paddle.
In this embodiment, the blade length is 2.1m, the camera is placed 5m below the blade, the size of the field of view of each binocular stereoscopic vision system is 4.6m × 2.3m, and the two binocular stereoscopic vision systems can cover the entire rotor field of view.
Step S12, designing a mark point:
in the embodiment, the circular mark points insensitive to noise are adopted, and the size and the distance of the circular mark points are as small as possible on the premise of meeting the positioning precision, so that a plurality of mark points can be arranged on the one-dimensional target, the effective data volume is increased, and the calibration precision is improved. The design of the marking points of the embodiment is as shown in fig. 2, the marking points are attached to the positions of 1/4 chord lines of the blade, the diameter of the marking points is 44mm, the distance between the two marking points is 90mm, and 18 marking points are arranged in total.
And S13, calibrating the high-speed cameras, and determining stereoscopic vision calibration parameters among each group of high-speed cameras.
Step S14, collecting blade rotation motion images at equal intervals through a frequency doubling device, and storing the images in a computer:
the blade rotation is schematically shown in FIG. 3, where O is the hub center position, { l 1 ,l 2 ,…,l m The position of the blade when it is rotating, { p } 11 ,p 21 ,…,p n1 ,…,p nm And the position of the mark point is. In the embodiment, the rotating speed of the blades is 50rpm, the blades collect blade rotation moving images once every 20 degrees, and the blades collect 18 positions in a circle.
Step S15, processing the paddle rotation motion image, extracting circle center pixel coordinates, and calculating the three-dimensional coordinates of the mark points in the field of view of each subsystem of the four-eye stereoscopic vision system by using the stereoscopic vision calibration parameters obtained in the step S13;
and S16, preprocessing the three-dimensional information of the mark points.
This embodiment eliminates random errors by acquiring 100 cycles of blade rotation motion images: firstly, sorting 100 coordinate values of the same marking point at the same position in a descending order, removing the first 20 percent and the second 20 percent, and taking the average value of the residual 60 percent of data as the coordinate value of the marking point.
And S2, according to the characteristics of a cone formed by the rotation motion of the one-dimensional target, enabling each phase mark point collected by the binocular stereo vision system to be on the conical surface, respectively carrying out cone fitting on the mark points collected by the two binocular stereo vision systems, and calculating the axis direction of the cone and the vertex coordinates of the cone. The method specifically comprises the following steps:
s21, selecting 9 uniformly distributed mark points in the first binocular stereo vision system, roughly fitting a cone to solve 9 parameters of a cone equation, and solving the cone axis of the first binocular stereo vision system
And the vertex O of the cone c1 The process of the initial solution of (c) is as follows:
setting the initial solution of the cone axis
Initial solution of the cone vertex O c1 (x 1 ,y 1 ,z 1 ) Then its equation is expressed as:
let the conic generatrix equation be:
and (2) combining the formula (1) and the formula (2) to obtain a cone surface equation:
F 1 (x,,)＝0#(3)
s22, substituting all mark points in the first binocular stereo vision system into an objective function through a Levenberg-Marquardt optimization algorithm to enable the sum of distances from the points to the fitting conical surface to be minimum, and obtaining an optimized conical axis
And the vertex O of the cone 1 ；
Step S23, carrying out cone fitting on the mark points acquired by the second binocular stereo vision system, and obtaining the cone axis according to the step S21 and the step S22
And the vertex O of the cone 2 。
And S3, calculating the angle deviation of the two binocular stereoscopic vision systems in the direction vertical to the axis of the cone by using the angle interval obtained by the angle sensor. Specifically, the method comprises the following steps:
the angular interval is recorded as
The azimuth angle of the ith phase in the coordinate system of the first binocular stereo vision system is recorded as theta i And the azimuth angle of the j-th phase in the second binocular stereo vision system coordinate system is recorded as theta j The angular deviation of the two binocular stereo vision systems in the direction perpendicular to the axis of the cone is recorded as
Then
In this embodiment, the 8 th phase and the 11 th phase are selected as the specific phases, i.e. the compartments between the two phases
Calculating according to the three-dimensional coordinates of the mark points obtained in the step S1 to obtain
And S4, converting the three-dimensional coordinates of the mark points in the second binocular stereoscopic vision system into the coordinate system of the first binocular stereoscopic vision system through rotation and translation transformation according to the relationship that the two conical surfaces physically belong to the same conical surface, and realizing the unification of the two binocular stereoscopic vision systems, thereby completing the global calibration. Specifically, the method comprises the following steps:
step S41, a
Rotate to and
parallel, its rotation matrix denoted R 1 In this embodiment, the
Step S42, adding O 2 Is translated to and O 1 Coincidence, with the translation matrix denoted as T, T = (-175.960, 2496.628,4.296 in this embodiment);
step S43, rotating the coordinate system of the second binocular stereo vision system around the conical axis of the first binocular stereo vision system
Its rotation matrix is denoted as R 2 In this embodiment, the
Step S44, combining the steps S41-S43, the three-dimensional homogeneous coordinate of any mark point in the second binocular stereo vision system
The homogeneous coordinate converted into the coordinate system of the first binocular stereo vision system is recorded as P ij ′：P ij ′＝R 2 [R 1 ,T]P ij 。
Although the present invention has been described with reference to the presently preferred embodiments, it will be understood by those skilled in the art that the foregoing description is illustrative only and is not intended to limit the scope of the invention, as claimed.
Claims (8)
1. A global calibration method for a non-target four-eye stereoscopic vision system is characterized by comprising the following steps:
the method comprises the following steps of S1, respectively obtaining three-dimensional coordinates of mark points on one-dimensional targets in the visual fields of two binocular stereoscopic vision systems, and obtaining an angle interval when a specific target image is shot by using an angle sensor;
s2, respectively carrying out cone fitting on the mark points acquired by the two binocular stereoscopic vision systems according to the characteristics of a cone formed by the rotation motion of the one-dimensional target, and calculating the axis direction of the cone and the vertex coordinates of the cone;
s3, calculating the angle deviation of the two binocular stereoscopic vision systems in the direction vertical to the axis of the cone by using the angle interval obtained by the angle sensor;
and S4, converting the three-dimensional coordinates of the mark points in the second binocular stereoscopic vision system into the coordinate system of the first binocular stereoscopic vision system through rotation and translation transformation according to the relationship that the two conical surfaces physically belong to the same conical surface, and finishing overall calibration.
2. The global calibration method for the non-target four-eye stereoscopic vision system according to claim 1, wherein the step S1 comprises:
s11, mounting an angle sensor and determining a measurement view field;
s12, designing a mark point;
s13, calibrating the high-speed cameras, and determining stereoscopic vision calibration parameters among each group of high-speed cameras;
s14, collecting blade rotation motion images at intervals;
step S15, processing the paddle rotation motion image, extracting circle center pixel coordinates, and calculating the three-dimensional coordinates of the mark points in the field of view of each subsystem of the four-eye stereoscopic vision system by using the stereoscopic vision calibration parameters obtained in the step S13;
and S16, preprocessing the three-dimensional information of the mark points.
3. The global calibration method for the non-target four-eye stereoscopic vision system according to claim 2, wherein the step S12 comprises:
the marking points are round marking points insensitive to noise, and the size and the distance of the marking points are as small as possible on the premise that the positioning accuracy can be met.
4. The global calibration method for the non-target four-eye stereoscopic vision system according to claim 3, wherein the step S16 comprises:
firstly, sorting the coordinate values of the same marking point at the same position in the order from small to large, removing the first 20 percent and the second 20 percent, and taking the average value of the residual 60 percent of data as the coordinate value of the marking point.
5. The global calibration method for the non-target four-eye stereoscopic vision system according to claim 4, wherein the step S2 comprises:
s21, selecting 9 uniformly distributed mark points in the first binocular stereo vision system, roughly fitting a cone to solve 9 parameters of a cone equation, and solving the cone axis of the first binocular stereo vision system
And the vertex O of the cone c1 The initial solution of (a);
s22, substituting all mark points in the first binocular stereo vision system into an objective function through a Levenberg-Marquardt optimization algorithm to enable the sum of distances from the points to the fitting conical surface to be minimum, and obtaining an optimized conical axis
And the vertex O of the cone 1 ；
Step S23, cone fitting is carried out on the mark points collected by the second binocular stereo vision system, and cone axes are obtained according to the step S21 and the step S22 in the same way
And the vertex O of the cone 2 。
6. The global calibration method for the non-target four-eye stereoscopic vision system according to claim 5, wherein the step S21 comprises:
with conical axisInitial solution
Initial solution of the cone vertex O c1 (x 1 ,y 1 ,z 1 ) The equation is expressed as:
let the conic generatrix equation be:
and (2) combining the formula (1) and the formula (2) to obtain a cone surface equation:
F 1 (x,,)＝0#(3)。
7. the global calibration method for the non-target four-eye stereoscopic vision system according to claim 6, wherein the step S3 comprises:
the angular interval is recorded as
The azimuth angle of the ith phase in the coordinate system of the first binocular stereo vision system is recorded as theta i And the azimuth angle of the j-th phase in the second binocular stereo vision system coordinate system is recorded as theta j The angular deviation of the two binocular stereo vision systems in the direction perpendicular to the axis of the cone is recorded as
Then
8. The global calibration method for the non-target four-eye stereoscopic vision system according to claim 7, wherein the step S4 comprises:
step S41, the
Rotate to and
parallel, its rotation matrix denoted R 1 ；
Step S42, adding O 2 Is translated to and O 1 Overlapping, and marking a translation matrix as T;
step S43, rotating the coordinate system of the second binocular stereo vision system around the conical axis of the first binocular stereo vision system
Its rotation matrix is denoted as R 2 ；
Step S44, combining the steps S41-S43, the three-dimensional homogeneous coordinate of any mark point in the second binocular stereo vision system
The homogeneous coordinate converted into the coordinate system of the first binocular stereo vision system is recorded as P ij ′ ：P ij ′ ＝R 2 [R 1 ,]P ij 。
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202211617024.2A CN115775281A (en) | 2022-12-15 | 2022-12-15 | Global calibration method for non-target four-eye stereoscopic vision system |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202211617024.2A CN115775281A (en) | 2022-12-15 | 2022-12-15 | Global calibration method for non-target four-eye stereoscopic vision system |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN115775281A true CN115775281A (en) | 2023-03-10 |
Family
ID=85392394
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202211617024.2A Pending CN115775281A (en) | 2022-12-15 | 2022-12-15 | Global calibration method for non-target four-eye stereoscopic vision system |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN115775281A (en) |
Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN106092057A (en) * | 2016-07-28 | 2016-11-09 | 南昌航空大学 | A kind of helicopter rotor blade dynamic trajectory measuring method based on four item stereo visions |
| CN110421562A (en) * | 2019-07-24 | 2019-11-08 | 中国地质大学（武汉） | Mechanical arm calibration system and scaling method based on four item stereo visions |
| CN111275770A (en) * | 2020-01-20 | 2020-06-12 | 南昌航空大学 | Global calibration method of four-eye stereoscopic vision system based on one-dimensional target rotation motion |
| WO2022134939A1 (en) * | 2020-12-24 | 2022-06-30 | 上海智能制造功能平台有限公司 | Data splicing and system calibration method for human body digital measurement device |
-
2022
- 2022-12-15 CN CN202211617024.2A patent/CN115775281A/en active Pending
Patent Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN106092057A (en) * | 2016-07-28 | 2016-11-09 | 南昌航空大学 | A kind of helicopter rotor blade dynamic trajectory measuring method based on four item stereo visions |
| CN110421562A (en) * | 2019-07-24 | 2019-11-08 | 中国地质大学（武汉） | Mechanical arm calibration system and scaling method based on four item stereo visions |
| CN111275770A (en) * | 2020-01-20 | 2020-06-12 | 南昌航空大学 | Global calibration method of four-eye stereoscopic vision system based on one-dimensional target rotation motion |
| WO2022134939A1 (en) * | 2020-12-24 | 2022-06-30 | 上海智能制造功能平台有限公司 | Data splicing and system calibration method for human body digital measurement device |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| Habib et al. | Automatic calibration of low-cost digital cameras | |
| US7680300B2 (en) | Visual object recognition and tracking | |
| CN109341668B (en) | Multi-camera measuring method based on refraction projection model and light beam tracking method | |
| CN111768486B (en) | Monocular camera three-dimensional reconstruction method and system based on rotating refraction sheet | |
| CN111811395A (en) | Monocular vision-based dynamic plane pose measurement method | |
| CN113592961B (en) | Point cloud splicing method based on precision control field and point cloud feature similarity | |
| Mei et al. | Monocular vision for pose estimation in space based on cone projection | |
| Bergström et al. | Virtual projective shape matching in targetless CAD-based close-range photogrammetry for efficient estimation of specific deviations | |
| CN111275770A (en) | Global calibration method of four-eye stereoscopic vision system based on one-dimensional target rotation motion | |
| CN111915685A (en) | Zoom camera calibration method | |
| Grudziński et al. | Stereovision tracking system for monitoring loader crane tip position | |
| Schwieger et al. | Image-based target detection and tracking using image-assisted robotic total stations | |
| Du et al. | Autonomous measurement and semantic segmentation of non-cooperative targets with deep convolutional neural networks | |
| Deng et al. | Joint calibration of dual lidars and camera using a circular chessboard | |
| Miao et al. | Coarse-to-fine hybrid 3d mapping system with co-calibrated omnidirectional camera and non-repetitive lidar | |
| Zhang et al. | Dynamic pose estimation of uncooperative space targets based on monocular vision | |
| Jiang et al. | A ball-shaped target development and pose estimation strategy for a tracking-based scanning system | |
| CN115775281A (en) | Global calibration method for non-target four-eye stereoscopic vision system | |
| Tjahjadi et al. | Single image orientation of UAV's imagery using orthogonal projection model | |
| Lu et al. | Image-based system for measuring objects on an oblique plane and its applications in 2-D localization | |
| CN112734838A (en) | Space target positioning method, equipment and storage medium | |
| CN113483879A (en) | Small satellite flutter high-speed video measurement method | |
| Eynard et al. | UAV Motion Estimation using Hybrid Stereoscopic Vision. | |
| Zhu et al. | Calibration of binocular cameras with non-overlapping fields of view based on planar mirrors | |
| Hrabar et al. | PTZ camera pose estimation by tracking a 3D target |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |