CN116681770A - Calibration method and calibration device for depth information acquisition equipment on rotating mechanism - Google Patents
Calibration method and calibration device for depth information acquisition equipment on rotating mechanism Download PDFInfo
- Publication number
- CN116681770A CN116681770A CN202210167055.6A CN202210167055A CN116681770A CN 116681770 A CN116681770 A CN 116681770A CN 202210167055 A CN202210167055 A CN 202210167055A CN 116681770 A CN116681770 A CN 116681770A
- Authority
- CN
- China
- Prior art keywords
- coordinate data
- depth information
- data set
- information acquisition
- transformation
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000000034 method Methods 0.000 title claims abstract description 67
- 230000007246 mechanism Effects 0.000 title claims abstract description 47
- 230000009466 transformation Effects 0.000 claims abstract description 109
- 239000003550 marker Substances 0.000 claims abstract description 45
- 230000004044 response Effects 0.000 claims abstract description 16
- 239000011159 matrix material Substances 0.000 claims description 41
- 238000004590 computer program Methods 0.000 claims description 16
- 238000006243 chemical reaction Methods 0.000 claims description 8
- 230000009471 action Effects 0.000 claims description 3
- 238000004891 communication Methods 0.000 description 9
- 230000000875 corresponding effect Effects 0.000 description 9
- 230000008569 process Effects 0.000 description 8
- 238000012545 processing Methods 0.000 description 7
- 238000010586 diagram Methods 0.000 description 5
- 230000006870 function Effects 0.000 description 4
- 238000013459 approach Methods 0.000 description 3
- 230000003287 optical effect Effects 0.000 description 3
- 238000009412 basement excavation Methods 0.000 description 2
- 238000004364 calculation method Methods 0.000 description 2
- 230000003993 interaction Effects 0.000 description 2
- 238000003491 array Methods 0.000 description 1
- 238000013473 artificial intelligence Methods 0.000 description 1
- 230000010267 cellular communication Effects 0.000 description 1
- 230000001413 cellular effect Effects 0.000 description 1
- 230000008859 change Effects 0.000 description 1
- 238000010276 construction Methods 0.000 description 1
- 230000002596 correlated effect Effects 0.000 description 1
- 238000007405 data analysis Methods 0.000 description 1
- 230000003247 decreasing effect Effects 0.000 description 1
- 238000001514 detection method Methods 0.000 description 1
- 238000005516 engineering process Methods 0.000 description 1
- 239000004973 liquid crystal related substance Substances 0.000 description 1
- 238000010801 machine learning Methods 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 239000013307 optical fiber Substances 0.000 description 1
- 238000012216 screening Methods 0.000 description 1
- 239000004065 semiconductor Substances 0.000 description 1
- 230000001953 sensory effect Effects 0.000 description 1
- 230000000007 visual effect Effects 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/80—Analysis of captured images to determine intrinsic or extrinsic camera parameters, i.e. camera calibration
-
- G—PHYSICS
- G01—MEASURING; TESTING
- G01B—MEASURING LENGTH, THICKNESS OR SIMILAR LINEAR DIMENSIONS; MEASURING ANGLES; MEASURING AREAS; MEASURING IRREGULARITIES OF SURFACES OR CONTOURS
- G01B11/00—Measuring arrangements characterised by the use of optical techniques
- G01B11/22—Measuring arrangements characterised by the use of optical techniques for measuring depth
-
- G—PHYSICS
- G01—MEASURING; TESTING
- G01B—MEASURING LENGTH, THICKNESS OR SIMILAR LINEAR DIMENSIONS; MEASURING ANGLES; MEASURING AREAS; MEASURING IRREGULARITIES OF SURFACES OR CONTOURS
- G01B21/00—Measuring arrangements or details thereof, where the measuring technique is not covered by the other groups of this subclass, unspecified or not relevant
- G01B21/02—Measuring arrangements or details thereof, where the measuring technique is not covered by the other groups of this subclass, unspecified or not relevant for measuring length, width, or thickness
- G01B21/04—Measuring arrangements or details thereof, where the measuring technique is not covered by the other groups of this subclass, unspecified or not relevant for measuring length, width, or thickness by measuring coordinates of points
- G01B21/042—Calibration or calibration artifacts
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/10—Image acquisition modality
- G06T2207/10028—Range image; Depth image; 3D point clouds
Landscapes
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Engineering & Computer Science (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Theoretical Computer Science (AREA)
- Length Measuring Devices By Optical Means (AREA)
Abstract
The disclosure provides a calibration method and a calibration device for depth information acquisition equipment on a rotating mechanism, relates to the technical field of machine vision, and particularly relates to calibration of image acquisition equipment. The implementation scheme is as follows: acquiring a first coordinate data set of a preset marker in a first coordinate system by a first depth information acquisition device in response to the rotatable body rotating to a first position; acquiring, by a second depth information acquisition device, a second set of coordinate data of the marker in a second coordinate system in response to the rotatable body rotating to a second position; converting the first coordinate data set into a reference coordinate system according to a predetermined first transformation relation to obtain a third coordinate data set; determining a rotation angle difference between the first position and the second position; and determining a second transformation relationship according to the second coordinate data set, the third coordinate data set and the rotation angle difference.
Description
Technical Field
The disclosure relates to the technical field of machine vision, in particular to calibration of image acquisition equipment, and specifically relates to a calibration method and device for depth information acquisition equipment on a rotating mechanism, electronic equipment, a computer readable storage medium and a computer program product.
Background
In the current unmanned excavating machinery, a plurality of side surfaces around the cockpit can be respectively provided with depth cameras for detecting the surrounding environment around the excavating machinery, so that the subsequent excavation is convenient. Since the setting positions of the plurality of depth cameras are different, it is necessary to calibrate each depth camera, that is, to determine the positional relationship between the position of the depth camera and the position of the cockpit.
In the prior art, for each depth camera to be calibrated, the position difference and the angle difference between the depth camera and the main body of the excavating machine need to be measured respectively, and then the transformation relation is obtained through a large amount of calculation. In case of a large number of depth cameras to be calibrated, the above method is very complex and time consuming.
The approaches described in this section are not necessarily approaches that have been previously conceived or pursued. Unless otherwise indicated, it should not be assumed that any of the approaches described in this section qualify as prior art merely by virtue of their inclusion in this section. Similarly, the problems mentioned in this section should not be considered as having been recognized in any prior art unless otherwise indicated.
Disclosure of Invention
The present disclosure provides a calibration method for a depth information acquisition device on a rotating mechanism, and a method, an apparatus, an electronic device, a computer readable storage medium and a computer program product for calibrating the apparatus.
According to an aspect of the present disclosure, there is provided a calibration method for a depth information collecting apparatus on a rotating mechanism, wherein the rotating mechanism includes a rotatable body, a calibrated first depth information collecting apparatus disposed on the rotatable body, and a second depth information collecting apparatus to be calibrated disposed on the rotatable body, the calibration method including: in response to the rotatable body rotating to a first position and a second position respectively, acquiring a first coordinate data set of a preset marker in a first coordinate system by a first depth information acquisition device and acquiring a second coordinate data set of the marker in a second coordinate system by a second depth information acquisition device respectively; converting the first coordinate data set into a reference coordinate system according to a predetermined first transformation relation to obtain a third coordinate data set; and determining a second transformation relationship for coordinate transformation between the second coordinate system and the reference coordinate system based on the second coordinate data set, the third coordinate data set, and the rotation angle difference between the first position and the second position.
According to another aspect of the present disclosure, there is provided a calibration apparatus for a depth information collecting device on a rotating mechanism, wherein the rotating mechanism includes a rotatable body, a calibrated first depth information collecting device disposed on the rotatable body, and a second depth information collecting device to be calibrated disposed on the rotatable body, the calibration apparatus including: the first depth information acquisition device is configured to acquire a first coordinate data set of a preset marker under a first coordinate system in response to the rotatable main body rotating to a first position; a second depth information acquisition device configured to acquire a second coordinate data set of the marker in a second coordinate system in response to rotation of the rotatable body to the second position; a first conversion unit configured to convert the first coordinate data set to a reference coordinate system according to a predetermined first conversion relation, to obtain a third coordinate data set; a first determination unit configured to determine a rotation angle difference between the first position and the second position; and a second determining unit configured to determine a second transformation relationship for coordinate transformation between the second coordinate system and the reference coordinate system based on the second coordinate data set, the third coordinate data set, and the rotation angle difference.
According to another aspect of the present disclosure, there is provided an electronic device including: at least one processor; and a memory communicatively coupled to the at least one processor; wherein the memory stores instructions executable by the at least one processor to enable the at least one processor to perform the method described above.
According to another aspect of the present disclosure, there is provided a non-transitory computer-readable storage medium storing computer instructions for causing a computer to perform the above-described method.
According to another aspect of the present disclosure, there is provided a computer program product comprising a computer program, wherein the computer program, when executed by a processor, implements the method described above.
According to one or more embodiments of the present disclosure, the second depth information collecting device to be calibrated is calibrated by means of the first depth information collecting device which is calibrated, so that the need of calibrating each depth information collecting device by independently measuring the position difference and the angle difference between each depth information collecting device and the rotatable main body in the related art is avoided, and therefore the calibration efficiency and the calibration speed are greatly improved by the method of the embodiments of the present disclosure.
It should be understood that the description in this section is not intended to identify key or critical features of the embodiments of the disclosure, nor is it intended to be used to limit the scope of the disclosure. Other features of the present disclosure will become apparent from the following specification.
Drawings
The accompanying drawings illustrate exemplary embodiments and, together with the description, serve to explain exemplary implementations of the embodiments. The illustrated embodiments are for exemplary purposes only and do not limit the scope of the claims. Throughout the drawings, identical reference numerals designate similar, but not necessarily identical, elements.
FIG. 1 illustrates a schematic top view of a rotating mechanism in which various methods described herein may be implemented, according to an embodiment of the present disclosure;
FIG. 2 illustrates a flow chart of a calibration method for a depth information gathering device on a rotating mechanism, according to an embodiment of the present disclosure;
FIG. 3 illustrates a flowchart of a method of acquiring a first coordinate data set and a second coordinate data set, according to an embodiment of the present disclosure;
FIG. 4 illustrates a flow chart of a calibration method for a depth information gathering device on a rotating mechanism according to an embodiment of the present disclosure;
FIG. 5 shows a block diagram of a calibration apparatus for a depth information gathering device on a rotating mechanism, according to an embodiment of the present disclosure;
fig. 6 illustrates a block diagram of an exemplary electronic device that can be used to implement embodiments of the present disclosure.
Detailed Description
Exemplary embodiments of the present disclosure are described below in conjunction with the accompanying drawings, which include various details of the embodiments of the present disclosure to facilitate understanding, and should be considered as merely exemplary. Accordingly, one of ordinary skill in the art will recognize that various changes and modifications of the embodiments described herein can be made without departing from the scope of the present disclosure. Also, descriptions of well-known functions and constructions are omitted in the following description for clarity and conciseness.
In the present disclosure, the use of the terms "first," "second," and the like to describe various elements is not intended to limit the positional relationship, timing relationship, or importance relationship of the elements, unless otherwise indicated, and such terms are merely used to distinguish one element from another element. In some examples, a first element and a second element may refer to the same instance of the element, and in some cases, they may also refer to different instances based on the description of the context.
The terminology used in the description of the various examples in this disclosure is for the purpose of describing particular examples only and is not intended to be limiting. Unless the context clearly indicates otherwise, the elements may be one or more if the number of the elements is not specifically limited. Furthermore, the term "and/or" as used in this disclosure encompasses any and all possible combinations of the listed items.
Embodiments of the present disclosure will be described in detail below with reference to the accompanying drawings.
Fig. 1 shows a schematic top view of a rotary mechanism 100 for implementing a method according to an embodiment of the present disclosure. In this embodiment, the rotation mechanism 100 may be an excavating machine, and in other embodiments, the rotation mechanism 100 may be another rotatable device such as an autonomous vehicle, a terrain detection device, or the like. As shown in fig. 1, the rotation mechanism 100 includes a rotatable body 110, a first depth information collecting apparatus 101, and a second depth information collecting apparatus 102. As shown in fig. 1, the rotatable body 110 may include a plurality of sides, and in the case where the rotating mechanism is an excavating machine, the rotatable body 110 may be a rotating cab of the excavating machine. As shown in fig. 1, the rotatable body 110 includes 4 sides, which can be bi-directionally rotated in the ω direction shown in fig. 1. The first depth information collecting apparatus 101 and the second depth information collecting apparatus 102 are disposed on different sides of the rotatable body 110, respectively, and collect depth information toward the outside of the rotatable body 110. In some embodiments, the first depth information acquisition device 101 and the second depth information acquisition device 102 may each be a depth camera. In the present embodiment, in addition to the first depth information collecting apparatus 101 and the second depth information collecting apparatus 102, a depth information collecting apparatus may be provided on the other side surface of the rotatable body 110, for example, as shown in fig. 1, and a third depth information collecting apparatus 103 may be provided on the opposite side of the second depth information collecting apparatus 102. A marker a may be preset outside the rotation mechanism 100 for a subsequent calibration process.
Fig. 2 shows a flowchart of a calibration method 200 for a depth information gathering device on a rotating mechanism according to an embodiment of the present disclosure. The rotating mechanism comprises a rotatable main body, calibrated first depth information acquisition equipment arranged on the rotatable main body and second depth information acquisition equipment to be calibrated arranged on the rotatable main body, and the calibrating method 200 comprises the following steps:
step 201, acquiring a first coordinate data set of a preset marker in a first coordinate system by a first depth information acquisition device in response to rotation of a rotatable main body to a first position;
step 202, acquiring a second coordinate data set of the marker in a second coordinate system by a second depth information acquisition device in response to the rotatable body rotating to a second position;
step 203, converting the first coordinate data set to a reference coordinate system according to a predetermined first transformation relation to obtain a third coordinate data set;
step 204, determining a rotation angle difference between the first position and the second position; and
step 205, determining a second transformation relation for coordinate transformation between the second coordinate system and the reference coordinate system according to the second coordinate data set, the third coordinate data set and the rotation angle difference.
The method of the embodiment of the disclosure calibrates the second depth information acquisition device to be calibrated by means of the calibrated first depth information acquisition device, and avoids the need of calibrating each depth information acquisition device by independently measuring the position difference and the angle difference between each depth information acquisition device and the rotatable main body in the related art, so that the method of the embodiment of the disclosure greatly improves the calibration efficiency and the calibration speed.
The marker in this embodiment may be a physical object having a specific geometric shape, such as a rectangular parallelepiped or a cylinder, etc. disposed around the rotation mechanism. In other embodiments, the marker may also be a pre-generated graphical pattern, for example, a display screen facing the rotating mechanism may be disposed around the rotating mechanism, and the graphical pattern may then be displayed on the display screen. The pattern may be a simple geometric pattern such as a rectangle or a circle, or a relatively complex pattern such as a checkerboard. The above-described depth information acquiring apparatus may be a depth camera for acquiring coordinates of an object to be photographed.
In step 201, the first position may be a position where the preset marker falls within the field of view of the first depth information collecting device, or a position where the rotating mechanism rotates to face the marker. As shown in fig. 1, the position shown in fig. 1 is the first position. The first depth information acquisition device may acquire depth information of a plurality of acquisition points on the marker, which may be presented in the form of a point cloud. It is understood that the above-mentioned "depth information" refers to information of a distance from the depth information collecting device to the collecting point. The associated processor of the subsequent depth information acquisition device may determine the coordinates of the acquisition points in the first coordinate system from the depth information in the form of the point clouds. The positions and the numbers of the plurality of acquisition points can be set by adjusting the acquisition parameters of the first depth information acquisition device, for example, the acquisition points can be increased by increasing the acquisition frequency of the first depth information acquisition device, or the acquisition points can be reduced by decreasing the acquisition frequency of the first depth information acquisition device.
Referring to fig. 1, a first coordinate system is shown with reference numeral C1, and the origin of coordinates of the first coordinate system is the center point of the first depth information collecting apparatus. The three directions of the first coordinate system and the orientation of the first depth information collecting device are associated, and the orientation of the first depth information collecting device may be set to the X direction of the first coordinate system, the Y-Z plane of the first coordinate system being perpendicular to the orientation of the first depth information collecting device. It will thus be appreciated that the first coordinate system is varied with the movement of the first depth information acquisition device.
In step 202, the second position may be a position where the preset marker falls within the field of view of the second depth information collecting device, or a position where the rotating mechanism rotates to face the marker. As shown in fig. 1, the second position may be a position rotated 90 ° counterclockwise from the first position shown in fig. 1. The second depth information collecting device may collect depth information of a plurality of collecting points on the marker, and a specific process of depth information collection is similar to that of the first depth information collecting device, which is not described herein again. The subsequent second depth information collecting device may also obtain coordinates of the plurality of collecting points in the second coordinate system as in the first depth information collecting device described above.
Referring to fig. 1, a second coordinate system is shown with reference numeral C2, and the origin of coordinates of the second coordinate system is the center point of the second depth information collecting apparatus. The three directions of the second coordinate system and the orientation of the second depth information collecting device are associated, and the orientation of the second depth information collecting device may be set to the X direction of the second coordinate system, the Y-Z plane of the second coordinate system being perpendicular to the orientation of the second depth information collecting device.
As described above, the first depth information collecting apparatus is calibrated in advance. By "calibrated" is meant here that the positional relationship between the first depth information acquisition device and the rotatable body of the rotation mechanism is known, in which case also the first transformation relationship for the coordinate transformation between the first coordinate system and the reference coordinate system is known. Referring to fig. 1, a reference coordinate system is shown with reference numeral C4, and the origin of coordinates of the reference coordinate system is the center point of the rotatable body of the rotation mechanism. The three directions of the reference coordinate system are associated with the posture of the rotatable body, and the height direction of the rotatable body may be set as the Z direction of the reference coordinate system, and the X direction of the reference coordinate system may be the front side direction of the rotatable body, for example, so that it is understood that the reference coordinate system rotates with the rotatable body during the rotation of the rotatable body, particularly, the X-Y direction of the reference coordinate system.
Then, in step 203, the first coordinate data set may be converted into a reference coordinate system according to a predetermined first transformation relationship, resulting in a third coordinate data set after the position transformation. The third coordinate data set represents coordinate data of a plurality of acquisition points of the marker in a reference coordinate system.
In step 204, a control device associated with the rotating mechanism (e.g., a memory of the excavation machine) may record the angle of rotation of the rotatable body of the rotating mechanism as it rotates from the first position to the second position, thereby determining a difference in rotation angle between the first position and the second position. It should be noted that the movement from the first position to the second position only includes a rotation process of the rotatable body, and no translational process of the rotation mechanism exists.
In step 205, a second transformation relationship is determined from the second coordinate data set, the third coordinate data set, and the rotation angle difference, the second transformation relationship being used for coordinate transformation between the second coordinate system and the reference coordinate system. After the second transformation relation is determined, the position relation between the second depth information acquisition equipment and the rotatable main body of the rotating mechanism can be further obtained, and the calibration of the second depth information acquisition equipment is completed.
Fig. 3 illustrates a flowchart of a method 300 of acquiring a first set of coordinate data and a second set of coordinate data, according to an embodiment of the present disclosure, as illustrated in fig. 3, the method 300 comprising:
step 301, respectively acquiring a plurality of first candidate coordinate data of a plurality of acquisition points of a marker under a first coordinate system by a first depth information acquisition device;
step 302, determining at least one key point from a plurality of acquisition points, wherein the at least one key point is a point where the geometric feature of the marker is located;
step 303, selecting first candidate coordinate data corresponding to at least one key point from a plurality of first candidate coordinate data to form a first coordinate data set;
step 304, respectively acquiring a plurality of second candidate coordinate data of a plurality of acquisition points of the marker under a second coordinate system by a second depth information acquisition device; and
and step 305, selecting second candidate coordinate data corresponding to at least one key point from the plurality of second candidate coordinate data to form a second coordinate data set.
As described above, the positions and the number of the plurality of acquisition points can be set by setting the acquisition parameters of the depth information acquisition apparatus, for example, 1000 acquisition points can be set, which are uniformly distributed on the marker. In step 301, a plurality of first candidate coordinate data of the acquisition points in a first coordinate system are acquired for subsequent screening.
In step 302, the at least one key point may be, for example, a point at which a geometric feature exists, such as a corner point of a marker, a contour point, or an intersection point of contour lines. The determination of the key points may be obtained by performing data analysis on a plurality of first candidate coordinate data. In this embodiment, if the coordinate range in the X direction of the plurality of first candidate coordinate data is between X1 and X2, points whose X-direction coordinates are X1 or X2 may be determined as contour points on the marker, and these points may be subsequently determined as key points. In other embodiments, other types of keypoints may be determined in other ways, and the methods of determining keypoints are well known to those skilled in the art and will not be described in detail here. After step 302, the IDs of these keypoints may also be recorded, which are used to mark which of the 1000 acquisition points described above are.
In this embodiment, the coordinate data corresponding to the key points where the geometric features of the selection markers are located form the first coordinate data set. The key points can calibrate the positions of the markers more accurately relative to the common acquisition points, so that the second transformation relation obtained later is more accurate.
In step 304, coordinate data of 1000 acquisition points may be acquired using the second depth information acquiring apparatus in the same manner as step 301. However, unlike step 304, the coordinate data acquired by the second depth information acquiring apparatus is data of the acquisition point in the second coordinate system.
In step 305, the coordinate data corresponding to the key point is determined from the plurality of second candidate coordinate data in step 304. As described above, after step 302, the IDs of the keypoints have been recorded, and then in step 305, the data of the acquisition points identical to those recorded with these IDs can be screened out, which correspond to the coordinate data of the keypoints. After step 305, each data in the first coordinate data set may be associated with corresponding data in the second coordinate data set, in particular for the same key point, the corresponding first and second coordinate data forming a coordinate pair for subsequent determination of the transformation relationship.
In the method of the present embodiment, the second coordinate data set also includes coordinate data of the key points of the markers, so that the data in the first coordinate data set can be subsequently correlated and form a coordinate pair. These coordinate pairs may then be used in a correlation calculation process to determine the transformation relationship.
Fig. 4 shows a flowchart of a method 400 of calibrating a depth information acquisition device on a rotating mechanism according to an embodiment of the present disclosure, as shown in fig. 4, the method 400 comprising:
step 401, acquiring a first coordinate data set of a preset marker in a first coordinate system by a first depth information acquisition device in response to rotation of a rotatable body to a first position;
step 402, acquiring a predetermined first transformation matrix;
step 403, applying the first transformation matrix to the first coordinate data set to obtain a third coordinate data set;
step 404, acquiring a second coordinate data set of the marker in a second coordinate system by a second depth information acquisition device in response to the rotatable body rotating to a second position;
step 405, determining a rotation transformation matrix according to the rotation angle difference, wherein the rotation transformation matrix is used for coordinate transformation of the reference coordinate system between the first position and the second position;
step 406 of determining a second transformation matrix such that the fourth set of coordinate data and the third set of coordinate data are equal, wherein the fourth set of coordinate data is obtained by applying the second transformation matrix and the rotation transformation matrix to the second set of coordinate data, respectively
In step 401, a first set of coordinate data k1= { p11, p12, …, p1n }, where n is the number of keypoints, is acquired. In step 402, a predetermined first transformation matrix T is obtained 14 The first transformation matrix T 14 For converting coordinates in the first coordinate system C1 to the current reference coordinate system C4. In step 403, T is set 14 Respectively acting on each data in K1 to obtain a third coordinate data set K3= { p31, p32, …, p3n }. The third coordinate data set represents coordinate data of the key point in a reference coordinate system when the rotatable body is located at the first position.
In step 404, a first set of coordinate data k2= { p21, p22, …, p2n }, where n is the number of keypoints, where the keypoints coincide with the keypoints in step 401.
In step 405, a rotational transformation matrix is used for coordinate transformation of the reference coordinate system between the first position and the second position. As described above, since the reference coordinate system is a coordinate system created with the rotatable body as the origin, the reference coordinate system and the coordinates of the points in the reference coordinate system also change as the rotatable body rotates, i.e., the rotational transformation matrix is used for coordinate transformation in the reference coordinate when the points are in the first position and in the reference coordinate when the same points are in the second position.
In step 406, the fourth coordinate data set is denoted as T 4 ＝T c ×T 24 ×P 2 Wherein T is 24 Representing the second transformation matrix, T 24 ×P 2 Coordinate data representing the key point in a reference coordinate system when the rotatable body is in the second position, T 4 Global representation pair T 24 ×P 2 Coordinate data after coordinate transformation from the second position to the first position is performed. Thus, it can be appreciated that the following equation exists:
T 4 ＝T c ×T 24 ×P 2 ＝T 3 ＝T 14 ×P 1
wherein P is 1 Represents any point in K1, where P 2 Represents any point in K2. The second transformation matrix T to be determined can be obtained by the above equation 24 A second transformation relation for the coordinate transformation between the second coordinate system and the reference coordinate system is obtained.
The second transformation matrix T obtained by the determination can also be utilized in the following specific application 24 Further calculating to obtain a transformation relation T between the first coordinate system and the second coordinate system 12 T can be calculated, for example, by the following formula 12 ：
T 12 ＝T 14 ×T 24 -1
After T is obtained 12 Then, the coordinate data of any object obtained by the first depth information collecting device can be converted into the coordinate system of the second depth information collecting device, so that the coordinate corresponding relation between the two depth information collecting devices is established. That is, after the first depth information collecting device collects the depth image of the target object, the target object can be estimated even if the target object is not present in the field of view of the second depth information collecting device A depth image of the body within the second depth information acquisition device.
The method 100 or the method 400 described above can also be extended to the case of a plurality of depth information acquisition devices to be calibrated. As shown in fig. 1, in addition to the second depth information acquisition device, there may be a third depth information acquisition device to be calibrated. The calibration may be performed using the method 100 or the method 400 described above for each of all of these depth information gathering devices to be calibrated. Therefore, under the condition that a plurality of depth cameras to be calibrated exist, only one calibrated first depth information acquisition device can be used for calibrating each of the plurality of depth information acquisition devices to be calibrated, and the calibration process is further simplified.
According to another aspect of the present disclosure, there is also provided a calibration apparatus 500 for a depth information collecting device on a rotating mechanism, and fig. 5 shows a block diagram of a structure of the calibration apparatus 500 for a depth information collecting device on a rotating mechanism according to an embodiment of the present disclosure. The rotary mechanism includes rotatable main part, sets up the first degree of depth information acquisition equipment that marks on rotatable main part and sets up the second degree of depth information acquisition equipment that waits to mark on rotatable main part, as shown in fig. 5, and calibration device includes: the first depth information acquisition device 510 is configured to acquire a first coordinate data set of a preset marker under a first coordinate system in response to rotation of the rotatable body to a first position, wherein the first coordinate system is established by taking the position of the first depth information acquisition device as an origin; a second depth information acquisition device 520 configured to acquire a second coordinate data set of the marker under a second coordinate system in response to the rotatable body rotating to a second position, wherein the second coordinate system is established with the position of the second depth information acquisition device as an origin; a first conversion unit 530 configured to convert the first coordinate data set into a reference coordinate system according to a predetermined first conversion relationship, to obtain a third coordinate data set, wherein the reference coordinate system is established with the position of the rotatable body as an origin, wherein the first conversion relationship is used for coordinate conversion between the first coordinate system and the reference coordinate system; a first determining unit 540 configured to determine a rotation angle difference between the first position and the second position; and a second determining unit 550 configured to determine a second transformation relationship according to the second coordinate data set, the third coordinate data set, and the rotation angle difference, wherein the second transformation relationship is used for coordinate transformation between the second coordinate system and the reference coordinate system.
In some embodiments, the first depth information acquisition device comprises: the first acquisition module is configured to respectively acquire a plurality of first candidate coordinate data of a plurality of acquisition points of the marker under a first coordinate system by the first depth information acquisition equipment; a first determination module configured to determine at least one keypoint from a plurality of acquisition points, wherein the at least one keypoint is a point at which a geometric feature of the marker is located; and a first selection module configured to select first candidate coordinate data corresponding to at least one key point from the plurality of first candidate coordinate data to compose a first coordinate data set.
In some embodiments, the second depth information collecting apparatus includes: a second acquisition module configured to acquire, by a second depth information acquisition device, a plurality of second candidate coordinate data of a plurality of acquisition points of the marker in a second coordinate system, respectively; and a second selection module configured to select second candidate coordinate data corresponding to at least one key point from the plurality of second candidate coordinate data to compose a second coordinate data set.
In some embodiments, the first transformation relationship comprises a first transformation matrix, wherein the first transformation unit comprises: a third acquisition module configured to acquire a predetermined first transformation matrix; and an action module configured to act on the first coordinate data set with the first transformation matrix to obtain a third coordinate data set.
In some embodiments, the second transformation relation comprises a second transformation matrix, wherein the second determining unit comprises: a second determination module configured to determine a rotational transformation matrix according to the rotational angle difference, wherein the rotational transformation matrix is used for coordinate transformation of the reference coordinate system between the first position and the second position; and a third determination module configured to determine the second transformation matrix such that the fourth coordinate data set and the third coordinate data set are equal, wherein the fourth coordinate data set is obtained by applying the second transformation matrix and the rotation transformation matrix to the second coordinate data set, respectively.
In some embodiments, the calibration device further comprises: a third determination unit configured to determine a second transformation relationship based on the second coordinate data set, the third coordinate data set, and the rotation angle difference: a third transformation relationship is determined from the first transformation relationship and the second transformation relationship, wherein the third transformation relationship is used for coordinate transformation between the first coordinate system and the second coordinate system.
According to embodiments of the present disclosure, there is also provided an electronic device, a readable storage medium and a computer program product.
In the technical scheme of the disclosure, the related processes of collecting, storing, using, processing, transmitting, providing, disclosing and the like of the personal information of the user accord with the regulations of related laws and regulations, and the public order colloquial is not violated.
Referring to fig. 6, a block diagram of an electronic device 600 that may be a server or a client of the present disclosure, which is an example of a hardware device that may be applied to aspects of the present disclosure, will now be described. Electronic devices are intended to represent various forms of digital electronic computer devices, such as laptops, desktops, workstations, personal digital assistants, servers, blade servers, mainframes, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital processing, cellular telephones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions, are meant to be exemplary only, and are not meant to limit implementations of the disclosure described and/or claimed herein.
As shown in fig. 6, the electronic device 600 includes a computing unit 601 that can perform various appropriate actions and processes according to a computer program stored in a Read Only Memory (ROM) 602 or a computer program loaded from a storage unit 608 into a Random Access Memory (RAM) 603. In the RAM603, various programs and data required for the operation of the electronic device 600 can also be stored. The computing unit 601, ROM 602, and RAM603 are connected to each other by a bus 604. An input/output (I/O) interface 605 is also connected to bus 604.
A number of components in the electronic device 600 are connected to the I/O interface 605, including: an input unit 606, an output unit 607, a storage unit 608, and a communication unit 609. The input unit 606 may be any type of device capable of inputting information to the electronic device 600, the input unit 606 may receive input numeric or character information and generate key signal inputs related to user settings and/or function control of the electronic device, and may include, but is not limited to, a mouse, a keyboard, a touch screen, a trackpad, a trackball, a joystick, a microphone, and/or a remote control. The output unit 607 may be any type of device capable of presenting information and may include, but is not limited to, a display, speakers, video/audio output terminals, vibrators, and/or printers. Storage unit 608 may include, but is not limited to, magnetic disks, optical disks. The communication unit 609 allows the electronic device 600 to exchange information/data with other devices through a computer network, such as the internet, and/or various telecommunications networks, and may include, but is not limited to, modems, network cards, infrared communication devices, wireless communication transceivers and/or chipsets, such as bluetooth (TM) devices, 802.11 devices, wiFi devices, wiMax devices, cellular communication devices, and/or the like.
The computing unit 601 may be a variety of general and/or special purpose processing components having processing and computing capabilities. Some examples of computing unit 601 include, but are not limited to, a Central Processing Unit (CPU), a Graphics Processing Unit (GPU), various specialized Artificial Intelligence (AI) computing chips, various computing units running machine learning model algorithms, a Digital Signal Processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit 601 performs the various methods and processes described above, such as calibration methods for a depth information acquisition device on a rotating mechanism. For example, in some embodiments, the calibration method for a depth information gathering device on a rotating mechanism may be implemented as a computer software program tangibly embodied on a machine-readable medium, such as storage unit 608. In some embodiments, part or all of the computer program may be loaded and/or installed onto the electronic device 600 via the ROM 602 and/or the communication unit 609. When the computer program is loaded into the RAM 603 and executed by the computing unit 601, one or more steps of the calibration method for a depth information collecting device on a rotating mechanism described above may be performed. Alternatively, in other embodiments, the computing unit 601 may be configured to perform the calibration method for the depth information acquisition device on the rotating mechanism by any other suitable means (e.g. by means of firmware).
Various implementations of the systems and techniques described here above may be implemented in digital electronic circuitry, integrated circuit systems, field Programmable Gate Arrays (FPGAs), application Specific Integrated Circuits (ASICs), application Specific Standard Products (ASSPs), systems On Chip (SOCs), complex Programmable Logic Devices (CPLDs), computer hardware, firmware, software, and/or combinations thereof. These various embodiments may include: implemented in one or more computer programs, the one or more computer programs may be executed and/or interpreted on a programmable system including at least one programmable processor, which may be a special purpose or general-purpose programmable processor, that may receive data and instructions from, and transmit data and instructions to, a storage system, at least one input device, and at least one output device.
Program code for carrying out methods of the present disclosure may be written in any combination of one or more programming languages. These program code may be provided to a processor or controller of a general purpose computer, special purpose computer, or other programmable data processing apparatus such that the program code, when executed by the processor or controller, causes the functions/operations specified in the flowchart and/or block diagram to be implemented. The program code may execute entirely on the machine, partly on the machine, as a stand-alone software package, partly on the machine and partly on a remote machine or entirely on the remote machine or server.
In the context of this disclosure, a machine-readable medium may be a tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device. The machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. The machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. More specific examples of a machine-readable storage medium would include an electrical connection based on one or more wires, a portable computer diskette, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.
To provide for interaction with a user, the systems and techniques described here can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to a user; and a keyboard and pointing device (e.g., a mouse or trackball) by which a user can provide input to the computer. Other kinds of devices may also be used to provide for interaction with a user; for example, feedback provided to the user may be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user may be received in any form, including acoustic input, speech input, or tactile input.
The systems and techniques described here can be implemented in a computing system that includes a background component (e.g., as a data server), or that includes a middleware component (e.g., an application server), or that includes a front-end component (e.g., a user computer having a graphical user interface or a web browser through which a user can interact with an implementation of the systems and techniques described here), or any combination of such background, middleware, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: local Area Networks (LANs), wide Area Networks (WANs), and the internet.
The computer system may include a client and a server. The client and server are typically remote from each other and typically interact through a communication network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other. The server may be a cloud server, a server of a distributed system, or a server incorporating a blockchain.
It should be appreciated that various forms of the flows shown above may be used to reorder, add, or delete steps. For example, the steps recited in the present disclosure may be performed in parallel, sequentially or in a different order, provided that the desired results of the disclosed aspects are achieved, and are not limited herein.
Although embodiments or examples of the present disclosure have been described with reference to the accompanying drawings, it is to be understood that the foregoing methods, systems, and apparatus are merely exemplary embodiments or examples, and that the scope of the present invention is not limited by these embodiments or examples but only by the claims following the grant and their equivalents. Various elements of the embodiments or examples may be omitted or replaced with equivalent elements thereof. Furthermore, the steps may be performed in a different order than described in the present disclosure. Further, various elements of the embodiments or examples may be combined in various ways. It is important that as technology evolves, many of the elements described herein may be replaced by equivalent elements that appear after the disclosure.
Claims (18)
1. A calibration method for a depth information acquisition device on a rotating mechanism, wherein the rotating mechanism comprises a rotatable body, a calibrated first depth information acquisition device arranged on the rotatable body, and a second depth information acquisition device to be calibrated arranged on the rotatable body, the calibration method comprising:
In response to the rotatable body rotating to a first position and a second position respectively, acquiring a first coordinate data set of a preset marker in a first coordinate system by the first depth information acquisition device and a second coordinate data set of the marker in a second coordinate system by the second depth information acquisition device respectively;
converting the first coordinate data set into a reference coordinate system according to a predetermined first transformation relation to obtain a third coordinate data set; and
a second transformation relationship for coordinate transformation between the second coordinate system and the reference coordinate system is determined from the second coordinate data set, the third coordinate data set, and a rotation angle difference between the first position and the second position.
2. The calibration method of claim 1, wherein the acquiring, by the first depth information acquisition device, a first set of coordinate data of a preset marker in a first coordinate system and acquiring, by the second depth information acquisition device, a second set of coordinate data of the marker in a second coordinate system in response to the rotatable body rotating to a first position and a second position, respectively, comprises:
The first depth information acquisition equipment respectively acquires a plurality of first candidate coordinate data of a plurality of acquisition points of the marker under the first coordinate system;
determining at least one key point from the plurality of acquisition points, wherein the at least one key point is a point where the geometric feature of the marker is located; and
and selecting first candidate coordinate data corresponding to the at least one key point from the plurality of first candidate coordinate data to form the first coordinate data set.
3. The calibration method of claim 2, wherein the acquiring, by the first depth information acquisition device, a first set of coordinate data of a preset marker in a first coordinate system and acquiring, by the second depth information acquisition device, a second set of coordinate data of the marker in a second coordinate system in response to the rotatable body rotating to a first position and a second position, respectively, further comprises:
respectively acquiring a plurality of second candidate coordinate data of a plurality of acquisition points of the marker under the second coordinate system by the second depth information acquisition equipment; and
and selecting second candidate coordinate data corresponding to the at least one key point from the plurality of second candidate coordinate data to form a second coordinate data set.
4. The calibration method according to claim 1, wherein the first transformation relation includes a first transformation matrix, wherein the converting the first coordinate data set to a reference coordinate system where the rotatable body is located according to the predetermined first transformation relation, to obtain a third coordinate data set includes:
acquiring a predetermined first transformation matrix; and
and the first transformation matrix acts on the first coordinate data set to obtain the third coordinate data set.
5. The calibration method of claim 4, wherein the second transformation relationship comprises a second transformation matrix, wherein the determining a second transformation relationship from the second coordinate data set, the third coordinate data set, and a rotation angle difference between the first position and the second position comprises:
determining a rotation transformation matrix according to the rotation angle difference, wherein the rotation transformation matrix is used for coordinate transformation of the reference coordinate system between the first position and the second position;
determining the second transformation matrix such that a fourth coordinate data set is equal to the third coordinate data set, wherein the fourth coordinate data set is obtained by applying the second transformation matrix and the rotation transformation matrix to the second coordinate data set, respectively.
6. The calibration method according to any one of claims 1 to 5, further comprising:
after determining a second transformation relation from the second coordinate data set, the third coordinate data set, and a rotation angle difference between the first position and the second position:
and determining a third transformation relation according to the first transformation relation and the second transformation relation, wherein the third transformation relation is used for coordinate transformation between the first coordinate system and the second coordinate system.
7. The calibration method according to any one of claims 1 to 5, wherein,
the first coordinate system is established by taking the position of the first depth information acquisition equipment as an origin;
the second coordinate system is established by taking the position of the second depth information acquisition equipment as an origin;
the reference coordinate system is established with the position of the rotatable body as an origin.
8. A calibration method for a depth information acquisition device on a rotating mechanism, wherein the rotating mechanism comprises a rotatable body, a calibrated first depth information acquisition device arranged on the rotatable body and a plurality of second depth information acquisition devices to be calibrated arranged on the rotatable body, wherein the method comprises:
For each of the plurality of second depth information collecting devices,
calibration of the second depth information collecting apparatus using the calibration method according to any one of claims 1 to 5.
9. The calibration method of claim 8, wherein the rotating mechanism comprises an excavating machine, the rotatable body of the excavating machine comprising a plurality of sides, the first depth information acquisition device being disposed on a first one of the plurality of sides, the plurality of second depth information acquisition devices being disposed on a plurality of second one of the plurality of sides, respectively, different from the first side.
10. A calibration device for a depth information acquisition device on a rotating mechanism, wherein the rotating mechanism comprises a rotatable main body, a calibrated first depth information acquisition device arranged on the rotatable main body and a second depth information acquisition device to be calibrated arranged on the rotatable main body, the calibration device comprises:
the first depth information acquisition device is configured to acquire a first coordinate data set of a preset marker under a first coordinate system in response to the rotatable main body rotating to a first position;
The second depth information acquisition device is configured to acquire a second coordinate data set of the marker in a second coordinate system in response to the rotatable body rotating to a second position;
a first conversion unit configured to convert the first coordinate data set into a reference coordinate system according to a predetermined first conversion relation, to obtain a third coordinate data set;
a first determination unit configured to determine a rotation angle difference between the first position and the second position; and
a second determining unit configured to determine a second transformation relationship for coordinate transformation between the second coordinate system and the reference coordinate system from the second coordinate data set, the third coordinate data set, and the rotation angle difference.
11. The calibration device of claim 10, wherein the first depth information acquisition apparatus comprises:
a first acquisition module configured to acquire, by the first depth information acquisition device, a plurality of first candidate coordinate data of a plurality of acquisition points of the marker in the first coordinate system, respectively;
a first determination module configured to determine at least one keypoint from the plurality of acquisition points, wherein the at least one keypoint is a point at which a geometric feature of the marker is located; and
And a first selecting module configured to select first candidate coordinate data corresponding to the at least one key point from the plurality of first candidate coordinate data to form the first coordinate data set.
12. The calibration device of claim 11, wherein the second depth information acquisition apparatus comprises:
a second acquisition module configured to acquire, by the second depth information acquisition device, a plurality of second candidate coordinate data of a plurality of acquisition points of the marker in the second coordinate system, respectively; and
and a second selecting module configured to select second candidate coordinate data corresponding to the at least one key point from the plurality of second candidate coordinate data to compose the second coordinate data set.
13. The calibration device of claim 10, wherein the first transformation relation comprises a first transformation matrix, wherein the first transformation unit comprises:
a third acquisition module configured to acquire a predetermined first transformation matrix; and
and the action module is configured to act the first transformation matrix on the first coordinate data set to obtain the third coordinate data set.
14. The calibration device of claim 13, wherein the second transformation relation comprises a second transformation matrix, wherein the second determination unit comprises:
A second determination module configured to determine a rotation transformation matrix from the rotation angle difference, wherein the rotation transformation matrix is used for coordinate transformation of the reference coordinate system between the first position and the second position;
and a third determination module configured to determine the second transformation matrix such that a fourth coordinate data set obtained by applying the second transformation matrix and the rotation transformation matrix to the second coordinate data set, respectively, and the third coordinate data set are equal.
15. The calibration device of any one of claims 10 to 14, further comprising:
a third determination unit configured to determine a second transformation relationship after determining the second coordinate data set, the third coordinate data set, and the rotation angle difference: and determining a third transformation relation according to the first transformation relation and the second transformation relation, wherein the third transformation relation is used for coordinate transformation between the first coordinate system and the second coordinate system.
16. An electronic device, comprising:
at least one processor; and
a memory communicatively coupled to the at least one processor; wherein the method comprises the steps of
The memory stores instructions executable by the at least one processor to enable the at least one processor to perform the method of any one of claims 1-9.
17. A non-transitory computer readable storage medium storing computer instructions for causing the computer to perform the method of any one of claims 1-9.
18. A computer program product comprising a computer program, wherein the computer program, when executed by a processor, implements the method of any of claims 1-9.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210167055.6A CN116681770A (en) | 2022-02-23 | 2022-02-23 | Calibration method and calibration device for depth information acquisition equipment on rotating mechanism |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210167055.6A CN116681770A (en) | 2022-02-23 | 2022-02-23 | Calibration method and calibration device for depth information acquisition equipment on rotating mechanism |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN116681770A true CN116681770A (en) | 2023-09-01 |
Family
ID=87787786
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202210167055.6A Pending CN116681770A (en) | 2022-02-23 | 2022-02-23 | Calibration method and calibration device for depth information acquisition equipment on rotating mechanism |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN116681770A (en) |
-
2022
- 2022-02-23 CN CN202210167055.6A patent/CN116681770A/en active Pending
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN110322500B (en) | Optimization method and device for instant positioning and map construction, medium and electronic equipment | |
| CN109903337B (en) | Method and apparatus for determining pose of bucket of excavator | |
| CN109946680B (en) | External parameter calibration method and device of detection system, storage medium and calibration system | |
| CN112258567B (en) | Visual positioning method and device for object grabbing point, storage medium and electronic equipment | |
| CN110163087B (en) | Face gesture recognition method and system | |
| CN110276774B (en) | Object drawing method, device, terminal and computer-readable storage medium | |
| CN110349212B (en) | Optimization method and device for instant positioning and map construction, medium and electronic equipment | |
| CN115578433B (en) | Image processing method, device, electronic equipment and storage medium | |
| CN111708033A (en) | Coordinate system calibration method and device, electronic equipment and storage medium | |
| EP4047556A2 (en) | Registration method and registration apparatus for autonomous vehicle, electronic device | |
| JP7351892B2 (en) | Obstacle detection method, electronic equipment, roadside equipment, and cloud control platform | |
| CN113610702B (en) | Picture construction method and device, electronic equipment and storage medium | |
| US10146331B2 (en) | Information processing system for transforming coordinates of a position designated by a pointer in a virtual image to world coordinates, information processing apparatus, and method of transforming coordinates | |
| CN117392241A (en) | Sensor calibration method and device in automatic driving and electronic equipment | |
| CN113436233A (en) | Registration method and device of automatic driving vehicle, electronic equipment and vehicle | |
| CN114926549B (en) | Three-dimensional point cloud processing method, device, equipment and storage medium | |
| CN108801226B (en) | Plane inclination testing method and equipment | |
| CN114882115B (en) | Vehicle pose prediction method and device, electronic equipment and storage medium | |
| CN116681770A (en) | Calibration method and calibration device for depth information acquisition equipment on rotating mechanism | |
| CN114674328B (en) | Map generation method, map generation device, electronic device, storage medium, and vehicle | |
| CN111932611B (en) | Object position acquisition method and device | |
| US11244470B2 (en) | Methods and systems for sensing obstacles in an indoor environment | |
| CN115049744A (en) | Robot hand-eye coordinate conversion method and device, computer equipment and storage medium | |
| JP2023503750A (en) | ROBOT POSITIONING METHOD AND DEVICE, DEVICE, STORAGE MEDIUM | |
| CN115272605A (en) | Method, device and equipment for picking up three-dimensional model component and storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |