CN113902814B - Unified calibration method for multiple cameras on automatic crane sling - Google Patents

Unified calibration method for multiple cameras on automatic crane sling Download PDF

Info

Publication number
CN113902814B
CN113902814B CN202111480198.4A CN202111480198A CN113902814B CN 113902814 B CN113902814 B CN 113902814B CN 202111480198 A CN202111480198 A CN 202111480198A CN 113902814 B CN113902814 B CN 113902814B
Authority
CN
China
Prior art keywords
camera
coordinate system
spreader
hanger
container
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Active
Application number
CN202111480198.4A
Other languages
Chinese (zh)
Other versions
CN113902814A (en
Inventor
石先城
万金建
李恒
曹志俊
张涛
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Wuhan Guide Intelligent Technology Co ltd
Original Assignee
Wuhan Guide Intelligent Technology Co ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Wuhan Guide Intelligent Technology Co ltd filed Critical Wuhan Guide Intelligent Technology Co ltd
Priority to CN202111480198.4A priority Critical patent/CN113902814B/en
Publication of CN113902814A publication Critical patent/CN113902814A/en
Application granted granted Critical
Publication of CN113902814B publication Critical patent/CN113902814B/en
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/80—Analysis of captured images to determine intrinsic or extrinsic camera parameters, i.e. camera calibration
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T5/00—Image enhancement or restoration
    • G06T5/80—Geometric correction
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/10—Image acquisition modality
    • G06T2207/10004—Still image; Photographic image
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/20—Special algorithmic details
    • G06T2207/20084—Artificial neural networks [ANN]

Landscapes

  • Engineering & Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Theoretical Computer Science (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Control And Safety Of Cranes (AREA)
  • Length Measuring Devices By Optical Means (AREA)

Abstract

The invention provides a unified calibration method for multiple cameras on an automatic crane sling, which comprises the following steps: s1: correspondingly installing hanger cameras near lock heads of the crane hangers respectively to ensure that the view finding directions of the hanger cameras are vertically downward and are respectively aligned with lock holes at one end of the container far away from the ground; s2: constructing a camera coordinate system of each lifting appliance and a world coordinate system of the container; distortion correction is carried out on the keyhole images acquired by the four lifting appliance cameras respectively to obtain an internal reference matrix of each lifting appliance camera; s3: respectively carrying out keyhole detection on four keyhole images acquired by a hanger camera, and carrying out keyhole positioning; s4: extracting the edge of the lockhole by adopting an ellipse fitting algorithm, and acquiring the virtual circumscribed rectangle of the lockhole ellipse and extracting the image coordinates of four vertexes of the virtual circumscribed rectangle; s5: resolving an external parameter matrix of each lifting appliance camera relative to a container world coordinate system; s6: and solving the relative pose of each hanger camera coordinate system constructed by the hanger cameras.

Description

Unified calibration method for multiple cameras on automatic crane sling
Technical Field
The invention relates to the technical field of port intelligent hoisting equipment, in particular to a unified calibration method for multiple cameras on an automatic crane sling.
Background
With the rapid development of port intellectualization, the demand of ports for automatic box stacking is increasingly urgent. However, in the automatic box stacking, subsystems such as first-layer box opening, automatic box grabbing, automatic box stacking and the like need to estimate the relative pose relationship between the lifting appliance cameras, and then the box grabbing and the box placing are achieved. The importance of external reference calibration among pinhole cameras on the lifting appliance is self-evident, how to quickly and accurately estimate the pose relationship among lifting appliance phases, and the efficiency and the effect of automatic box stacking are directly related. The traditional calibration method is mainly used for calibrating external parameters among cameras in pin holes on a lifting appliance in modes of a calibration plate, obvious markers placed on the ground and the like. This method is relatively cumbersome.
Aiming at the traditional multi-camera calibration method, the calibration process is necessary to be simplified, and the camera calibration efficiency and the automatic box stacking efficiency of a crane are improved.
Disclosure of Invention
In view of the above, the present invention provides a unified calibration method for multiphase external parameters on an automatic crane spreader.
The technical scheme of the invention is realized as follows: the invention provides a unified calibration method for multiple cameras on an automatic crane sling, which comprises the following steps:
s1: correspondingly installing a spreader camera near a lock head of a crane spreader respectively, enabling the view finding direction of each spreader camera to be vertically downward and to be respectively aligned with a lock hole at one end of a container far away from the ground, and respectively acquiring images corresponding to four different lock holes on the end surface of the same container by the four spreader cameras;
s2: constructing a coordinate system of each hanger camera, and constructing a world coordinate system of the container; distortion correction is carried out on the keyhole images acquired by the four lifting appliance cameras respectively to obtain an internal reference matrix of each lifting appliance camera;
s3: respectively carrying out keyhole detection on four keyhole images acquired by a hanger camera, and carrying out keyhole positioning;
s4: extracting the edge of the lock hole from the positioned graph of the lock hole by adopting an ellipse fitting algorithm, acquiring a virtual circumscribed rectangle of the ellipse of the lock hole, and extracting image coordinates of four vertexes of the virtual circumscribed rectangle from an image coordinate system corresponding to each hanger camera;
s5: extracting image coordinates of each vertex of the virtual circumscribed rectangle from the internal reference matrix of the spreader camera obtained in the step S2 and the image coordinate system obtained in the step S4, and calculating the external reference matrix of each spreader camera relative to the container world coordinate system;
s6: and solving the relative pose of each hanger camera coordinate system constructed by the hanger cameras according to a coordinate transformation theory.
Based on the above technical solution, preferably, the obtaining of the internal reference matrix of each spreader camera in step S2 is to establish a camera coordinate system of each spreader camera, respectively, and let four spreader cameras be A, B, C and D, and the corresponding camera coordinate systems be A, B, C and D, respectively
Figure DEST_PATH_IMAGE002
、
Figure DEST_PATH_IMAGE004
、
Figure DEST_PATH_IMAGE006
And
Figure DEST_PATH_IMAGE008
(ii) a The Z-axis direction of each camera coordinate system is the direction that the optical axis of the hanger camera points to the lock hole at one end of the container far away from the ground, and the X-axis and the Y-axis of each camera coordinate system are respectively vertical to the Z-axis direction; constructing a container world coordinate system by using the geometric center of the upper surface of the container as an origin and the length extension direction, the width extension direction and the vertical downward direction of the container as coordinate axes
Figure DEST_PATH_IMAGE010
(ii) a Distortion correction is carried out on the keyhole images acquired by the four lifting appliance cameras respectively by adopting a Zhang friend calibration method: transforming the keyhole image in the world coordinate system of the container into a point in the pixel coordinate system of each hanger camera by each hanger camera to enable the internal reference matrix of each hanger camera
Figure DEST_PATH_IMAGE012
Wherein, in the step (A),
Figure DEST_PATH_IMAGE014
，
Figure DEST_PATH_IMAGE016
，
Figure DEST_PATH_IMAGE018
and
Figure DEST_PATH_IMAGE020
the number of pixels representing a unit distance in two axial directions of a two-dimensional coordinate in an image coordinate system;
Figure DEST_PATH_IMAGE022
is the focal length of each sling camera,
Figure DEST_PATH_IMAGE024
and
Figure DEST_PATH_IMAGE026
is the focal length
Figure 736080DEST_PATH_IMAGE022
Blending with pixel aspect ratio;
Figure DEST_PATH_IMAGE028
the radial distortion parameter indicates the included angle of the board edge calibrated by the chessboard adopted by the calibration method of shooting in the image coordinate system;
Figure DEST_PATH_IMAGE030
and
Figure DEST_PATH_IMAGE032
the coordinates of the center of the camera light-sensitive plate of the lifting appliance in a pixel coordinate system; the corresponding internal reference matrix is obtained for each of the spreader cameras A, B, C and D.
Preferably, in the step S3, the detection of the four keyhole images acquired by the spreader camera is performed by using a target detection algorithm YOLO V5 to detect and identify the region where the keyhole is located.
Preferably, the extracting of the image coordinates of the four vertexes of the virtual circumscribed rectangle is to select the image coordinates of points located at the edge of the keyhole area in an image coordinate system, construct an ellipse in a binomial fitting manner, and solve the central position, the major axis and the minor axis of the ellipse; and respectively offsetting the axial directions of the long axis and the short axis by the distances of the semi-short axis and the semi-long axis by taking the center of the ellipse as a reference, wherein coordinates of four vertexes of the virtual circumscribed rectangle enclosed by offsetting under the image coordinate system are the image coordinates of the vertexes to be extracted.
Preferably, the external reference matrix of each lifting appliance camera relative to the container world coordinate system is solved, and the extracted image coordinate of each keyhole virtual external rectangle vertex is
Figure DEST_PATH_IMAGE034
、
Figure DEST_PATH_IMAGE036
、
Figure DEST_PATH_IMAGE038
And
Figure DEST_PATH_IMAGE040
16 sets of coordinates in total; the 16 groups of image coordinates correspond to a container world coordinate system
Figure DEST_PATH_IMAGE041
Is defined as a three-dimensional coordinate of
Figure DEST_PATH_IMAGE043
、
Figure DEST_PATH_IMAGE045
、
Figure DEST_PATH_IMAGE047
And
Figure DEST_PATH_IMAGE049
(ii) a According to the hanger camera pinhole model, the following relationship exists for the hanger camera A:
Figure DEST_PATH_IMAGE051
formula 1;
Figure DEST_PATH_IMAGE053
formula 2;
Figure DEST_PATH_IMAGE055
formula 3;
Figure DEST_PATH_IMAGE057
formula 4;
since the internal reference matrix of each spreader camera has been obtained in step S2, the first term on the right side of the above equations 1, 2, 3 and 4
Figure DEST_PATH_IMAGE059
、
Figure DEST_PATH_IMAGE061
、
Figure DEST_PATH_IMAGE063
、
Figure DEST_PATH_IMAGE065
And
Figure DEST_PATH_IMAGE067
are all known in the art and are all known,
Figure DEST_PATH_IMAGE069
and
Figure DEST_PATH_IMAGE071
respectively represent two-dimensional image coordinates of the vertex of a virtual circumscribed rectangle of the lock hole corresponding to the hanger camera A,
Figure 100002_DEST_PATH_IMAGE073
、
Figure 100002_DEST_PATH_IMAGE075
and
Figure 100002_DEST_PATH_IMAGE077
respectively represent the three-dimensional coordinates of the vertex of the virtual circumscribed rectangle of the lock hole corresponding to the hanger camera A under the world coordinate system of the container,
Figure 100002_DEST_PATH_IMAGE079
；
Figure DEST_PATH_IMAGE081
scale factors from the container world coordinate system to the spreader camera; the equations are solved by a PnP algorithm to obtain the pose of the spreader camera A relative to the container
Figure DEST_PATH_IMAGE083
And
Figure DEST_PATH_IMAGE085
(ii) a In the same way, the positions and postures of the spreader cameras B, C and D relative to the container are respectively obtained
Figure DEST_PATH_IMAGE087
And
Figure DEST_PATH_IMAGE089
、
Figure DEST_PATH_IMAGE091
and
Figure DEST_PATH_IMAGE093
and
Figure DEST_PATH_IMAGE095
and
Figure DEST_PATH_IMAGE097
，
Figure DEST_PATH_IMAGE099
is the transpose of the zero matrix.
Preferably, the equation set is solved by adopting a PnP algorithm, namely, a P3P algorithm is adopted, that is, 3 of the image coordinates of the virtual circumscribed rectangle vertexes of each lock hole are selected, three-dimensional coordinates of the virtual circumscribed rectangle vertexes of the lock hole corresponding to the three points under a container world coordinate system are selected, three groups of corresponding image coordinates and three-dimensional coordinates are formed as feature points for calculation, and results of solving four groups of poses of the spreader camera relative to the container are formed, and the remaining 1 virtual circumscribed rectangle vertexes of the lock hole and the three-dimensional coordinates of the vertexes under the corresponding container world coordinate system are used as verification points; and respectively calculating the reprojection errors of the verification points according to the calculated result of each pose, and selecting the result of the pose with the minimum reprojection error as the pose of the spreader camera relative to the container.
Further preferably, the calculation of the relative pose of each spreader camera coordinate system in step S6 adopts the following method: according to the coordinate transformation theory, a point exists in the world coordinate system of the container
Figure DEST_PATH_IMAGE101
Then, the coordinate transformation relationship of the point in the spreader camera a is as follows:
Figure DEST_PATH_IMAGE103
，
Figure DEST_PATH_IMAGE104
and
Figure 78681DEST_PATH_IMAGE085
the same can obtain the point for the pose of the spreader camera A relative to the container
Figure 432302DEST_PATH_IMAGE101
Coordinate transformation relationships among spreader cameras B, C and D are
Figure DEST_PATH_IMAGE106
，
Figure DEST_PATH_IMAGE108
，
Figure DEST_PATH_IMAGE110
；
Figure DEST_PATH_IMAGE112
、
Figure DEST_PATH_IMAGE114
、
Figure DEST_PATH_IMAGE116
And
Figure DEST_PATH_IMAGE118
is a point
Figure 417445DEST_PATH_IMAGE101
Three-dimensional coordinates in a coordinate system corresponding to spreader cameras A, B, C and D; the above equations are combined, and the conversion relation between the hanger camera coordinate system corresponding to the hanger camera B and the hanger camera coordinate system corresponding to the hanger camera A is as follows:
Figure DEST_PATH_IMAGE120
then, the external reference matrix of the spreader camera B relative to the spreader camera a is: rotation matrix
Figure DEST_PATH_IMAGE122
Translation vector
Figure DEST_PATH_IMAGE124
Wherein
Figure DEST_PATH_IMAGE126
Is composed of
Figure 6558DEST_PATH_IMAGE087
The inverse matrix of (d); similarly, the transformation relationship between the spreader camera C and the spreader camera coordinate system of the spreader camera B is:
Figure DEST_PATH_IMAGE128
，
Figure DEST_PATH_IMAGE130
is that
Figure 127967DEST_PATH_IMAGE091
The inverse matrix of (d); the conversion relation between the coordinate systems of the hanger camera D relative to the hanger camera C is as follows:
Figure DEST_PATH_IMAGE132
，
Figure DEST_PATH_IMAGE134
is that
Figure DEST_PATH_IMAGE135
The inverse matrix of (c).
Compared with the prior art, the unified calibration method for the multiple cameras on the automatic crane lifting appliance has the following beneficial effects:
(1) according to the scheme, the hanger camera is correspondingly arranged at the position of the hanger lock head, images of each lock hole of the container are obtained, and target detection is realized on the lock holes through a deep learning algorithm; then accurately extracting boundary points of the lock hole through ellipse fitting, constructing an ellipse virtual circumscribed rectangle and a vertex thereof, solving the pose of each hanger camera relative to the container through the actual size of the lock hole, further acquiring the relative pose of each hanger camera, simplifying the calibration process of external parameters of the multi-phase machine, avoiding arranging auxiliary marks on the ground, and improving the efficiency of automatic container stacking of the container;
(2) solving an internal reference matrix of each spreader camera, and then solving the pose of the most reasonable spreader camera with the minimum error relative to the container by combining the projection relation between the virtual circumscribed rectangle vertex coordinate of the lock hole under the image coordinate system and the coordinate under the container world coordinate system;
(3) the method combining the image distortion correction, the keyhole elliptical contour fitting, the PnP algorithm and other solving means is adopted, so that the calculation process has better efficiency and precision.
Drawings
In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the embodiments or the prior art will be briefly described below, it is obvious that the drawings in the following description are only some embodiments of the present invention, and for those skilled in the art, other drawings can be obtained according to the drawings without creative efforts.
FIG. 1 is a flow chart of a unified calibration method for multiple cameras on an automated crane spreader according to the present invention;
FIG. 2 is a layout plan view of spreader cameras on the spreader in the unified calibration method for multiple cameras on the spreader of the automated crane according to the present invention;
FIG. 3 is a schematic diagram of a lock hole image acquired by a spreader camera of the unified calibration method for multiple cameras on an automatic crane spreader of the present invention;
FIG. 4 is a schematic diagram of the coordinate systems of the cameras of the unified calibration method for multiple cameras on the lifting appliance of the automatic crane according to the present invention
FIG. 5 is a schematic diagram of a world coordinate system of a container, lock holes, and a virtual rectangle circumscribing each lock hole according to the unified calibration method for multiple cameras on an automated crane spreader of the present invention;
FIG. 6 is a schematic diagram of the unified calibration method for multiple cameras on the lifting appliance of the automatic crane according to the present invention, wherein 20-foot cabinets are used for calibration;
FIG. 7 is a schematic diagram of the unified calibration method for multiple cameras on the automatic crane spreader according to the present invention, wherein a 40-foot cabinet is used for calibration verification.
Detailed Description
The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the embodiments of the present invention, and it is obvious that the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments. All other embodiments, which can be obtained by a person skilled in the art without any inventive step based on the embodiments of the present invention, are within the scope of the present invention.
As shown in fig. 1-5, the present invention provides a unified calibration method for multiple cameras on an automatic crane spreader, comprising the following steps:
s1: correspondingly installing a spreader camera near a lock head of a crane spreader respectively, enabling the view finding direction of each spreader camera to be vertically downward and to be respectively aligned with a lock hole at one end of a container far away from the ground, and respectively acquiring images corresponding to four different lock holes on the end surface of the same container by the four spreader cameras;
s2: constructing a coordinate system of each hanger camera, and constructing a world coordinate system of the container; distortion correction is carried out on the keyhole images acquired by the four lifting appliance cameras respectively to obtain an internal reference matrix of each lifting appliance camera;
the specific method comprises the following steps: as shown in fig. 4, the camera coordinate systems of the spreader cameras are respectively established, the four spreader cameras are A, B, C and D, and the corresponding camera coordinate systems are respectively
Figure DEST_PATH_IMAGE136
、
Figure DEST_PATH_IMAGE137
、
Figure 207085DEST_PATH_IMAGE006
And
Figure DEST_PATH_IMAGE138
(ii) a The Z-axis direction of each camera coordinate system is the direction that the optical axis of the hanger camera points to the lock hole at one end of the container far away from the ground, and the X-axis and the Y-axis of each camera coordinate system are respectively vertical to the Z-axis direction; as shown in fig. 5, a container world coordinate system is constructed with the geometric center of the upper surface of the container as the origin and the length extending direction, the width extending direction, and the vertically downward direction of the container as coordinate axes
Figure 148365DEST_PATH_IMAGE041
(ii) a Distortion correction is carried out on the keyhole images acquired by the four lifting appliance cameras respectively by adopting a Zhang friend calibration method: transforming the keyhole image in the world coordinate system of the container into a point in the pixel coordinate system of each hanger camera by each hanger camera to enable the internal reference matrix of each hanger camera
Figure DEST_PATH_IMAGE139
Wherein, in the step (A),
Figure 823060DEST_PATH_IMAGE014
，
Figure 2237DEST_PATH_IMAGE016
，
Figure DEST_PATH_IMAGE141
and
Figure DEST_PATH_IMAGE142
the number of pixels representing a unit distance in two axial directions of a two-dimensional coordinate in an image coordinate system;
Figure 635344DEST_PATH_IMAGE022
is the focal length of each sling camera,
Figure 408128DEST_PATH_IMAGE024
and
Figure DEST_PATH_IMAGE143
is the focal length
Figure 604623DEST_PATH_IMAGE022
Blending with pixel aspect ratio;
Figure 920198DEST_PATH_IMAGE028
the radial distortion parameter indicates the included angle of the board edge calibrated by the chessboard adopted by the calibration method of shooting in the image coordinate system;
Figure 520943DEST_PATH_IMAGE030
and
Figure 905657DEST_PATH_IMAGE032
the coordinates of the center of the camera light-sensitive plate of the lifting appliance in a pixel coordinate system; the corresponding internal reference matrix is obtained for each of the spreader cameras A, B, C and D. Such as
Figure DEST_PATH_IMAGE145
、
Figure DEST_PATH_IMAGE147
、
Figure DEST_PATH_IMAGE149
、
Figure DEST_PATH_IMAGE151
And
Figure DEST_PATH_IMAGE153
are respectively provided withThe internal parameters of the internal parameter matrix of the hanger camera A are represented, different cameras are distinguished by different footmarks, and the marking method of the internal parameters of the hanger cameras B, C is similar to that of the hanger camera D. Distortion correction is performed by adopting a Zhangyingyou calibration method, and belongs to the technical means known by the technical personnel in the field.
S3: respectively carrying out keyhole detection on four keyhole images acquired by a hanger camera, and carrying out keyhole positioning; in this embodiment, the four keyhole images acquired by the spreader camera are used for keyhole detection, and a target detection algorithm YOLO V5 is used for detecting and identifying the areas where the keyholes are located. The target detection algorithm YOLO is an abbreviation of You Only Look one, and the core idea is to predict the position and the type of a detected target, divide an image into a plurality of grids, enable each grid to be responsible for target detection in the grid, and predict the positioning confidence coefficient and the type probability vector of the target contained in all the grids at one time to complete the detection and the identification of the target at one time. The target detection algorithm YOLO V5 is an open source algorithm, and can obtain codes at websites such as gitubs, and belongs to the prior art known by those skilled in the art, and is not described herein again.
S4: extracting the edge of the lock hole from the positioned graph of the lock hole by adopting an ellipse fitting algorithm, acquiring a virtual circumscribed rectangle of the ellipse of the lock hole, and extracting image coordinates of four vertexes of the virtual circumscribed rectangle from an image coordinate system corresponding to each hanger camera;
after identifying the lockhole area, selecting the image coordinates of points positioned at the edge of the lockhole area in an image coordinate system, constructing an ellipse by a binomial fitting mode, fitting an elliptic curve according to the known image coordinates, and selecting an nlinfit function for fitting in MATLAB; solving the central position, the major axis and the minor axis of the ellipse; and respectively offsetting the distances of the semi-short axis and the semi-long axis of the long axis and the short axis to the axial direction of the image coordinate system symmetrically by taking the center of the ellipse as a reference, wherein the coordinates of four vertexes of the virtual circumscribed rectangle surrounded by offset under the image coordinate system are the image coordinates of the vertexes to be extracted.
S5: extracting image coordinates of each vertex of the virtual circumscribed rectangle from the internal reference matrix of the spreader camera obtained in the step S2 and the image coordinate system obtained in the step S4, and calculating the external reference matrix of each spreader camera relative to the container world coordinate system;
the specific method comprises the following steps: in step S4, the extracted image coordinates of the vertices of the virtual circumscribed rectangle for each keyhole are set to
Figure DEST_PATH_IMAGE154
、
Figure DEST_PATH_IMAGE155
、
Figure 843526DEST_PATH_IMAGE038
And
Figure DEST_PATH_IMAGE156
16 sets of coordinates in total; the 16 groups of image coordinates correspond to a container world coordinate system
Figure 262875DEST_PATH_IMAGE041
Is defined as a three-dimensional coordinate of
Figure DEST_PATH_IMAGE157
、
Figure 972205DEST_PATH_IMAGE045
、
Figure 578636DEST_PATH_IMAGE047
And
Figure DEST_PATH_IMAGE158
(ii) a According to the hanger camera pinhole model, the following relationship exists for the hanger camera A:
Figure 257879DEST_PATH_IMAGE051
formula 1;
Figure 16887DEST_PATH_IMAGE053
formula 2;
Figure 490594DEST_PATH_IMAGE055
formula 3;
Figure DEST_PATH_IMAGE159
formula 4;
since the internal reference matrix of each spreader camera has been obtained in step S2, the first term on the right side of the above equations 1, 2, 3 and 4
Figure 115479DEST_PATH_IMAGE059
、
Figure 942621DEST_PATH_IMAGE061
、
Figure 946349DEST_PATH_IMAGE063
、
Figure 715591DEST_PATH_IMAGE065
And
Figure 906401DEST_PATH_IMAGE067
are all known in the art and are all known,
Figure DEST_PATH_IMAGE160
and
Figure 68392DEST_PATH_IMAGE071
respectively represent two-dimensional image coordinates of the vertex of a virtual circumscribed rectangle of the lock hole corresponding to the hanger camera A,
Figure 785681DEST_PATH_IMAGE073
、
Figure 601190DEST_PATH_IMAGE075
and
Figure 279296DEST_PATH_IMAGE077
respectively represent the three-dimensional coordinates of the vertex of the virtual circumscribed rectangle of the lock hole corresponding to the hanger camera A under the world coordinate system of the container,
Figure 713820DEST_PATH_IMAGE079
；
Figure 426561DEST_PATH_IMAGE081
scale factors from the container world coordinate system to the spreader camera; the equations are solved by a PnP algorithm to obtain the pose of the spreader camera A relative to the container
Figure DEST_PATH_IMAGE161
And
Figure DEST_PATH_IMAGE162
(ii) a In the same way, the positions and postures of the spreader cameras B, C and D relative to the container are respectively obtained
Figure DEST_PATH_IMAGE163
And
Figure DEST_PATH_IMAGE164
、
Figure 131081DEST_PATH_IMAGE091
and
Figure DEST_PATH_IMAGE165
and
Figure DEST_PATH_IMAGE166
and
Figure DEST_PATH_IMAGE167
obtaining an external reference matrix of each sling camera,
Figure 765324DEST_PATH_IMAGE099
is the transpose of the zero matrix.
As a preferred mode, the PnP algorithm is a P3P algorithm, that is, 3 of the image coordinates of the vertices of the virtual circumscribed rectangles of each keyhole are selected, three-dimensional coordinates of the vertices of the virtual circumscribed rectangles of the keyhole corresponding to the three points in the world coordinate system of the container are selected, three sets of corresponding image coordinates and three-dimensional coordinates are formed as feature points for calculation, the results of four sets of poses of the spreader camera relative to the container are solved, and the remaining vertices of the virtual circumscribed rectangles of the keyhole and the three-dimensional coordinates of the vertices in the world coordinate system of the container corresponding to the three sets of corresponding image coordinates and three-dimensional coordinates are used as verification points; and respectively calculating the reprojection errors of the verification points according to the calculated result of each pose, and selecting the result of the pose with the minimum reprojection error as the pose of the spreader camera relative to the container.
S6: solving the relative pose of each hanger camera coordinate system constructed by hanger cameras according to a coordinate transformation theory; the method specifically comprises the following steps: let a point exist in the world coordinate system of the container
Figure 987227DEST_PATH_IMAGE101
Then, the coordinate transformation relationship of the point in the spreader camera a is as follows:
Figure DEST_PATH_IMAGE168
，
Figure DEST_PATH_IMAGE169
and
Figure DEST_PATH_IMAGE170
the same can obtain the point for the pose of the spreader camera A relative to the container
Figure 147950DEST_PATH_IMAGE101
Coordinate transformation relationships among spreader cameras B, C and D are
Figure DEST_PATH_IMAGE171
，
Figure DEST_PATH_IMAGE172
，
Figure 711786DEST_PATH_IMAGE110
；
Figure 630064DEST_PATH_IMAGE112
、
Figure 655658DEST_PATH_IMAGE114
、
Figure DEST_PATH_IMAGE173
And
Figure 483936DEST_PATH_IMAGE118
is a point
Figure 546570DEST_PATH_IMAGE101
Three-dimensional coordinates in a coordinate system corresponding to spreader cameras A, B, C and D; the above equations are combined, and the conversion relation between the hanger camera coordinate system corresponding to the hanger camera B and the hanger camera coordinate system corresponding to the hanger camera A is as follows:
Figure DEST_PATH_IMAGE174
then, the external reference matrix of the spreader camera B relative to the spreader camera a is: rotation matrix
Figure DEST_PATH_IMAGE175
Translation vector
Figure DEST_PATH_IMAGE176
Wherein
Figure DEST_PATH_IMAGE177
Is composed of
Figure DEST_PATH_IMAGE178
The inverse matrix of (d); similarly, the transformation relationship between the spreader camera C and the spreader camera coordinate system of the spreader camera B is:
Figure DEST_PATH_IMAGE179
，
Figure DEST_PATH_IMAGE181
in order to be a vector of rotation,
Figure DEST_PATH_IMAGE183
in order to translate the vector, the vector is translated,
Figure DEST_PATH_IMAGE185
is that
Figure DEST_PATH_IMAGE186
The inverse matrix of (d); the conversion relation between the coordinate systems of the hanger camera D relative to the hanger camera C is as follows:
Figure 857204DEST_PATH_IMAGE132
，
Figure DEST_PATH_IMAGE188
in order to be a matrix of rotations,
Figure DEST_PATH_IMAGE190
in order to translate the vector, the vector is translated,
Figure DEST_PATH_IMAGE191
is that
Figure DEST_PATH_IMAGE192
The inverse matrix of (c).
As shown in fig. 6-7, the multi-camera unified calibration method of the present embodiment is now verified, and the above calibration process is completed by using 20 containers, and the 20 containers are placed in the central position of the world coordinate system of the container, so that the external parameters of the spreader camera B relative to the spreader camera a are: rotation matrix
Figure DEST_PATH_IMAGE194
Translation vector
Figure DEST_PATH_IMAGE196
(ii) a Making the external parameters of the hanger camera C relative to the hanger camera B as follows: rotation matrix
Figure DEST_PATH_IMAGE198
Translation vector
Figure DEST_PATH_IMAGE200
(ii) a Making the external parameters of the hanger camera D relative to the hanger camera C as follows: rotation matrix
Figure DEST_PATH_IMAGE202
Translation vector
Figure DEST_PATH_IMAGE204
；
Then, the calibration verification is carried out by using 40-foot containers, and the 40-foot containers are arranged at the central position of the world coordinate system of the containers: the width of the flexible both wings of hoist removes the length of matching the container, when current 40 chi container calibration is verified, makes hoist camera B be for hoist camera A's external parameter: rotation matrix
Figure DEST_PATH_IMAGE206
Translation vector
Figure DEST_PATH_IMAGE208
(ii) a Making the external parameters of the hanger camera C relative to the hanger camera B as follows: rotation matrix
Figure DEST_PATH_IMAGE210
Translation vector
Figure DEST_PATH_IMAGE212
(ii) a Making the external parameters of the hanger camera D relative to the hanger camera C as follows: rotation matrix
Figure DEST_PATH_IMAGE214
Translation vector
Figure DEST_PATH_IMAGE216
；
The four spreader cameras A, B, C and D only generate horizontal displacement, and do not generate relative rotation between the spreader cameras, the spreader camera A or C at one end of the spreader in the horizontal direction is only in the X of the container world coordinate system relative to the spreader camera B or D at the other endWTranslating in the axial direction; the rotation matrix in the extrinsic parameters is substantially unchanged; x of translation vector of spreader camera A or C relative to spreader camera B or D in world coordinate system of containerWThe direction is twice of the translation vector when the 20-foot container is calibrated, and the translation amount in other axial directions is unchanged; therefore, the following relationship exists:
Figure DEST_PATH_IMAGE218
；
Figure DEST_PATH_IMAGE220
；
Figure DEST_PATH_IMAGE222
；
Figure DEST_PATH_IMAGE224
；
Figure DEST_PATH_IMAGE226
；
Figure DEST_PATH_IMAGE228
；
Figure DEST_PATH_IMAGE230
；
Figure DEST_PATH_IMAGE232
；
Figure DEST_PATH_IMAGE234
；
Figure DEST_PATH_IMAGE236
；
and the suffix of each translation vector respectively represents the components of the translation vector in the directions of three coordinate axes of the container world coordinate system. If the above relations are met, the calibration verification is successful, otherwise, the verification fails, and the verification needs to be performed again from the steps S2 to S6, and then the verification is performed again between the 20-size container and the 40-size container. The deviation of two sides of the approximate equal sign in the relation is not more than 5 percent, and the requirement can be considered to be met.
The above description is only for the purpose of illustrating the preferred embodiments of the present invention and is not to be construed as limiting the invention, and any modifications, equivalents, improvements and the like that fall within the spirit and principle of the present invention are intended to be included therein.

Claims (2)

1. The unified calibration method for the multiple cameras on the automatic crane sling is characterized by comprising the following steps of: the method comprises the following steps:
s1: correspondingly installing a spreader camera near a lock head of a crane spreader respectively, enabling the view finding direction of each spreader camera to be vertically downward and to be respectively aligned with a lock hole at one end of a container far away from the ground, and respectively acquiring images corresponding to four different lock holes on the end surface of the same container by the four spreader cameras;
s2: constructing a coordinate system of each hanger camera, and constructing a world coordinate system of the container; distortion correction is carried out on the keyhole images acquired by the four lifting appliance cameras respectively to obtain an internal reference matrix of each lifting appliance camera;
s3: respectively carrying out keyhole detection on four keyhole images acquired by a hanger camera, and carrying out keyhole positioning;
s4: extracting the edge of the lock hole from the positioned graph of the lock hole by adopting an ellipse fitting algorithm, acquiring a virtual circumscribed rectangle of the ellipse of the lock hole, and extracting image coordinates of four vertexes of the virtual circumscribed rectangle from an image coordinate system corresponding to each hanger camera;
s5: extracting image coordinates of each vertex of the virtual circumscribed rectangle from the internal reference matrix of the spreader camera obtained in the step S2 and the image coordinate system obtained in the step S4, and calculating the external reference matrix of each spreader camera relative to the container world coordinate system;
s6: solving the relative pose of each hanger camera coordinate system constructed by hanger cameras according to a coordinate transformation theory;
in the step S2, the obtaining of the internal reference matrix of each spreader camera is to establish a camera coordinate system of each spreader camera, where the four spreader cameras are A, B, C and D, and the corresponding camera coordinate systems are A, B, C and D, respectively
Figure 769129DEST_PATH_IMAGE001
、
Figure 51206DEST_PATH_IMAGE002
、
Figure 18025DEST_PATH_IMAGE003
And
Figure 727355DEST_PATH_IMAGE004
(ii) a The Z-axis direction of each camera coordinate system is the direction that the optical axis of the hanger camera points to the lock hole at one end of the container far away from the ground, and the X-axis and the Y-axis of each camera coordinate system are respectively vertical to the Z-axis direction; constructing a container world coordinate system by using the geometric center of the upper surface of the container as an origin and the length extension direction, the width extension direction and the vertical downward direction of the container as coordinate axes
Figure 143905DEST_PATH_IMAGE005
(ii) a Distortion correction is carried out on the keyhole images acquired by the four lifting appliance cameras respectively by adopting a Zhang friend calibration method: transforming the keyhole image in the world coordinate system of the container into a point in the pixel coordinate system of each hanger camera by each hanger camera to enable the internal reference matrix of each hanger camera
Figure 229673DEST_PATH_IMAGE006
Wherein
Figure 50998DEST_PATH_IMAGE007
，
Figure 931230DEST_PATH_IMAGE008
，
Figure 572427DEST_PATH_IMAGE009
And
Figure 461885DEST_PATH_IMAGE010
the number of pixels representing a unit distance in two axial directions of a two-dimensional coordinate in an image coordinate system;
Figure 137717DEST_PATH_IMAGE011
is the focal length of each sling camera,
Figure 454429DEST_PATH_IMAGE012
and
Figure 582922DEST_PATH_IMAGE013
is the focal length
Figure 10492DEST_PATH_IMAGE011
Blending with pixel aspect ratio;
Figure 540831DEST_PATH_IMAGE014
the radial distortion parameter indicates the included angle of the board edge calibrated by the chessboard adopted by the calibration method of shooting in the image coordinate system;
Figure 294023DEST_PATH_IMAGE015
and
Figure 664741DEST_PATH_IMAGE016
the coordinates of the center of the camera light-sensitive plate of the lifting appliance in a pixel coordinate system; respectively solving corresponding internal reference matrixes for the sling cameras A, B, C and D;
in the step S3, the detection of the four keyhole images acquired by the spreader camera is performed by using a target detection algorithm YOLO V5 to detect and identify the region where the keyhole is located;
the extraction of the image coordinates of the four vertexes of the virtual circumscribed rectangle is to select the image coordinates of points positioned at the edge of the lockhole area in an image coordinate system, construct an ellipse in a binomial fitting mode, and solve the central position, the major axis and the minor axis of the ellipse; respectively offsetting the axial directions of the long axis and the short axis by the distances of the semi-short axis and the semi-long axis by taking the center of the ellipse as a reference, wherein coordinates of four vertexes of a virtual circumscribed rectangle enclosed by offsetting under an image coordinate system are image coordinates of vertexes to be extracted;
each lifting appliance is solvedThe camera makes the image coordinate of the vertex of the virtual circumscribed rectangle of each keyhole as the extracted image coordinate of the external reference matrix of the world coordinate system of the container in step S4
Figure 896002DEST_PATH_IMAGE018
、
Figure 280847DEST_PATH_IMAGE020
、
Figure 204941DEST_PATH_IMAGE021
And
Figure 42447DEST_PATH_IMAGE023
16 sets of coordinates in total; the 16 groups of image coordinates correspond to a container world coordinate system
Figure 811820DEST_PATH_IMAGE025
Is defined as a three-dimensional coordinate of
Figure 51171DEST_PATH_IMAGE027
、
Figure 349428DEST_PATH_IMAGE029
、
Figure 674230DEST_PATH_IMAGE031
And
Figure 247294DEST_PATH_IMAGE033
(ii) a According to the hanger camera pinhole model, the following relationship exists for the hanger camera A:
Figure 338222DEST_PATH_IMAGE034
formula 1;
Figure 604119DEST_PATH_IMAGE035
formula 2;
Figure 416217DEST_PATH_IMAGE036
formula 3;
Figure 527392DEST_PATH_IMAGE037
formula 4;
since the internal reference matrix of each spreader camera has been obtained in step S2, the first term on the right side of the above equations 1, 2, 3 and 4
Figure 741336DEST_PATH_IMAGE038
、
Figure 912554DEST_PATH_IMAGE039
、
Figure 477528DEST_PATH_IMAGE040
、
Figure 392394DEST_PATH_IMAGE041
And
Figure 726424DEST_PATH_IMAGE042
are all known in the art and are all known,
Figure 334123DEST_PATH_IMAGE043
and
Figure 855234DEST_PATH_IMAGE044
respectively represent two-dimensional image coordinates of the vertex of a virtual circumscribed rectangle of the lock hole corresponding to the hanger camera A,
Figure 570861DEST_PATH_IMAGE045
、
Figure 759397DEST_PATH_IMAGE046
and
Figure 537997DEST_PATH_IMAGE047
respectively represent the three-dimensional coordinates of the vertex of the virtual circumscribed rectangle of the lock hole corresponding to the hanger camera A under the world coordinate system of the container,
Figure 546405DEST_PATH_IMAGE048
；
Figure 803074DEST_PATH_IMAGE049
scale factors from the container world coordinate system to the spreader camera; the equations are solved by a PnP algorithm to obtain the pose of the spreader camera A relative to the container
Figure 846116DEST_PATH_IMAGE050
And
Figure 795617DEST_PATH_IMAGE051
(ii) a In the same way, the positions and postures of the spreader cameras B, C and D relative to the container are respectively obtained
Figure 556900DEST_PATH_IMAGE052
And
Figure 351681DEST_PATH_IMAGE053
、
Figure 514809DEST_PATH_IMAGE054
and
Figure 635212DEST_PATH_IMAGE055
and
Figure 618211DEST_PATH_IMAGE056
and
Figure 213753DEST_PATH_IMAGE057
，
Figure 496967DEST_PATH_IMAGE058
is the transpose of the zero matrix;
in the step S6, the calculation of the relative pose of each spreader camera coordinate system is performed by the following method: according to the coordinate transformation theory, a point exists in the world coordinate system of the container
Figure 522692DEST_PATH_IMAGE059
Then, the coordinate transformation relationship of the point in the spreader camera a is as follows:
Figure 992987DEST_PATH_IMAGE060
，
Figure 395150DEST_PATH_IMAGE061
and
Figure 267291DEST_PATH_IMAGE062
the same can obtain the point for the pose of the spreader camera A relative to the container
Figure 729496DEST_PATH_IMAGE059
Coordinate transformation relationships among spreader cameras B, C and D are
Figure 687088DEST_PATH_IMAGE063
，
Figure 892941DEST_PATH_IMAGE064
，
Figure 885168DEST_PATH_IMAGE065
；
Figure 252695DEST_PATH_IMAGE066
、
Figure 697583DEST_PATH_IMAGE067
、
Figure 438619DEST_PATH_IMAGE068
And
Figure 285352DEST_PATH_IMAGE069
is a point
Figure 89360DEST_PATH_IMAGE059
Three-dimensional coordinates in a coordinate system corresponding to spreader cameras A, B, C and D; the above equations are combined, and the conversion relation between the hanger camera coordinate system corresponding to the hanger camera B and the hanger camera coordinate system corresponding to the hanger camera A is as follows:
Figure 755965DEST_PATH_IMAGE070
then, the external reference matrix of the spreader camera B relative to the spreader camera a is: rotation matrix
Figure 569200DEST_PATH_IMAGE071
Translation vector
Figure 4860DEST_PATH_IMAGE072
Wherein
Figure DEST_PATH_IMAGE073
Is composed of
Figure 448611DEST_PATH_IMAGE074
The inverse matrix of (d); similarly, the transformation relationship between the spreader camera C and the spreader camera coordinate system of the spreader camera B is:
Figure DEST_PATH_IMAGE075
，
Figure 805774DEST_PATH_IMAGE076
is that
Figure DEST_PATH_IMAGE077
The inverse matrix of (d); the conversion relation between the coordinate systems of the hanger camera D relative to the hanger camera C is as follows:
Figure 91875DEST_PATH_IMAGE078
，
Figure DEST_PATH_IMAGE079
is that
Figure 850883DEST_PATH_IMAGE080
The inverse matrix of (c).
2. The method for unified calibration of multiple cameras on an automated crane spreader according to claim 1, wherein: the equation set is solved by adopting a PnP algorithm, namely, a P3P algorithm is adopted, namely 3 image coordinates of the vertexes of the virtual circumscribed rectangles of the lock holes are selected, three-dimensional coordinates of the vertexes of the virtual circumscribed rectangles of the lock holes corresponding to the three points under a container world coordinate system are selected, three groups of corresponding image coordinates and three-dimensional coordinates are formed as feature points for calculation, the result of solving the pose of the spreader camera relative to the container is obtained, and the remaining 1 vertex of the virtual circumscribed rectangles of the lock holes and the three-dimensional coordinates of the vertexes under the corresponding container world coordinate system are used as verification points; and respectively calculating the reprojection errors of the verification points according to the calculated result of each pose, and selecting the result of the pose with the minimum reprojection error as the pose of the spreader camera relative to the container.
CN202111480198.4A 2021-12-07 2021-12-07 Unified calibration method for multiple cameras on automatic crane sling Active CN113902814B (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202111480198.4A CN113902814B (en) 2021-12-07 2021-12-07 Unified calibration method for multiple cameras on automatic crane sling

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202111480198.4A CN113902814B (en) 2021-12-07 2021-12-07 Unified calibration method for multiple cameras on automatic crane sling

Publications (2)

Publication Number Publication Date
CN113902814A CN113902814A (en) 2022-01-07
CN113902814B true CN113902814B (en) 2022-03-15

Family

ID=79025525

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202111480198.4A Active CN113902814B (en) 2021-12-07 2021-12-07 Unified calibration method for multiple cameras on automatic crane sling

Country Status (1)

Country Link
CN (1) CN113902814B (en)

Families Citing this family (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN115578237A (en) * 2022-09-22 2023-01-06 中车资阳机车有限公司 Lock hole positioning system and method for split container spreader

Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN101251381A (en) * 2007-12-29 2008-08-27 武汉理工大学 Dual container positioning system based on machine vision
WO2019188521A1 (en) * 2018-03-28 2019-10-03 京セラ株式会社 Image capture device, moving body, and image capture method
CN111704036A (en) * 2020-07-03 2020-09-25 上海驭矩信息科技有限公司 Lifting equipment alignment system and method
CN113012235A (en) * 2021-03-18 2021-06-22 武汉港迪智能技术有限公司 Port crane lifting appliance attitude control system and control method thereof
JP2021099722A (en) * 2019-12-23 2021-07-01 京セラ株式会社 Calibration device, imaging apparatus, movable body, and calibration method

Patent Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN101251381A (en) * 2007-12-29 2008-08-27 武汉理工大学 Dual container positioning system based on machine vision
WO2019188521A1 (en) * 2018-03-28 2019-10-03 京セラ株式会社 Image capture device, moving body, and image capture method
JP2021099722A (en) * 2019-12-23 2021-07-01 京セラ株式会社 Calibration device, imaging apparatus, movable body, and calibration method
CN111704036A (en) * 2020-07-03 2020-09-25 上海驭矩信息科技有限公司 Lifting equipment alignment system and method
CN113012235A (en) * 2021-03-18 2021-06-22 武汉港迪智能技术有限公司 Port crane lifting appliance attitude control system and control method thereof

Non-Patent Citations (1)

* Cited by examiner, † Cited by third party
Title
基于机器视觉的吊具位姿检测系统;倪菲;《起重运输机械》;20111031;43-47 *

Also Published As

Publication number Publication date
CN113902814A (en) 2022-01-07

Similar Documents

Publication Publication Date Title
WO2019200837A1 (en) Method and system for measuring volume of parcel, and storage medium and mobile terminal
CN106249881B (en) Augmented reality view field space and virtual three-dimensional target dynamic method for registering
US10846844B1 (en) Collaborative disparity decomposition
US8600192B2 (en) System and method for finding correspondence between cameras in a three-dimensional vision system
CN108734744A (en) A kind of remote big field-of-view binocular scaling method based on total powerstation
CN113902814B (en) Unified calibration method for multiple cameras on automatic crane sling
CN105096317A (en) Fully automatic calibration method for high performance camera under complicated background
JP2004213332A (en) Calibration device, calibration method, program for calibration, and calibration mending tool
JP5432835B2 (en) How to calibrate the camera
KR20130026741A (en) Method and apparatus for providing automated detection of calibration
CN103049731B (en) Decoding method for point-distributed color coding marks
CN104766309A (en) Plane feature point navigation and positioning method and device
CN109784250A (en) The localization method and device of automatically guiding trolley
Liu et al. Pose estimation in heavy clutter using a multi-flash camera
WO2023273108A1 (en) Monocular distance measurement method and apparatus, and intelligent apparatus
CN115880372A (en) Unified calibration method and system for external hub positioning camera of automatic crane
CN116077184A (en) Positioning device and method of AR operation navigation system
CN104881641B (en) Mobile device-based questionnaire and table Digital identification method and system
CN110223356A (en) A kind of monocular camera full automatic calibration method based on energy growth
US20230245346A1 (en) Method and device for corner marking and parameter calibration
WO2023005195A1 (en) Map data processing method and apparatus, and household appliance and readable storage medium
CN108205641A (en) Images of gestures processing method and processing device
Karimi et al. A new method for automatic and accurate coded target recognition in oblique images to improve augmented reality precision
WO2023077827A1 (en) Three-dimensional tower-type checkerboard for multi-sensor calibration, and lidar-camera joint calibration method based on checkerboard
CN103177416B (en) A kind of QR code image position method based on least square method

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
GR01 Patent grant
GR01 Patent grant