CN115830103A - Monocular color-based transparent object positioning method and device and storage medium - Google Patents
Monocular color-based transparent object positioning method and device and storage medium Download PDFInfo
- Publication number
- CN115830103A CN115830103A CN202211501197.8A CN202211501197A CN115830103A CN 115830103 A CN115830103 A CN 115830103A CN 202211501197 A CN202211501197 A CN 202211501197A CN 115830103 A CN115830103 A CN 115830103A
- Authority
- CN
- China
- Prior art keywords
- distortion
- coordinate system
- image
- transparent object
- pixel
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Landscapes
- Length Measuring Devices By Optical Means (AREA)
Abstract
The invention relates to a monocular color-based transparent object positioning method, a monocular color-based transparent object positioning device and a storage medium, which are applied to the technical field of visual positioning and comprise the following steps: the method comprises the steps of shooting a calibration image of a calibration plate, obtaining angular point information of the calibration image in multiple angles, solving internal reference, external reference and distortion parameters of a camera through world coordinates and pixel coordinates according to the obtained angular point information, namely the world coordinates and the pixel coordinates of the calibration image, obtaining an image of a transparent object through a monocular camera, correcting the transparent image through the calculated parameters, and reversely pushing world system coordinates of the transparent object according to the mapping relation between the world system coordinates and the pixel coordinates according to the pixel coordinates of the shot transparent image, so that the accurate positioning of the transparent object through the monocular camera is realized, and the problems that in the prior art, the depth information cannot be accurately obtained through a method based on a depth monocular camera and a method based on binocular vision, and the positioning is not accurate are solved.
Description
Technical Field
The invention relates to the technical field of visual positioning, in particular to a monocular color based transparent object positioning method and device and a storage medium.
Background
The vision technology changes the mode of the robot in industrial production by virtue of the characteristics of the robot. By adopting the image processing technology in machine vision, the target can be identified and positioned without a high-precision tool clamp, so that the flexibility of the industrial robot in the use process is greatly improved, and the application range and the practical application value of the industrial robot are expanded. The mainstream positioning detection methods at present are classified into a method based on a depth monocular camera and a method based on binocular vision, but the methods have defects in practical application, and depth information cannot be captured for a transparent object, so that the application in industry is influenced to a great extent;
based on object identification and pose estimation of the depth monocular camera, the monocular camera is calibrated firstly, then a final point cloud picture is obtained through a mapping relation of the depth picture and the point cloud picture, and finally preprocessing is carried out according to an actual scene. The method can accurately and reliably acquire the three-dimensional information of the object.
The principle of mainstream depth cameras such as KinectV2 and Realsense is depth estimation based on an infrared sensor. The infrared projector actively projects near infrared spectrum, and random reflection spots can be formed after the near infrared spectrum irradiates a rough object or penetrates through frosted glass, so that the near infrared spectrum can be collected by the infrared monocular camera. And calculating the depth map of the human body or the object in the visible range by analyzing the infrared image. However, for the application scene in the invention, the transparent object cannot correctly obtain the depth information thereof, so that the positioning cannot be carried out;
however, the binocular monocular camera has higher cost compared with the monocular camera, and has difficulty in a feature point matching algorithm, so that the application of binocular stereoscopic vision in the field of industrial robots is restricted to a great extent.
Disclosure of Invention
In view of the above, the present invention is directed to a monocular color based transparent object positioning method, apparatus and storage medium, so as to solve the problems in the prior art that depth information cannot be captured for a transparent object, and positioning is not accurate.
According to a first aspect of the embodiments of the present invention, there is provided a monocular color based transparent object positioning method, including:
selecting a plurality of calibration images of the checkerboard calibration plate shot at any angle, and calculating world coordinates and pixel coordinates of each checkerboard corner point on all the calibration images;
solving an internal reference matrix with distortion, an external reference matrix with distortion and distortion parameters according to world coordinates and pixel coordinates of each checkerboard corner point on the calibrated image;
solving the distortion-free internal reference matrix according to the solved distorted internal reference matrix and the distortion parameter;
shooting an image of a transparent object on a fixed plane through a monocular camera, and correcting the image of the transparent object by using an internal reference matrix with distortion and distortion parameters;
and obtaining the pixel coordinates of the transparent object in the corrected image of the transparent object, reversely deducing the world coordinates of the transparent object according to the mapping relation between the world coordinate system and the pixel coordinate system and the solved distortion-free internal reference matrix and external reference matrix, and realizing the positioning of the transparent object through the world coordinates.
Preferably, the first and second electrodes are formed of a metal,
the photographing of the image of the transparent object on the fixed plane by the monocular camera includes:
shooting images of the lodging transparent objects at all angles through fixing the view angle of the monocular camera to form a data set;
adjusting the brightness of each image in the data set up and down on the basis of the original image, respectively rotating each image in the data set clockwise by a certain angle and counterclockwise by a certain angle, adding the adjusted images into the data set together, and expanding the data set;
the data set is trained through an object detection algorithm to obtain a detection model, and the detection model can identify the transparent object in the lodging state.
Preferably, the first and second electrodes are formed of a metal,
the mapping relation from the world coordinate system to the pixel coordinate system comprises:
the world coordinate system obtains a camera coordinate system through rigid body transformation, the camera coordinate system obtains an image coordinate system through perspective projection, and the image coordinate system obtains a pixel coordinate system through affine transformation;
the rigid body transformation includes: multiplying the world coordinate system by a rotation matrix and adding a translation matrix to obtain a camera coordinate system;
the perspective projection includes: converting a camera coordinate system into a planar two-dimensional coordinate system, namely an image coordinate system, according to the relation among the focal length, the object distance and the distance of lens imaging under the pinhole model;
the affine transformation includes: in a planar two-dimensional coordinate system, the coordinate origin of the image coordinate system is converted into the midpoint of the pixel coordinate system, so that the conversion from the image coordinate system to the pixel coordinate system is realized.
Preferably, the first and second electrodes are formed of a metal,
the distorted internal reference matrix comprises:
combining a mapping matrix from a monocular camera coordinate system to an image coordinate system and a mapping matrix from the image coordinate system to a pixel coordinate system to obtain an initial reference matrix;
setting an unknown number as a distortion parameter of the monocular camera, and determining a radial distortion formula and a tangential distortion formula according to the distortion parameter and a coordinate point of an image coordinate system;
combining a radial distortion formula and a tangential distortion formula to obtain a coordinate point of a distorted image coordinate system projected to a pixel coordinate system;
substituting the coordinate points of the distorted image coordinate system projected to the pixel coordinate system into the initial internal reference matrix to obtain an internal reference matrix with distortion;
and obtaining an external parameter matrix according to the mapping matrix from the world coordinate system to the camera coordinate system.
Preferably, the first and second liquid crystal display panels are,
the method for solving the internal reference matrix with distortion, the external reference matrix with distortion and the distortion parameter according to the world coordinates and the pixel coordinates of each checkerboard corner point on the calibration image comprises the following steps:
combining the distorted internal reference matrix and the distorted external reference matrix to obtain a single-point distorted camera imaging model;
inputting the world coordinates and the pixel coordinates of each checkerboard corner point into a single-point camera imaging model with distortion;
setting initial values for an internal parameter matrix with distortion, an external parameter matrix and distortion parameters, gradually minimizing the reprojection error through a linear optimization method to enable the error of the initial values to be minimum, and taking the initial values after error minimization as the solved internal parameter matrix with distortion, external parameter matrix and distortion parameters.
Preferably, the first and second electrodes are formed of a metal,
the obtaining of the distortion-free internal reference matrix according to the solved distorted internal reference matrix and the distortion parameters comprises:
solving the distorted image coordinate points according to the solved distorted internal reference matrix and the mapping relation of the distorted image coordinate points to the coordinate points of the pixel coordinate system;
solving an image coordinate point before distortion according to the mapping relation between the image coordinate point before distortion and the image coordinate point after distortion because the distortion parameter is known;
solving the physical sizes of the horizontal axis and the vertical axis of each pixel in a pixel coordinate system according to the image coordinate points;
and solving the distortion-free internal reference matrix according to the physical sizes of the horizontal axis and the vertical axis of each pixel in the pixel coordinate system.
According to a second aspect of embodiments of the present invention, there is provided a monocular color based transparent object positioning device, comprising:
a calibration module: the system is used for selecting a plurality of calibration images of the checkerboard calibration board shot at any angle, and calculating world coordinates and pixel coordinates of each checkerboard corner point on all the calibration images;
a distortion solving module: the system comprises an image acquisition module, a parameter calculation module and a parameter calculation module, wherein the image acquisition module is used for acquiring world coordinates and pixel coordinates of each checkerboard corner point on a calibrated image;
a distortion-free solving module: the distortion parameter calculation module is used for calculating a distortion-free internal reference matrix according to the calculated internal reference matrix with distortion and the distortion parameters;
a correction module: the image correction device is used for shooting an image of a transparent object on a fixed plane through a monocular camera and correcting the image of the transparent object by using an internal parameter matrix with distortion and distortion parameters;
a positioning module: the method is used for obtaining the pixel coordinates of the transparent object in the corrected image of the transparent object, reversely pushing the world coordinates of the transparent object according to the mapping relation between the world coordinate system and the pixel coordinate system and the solved distortion-free internal reference matrix and external reference matrix, and realizing the positioning of the transparent object through the world coordinates.
According to a third aspect of embodiments of the present invention, there is provided a storage medium storing a computer program which, when executed by a master, performs the steps of the above-described method.
The technical scheme provided by the embodiment of the invention can have the following beneficial effects:
according to the method, the calibration image of the calibration plate is shot, the angular point information of the multi-angle calibration image is obtained, according to the obtained angular point information, namely the world coordinate and the pixel coordinate of the calibration image, the internal reference, the external reference and the distortion parameter of the camera are solved through the world coordinate and the pixel coordinate, the monocular camera is used for obtaining the image of the transparent object, the transparent image is corrected through the calculated parameter, then according to the pixel coordinate of the shot transparent image, the world system coordinate of the transparent object is reversely pushed according to the mapping relation between the world system coordinate and the pixel coordinate, so that the precise positioning of the transparent object through the monocular camera is realized, and the problems that in the prior art, the depth information cannot be accurately obtained through a method based on a depth monocular camera and a method based on binocular vision, and the positioning is not precise are solved.
It is to be understood that both the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the invention, as claimed.
Drawings
The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the invention and together with the description, serve to explain the principles of the invention.
FIG. 1 is a schematic flow chart diagram illustrating a monocular color based transparent object positioning method in accordance with one exemplary embodiment;
FIG. 2 is a schematic diagram illustrating a principle of camera coordinate transformation in accordance with an exemplary embodiment;
FIG. 3 is a schematic diagram of a perspective projection shown in accordance with another exemplary embodiment;
FIG. 4 is a schematic diagram of an affine transformation shown in accordance with another exemplary embodiment;
FIG. 5 is a schematic diagram illustrating the use of the apparatus according to another exemplary embodiment;
FIG. 6 is a system diagram illustrating a monocular color based transparent object locating device according to another exemplary embodiment;
in the drawings: the system comprises a 1-calibration module, a 2-distortion solving module, a 3-distortion-free solving module, a 4-correction module and a 5-positioning module.
Detailed Description
Reference will now be made in detail to the exemplary embodiments, examples of which are illustrated in the accompanying drawings. When the following description refers to the accompanying drawings, like numbers in different drawings represent the same or similar elements unless otherwise indicated. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present invention. Rather, they are merely examples of apparatus and methods consistent with certain aspects of the invention, as detailed in the appended claims.
Example one
Fig. 1 is a flowchart illustrating a monocular color based transparent object positioning method according to an exemplary embodiment, as shown in fig. 1, the method including:
s1, selecting a plurality of calibration images of the checkerboard calibration plate shot at any angle, and calculating world coordinates and pixel coordinates of each checkerboard angular point on all the calibration images;
s2, solving an internal reference matrix with distortion, an external reference matrix with distortion and distortion parameters according to world coordinates and pixel coordinates of each checkerboard corner point on the calibrated image;
s3, solving the distortion-free internal reference matrix according to the solved distorted internal reference matrix and the distortion parameter;
s4, shooting the image of the transparent object on the fixed plane through a monocular camera, and correcting the image of the transparent object by using the internal reference matrix with distortion and the distortion parameters;
s5, obtaining pixel coordinates of the transparent object in the corrected image of the transparent object, reversely pushing the world coordinates of the transparent object according to a mapping relation between a world coordinate system and the pixel coordinate system and the solved distortion-free internal reference matrix and external reference matrix, and realizing the positioning of the transparent object through the world coordinates;
it will be appreciated that it is often necessary to establish a controlled indoor environment prior to acquiring image data: placing the monocular camera with a fixed focal length at a position approximately 1.9m from the ground; the transparent bottle conveying plate shoots a calibration data set at each angle at a position 0.2m to 0.8m away from the monocular camera, as shown in figure 5, a calibration result of any one of the calibration images is used as a three-dimensional coordinate reference system, angular point information of the calibration images is extracted, as the calibration plate is used, world coordinates and pixel coordinates can be simultaneously obtained, an initial internal reference matrix and an external reference matrix of the monocular camera can be determined according to a mapping conversion relation between the world coordinate system and the pixel coordinate system, distortion parameters are set, distorted coordinates in the pixel coordinate system are determined according to distortion parameters of the monocular camera, the distorted coordinates are substituted into the initial internal reference matrix to obtain an internal reference matrix with distortion, and the internal reference matrix, the external reference matrix and the distortion parameters with distortion are solved according to the world coordinates and the pixel coordinates of each angular point on the calibration images; acquiring a distortion-free internal reference matrix according to the solved distortion-free internal reference matrix and distortion parameters, shooting an image of the transparent object through a monocular camera, correcting the image of the transparent object through the distortion-free internal reference matrix and the distortion parameters, acquiring pixel coordinates of the transparent object, reversely pushing world coordinates of the transparent object according to a mapping relation between a world coordinate system and the pixel coordinate system and the solved distortion-free internal reference matrix and the solved external reference matrix, and realizing the positioning of the transparent object through the world coordinates; according to the method, the monocular camera is used for obtaining the image of the transparent object, the transparent image is corrected through the calculated parameters, the world system coordinate of the transparent object is reversely pushed according to the mapping relation between the world system coordinate and the pixel coordinate of the transparent image obtained through shooting, so that the transparent object can be accurately positioned through the monocular camera, and the problems that in the prior art, the depth information cannot be accurately obtained through a method based on the depth monocular camera and a method based on binocular vision, and the positioning is not accurate are solved.
Preferably, the first and second electrodes are formed of a metal,
the photographing of the image of the transparent object by the monocular camera includes:
shooting images of the transparent object falling at all angles through fixing the visual angle of the monocular camera to form a data set;
adjusting the brightness of each image in the data set up and down on the basis of an original image, respectively rotating each image in the data set clockwise by a certain angle and counterclockwise by a certain angle, adding the adjusted images into the data set together, and expanding the data set;
training the data set through an object algorithm to obtain a detection model, wherein the detection model can identify a transparent object in a lodging state;
it can be understood that the object detection algorithm adopts a YOLO (young Only Look one) series algorithm, and the YOLO 5 algorithm is a typical representative of the current target detection algorithm, and the main idea is to convert the target detection problem into a regression problem. A YOLOv5m model of a YOLOv5 algorithm is selected, the relationship between the scale of an algorithm network and the detection precision is well compatible, and before the YOLO is used for detection, a data set of a transparent object in a lodging state needs to be collected. And shooting the transparent object falling at each angle under the fixed camera view angle. To further enhance the robustness of the data set, the acquired data set is augmented. Firstly, in order to avoid the influence of illumination, the brightness of each image is respectively adjusted to be 0.6 times and 1.4 times of the original image; secondly, simulating the lodging condition of the transparent object more, rotating each image by 45 degrees clockwise and 45 degrees anticlockwise respectively, training the data set by using YOLO (YOLO), opening the monocular camera after distortion correction to detect the transparent object in the field of vision, displaying the transparent object at the terminal if the lodging state of the transparent object is detected, and calculating the two-dimensional pixel coordinate of the center point of the Bounding Box.
Preferably, the first and second liquid crystal display panels are,
the mapping relation from the world coordinate system to the pixel coordinate system comprises:
the world coordinate system obtains a camera coordinate system through rigid body transformation, the camera coordinate system obtains an image coordinate system through perspective projection, and the image coordinate system obtains a pixel coordinate system through affine transformation;
the rigid body transformation includes: multiplying the world coordinate system by a rotation matrix and adding a translation matrix to obtain a camera coordinate system;
the perspective projection includes: converting a camera coordinate system into a planar two-dimensional coordinate system, namely an image coordinate system, according to the relation among the focal length, the object distance and the distance of the lens imaging under the pinhole model;
the affine transformation includes: in a planar two-dimensional coordinate system, converting the origin of coordinates of an image coordinate system into the midpoint of a pixel coordinate system, and realizing the conversion from the image coordinate system to the pixel coordinate system;
it is understood that, as shown in fig. 2, there is a fixed mapping relationship between the world coordinate system and the pixel coordinate system, for example, the world coordinate system can be converted into the camera coordinate system through rigid body transformation, which is specifically shown as follows: the process of rigid transformation, which is from a point in the world coordinate system to a point in the camera coordinate system, can be described by a rotation matrix R and a translation matrix T, and the following rigid transformation formula exists:
(1)
where (U, V, W) represents a coordinate point in the world coordinate system and (X, Y, Z) represents a coordinate point in the camera coordinate system, where R is a 3X3 rotation matrix (orthogonal rotation matrix) and T is a three-dimensional translation vector, in homogeneous coordinate form:
(2)
the camera coordinate system can be transformed into an image coordinate system through perspective projection, and according to the relationship between the imaging focal length, the object distance and the distance of the lens under the pinhole model, as shown in fig. 3, the following can be obtained:
(3)
wherein f represents the focal length of the monocular camera, and x and y represent coordinate points in an image coordinate system; and (4) forming the relation of the formula (3) into a homogeneous coordinate formula to obtain:
(4)
the image coordinate system can be converted into a pixel coordinate system through affine transformation, i.e. in a two-dimensional plane, the object performs operations such as Translation (Translation), scaling (Scale), flip (Flip), rotation (Rotation) and cut (Shear), u corresponds to x, v corresponds to y, a coordinate origin O' (0,0) of the image coordinate system is a midpoint of the pixel coordinate system, as shown in fig. 4, the image coordinate system is in mm, dx and dy respectively represent the physical size, i.e. resolution, of each pixel on a horizontal axis x and a vertical axis y, and represent the actual distance (mm) corresponding to each pixel, and dx and dy are shown in formula (5) and formula (6), where u is u corresponds to x, v corresponds to y, and u is shown in formula (6) 0 And v 0 Respectively the central position of the pixel coordinate system;
(5)
(6)
therefore, the relationship between the image coordinate system and the pixel coordinate system is shown in formula (7):
(7)
in summary, the conversion relationship from the world coordinate system to the pixel coordinate system is shown in equation (8):
(8)
wherein Z represents a scale factor;
preferably, the first and second electrodes are formed of a metal,
the determining of the initial internal reference matrix and the external reference matrix of the monocular camera according to the mapping relationship from the world coordinate system to the pixel coordinate system comprises:
combining a mapping matrix from a monocular camera coordinate system to an image coordinate system and a mapping matrix from the image coordinate system to a pixel coordinate system to obtain an initial internal reference matrix;
obtaining an external parameter matrix according to a mapping matrix from a world coordinate system to a monocular camera coordinate system;
it can be understood that the first two matrices of equation 8 in the above are combined to obtain equation 9, that is, the mapping matrix from the monocular camera coordinate system to the image coordinate system and the mapping matrix from the image coordinate system to the pixel coordinate system are combined:
(9)
in the formula, K is the initial internal reference matrix;
the external parameter matrix depends on the relative position of the camera coordinate system and the world coordinate system, that is, the mapping matrix from the world coordinate system to the monocular camera coordinate system, and the external parameter matrix 10 is obtained:
(10)
then the imaging model of the single-point undistorted camera is shown as formula 11:
(11)
preferably, the first and second electrodes are formed of a metal,
the distorted internal reference matrix comprises:
combining a mapping matrix from a monocular camera coordinate system to an image coordinate system and a mapping matrix from the image coordinate system to a pixel coordinate system to obtain an initial internal reference matrix;
setting an unknown number as a distortion parameter of the monocular camera, and determining a radial distortion formula and a tangential distortion formula according to the distortion parameter and a coordinate point of an image coordinate system;
combining a radial distortion formula and a tangential distortion formula to obtain a coordinate point of a distorted image coordinate system projected to a pixel coordinate system;
substituting coordinate points of the distorted image coordinate system projected to the pixel coordinate system into the initial internal reference matrix to obtain a distorted internal reference matrix;
obtaining an external parameter matrix according to a mapping matrix from a world coordinate system to a camera coordinate system;
it can be understood that, in the process of deriving all the above coordinate system formulas, the linear camera model is followed, but the actual camera cannot reach the linear model due to lens manufacturing process and the like, so that the original image acquired by the camera contains distortion, and the camera distortion occurs in a phaseDuring the conversion of the machine coordinate system to the image coordinate system, the distorted camera coordinate system is thus transformed into: world coordinate system->Camera coordinate system->(distortion free) image coordinate system->(with distortion) image coordinate system->A pixel coordinate system; the distortion includes radial distortion and tangential distortion, for this reason, the distortion parameter has been introduced to this application: k is a radical of 1 ，k 2 ，k 3 ，p 1 ，p 2 ，
The radial distortion equation is shown in equation 12:
(12)
wherein (x, y) is a coordinate point in a linear, undistorted normalized image coordinate system,
for a corresponding distorted normalized image coordinate point, r is the distance from the image coordinate of the coordinate point to the image center point o', i.e. r 2 ＝x 2 +y 2 ；
The tangential distortion equation is shown in equation 13:
(13)
combinations (12) and (13), for a point (x, y) in the image coordinate system, the position of this point on the pixel plane can be calculated;
the distorted image coordinate transformation formula is shown as formula (14):
(14)
projecting the distorted point to a pixel plane through an internal reference matrix to obtain the correct position of the point on the pixel plane, as shown in formula (15):
(15)
wherein the content of the first and second substances,
representing the coordinates of the center of the camera light-sensing plate under a pixel coordinate system after being corrected;
(16)
(17)
then the single point camera imaging model with distortion is shown as equation (18):
(18)
wherein the first term of the product is camera internal parameter with distortion, the second term is camera external parameter, and f is camera focal length with distortion.
Preferably, the first and second electrodes are formed of a metal,
the method for solving the internal reference matrix with distortion, the external reference matrix with distortion and the distortion parameter according to the world coordinates and the pixel coordinates of each checkerboard corner point on the calibration image comprises the following steps:
combining the internal reference matrix with distortion and the external reference matrix to obtain a single-point camera imaging model with distortion;
inputting the world coordinates and the pixel coordinates of each checkerboard corner point into a single-point camera imaging model with distortion;
setting initial values for an internal parameter matrix with distortion, an external parameter matrix and distortion parameters, gradually minimizing a reprojection error through a linear optimization method to enable the error of the initial values to be minimum, and taking the initial values after error minimization as the solved internal parameter matrix with distortion, the external parameter matrix and the distortion parameters;
it can be understood that the formula (18) is a single-point distorted camera imaging model obtained by combining the distorted internal reference matrix and the distorted external reference matrix, since the world coordinates (U, V, W) of each checkerboard corner point are known, and the distorted pixel coordinates are known
As is known, the world coordinates of each checkerboard corner point and the distorted pixel coordinates are substituted into a formula (18), initial values are set for the internal parameter matrix with distortion, the external parameter matrix with distortion and distortion parameters, the reprojection error is gradually minimized through a nonlinear optimization method to minimize the error of the initial values, the initial values with the minimized error are used as unknowns in the internal parameter matrix with distortion and the external parameter matrix with distortion to solve the internal parameter matrix with distortion and the external parameter matrix with distortion, and the initial values with the minimized error are the solved distortion parameters for the distortion parameters.
Preferably, the first and second electrodes are formed of a metal,
the obtaining of the distortion-free internal reference matrix according to the solved distorted internal reference matrix and the distortion parameters comprises:
solving the distorted image coordinate points according to the solved distorted internal reference matrix and the mapping relation of the distorted image coordinate points to the coordinate points of the pixel coordinate system;
solving an image coordinate point before distortion according to the mapping relation between the image coordinate point before distortion and the image coordinate point after distortion because the distortion parameter is known;
solving the physical sizes of a horizontal axis and a vertical axis of each pixel in a pixel coordinate system according to the image coordinate points;
solving an undistorted internal reference matrix according to the physical sizes of the horizontal axis and the vertical axis of each pixel in a pixel coordinate system;
it can be understood that the distorted image, video or real-time shot is corrected according to the distorted internal reference matrix and the distortion coefficient, and the corrected pixel points (u, v) correspond to the pixel points before correction
According to the above formula (15), since the distorted internal reference matrix is solved, the distorted image coordinate points can be solved according to the formula (15)
Because the distortion parameters are known, the coordinate point (x, y) of the image before distortion can be solved according to the formula (14), and then a corrected new camera internal parameter matrix, namely an undistorted internal parameter matrix, can be calculated according to the formulas (5), (6) and (9), wherein the undistorted internal parameter matrix is the initial internal parameter matrix.
It can be understood that, the obtaining of the pixel coordinates of the transparent object and the inverse pushing of the world coordinates of the transparent object according to the mapping relationship between the world coordinate system and the pixel coordinate system, the positioning of the transparent object by the world coordinates is specifically as follows:
it is known that: pixel coordinates (u, v) of the transparent object;
derived image coordinates (x, y):
deriving camera seatTarget (X, Y, Z):
deriving world coordinates (U, V, W):
to sum up:
the application is verified by the scheme, and data shown in the following table are obtained:
example two
FIG. 6 is a system diagram illustrating a monocular color based transparent object positioning device, according to another exemplary embodiment, including:
a calibration module 1: the system is used for selecting a plurality of calibration images of the checkerboard calibration board shot at any angle, and calculating world coordinates and pixel coordinates of each checkerboard corner point on all the calibration images;
distortion solving module 2: the system comprises an image acquisition module, a parameter calculation module and a parameter calculation module, wherein the image acquisition module is used for acquiring world coordinates and pixel coordinates of each checkerboard corner point on a calibrated image;
distortion-free solving module 3: the distortion parameter calculation module is used for calculating a distortion-free internal reference matrix according to the calculated internal reference matrix with distortion and the distortion parameters;
the correction module 4: the image correction device is used for shooting an image of a transparent object on a fixed plane through a monocular camera and correcting the image of the transparent object by using an internal parameter matrix with distortion and distortion parameters;
the positioning module 5: the method is used for obtaining the pixel coordinates of the transparent object in the corrected image of the transparent object, reversely pushing the world coordinates of the transparent object according to the mapping relation between the world coordinate system and the pixel coordinate system and the solved distortion-free internal reference matrix and external reference matrix, and realizing the positioning of the transparent object through the world coordinates;
it can be understood that, in the present application, a calibration module 1 selects a plurality of calibration images of the checkerboard calibration board shot at any angle, and calculates world coordinates and pixel coordinates of each checkerboard corner point on all the calibration images; solving an internal reference matrix with distortion, an external reference matrix and distortion parameters according to world coordinates and pixel coordinates of each checkerboard angular point on the calibrated image through a distortion solving module 2; the distortion-free solving module 3 is used for solving a distortion-free internal reference matrix according to the solved internal reference matrix with distortion and the distortion parameters; the correction module 4 shoots an image of a transparent object on a fixed plane through a monocular camera, and corrects the image of the transparent object by using an internal parameter matrix with distortion and distortion parameters; the positioning module 5 acquires pixel coordinates of the transparent object in the corrected image of the transparent object, reversely pushes the world coordinates of the transparent object according to the mapping relation between a world coordinate system and the pixel coordinate system and the solved distortion-free internal reference matrix and external reference matrix, and realizes the positioning of the transparent object through the world coordinates; according to the method, the monocular camera is used for obtaining the image of the transparent object, the transparent image is corrected through the calculated parameters, the world system coordinate of the transparent object is reversely pushed according to the mapping relation between the world system coordinate and the pixel coordinate of the transparent image obtained through shooting, so that the transparent object can be accurately positioned through the monocular camera, and the problems that in the prior art, the depth information cannot be accurately obtained through a method based on the depth monocular camera and a method based on binocular vision, and the positioning is not accurate are solved.
Example three:
the present embodiment provides a storage medium, which stores a computer program, and when the computer program is executed by a master controller, the computer program implements the steps of the method;
it will be appreciated that the storage medium referred to above may be a read-only memory, a magnetic or optical disk, or the like.
It is understood that the same or similar parts in the above embodiments may be mutually referred to, and the same or similar parts in other embodiments may be referred to for the content which is not described in detail in some embodiments.
It should be noted that, in the description of the present invention, the terms "first", "second", etc. are used for descriptive purposes only and are not to be construed as indicating or implying relative importance. Further, in the description of the present invention, the meaning of "a plurality" means at least two unless otherwise specified.
Any process or method descriptions in flow charts or otherwise described herein may be understood as representing modules, segments, or portions of code which include one or more executable instructions for implementing specific logical functions or steps of the process, and alternate implementations are included within the scope of the preferred embodiment of the present invention in which functions may be executed out of order from that shown or discussed, including substantially concurrently or in reverse order, depending on the functionality involved, as would be understood by those reasonably skilled in the art of the present invention.
It should be understood that portions of the present invention may be implemented in hardware, software, firmware, or a combination thereof. In the above embodiments, various steps or methods may be implemented in software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented in hardware, as in another embodiment, any one or combination of the following technologies, which are well known in the art, may be used: a discrete logic circuit having a logic gate circuit for implementing a logic function on a data signal, an application specific integrated circuit having an appropriate combinational logic gate circuit, a Programmable Gate Array (PGA), a Field Programmable Gate Array (FPGA), or the like.
It will be understood by those skilled in the art that all or part of the steps carried by the method for implementing the above embodiments may be implemented by hardware related to instructions of a program, which may be stored in a computer readable storage medium, and when the program is executed, the program includes one or a combination of the steps of the method embodiments.
In addition, functional units in the embodiments of the present invention may be integrated into one processing module, or each unit may exist alone physically, or two or more units are integrated into one module. The integrated module can be realized in a hardware mode, and can also be realized in a software functional module mode. The integrated module, if implemented in the form of a software functional module and sold or used as a stand-alone product, may also be stored in a computer readable storage medium.
The storage medium mentioned above may be a read-only memory, a magnetic or optical disk, etc.
In the description herein, references to the description of the term "one embodiment," "some embodiments," "an example," "a specific example," or "some examples," etc., mean that a particular feature, structure, material, or characteristic described in connection with the embodiment or example is included in at least one embodiment or example of the invention. In this specification, the schematic representations of the terms used above do not necessarily refer to the same embodiment or example. Furthermore, the particular features, structures, materials, or characteristics described may be combined in any suitable manner in any one or more embodiments or examples.
Although embodiments of the present invention have been shown and described above, it is understood that the above embodiments are exemplary and should not be construed as limiting the present invention, and that variations, modifications, substitutions and alterations can be made to the above embodiments by those of ordinary skill in the art within the scope of the present invention.
Claims (8)
1. A monocular color based transparent object positioning method, the method comprising:
selecting a plurality of calibration images of the checkerboard calibration plate shot at any angle, and calculating world coordinates and pixel coordinates of each checkerboard corner point on all the calibration images;
solving an internal reference matrix with distortion, an external reference matrix with distortion and distortion parameters according to world coordinates and pixel coordinates of each checkerboard corner point on the calibrated image;
solving a distortion-free internal reference matrix according to the solved internal reference matrix with distortion and the distortion parameters;
shooting an image of a transparent object on a fixed plane through a monocular camera, and correcting the image of the transparent object by using an internal reference matrix with distortion and distortion parameters;
and obtaining the pixel coordinates of the transparent object in the corrected image of the transparent object, reversely deducing the world coordinates of the transparent object according to the mapping relation between the world coordinate system and the pixel coordinate system and the solved distortion-free internal reference matrix and external reference matrix, and realizing the positioning of the transparent object through the world coordinates.
2. The method of claim 1,
the photographing of the image of the transparent object on the fixed plane by the monocular camera includes:
shooting images of the transparent object falling at all angles through fixing the visual angle of the monocular camera to form a data set;
adjusting the brightness of each image in the data set up and down on the basis of an original image, respectively rotating each image in the data set clockwise by a certain angle and counterclockwise by a certain angle, adding the adjusted images into the data set together, and expanding the data set;
the data set is trained through an object detection algorithm to obtain a detection model, and the detection model can identify the transparent object in the lodging state.
3. The method of claim 1,
the mapping relation from the world coordinate system to the pixel coordinate system comprises the following steps:
the world coordinate system obtains a camera coordinate system through rigid body transformation, the camera coordinate system obtains an image coordinate system through perspective projection, and the image coordinate system obtains a pixel coordinate system through affine transformation;
the rigid body transformation includes: multiplying the world coordinate system by a rotation matrix and adding a translation matrix to obtain a camera coordinate system;
the perspective projection includes: converting a camera coordinate system into a planar two-dimensional coordinate system, namely an image coordinate system, according to the relation among the focal length, the object distance and the distance of the lens imaging under the pinhole model;
the affine transformation includes: in a planar two-dimensional coordinate system, the coordinate origin of the image coordinate system is converted into the midpoint of the pixel coordinate system, so that the conversion from the image coordinate system to the pixel coordinate system is realized.
4. The method of claim 3,
the distorted internal reference matrix comprises:
combining a mapping matrix from a monocular camera coordinate system to an image coordinate system and a mapping matrix from the image coordinate system to a pixel coordinate system to obtain an initial internal reference matrix;
setting an unknown number as a distortion parameter of the monocular camera, and determining a radial distortion formula and a tangential distortion formula according to the distortion parameter and a coordinate point of an image coordinate system;
combining a radial distortion formula and a tangential distortion formula to obtain a coordinate point of a distorted image coordinate system projected to a pixel coordinate system;
substituting the coordinate points of the distorted image coordinate system projected to the pixel coordinate system into the initial internal reference matrix to obtain an internal reference matrix with distortion;
and obtaining an external parameter matrix according to the mapping matrix from the world coordinate system to the camera coordinate system.
5. The method of claim 4,
the method for solving the internal reference matrix with distortion, the external reference matrix with distortion and the distortion parameter according to the world coordinates and the pixel coordinates of each checkerboard corner point on the calibration image comprises the following steps:
combining the internal reference matrix with distortion and the external reference matrix to obtain a single-point camera imaging model with distortion;
inputting the world coordinates and the pixel coordinates of each checkerboard corner point into a single-point camera imaging model with distortion;
setting initial values for an internal parameter matrix with distortion, an external parameter matrix and distortion parameters, gradually minimizing the reprojection error through a linear optimization method to enable the error of the initial values to be minimum, and taking the initial values after error minimization as the solved internal parameter matrix with distortion, external parameter matrix and distortion parameters.
6. The method of claim 5,
the obtaining of the distortion-free internal reference matrix according to the solved distorted internal reference matrix and the distortion parameter comprises:
solving the distorted image coordinate points according to the solved distorted internal reference matrix and the mapping relation of the distorted image coordinate points to the coordinate points of the pixel coordinate system;
because the distortion parameters are known, solving the image coordinate points before distortion according to the mapping relation between the image coordinate points before distortion and the image coordinate points after distortion;
solving the physical sizes of a horizontal axis and a vertical axis of each pixel in a pixel coordinate system according to the image coordinate points;
and solving the distortion-free internal reference matrix according to the physical sizes of the horizontal axis and the vertical axis of each pixel in the pixel coordinate system.
7. A monocular color based transparent object positioning device, the device comprising:
a calibration module: the system is used for selecting a plurality of calibration images of the checkerboard calibration board shot at any angle, and calculating world coordinates and pixel coordinates of each checkerboard corner point on all the calibration images;
a distortion solving module: the system comprises an image acquisition module, a parameter calculation module and a parameter calculation module, wherein the image acquisition module is used for acquiring world coordinates and pixel coordinates of each checkerboard corner point on a calibrated image;
a distortion-free solving module: the distortion parameter calculation module is used for calculating a distortion-free internal reference matrix according to the calculated internal reference matrix with distortion and the distortion parameters;
a correction module: the image correction device is used for shooting an image of a transparent object on a fixed plane through a monocular camera and correcting the image of the transparent object by using an internal parameter matrix with distortion and distortion parameters;
a positioning module: the method is used for obtaining the pixel coordinates of the transparent object in the corrected image of the transparent object, reversely pushing the world coordinates of the transparent object according to the mapping relation between the world coordinate system and the pixel coordinate system and the solved distortion-free internal reference matrix and external reference matrix, and realizing the positioning of the transparent object through the world coordinates.
8. A storage medium, characterized in that the storage medium stores a computer program which, when executed by a master controller, performs the steps of a monocular color based transparent object positioning method according to any one of claims 1-6.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202211501197.8A CN115830103A (en) | 2022-11-28 | 2022-11-28 | Monocular color-based transparent object positioning method and device and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202211501197.8A CN115830103A (en) | 2022-11-28 | 2022-11-28 | Monocular color-based transparent object positioning method and device and storage medium |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN115830103A true CN115830103A (en) | 2023-03-21 |
Family
ID=85532119
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202211501197.8A Pending CN115830103A (en) | 2022-11-28 | 2022-11-28 | Monocular color-based transparent object positioning method and device and storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN115830103A (en) |
Cited By (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN116433756A (en) * | 2023-06-15 | 2023-07-14 | 浪潮智慧科技有限公司 | Surface object space analysis method, device and medium of monocular camera |
| CN116485918A (en) * | 2023-06-25 | 2023-07-25 | 天府兴隆湖实验室 | Calibration method, calibration system and computer readable storage medium |
| CN116630442A (en) * | 2023-07-19 | 2023-08-22 | 绘见科技(深圳)有限公司 | Visual SLAM pose estimation precision evaluation method and device |
| CN117576228A (en) * | 2024-01-16 | 2024-02-20 | 成都合能创越软件有限公司 | Real-time scene-based camera coordinate calibration method and system |
| CN117934600A (en) * | 2024-03-25 | 2024-04-26 | 南京信息工程大学 | Method for quickly identifying remote markers and resolving three-dimensional positions based on unmanned aerial vehicle |
-
2022
- 2022-11-28 CN CN202211501197.8A patent/CN115830103A/en active Pending
Cited By (9)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN116433756A (en) * | 2023-06-15 | 2023-07-14 | 浪潮智慧科技有限公司 | Surface object space analysis method, device and medium of monocular camera |
| CN116433756B (en) * | 2023-06-15 | 2023-08-18 | 浪潮智慧科技有限公司 | Surface object space analysis method, device and medium of monocular camera |
| CN116485918A (en) * | 2023-06-25 | 2023-07-25 | 天府兴隆湖实验室 | Calibration method, calibration system and computer readable storage medium |
| CN116485918B (en) * | 2023-06-25 | 2023-09-08 | 天府兴隆湖实验室 | Calibration method, calibration system and computer readable storage medium |
| CN116630442A (en) * | 2023-07-19 | 2023-08-22 | 绘见科技(深圳)有限公司 | Visual SLAM pose estimation precision evaluation method and device |
| CN116630442B (en) * | 2023-07-19 | 2023-09-22 | 绘见科技(深圳)有限公司 | Visual SLAM pose estimation precision evaluation method and device |
| CN117576228A (en) * | 2024-01-16 | 2024-02-20 | 成都合能创越软件有限公司 | Real-time scene-based camera coordinate calibration method and system |
| CN117576228B (en) * | 2024-01-16 | 2024-04-16 | 成都合能创越软件有限公司 | Real-time scene-based camera coordinate calibration method and system |
| CN117934600A (en) * | 2024-03-25 | 2024-04-26 | 南京信息工程大学 | Method for quickly identifying remote markers and resolving three-dimensional positions based on unmanned aerial vehicle |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN111210468B (en) | Image depth information acquisition method and device | |
| CN115830103A (en) | Monocular color-based transparent object positioning method and device and storage medium | |
| CN106875339B (en) | Fisheye image splicing method based on strip-shaped calibration plate | |
| CN110689581B (en) | Structured light module calibration method, electronic device and computer readable storage medium | |
| CN110809786B (en) | Calibration device, calibration chart, chart pattern generation device, and calibration method | |
| JP5580164B2 (en) | Optical information processing apparatus, optical information processing method, optical information processing system, and optical information processing program | |
| US5878174A (en) | Method for lens distortion correction of photographic images for texture mapping | |
| JP4245963B2 (en) | Method and system for calibrating multiple cameras using a calibration object | |
| US5898438A (en) | Texture mapping of photographic images to CAD surfaces | |
| CN111815716A (en) | Parameter calibration method and related device | |
| US10552984B2 (en) | Capture device calibration methods and systems | |
| Von Gioi et al. | Towards high-precision lens distortion correction | |
| CN111009030A (en) | Multi-view high-resolution texture image and binocular three-dimensional point cloud mapping method | |
| CN114299156A (en) | Method for calibrating and unifying coordinates of multiple cameras in non-overlapping area | |
| JP2002516443A (en) | Method and apparatus for three-dimensional display | |
| RU2479828C2 (en) | Stereoscopic measurement system and method | |
| CN113298886A (en) | Calibration method of projector | |
| CN114485953A (en) | Temperature measuring method, device and system | |
| CN113963065A (en) | Lens internal reference calibration method and device based on external reference known and electronic equipment | |
| CN112419427A (en) | Method for improving time-of-flight camera accuracy | |
| JP3704494B2 (en) | How to check camera viewpoint and focal length | |
| CN115880369A (en) | Device, system and method for jointly calibrating line structured light 3D camera and line array camera | |
| CN115307865A (en) | Model deformation measurement method for high-temperature hypersonic flow field | |
| CN115631245A (en) | Correction method, terminal device and storage medium | |
| Oniga et al. | Metric and Non-Metric Cameras Calibration for the Improvement of Real-Time Monitoring Process Results. |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |