CN113989428A - Metallurgical reservoir area global three-dimensional reconstruction method and device based on depth vision - Google Patents

Metallurgical reservoir area global three-dimensional reconstruction method and device based on depth vision Download PDF

Info

Publication number
CN113989428A
CN113989428A CN202111034419.5A CN202111034419A CN113989428A CN 113989428 A CN113989428 A CN 113989428A CN 202111034419 A CN202111034419 A CN 202111034419A CN 113989428 A CN113989428 A CN 113989428A
Authority
CN
China
Prior art keywords
depth
original image
information
dimensional
global
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Pending
Application number
CN202111034419.5A
Other languages
Chinese (zh)
Inventor
徐冬
闫贺
杨荃
刘洋
王晓晨
何海楠
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
University of Science and Technology Beijing USTB
Original Assignee
University of Science and Technology Beijing USTB
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by University of Science and Technology Beijing USTB filed Critical University of Science and Technology Beijing USTB
Priority to CN202111034419.5A priority Critical patent/CN113989428A/en
Publication of CN113989428A publication Critical patent/CN113989428A/en
Pending legal-status Critical Current

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T17/00—Three dimensional [3D] modelling, e.g. data description of 3D objects
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/30—Determination of transform parameters for the alignment of images, i.e. image registration
    • G06T7/33—Determination of transform parameters for the alignment of images, i.e. image registration using feature-based methods
    • G06T7/337—Determination of transform parameters for the alignment of images, i.e. image registration using feature-based methods involving reference images or patches
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/30—Determination of transform parameters for the alignment of images, i.e. image registration
    • G06T7/35—Determination of transform parameters for the alignment of images, i.e. image registration using statistical methods
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2200/00—Indexing scheme for image data processing or generation, in general
    • G06T2200/32—Indexing scheme for image data processing or generation, in general involving image mosaicing
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/10—Image acquisition modality
    • G06T2207/10016—Video; Image sequence
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/10—Image acquisition modality
    • G06T2207/10028—Range image; Depth image; 3D point clouds

Landscapes

  • Engineering & Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Theoretical Computer Science (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Geometry (AREA)
  • Computer Graphics (AREA)
  • Software Systems (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • Bioinformatics & Cheminformatics (AREA)
  • Bioinformatics & Computational Biology (AREA)
  • Evolutionary Biology (AREA)
  • Probability & Statistics with Applications (AREA)
  • Image Analysis (AREA)
  • Length Measuring Devices By Optical Means (AREA)

Abstract

The invention provides a metallurgical reservoir area global three-dimensional reconstruction method and device based on depth vision, and relates to the technical field of intelligent warehouse logistics. The method comprises the following steps: the global three-dimensional reconstruction system takes an original image and a depth image acquired by a camera as input, calculates an inter-frame pose transformation relation by matching feature points between adjacent images and depth information, and accordingly splices local space three-dimensional information acquired at different positions. According to the invention, by using the metallurgy reservoir area global three-dimensional reconstruction system and method integrating depth vision, the spatial three-dimensional information of the metallurgy reservoir area global can be obtained in a spatial three-dimensional information splicing mode, so that the stack is more completely measured.

Description

Metallurgical reservoir area global three-dimensional reconstruction method and device based on depth vision
Technical Field
The invention relates to the technical field of intelligent warehouse logistics, in particular to a metallurgical reservoir area global three-dimensional reconstruction method and device based on depth vision.
Background
The overhead traveling crane in the unmanned overhead traveling crane storehouse area mainly relies on the gray bus measuring device to self location, realizes through PLC at the overhead traveling crane motion in-process and feeds back the callback system and obtain current overhead traveling crane position. In the prior art, for three-dimensional reconstruction of piles and vehicles in a large-field-of-view metallurgical reservoir area, detection information obtained by using fixed position detection is extremely limited.
Disclosure of Invention
The invention provides a metallurgical reservoir area global three-dimensional reconstruction method and device based on depth vision, aiming at the problem that in the prior art, for three-dimensional reconstruction of a stack and a vehicle in a large-field-of-view metallurgical reservoir area, detection information obtained by using fixed position detection is extremely limited.
In order to solve the technical problems, the invention provides the following technical scheme:
in one aspect, a depth vision-based metallurgical reservoir global three-dimensional reconstruction method is provided, and includes:
s1: acquiring original images of continuous frames and depth images corresponding to the original images through a depth camera to form an original image sequence and a depth image sequence; respectively detecting ORB characteristic points in two adjacent frames of original images;
s2: matching ORB characteristic points between two adjacent frames of original images to obtain an initial matching point pair; screening the initial matching point pairs to obtain matching information of the characteristic point pairs and pixel coordinates of the characteristic point pairs;
s3: extracting depth information of a depth image corresponding to the original image through the depth image sequence; combining the matched characteristic point pairs with pixel coordinates and corresponding depth information to obtain local space three-dimensional information;
s4: calculating an interframe pose transformation matrix of the original image;
s5: repeatedly executing S1-S4 until all the inter-frame pose transformation matrixes in the original image sequence are calculated; splicing the local space three-dimensional information collected at different positions by combining a pose transformation matrix to form global space three-dimensional information;
s6: and converting the coordinate of the global space three-dimensional information to a library area crown block coordinate system through a coordinate system, and providing the coordinate for the crown block.
Optionally, in step S1, acquiring, by a depth camera, original images of consecutive frames and depth images corresponding to the original images to form an original image sequence and a depth image sequence; respectively detecting ORB characteristic points in two adjacent frames of original images, comprising:
s11: acquiring original images and depth images of continuous frames by a depth camera arranged on an overhead traveling crane to move along with the overhead traveling crane; forming an original image sequence and a depth image sequence;
s12: selecting a current original image and a previous frame of original image to extract ORB characteristic points to obtain pixel coordinates of the ORB characteristic points;
s13: taking the ORB characteristic point as a center, selecting 13 pixel points according to the size of a window, selecting 128 groups of comparison point pairs in the window, and calculating a binary coding string of an ORB characteristic point descriptor by comparing the gray value of the central pixel with the gray value of the pixel points in the window through a formula (1):
Figure BDA0003246401200000021
wherein, I (p), I (q) represent the gray values of the central pixel and the pixels in the window; n is a radical ofpRepresenting pixel points contained in a central pixel neighborhood within a set window size range;
if I (q) ≧ I (p), the binary string is marked as 0, and if I (q) < I (p), the binary string is marked as 1.
Optionally, in step S2, ORB feature points between two adjacent frames of original images are matched to obtain an initial matching point pair; screening the initial matching point pairs to acquire the matching information of the characteristic point pairs and the pixel coordinates of the characteristic point pairs, and the method comprises the following steps:
s21: matching ORB characteristic points detected in the current original image and the previous frame original image to form a matching point pair; using Hamming distance percentage to measure the consistency of the matching point pairs;
s22: substituting the initial matching point pair obtained in S21 into formula (2) by using a random consensus algorithm RANSAC:
Figure BDA0003246401200000031
wherein, (x ', y') represents the initial matching point pair position in the current original image; (x, y) represents the initial matching point pair position of the previous frame original image; s represents a scale parameter;
determining an optimal transformation matrix meeting the maximum matching point time pair, and eliminating the cluster points;
s23: collecting position coordinate information of crown blocks at different images and different moments, calculating the motion direction of the crown blocks, and removing characteristic point pairs of which the relative positions of the current original image and the previous original image do not accord with the motion trend of the crown blocks; and obtaining final feature point pair matching information and feature point pair pixel coordinates.
Optionally, in step S3, extracting depth information of a depth image corresponding to the original image through the sequence of depth images; combining the matched characteristic point pairs with pixel coordinates and corresponding depth information to obtain local space three-dimensional information, wherein the local space three-dimensional information comprises the following steps:
s31: extracting a depth value D of a depth image corresponding to the original image through the depth image sequence;
s32: substituting the pixel coordinates of the feature point pair and the depth value D into formula (3) according to the pixel coordinates of the feature point pair obtained in step S2 in combination with the depth value D, calculating the local space three-dimensional coordinates of the feature point, and obtaining local space three-dimensional information:
Figure BDA0003246401200000032
wherein: (u, v) represents feature point pixel coordinates; cx,CyCalibrating parameters for the camera, and respectively representing the position coordinates of the image principal points; f. ofx,fyFor camera calibration parameters, fxRepresenting the component of the focal length in the direction of the X-axis in the image pixel coordinate system；fyRepresents the component of the focal length along the Y-axis direction in the image pixel coordinate system; x, Y, D represent the calculated spatial three-dimensional coordinates.
Optionally, in step S4, calculating an inter-frame pose transformation matrix of the original image includes:
s41: constructing a spatial point set error item between the current original image and the previous frame of original image, calculating a group of spatial transformation matrixes with minimum spatial position errors through Singular Value Decomposition (SVD) of a formula (4),
Figure BDA0003246401200000033
wherein: p is a radical ofiRepresenting a space three-dimensional point set q obtained by matching the characteristic points of the current original imageiRepresenting a space three-dimensional point set obtained after the previous frame of original image is matched with the feature points; eiRepresenting a spatial transformation matrix [ R | t]A spatial position error set corresponding to the characteristic point pairs; r represents a spatial rotation matrix that minimizes spatial position error, and t represents a spatial transformation matrix;
s42: will EiAnd converting the pose information into a quaternion format for storage to obtain an interframe pose transformation matrix.
Optionally, in step S5, the stitching the local spatial three-dimensional information collected at different positions with the pose transformation matrix to form global spatial three-dimensional information, including:
combining the local space three-dimensional information under the images at different moments with the interframe pose transformation matrix; and (3) splicing the spatial three-dimensional information by using a spatial transformation matrix through a formula (5) to form global spatial three-dimensional information:
Pi＝T*P' (5)
wherein: piRepresenting a set of spatial three-dimensional points, P ', in the current original image'iRepresenting a spatial three-dimensional point set in the previous frame of image; and T represents the calculated inter-frame pose transformation matrix.
Optionally, in step S6, converting the coordinates in the global space three-dimensional information to a location below the coordinate system of the overhead traveling crane through the coordinate system, and providing the location for the overhead traveling crane, including:
performing coordinate system fitting by acquiring position coordinate information of the same detection point in the same scene, and calculating a coordinate transformation relation; and converting the coordinates in the three-dimensional information of the global space into a library area coordinate system, and providing the coordinates for the overhead travelling crane for use.
In one aspect, a depth vision-based metallurgical reservoir global three-dimensional reconstruction apparatus is provided, and the apparatus is applied to the method in any one of the above items, and the apparatus includes:
the ORB detection module is used for acquiring original images of continuous frames and depth images corresponding to the original images through a depth camera to form an original image sequence and a depth image sequence; respectively detecting ORB characteristic points in two adjacent frames of original images;
the similarity matching module is used for matching ORB characteristic points between two adjacent frames of original images to obtain an initial matching point pair; screening the initial matching point pairs to obtain matching information of the characteristic point pairs and pixel coordinates of the characteristic point pairs;
the local information calculation module is used for extracting the depth information of the depth image corresponding to the original image through the depth image sequence; combining the matched characteristic point pairs with pixel coordinates and corresponding depth information to obtain local space three-dimensional information;
the pose transformation calculation module is used for calculating an interframe pose transformation matrix;
the global information splicing module is used for calculating all inter-frame pose transformation matrixes in the original image sequence; splicing the local space three-dimensional information collected at different positions by combining a pose transformation matrix to form global space three-dimensional information;
and the coordinate system conversion module is used for converting the coordinates in the three-dimensional information of the global space to a coordinate system of the overhead crane in the storage area through the coordinate system and providing the coordinates for the overhead crane.
Optionally, the ORB detection module is configured to acquire an original image and a depth image of consecutive frames by a depth camera arranged on the overhead traveling crane moving with the overhead traveling crane; forming an original image sequence and a depth image sequence;
selecting a current original image and a previous frame of original image to extract ORB characteristic points to obtain pixel coordinates of the ORB characteristic points;
after determining the pixel coordinate position of the ORB feature point, the ORB feature point is taken as the center, the window size is selected to be 13 pixel points, 128 groups of comparison point pairs are selected in the window, and the binary coding string of the ORB feature point descriptor is calculated by comparing the gray value of the central pixel with the gray value of the pixel points in the window.
Optionally, the similarity matching module is configured to match ORB feature points detected in the current original image and the previous original image to form a matching point pair; using Hamming distance percentage to measure the consistency of the matching point pairs;
selecting matching point pairs of which the code exclusive OR value is smaller than the Hamming distance percentage threshold value in the binary code string, and setting the group of characteristic points as initial matching point pairs:
determining an optimal transformation matrix meeting the maximum matching point time pair through a random consensus (RANSAC) algorithm, and removing the outliers;
collecting position coordinate information of crown blocks at different images and different moments, calculating the motion direction of the crown blocks, and removing characteristic point pairs of which the relative positions of the current original image and the previous original image do not accord with the motion trend of the crown blocks; and obtaining final feature point pair matching information and feature point pair pixel coordinates.
The technical scheme of the embodiment of the invention at least has the following beneficial effects:
in the scheme, the method and the device for the global three-dimensional reconstruction of the metallurgical reservoir area with the depth vision integrated are adopted, the original image and the depth image covering the whole metallurgical reservoir area are collected in real time in a surface scanning mode, and the global spatial three-dimensional information of the metallurgical reservoir area is obtained in a spatial three-dimensional information splicing mode, so that the effect of more comprehensively measuring the pile is achieved.
Drawings
In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings needed to be used in the description of the embodiments will be briefly introduced below, and it is obvious that the drawings in the following description are only some embodiments of the present invention, and it is obvious for those skilled in the art to obtain other drawings based on these drawings without creative efforts.
FIG. 1 is a flowchart of a depth vision-based metallurgical reservoir global three-dimensional reconstruction method according to an embodiment of the present invention;
FIG. 2 is a feature point detection flow chart of a metallurgical reservoir area global three-dimensional reconstruction method based on depth vision according to an embodiment of the present invention;
FIG. 3 is a feature point matching flow chart of a deep vision-based metallurgical reservoir global three-dimensional reconstruction method according to an embodiment of the present invention;
FIG. 4 shows an effect of a feature point matching algorithm of the deep vision-based metallurgical reservoir area global three-dimensional reconstruction method according to the embodiment of the present invention;
FIG. 5 is a flow chart of a local space three-dimensional information calculation of a metallurgical reservoir area global three-dimensional reconstruction method based on depth vision according to an embodiment of the present invention;
FIG. 6 is a flow chart of inter-frame pose transformation matrix calculation of a metallurgical reservoir global three-dimensional reconstruction method based on depth vision according to an embodiment of the present invention;
fig. 7 is a device block diagram of a device for reconstructing a metallurgical reservoir global three-dimensional area based on depth vision according to an embodiment of the present invention.
Detailed Description
In order to make the technical problems, technical solutions and advantages of the present invention more apparent, the following detailed description is given with reference to the accompanying drawings and specific embodiments.
As shown in fig. 1, an embodiment of the present invention provides a depth vision-based global three-dimensional reconstruction method for a metallurgical reservoir area, including:
s1: acquiring original images of continuous frames and depth images corresponding to the original images through a depth camera to form an original image sequence and a depth image sequence; the method detects ORB (ordered FAST and Rotated BRIEF, FAST feature point extraction and description algorithm) feature points in two adjacent frames of original images respectively.
S2: matching ORB characteristic points between two adjacent frames of original images to obtain an initial matching point pair; and screening the initial matching point pairs to obtain matching information of the characteristic point pairs and pixel coordinates of the characteristic point pairs.
S3: extracting depth information of a depth image corresponding to the original image through the depth image sequence; and combining the pixel coordinates of the matched characteristic points and the corresponding depth information to obtain local space three-dimensional information.
S4: and calculating an interframe pose transformation matrix of the original image.
S5: repeatedly executing S1-S4 until all the inter-frame pose transformation matrixes in the original image sequence are calculated; and splicing the local space three-dimensional information collected at different positions by combining the pose transformation matrix to form global space three-dimensional information.
S6: and converting the coordinate of the global space three-dimensional information to a library area crown block coordinate system through a coordinate system, and providing the coordinate for the crown block.
In this embodiment, by using the metallurgical reservoir area global three-dimensional reconstruction system and method that incorporate depth vision, spatial three-dimensional information of the metallurgical reservoir area global can be obtained in a manner of spatial three-dimensional information stitching, so that the pile can be measured more completely.
As shown in fig. 2, in step S1, acquiring, by a depth camera, original images of consecutive frames and depth images corresponding to the original images to form an original image sequence and a depth image sequence; respectively detecting ORB characteristic points in two adjacent frames of original images, comprising:
s11: acquiring original images and depth images of continuous frames by a depth camera arranged on an overhead traveling crane to move along with the overhead traveling crane; constituting an original image sequence and a depth image sequence.
S12: and selecting the current original image and the previous frame of original image to extract ORB characteristic points to obtain pixel coordinates of the ORB characteristic points.
S13: taking the ORB characteristic point as a center, selecting 13 pixel points according to the size of a window, selecting 128 groups of comparison point pairs in the window, and calculating a binary coding string of an ORB characteristic point descriptor by comparing the gray value of the central pixel with the gray value of the pixel points in the window through a formula (1):
Figure BDA0003246401200000071
wherein, I(p)Representing the gray value of the central pixel, I(q)Representing the gray values of the pixels within the window; n is a radical ofpRepresenting pixel points contained in a central pixel neighborhood within a set window size range;
if I (q) ≧ I (p), the binary string is marked as 0, and if I (q) < I (p), the binary string is marked as 1.
In this embodiment, it is assumed that the camera is an area array CCD camera that can acquire an original image and a depth image at the same time.
By calibrating camera parameters and integrating depth image information, a corresponding relationship between an image pixel plane and spatial three-dimensional information can be established, and in this embodiment, an internal reference matrix of the depth camera can be expressed as:
Figure BDA0003246401200000081
wherein f isx,fyFor camera calibration parameters, fxRepresenting the component of the focal length along the X-axis direction in the image pixel coordinate system; f. ofyRepresents the component of the focal length along the Y-axis direction in the image pixel coordinate system; cx,CyAnd representing camera calibration parameters which respectively represent the position coordinates of the image principal points.
In this embodiment, the internal reference calibration result of the depth camera is as follows:
Figure BDA0003246401200000082
for ORB feature point extraction: firstly, the current original image and the previous frame original image are selected to carry out ORB feature point extraction, and feature point pixel coordinates (u, v) are obtained.
In this embodiment, if 16 neighborhood pixels q on a circle with a radius of 3 are selected corresponding to the center pixel p, if the neighborhood is qualified, the neighborhood can be used as a feature point, and the judgment expression is as follows;
Figure BDA0003246401200000083
wherein N represents the number of conforming pixel points; lambda [ alpha ]dRepresenting a pixel difference threshold.
In this embodiment, the number of coincident pixels is set to 12, and the pixel difference threshold is set to 15.
In this embodiment, by extracting ORB feature points in each frame of image, the number of ORB feature points in each frame of image is shown in table 1:
TABLE 1
Figure BDA0003246401200000084
Figure BDA0003246401200000091
As shown in fig. 3, in step S2, ORB feature points between two adjacent frames of original images are matched to obtain an initial matching point pair; screening the initial matching point pairs to acquire the matching information of the characteristic point pairs and the pixel coordinates of the characteristic point pairs, and the method comprises the following steps:
s21: matching ORB characteristic points detected in the current original image and the previous frame original image to form a matching point pair; the hamming distance percentage is used to measure the consistency of the matching point pairs.
S22: substituting the initial matching point pair obtained in S21 into formula (2) by RANSAC (RANdom SAmple Consensus, RANdom Consensus algorithm):
Figure BDA0003246401200000092
wherein, (x ', y') represents the initial matching point pair position in the current original image; (x, y) represents the initial matching point pair position of the previous frame image; s represents a scale parameter.
And determining an optimal transformation matrix meeting the maximum matching point time pair, and removing the cluster points.
S23: collecting position coordinate information of crown blocks at different images and different moments, calculating the motion direction of the crown blocks, and removing characteristic point pairs of which the relative positions of the current original image and the previous original image do not accord with the motion trend of the crown blocks; and obtaining final feature point pair matching information and feature point pair pixel coordinates.
In this embodiment, the threshold of hamming distance percentage is set to 0.6, the feature points in the current original image and the previous frame original image are matched, and the hamming distance percentage is used to measure the consistency of the feature points:
η＝Hamming(Sl(x,y),Sr(x',y'))
if the descriptor code exclusive or value is smaller than a certain threshold value, the group of feature points is judged to be a matching point pair;
wherein S isl(x, y) represents the ORB feature point descriptor of the current original image, Sr(x ', y') represents the ORB feature point descriptor of the previous frame image; η represents the hamming distance percentage threshold.
The number of ORB feature point pairs initially matched by hamming distance in this embodiment is shown in table 2:
TABLE 2
Figure BDA0003246401200000093
Figure BDA0003246401200000101
In the embodiment, through RANSAC, the initial matching point pairs are brought into the matrix in an iterative mode, the optimal transformation matrix meeting the maximum matching point pairs is determined, and the cluster points are removed; and obtaining final feature point pair matching information and feature point pair pixel coordinates through motion trend constraint. In this embodiment, the effect of the ORB feature point matching algorithm based on the motion trend constraint is shown in fig. 4.
The number of matching point pairs of completely consistent ORB feature points obtained after the RANSAC immediate consistency constraint and the motion trend constraint in this embodiment is shown in table 3:
TABLE 3
Matching images Match characteristic point (pair) Matching images Match characteristic point (pair)
1-2 22 11-12 10
2-3 20 12-13 18
3-4 22 13-14 20
4-5 30 14-15 18
5-6 32 15-16 41
6-7 11 16-17 54
7-8 9 17-18 85
8-9 6 18-19 99
9-10 10 19-20 180
10-11 8
As shown in fig. 5, in step S3, depth information of a depth image corresponding to an original image is extracted through a sequence of depth images; combining the matched characteristic point pairs with pixel coordinates and corresponding depth information to obtain local space three-dimensional information, wherein the local space three-dimensional information comprises the following steps:
s31: the depth value D of the depth image corresponding to the original image is extracted through the sequence of depth images.
S32: substituting the pixel coordinates of the feature point pair and the depth value D into formula (3) according to the pixel coordinates of the feature point pair obtained in step S2 in combination with the depth value D, calculating the local space three-dimensional coordinates of the feature point, and obtaining local space three-dimensional information:
Figure BDA0003246401200000111
wherein: (u, v) represents feature point pixel coordinates; cx,CyCalibrating parameters for the camera, and respectively representing the position coordinates of the image principal points; f. ofx,fyFor camera calibration parameters, fxRepresenting the component of the focal length along the X-axis direction in the image pixel coordinate system; f. ofyRepresents the component of the focal length along the Y-axis direction in the image pixel coordinate system; x, Y, D represent the calculated spatial three-dimensional coordinates.
In this embodiment, the spatial transformation matrix is converted into a quaternion form by using the rodriess formula and stored.
The inter-frame pose transformation values between each group of images can be expressed by using quaternions as shown in table 4:
TABLE 4
Figure BDA0003246401200000112
Figure BDA0003246401200000121
As shown in fig. 6, in step S4, calculating an inter-frame pose transformation matrix of the original image includes:
s41: constructing a spatial point set error term between the current original image and the previous frame original image, calculating a group of spatial transformation matrixes with minimum spatial position errors through SVD (Singular Value Decomposition) in formula (4),
Figure BDA0003246401200000122
wherein: p is a radical ofiRepresenting a space three-dimensional point set q obtained by matching the characteristic points of the current original imageiRepresenting a space three-dimensional point set obtained after the previous frame of original image is matched with the feature points; eiRepresenting a spatial transformation matrix [ R | t]A spatial position error set corresponding to the characteristic point pairs; r denotes a spatial rotation matrix that minimizes spatial position error, and t denotes a spatial transformation matrix.
S42: will EiAnd converting the pose information into a quaternion format for storage to obtain an interframe pose transformation matrix.
In step S5, the local spatial three-dimensional information collected at different positions is spliced with the pose transformation matrix to form global spatial three-dimensional information, including:
combining the local space three-dimensional information under the images at different moments with the interframe pose transformation matrix; and (3) splicing the spatial three-dimensional information by using a spatial transformation matrix through a formula (5) to form global spatial three-dimensional information:
Pi＝T*P' (5)
wherein: piRepresenting a set of spatial three-dimensional points, P, in the current original imagei' represents a set of spatial three-dimensional points in the previous frame of image; and T represents the calculated inter-frame pose transformation matrix.
In this embodiment, the image acquisition resolution is 1280 × 720, the maximum number of spatial three-dimensional points that can be acquired by a single image is 921600, and the corresponding actual coverage area is 10m × 7m according to the height of the installation position.
Splicing the control three-dimensional points in each group of images, and counting the spatial three-dimensional point information under each group of images as shown in table 5:
TABLE 5
Figure BDA0003246401200000123
Figure BDA0003246401200000131
In this embodiment, the detection coverage is increased to 20m × 7m by using the global point cloud image obtained by spatial three-dimensional information stitching.
In step S6, the converting the global space three-dimensional information into the coordinate system of the overhead traveling crane in the storage area through the coordinate system, and providing the information to the overhead traveling crane for use, includes:
performing coordinate system fitting by acquiring position coordinate information of the same detection point in the same scene, and calculating a coordinate transformation relation; and converting the global space three-dimensional information into a library area coordinate system, and providing the library area coordinate system for the overhead travelling crane to use.
In this embodiment, coordinate information of the same detection point of the visual detection system and the spatial detection system in the same scene is obtained, coordinate system fitting is performed, a coordinate transformation relationship is calculated, and finally, a detection result based on depth vision is transformed to a library area coordinate system and provided to the overhead traveling crane for operation.
In this embodiment, coordinate values of the same point in the library region coordinate system and the first image in the continuous image sequence acquired by the depth camera in the respective coordinate systems are obtained, as shown in table 6:
TABLE 6
Figure BDA0003246401200000132
Figure BDA0003246401200000141
In this embodiment, the fitting result of the spatial transformation matrix between the visual coordinate system and the library area coordinate system is:
Figure BDA0003246401200000142
the embodiment of the invention provides a depth vision-based metallurgical reservoir area global three-dimensional reconstruction device 700, wherein the device 700 is used for realizing the depth vision-based metallurgical reservoir area global three-dimensional reconstruction method, and as shown in fig. 7, the device comprises:
an ORB detection module 701, configured to acquire an original image of consecutive frames and a depth image corresponding to the original image by using a depth camera, so as to form an original image sequence and a depth image sequence; respectively detecting ORB characteristic points in two adjacent frames of original images;
a similarity matching module 702, configured to match ORB feature points between two adjacent original images to obtain an initial matching point pair; and screening the initial matching point pairs to obtain matching information of the characteristic point pairs and pixel coordinates of the characteristic point pairs.
A local information calculation module 703, configured to extract depth information of a depth image corresponding to an original image through a depth image sequence; and combining the pixel coordinates of the matched characteristic points and the corresponding depth information to obtain local space three-dimensional information.
And the pose transformation calculating module 704 is used for calculating an interframe pose transformation matrix.
The global information splicing module 705 is configured to calculate all inter-frame pose transformation matrices in the original image sequence; and splicing the local space three-dimensional information collected at different positions by combining the pose transformation matrix to form global space three-dimensional information.
And the coordinate system conversion module 706 is configured to convert the global space three-dimensional information into a library area overhead crane coordinate system through a coordinate system, and provide the library area overhead crane with the global space three-dimensional information.
In this embodiment, the depth camera frame is installed on the overhead traveling crane and connected to the ground server via the ethernet, and the original image and the depth image covering the entire metallurgical storage area are collected in real time in a surface scanning manner along with the movement of the overhead traveling crane.
The ORB detection module 701 is used for acquiring original images and depth images of continuous frames by a depth camera arranged on the overhead traveling crane to move along with the overhead traveling crane; forming an original image sequence and a depth image sequence;
selecting a current original image and a previous frame of original image to extract ORB characteristic points to obtain pixel coordinates of the ORB characteristic points;
after determining the pixel coordinate position of the ORB feature point, the ORB feature point is taken as the center, the window size is selected to be 13 pixel points, 128 groups of comparison point pairs are selected in the window, and the binary coding string of the ORB feature point descriptor is calculated by comparing the gray value of the central pixel with the gray value of the pixel points in the window.
A similarity matching module 702, configured to match ORB feature points detected in the current original image and the previous original image to form a matching point pair; using Hamming distance percentage to measure the consistency of the matching point pairs;
selecting matching point pairs of which the code exclusive OR value is smaller than the Hamming distance percentage threshold value in the binary code string, and setting the group of characteristic points as initial matching point pairs:
determining an optimal transformation matrix meeting the maximum matching point time pair through a random consensus (RANSAC) algorithm, and removing the outliers;
collecting position coordinate information of crown blocks at different images and different moments, calculating the motion direction of the crown blocks, and removing characteristic point pairs of which the relative positions of the current original image and the previous original image do not accord with the motion trend of the crown blocks; and obtaining final feature point pair matching information and feature point pair pixel coordinates.
According to the invention, by using the metallurgy reservoir area global three-dimensional reconstruction system and method integrating depth vision, the spatial three-dimensional information of the metallurgy reservoir area global can be obtained in a spatial three-dimensional information splicing mode, so that the stack is more completely measured.
It will be understood by those skilled in the art that all or part of the steps for implementing the above embodiments may be implemented by hardware, or may be implemented by a program instructing relevant hardware, where the program may be stored in a computer-readable storage medium, and the above-mentioned storage medium may be a read-only memory, a magnetic disk or an optical disk, etc.
The above description is only for the purpose of illustrating the preferred embodiments of the present invention and is not to be construed as limiting the invention, and any modifications, equivalents, improvements and the like that fall within the spirit and principle of the present invention are intended to be included therein.

Claims (10)

1. A metallurgy reservoir global three-dimensional reconstruction method based on depth vision is characterized by comprising the following steps:
s1: acquiring original images of continuous frames and depth images corresponding to the original images through a depth camera to form an original image sequence and a depth image sequence; respectively detecting ORB characteristic points in two adjacent frames of original images;
s2: matching ORB characteristic points between two adjacent frames of original images to obtain an initial matching point pair; screening the initial matching point pairs to obtain matching information of the characteristic point pairs and pixel coordinates of the characteristic point pairs;
s3: extracting depth information of a depth image corresponding to the original image through the depth image sequence; combining the matched characteristic point pairs with pixel coordinates and corresponding depth information to obtain local space three-dimensional information;
s4: calculating an interframe pose transformation matrix of the original image;
s5: repeatedly executing S1-S4 until all the inter-frame pose transformation matrixes in the original image sequence are calculated; splicing the local space three-dimensional information collected at different positions by combining a pose transformation matrix to form global space three-dimensional information;
s6: and converting the coordinate of the global space three-dimensional information to a library area crown block coordinate system through a coordinate system, and providing the coordinate for the crown block.
2. The depth vision-based metallurgical reservoir global three-dimensional reconstruction method according to claim 1, wherein in step S1, acquiring original images of consecutive frames and depth images corresponding to the original images by a depth camera, and forming an original image sequence and a depth image sequence; respectively detecting ORB characteristic points in two adjacent frames of original images, comprising:
s11: acquiring original images and depth images of continuous frames by a depth camera arranged on an overhead traveling crane to move along with the overhead traveling crane; forming an original image sequence and a depth image sequence;
s12: selecting a current original image and a previous frame of original image to extract ORB characteristic points to obtain pixel coordinates of the ORB characteristic points;
s13: taking the ORB feature point as a center, selecting 13 pixel points according to the size of the window, selecting 128 groups of comparison point pairs in the window, and calculating a binary coding string of the ORB feature point descriptor according to a formula (1) by comparing the gray value of the central pixel with the gray value of the pixel points in the window:
Figure FDA0003246401190000021
wherein, I(p)Representing the gray value of the central pixel, I(q)Representing the gray values of the pixels within the window; n is a radical ofpRepresenting pixel points contained in a central pixel neighborhood within a set window size range;
if I (q) ≧ I (p), the binary encoding string is marked as 0, if I (q) < I (p), the binary encoding string is marked as 1.
3. The depth vision-based metallurgical library area global three-dimensional reconstruction method according to claim 2, wherein in the step S2, ORB feature points between two adjacent frames of original images are matched to obtain an initial matching point pair; screening the initial matching point pairs to acquire the matching information of the characteristic point pairs and the pixel coordinates of the characteristic point pairs, and the method comprises the following steps:
s21: matching ORB characteristic points detected in the current original image and the previous frame original image to form a matching point pair; using Hamming distance percentage to measure the consistency of the matching point pairs;
s22: substituting the initial matching point pair obtained in S21 into formula (2) by using a random consensus algorithm RANSAC:
Figure FDA0003246401190000022
wherein, (x ', y') represents the initial matching point pair position in the current original image; (x, y) represents the initial matching point pair position of the previous frame image; s represents a scale parameter;
determining an optimal transformation matrix meeting the maximum matching point time pair, and eliminating the cluster points;
s23: collecting position coordinate information of crown blocks at different images and different moments, calculating the motion direction of the crown blocks, and removing characteristic point pairs of which the relative positions of the current original image and the previous original image do not accord with the motion trend of the crown blocks; and obtaining final feature point pair matching information and feature point pair pixel coordinates.
4. The method for the global three-dimensional reconstruction of the metallurgy library area based on the depth vision of claim 1, wherein in the step S3, the depth information of the depth image corresponding to the original image is extracted through a depth image sequence; combining the matched characteristic point pairs with pixel coordinates and corresponding depth information to obtain local space three-dimensional information, wherein the local space three-dimensional information comprises the following steps:
s31: extracting a depth value D of a depth image corresponding to the original image through the depth image sequence;
s32: substituting the pixel coordinates of the feature point pair and the depth value D into formula (3) according to the pixel coordinates of the feature point pair obtained in step S2 in combination with the depth value D, calculating the local space three-dimensional coordinates of the feature point, and obtaining local space three-dimensional information:
Figure FDA0003246401190000032
wherein: (u, v) represents feature point pixel coordinates; cx,CyRepresenting camera calibration parameters and respectively representing position coordinates of image principal points; f. ofx,fyFor camera calibration parameters, fxRepresenting the component of the focal length along the X-axis direction in the image pixel coordinate system; f. ofyRepresents the component of the focal length along the Y-axis direction in the image pixel coordinate system; x, Y, D represent the calculated spatial three-dimensional coordinates.
5. The metallurgy reservoir global three-dimensional reconstruction method based on depth vision according to claim 1, wherein in the step S4, calculating an inter-frame pose transformation matrix of an original image comprises:
s41: constructing a spatial point set error item between the current original image and the previous frame original image, and calculating a group of spatial transformation matrixes with minimum spatial position errors through Singular Value Decomposition (SVD) of a formula (4):
Figure FDA0003246401190000031
wherein: p is a radical ofiRepresenting a space three-dimensional point set q obtained by matching the characteristic points of the current original imageiRepresenting a space three-dimensional point set obtained after the previous frame of original image is matched with the feature points; eiRepresenting a spatial transformation matrix [ R | t]A spatial position error set corresponding to the characteristic point pairs; r represents a spatial rotation matrix that minimizes spatial position error, and t represents a spatial transformation matrix;
s42: will EiAnd converting the pose information into a quaternion format for storage to obtain an interframe pose transformation matrix.
6. The metallurgy reservoir global three-dimensional reconstruction method based on depth vision according to claim 5, wherein in step S5, the stitching of the local space three-dimensional information collected at different positions with the pose transformation matrix to form global space three-dimensional information comprises:
combining the local space three-dimensional information under the images at different moments with the interframe pose transformation matrix; and (3) splicing the spatial three-dimensional information by using a spatial transformation matrix through a formula (5) to form global spatial three-dimensional information:
Pi＝T*P' (5)
wherein: piRepresenting a set of spatial three-dimensional points, P ', in the current original image'iRepresenting a spatial three-dimensional point set in the previous frame of image; and T represents the calculated inter-frame pose transformation matrix.
7. The metallurgy reservoir global three-dimensional reconstruction method based on depth vision according to claim 1, wherein in step S6, the coordinates in the global space three-dimensional information are converted to a position below a reservoir crown block coordinate system through a coordinate system and provided for a crown block to use, and the method comprises:
performing coordinate system fitting by acquiring position coordinate information of the same detection point in the same scene, and calculating a coordinate transformation relation; and converting the coordinates in the three-dimensional information of the global space into a library area coordinate system, and providing the coordinates for the overhead travelling crane for use.
8. A depth vision based global three-dimensional reconstruction device for a metallurgical reservoir, which is applied to the method according to any one of claims 1 to 7, and comprises:
the ORB detection module is used for acquiring original images of continuous frames and depth images corresponding to the original images through a depth camera to form an original image sequence and a depth image sequence; respectively detecting ORB characteristic points in two adjacent frames of original images;
the similarity matching module is used for matching ORB characteristic points between two adjacent frames of original images to obtain an initial matching point pair; screening the initial matching point pairs to obtain matching information of the characteristic point pairs and pixel coordinates of the characteristic point pairs;
the local information calculation module is used for extracting the depth information of the depth image corresponding to the original image through the depth image sequence; combining the matched characteristic point pairs with pixel coordinates and corresponding depth information to obtain local space three-dimensional information;
the pose transformation calculation module is used for calculating an interframe pose transformation matrix;
the global information splicing module is used for calculating all interframe pose transformation matrixes in the original image sequence; splicing the local space three-dimensional information collected at different positions by combining a pose transformation matrix to form global space three-dimensional information;
and the coordinate system conversion module is used for converting the coordinates in the three-dimensional information of the global space to a coordinate system of the overhead crane in the storage area through the coordinate system and providing the coordinates for the overhead crane.
9. The metallurgy reservoir global three-dimensional reconstruction device based on depth vision according to claim 8, wherein the ORB detection module is used for acquiring the original images and the depth images of the continuous frames by a depth camera arranged on the crown block moving along with the crown block; forming an original image sequence and a depth image sequence;
selecting a current original image and a previous frame of original image to extract ORB characteristic points to obtain pixel coordinates of the ORB characteristic points;
after determining the pixel coordinate position of the ORB feature point, selecting 13 pixel points by taking the ORB feature point as a center and the window size, selecting 128 groups of comparison point pairs in the window, and calculating a binary coding string of an ORB feature point descriptor by comparing the gray value of the central pixel with the gray value of the pixel points in the window.
10. The device for reconstructing metallurgy library area global three-dimensional based on depth vision according to claim 8, wherein the similarity matching module is configured to match ORB feature points detected in a current original image and a previous original image to form a matching point pair; using Hamming distance percentage to measure the consistency of the matching point pairs;
determining an optimal transformation matrix meeting the maximum matching point time pair through a random consensus (RANSAC) algorithm, and removing the outliers;
collecting position coordinate information of crown blocks at different images and different moments, calculating the motion direction of the crown blocks, and removing characteristic point pairs of which the relative positions of the current original image and the previous original image do not accord with the motion trend of the crown blocks; and obtaining final feature point pair matching information and feature point pair pixel coordinates.
CN202111034419.5A 2021-09-03 2021-09-03 Metallurgical reservoir area global three-dimensional reconstruction method and device based on depth vision Pending CN113989428A (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202111034419.5A CN113989428A (en) 2021-09-03 2021-09-03 Metallurgical reservoir area global three-dimensional reconstruction method and device based on depth vision

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202111034419.5A CN113989428A (en) 2021-09-03 2021-09-03 Metallurgical reservoir area global three-dimensional reconstruction method and device based on depth vision

Publications (1)

Publication Number Publication Date
CN113989428A true CN113989428A (en) 2022-01-28

Family

ID=79735351

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202111034419.5A Pending CN113989428A (en) 2021-09-03 2021-09-03 Metallurgical reservoir area global three-dimensional reconstruction method and device based on depth vision

Country Status (1)

Country Link
CN (1) CN113989428A (en)

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN116952413A (en) * 2023-09-19 2023-10-27 徐州凌南生物科技有限公司 Temperature measurement method and system for biomass fuel stack

Cited By (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN116952413A (en) * 2023-09-19 2023-10-27 徐州凌南生物科技有限公司 Temperature measurement method and system for biomass fuel stack
CN116952413B (en) * 2023-09-19 2023-11-24 徐州凌南生物科技有限公司 Temperature measurement method and system for biomass fuel stack

Similar Documents

Publication Publication Date Title
CN111145238B (en) Three-dimensional reconstruction method and device for monocular endoscopic image and terminal equipment
CN110569704B (en) Multi-strategy self-adaptive lane line detection method based on stereoscopic vision
JP6126809B2 (en) Egomotion estimation system and method
CN104484648B (en) Robot variable visual angle obstacle detection method based on outline identification
CN106875437B (en) RGBD three-dimensional reconstruction-oriented key frame extraction method
CN111340867A (en) Depth estimation method and device for image frame, electronic equipment and storage medium
CN105335955A (en) Object detection method and object detection apparatus
CN109376641B (en) Moving vehicle detection method based on unmanned aerial vehicle aerial video
CN112184792B (en) Road gradient calculation method and device based on vision
CN111996883B (en) Method for detecting width of road surface
CN112929626B (en) Three-dimensional information extraction method based on smartphone image
CN109215118B (en) Incremental motion structure recovery optimization method based on image sequence
CN114693619A (en) Rigid contact net abrasion detection method, equipment and storage medium
CN111798505A (en) Monocular vision-based dense point cloud reconstruction method and system for triangularized measurement depth
CN112541423A (en) Synchronous positioning and map construction method and system
CN112630469A (en) Three-dimensional detection method based on structured light and multi-light-field camera
CN113989428A (en) Metallurgical reservoir area global three-dimensional reconstruction method and device based on depth vision
CN111325828A (en) Three-dimensional face acquisition method and device based on three-eye camera
CN112132971B (en) Three-dimensional human modeling method, three-dimensional human modeling device, electronic equipment and storage medium
CN111476062A (en) Lane line detection method and device, electronic equipment and driving system
Sheng et al. Mobile robot localization and map building based on laser ranging and PTAM
CN112164044A (en) Wear analysis method of rigid contact net based on binocular vision
CN113884025B (en) Method and device for detecting optical loop of additive manufacturing structure, electronic equipment and storage medium
CN113674407B (en) Three-dimensional terrain reconstruction method, device and storage medium based on binocular vision image
CN109089100B (en) Method for synthesizing binocular stereo video

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination