CN117830385A - Material pile volume measurement method, device, electronic equipment and storage medium - Google Patents

Abstract

The invention provides a method and a device for measuring the volume of a material pile, electronic equipment and a storage medium. The method comprises the following steps: respectively acquiring material pile images under different visual angles; respectively carrying out background segmentation on each material pile image to obtain a frame image containing the material pile corresponding to each material pile image; dividing windows of each frame image respectively, and carrying out three-dimensional matching on the frame images based on gray average values and gradient information corresponding to each window to obtain optimal parallax of each corresponding window in each frame image; and determining three-dimensional point cloud data of the material pile based on the optimal parallax, and triangulating the three-dimensional point cloud data to obtain the volume of the material pile. The invention can improve the measurement precision of the volume of the material pile.

Description

Material pile volume measurement method, device, electronic equipment and storage medium

Technical Field

The present invention relates to the field of machine vision technology, and in particular to a material pile volume measurement method, device, electronic equipment and storage medium.

Background technique

With the rapid development of the world's circulation field, the circulation speed of goods is getting faster and faster, and enterprises have a higher demand for accurate inventory. At present, there are a large number of material piles in thermal power plants, construction sites, granaries and ports, and it is necessary to accurately calculate the volume of the material piles so as to reasonably allocate space resources. However, the material piles are large in size and irregular in shape, and it is impossible to achieve accurate measurement with traditional scales alone.

There are two common methods for measuring the volume of material piles, namely laser measurement and photogrammetry. The laser measurement method measures the distance of the material pile by emitting laser pulses, and then calculates the volume of the material pile. The laser measurement method has high measurement accuracy, but the measurement process is complicated, the speed is slow, and it requires operators to have a high level of professionalism. The cost is also relatively expensive.

Photogrammetry is a method of photographing a material pile with a multi-camera and measuring the volume of the material pile based on the image of the material pile. Photogrammetry has the advantages of simple operation, high efficiency and low cost. However, the material pile has weak texture, and the shooting process is easily affected by light, resulting in low volume measurement accuracy.

Summary of the invention

Embodiments of the present invention provide a material pile volume measurement method, device, electronic device and storage medium to solve the problem of low volume measurement accuracy when measuring the material pile volume based on a material pile image.

In a first aspect, an embodiment of the present invention provides a method for measuring the volume of a material pile, comprising:

Acquire images of the material pile at different viewing angles respectively;

Perform background segmentation on each material pile image respectively to obtain a frame image containing the material pile corresponding to each material pile image;

Divide each frame image into windows respectively, and perform stereo matching on the frame images based on the grayscale mean and gradient information corresponding to each window to obtain the optimal disparity of each corresponding window in each frame image;

Based on the optimal parallax, three-dimensional point cloud data of the material pile is determined, and the three-dimensional point cloud data is triangulated to obtain the volume of the material pile.

In a possible implementation, the frame image includes a first frame image and a second frame image;

The step of dividing each frame image into windows and performing stereo matching on the frame images based on the grayscale mean and gradient information corresponding to each window to obtain the optimal disparity of each corresponding window in each frame image includes:

According to a preset window size, the first frame image and the second frame image are divided into a plurality of windows respectively;

Calculating the grayscale mean and gradient information corresponding to each window respectively, and calculating the matching cost between each window in the first frame image and each window in the second frame image based on the grayscale mean and gradient information corresponding to each window;

Cost aggregation, disparity calculation and disparity optimization are performed based on the matching cost to obtain an optimal disparity map; wherein the optimal disparity map includes the optimal disparity between each window in the first frame image and the corresponding window in the second frame image.

In a possible implementation, calculating the grayscale mean corresponding to each window includes:

according to Calculate the grayscale mean corresponding to each window;

Where, I _avg (x, y) represents the grayscale mean corresponding to the window with the coordinates of the upper left corner vertex (x, y), ω ₁ represents the length of the window, ω ₂ represents the width of the window, i represents the length variable, j represents the width variable, and I(x+i, y+j) represents the grayscale value of the pixel point with coordinates (x+i, y+j) in the window;

The gradient information includes: normalized gradient amplitude; calculating the gradient information corresponding to each window, including:

Obtaining the horizontal gradient vector and the vertical gradient vector corresponding to each window respectively, and calculating the gradient amplitude corresponding to each window based on the horizontal gradient vector and the vertical gradient vector corresponding to each window;

The gradient amplitude corresponding to each window is normalized respectively to obtain the normalized gradient amplitude corresponding to each window.

In a possible implementation, based on the grayscale mean and gradient information corresponding to each window, the matching cost between each window in the first frame image and each window in the second frame image is calculated, including:

Calculating the grayscale differences between each window in the first border image and each window in the second border image according to the grayscale mean values corresponding to each window in the first border image and the second border image;

Calculating local structural information between each window in the first frame image and each window in the second frame image according to gradient information corresponding to each window in the first frame image and the second frame image;

According to the grayscale difference and the local structure information, the matching cost between each window in the first frame image and each window in the second frame image is calculated respectively.

In a possible implementation, calculating the grayscale difference between each window in the first border image and each window in the second border image according to the grayscale mean corresponding to each window in the first border image and the second border image, respectively, includes:

According to D(x,y)=|I _lavg (x,y)-I _ravg (x,y)|, the grayscale difference between each window in the first frame image and each window in the second frame image is calculated respectively;

Among them, D(x,y) represents the grayscale difference between the window with the upper left corner vertex coordinates (x,y) in the first border image and the window with the upper left corner vertex coordinates (x,y) in the second border image, I _lavg (x,y) represents the grayscale mean corresponding to the window with the upper left corner vertex coordinates (x,y) in the first border image, and I _lavg (x,y) represents the grayscale mean corresponding to the window with the upper left corner vertex coordinates (x,y) in the second border image.

In a possible implementation, respectively calculating the local structural information between each window in the first frame image and each window in the second frame image according to the gradient information corresponding to each window in the first frame image and the second frame image includes:

According to S(x,y)=N _l (x,y)+N _r (x,y), local structural information between each window in the first frame image and each window in the second frame image is calculated respectively;

Among them, S(x,y) represents the local structural information between the window with the upper left corner vertex coordinates (x,y) in the first border image and the window with the upper left corner vertex coordinates (x,y) in the second border image, N _l (x,y) represents the gradient information corresponding to the window with the upper left corner vertex coordinates (x,y) in the first border image, and N _r (x,y) represents the gradient information corresponding to the window with the upper left corner vertex coordinates (x,y) in the second border image.

In a possible implementation, respectively calculating the matching cost between each window in the first frame image and each window in the second frame image according to the grayscale difference and the local structure information includes:

According to C(x,y)=αD(x,y)+βS(x,y), the matching cost between each window in the first frame image and each window in the second frame image is calculated respectively;

Among them, C(x,y) represents the matching cost between the window with the upper left corner vertex coordinates (x,y) in the first border image and the window with the upper left corner vertex coordinates (x,y) in the second border image, α represents the first weight, β represents the second weight, D(x,y) represents the grayscale difference between the window with the upper left corner vertex coordinates (x,y) in the first border image and the window with the upper left corner vertex coordinates (x,y) in the second border image, and S(x,y) represents the local structural information between the window with the upper left corner vertex coordinates (x,y) in the first border image and the window with the upper left corner vertex coordinates (x,y) in the second border image.

In a second aspect, an embodiment of the present invention provides a material pile volume measuring device, comprising:

An acquisition module, used to respectively acquire images of the material pile at different viewing angles;

A processing module, used to perform background segmentation on each material pile image respectively, to obtain a frame image containing the material pile corresponding to each material pile image;

The processing module is further used to divide each frame image into windows respectively, and perform stereo matching on the frame images based on the grayscale mean and gradient information corresponding to each window to obtain the optimal disparity of each corresponding window in each frame image;

A measurement module is used to determine the three-dimensional point cloud data of the material pile based on the optimal parallax, and triangulate the three-dimensional point cloud data to obtain the volume of the material pile.

In a third aspect, an embodiment of the present invention provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the processor implements the steps of the method described in the first aspect or any possible implementation manner of the first aspect.

In a fourth aspect, an embodiment of the present invention provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the method described in the first aspect or any possible implementation method of the first aspect are implemented.

The embodiment of the present invention provides a material pile volume measurement method, device, electronic device and storage medium, which divides the frame image containing the material pile corresponding to the material pile image under different viewing angles into windows, and stereo matching the frame image based on the grayscale mean and gradient information corresponding to each window to obtain the optimal disparity, thereby determining the volume of the material pile based on the optimal disparity. Among them, by dividing the window and performing stereo matching based on the grayscale mean and gradient information corresponding to each window, on the one hand, it is possible to capture slight changes in the frame image, and on the other hand, it is possible to improve the texture poverty caused by the influence of lighting, so as to improve the stereo matching accuracy, and then improve the material pile volume measurement accuracy.

BRIEF DESCRIPTION OF THE DRAWINGS

In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

FIG1 is a flow chart of a method for measuring the volume of a material pile provided in an embodiment of the present invention;

FIG2 is a flowchart of an implementation of determining an optimal disparity provided by an embodiment of the present invention;

FIG3 is a flowchart of an implementation of calculating a matching cost provided by an embodiment of the present invention;

4 is a schematic diagram of the structure of a material pile volume measuring device provided in an embodiment of the present invention;

FIG. 5 is a schematic diagram of an electronic device provided by an embodiment of the present invention.

Detailed ways

In the following description, specific details such as specific system structures, technologies, etc. are provided for the purpose of illustration rather than limitation, so as to provide a thorough understanding of the embodiments of the present invention. However, it should be clear to those skilled in the art that the present invention may be implemented in other embodiments without these specific details. In other cases, detailed descriptions of well-known systems, devices, circuits, and methods are omitted to prevent unnecessary details from obstructing the description of the present invention.

In the related art, there are two methods for measuring the volume of material piles, namely laser measurement and photogrammetry. The laser measurement method measures the distance of the material pile by emitting laser pulses, and then calculates the volume of the material pile. The laser measurement method has high measurement accuracy, but the measurement process is complicated, the speed is slow, and the operator is required to have a high level of professionalism, and the cost is relatively expensive. The photogrammetry method is to shoot the material pile through a multi-eye camera and measure the volume of the material pile based on the image of the material pile. Photogrammetry has the advantages of simple operation, high efficiency and low cost. However, the material pile has weak texture, and the shooting process is easily affected by light, resulting in low volume measurement accuracy.

In order to improve the measurement accuracy of the material pile volume, in this embodiment, the border image containing the material pile corresponding to the material pile image is divided into windows, and stereo matching is performed based on the grayscale mean and gradient information corresponding to each window. This can not only capture slight changes in the border image, but also improve the texture poverty caused by lighting, so as to improve the stereo matching accuracy and thus improve the measurement accuracy of the material pile volume.

In order to make the purpose, technical solutions and advantages of the present invention more clear, specific embodiments will be described below in conjunction with the accompanying drawings.

FIG1 is a flow chart of a method for measuring the volume of a material pile provided by an embodiment of the present invention, which is described in detail as follows:

Step 101 : acquiring images of a material pile at different viewing angles respectively.

When acquiring images of the material pile at different viewing angles, the present embodiment may respectively use a plurality of pre-calibrated cameras, such as a first camera and a second camera, to acquire images of the material pile from different viewing angles, or may use a binocular camera to acquire images of the material pile at different viewing angles.

In this embodiment, before using a binocular camera to obtain images of a material pile at different viewing angles, the camera may be calibrated and stereo corrected in advance.

Taking the calibration of a binocular camera as an example, this embodiment can use a preset calibration method, such as Zhang Zhengyou's calibration method, when calibrating the binocular camera. Optionally, this embodiment can use the binocular camera to be calibrated to shoot a checkerboard calibration plate multiple times, and continuously adjust the angle and distance of the checkerboard during the process. The intersection detection algorithm is used to extract the coordinates of the corner points in the captured image. The homography matrix is solved based on the corner point coordinates to calculate the internal and external parameters of the binocular camera, and the distortion coefficient of the binocular camera is obtained using the least squares method. The internal and external parameters and distortion coefficient of the binocular camera are optimized to obtain the optimized internal and external parameters and distortion coefficient, and the calibration is completed.

Here, this embodiment can use the Bouguet algorithm to make the plane origin coordinates of the left and right images taken by the binocular camera the same based on the optimized internal and external parameters and distortion coefficients, so as to achieve surface alignment and row alignment, thereby completing the stereo calibration of the binocular camera. The left and right images after calibration eliminate lens distortion, the epipolar lines of the left and right views are parallel to each other, and the ordinates of the corresponding points are consistent.

Step 102 : performing background segmentation on each material pile image respectively to obtain a frame image including the material pile corresponding to each material pile image.

Among them, this embodiment can remove the background in the material pile image by performing background segmentation on the material pile image, and only retain the material pile, thereby improving the accuracy of subsequent processing results.

When performing background segmentation on the material pile image, this embodiment may adopt a preset segmentation method, such as the maximum inter-class variance method. This embodiment may obtain the pixel threshold that maximizes the inter-class variance by a traversal method. Based on the size relationship between the pixel value of each pixel point and the pixel threshold, each pixel point belonging to the material pile is further determined.

It is understandable that the shape of the material pile is irregular. To facilitate subsequent calculations, the embodiment of the present invention determines the rectangular area corresponding to the material pile as a frame image, and performs subsequent steps based on the frame image. The rectangular area corresponding to the material pile is the smallest rectangular area containing the outer boundary of the material pile.

Step 103 , divide each frame image into windows respectively, and perform stereo matching on the frame images based on the grayscale mean and gradient information corresponding to each window, so as to obtain the optimal disparity of each corresponding window in each frame image.

The embodiment of the present invention can capture small changes in the frame image by dividing each frame image into windows and performing stereo matching of the frame image based on the grayscale mean and gradient information corresponding to each window, thereby improving the accuracy of stereo matching and reducing mismatching. In particular, the accuracy of stereo matching can be greatly improved for areas with rich details.

The frame image includes a first frame image and a second frame image.

In some embodiments, referring to FIG. 2 , the above-mentioned window division of each frame image, and stereo matching of the frame images based on the grayscale mean and gradient information corresponding to each window, to obtain the optimal disparity of each corresponding window in each frame image, may include:

Step 201 : dividing the first frame image and the second frame image into a plurality of windows respectively according to preset window sizes.

Exemplarily, the preset window size may be 5×5 or 7×7, that is, 5 pixels×5 pixels, or 7 pixels×7 pixels. Based on the preset window size, the first frame image and the second frame image may be divided into a plurality of windows respectively.

Step 202, respectively calculating the grayscale mean and gradient information corresponding to each window, and based on the grayscale mean and gradient information corresponding to each window, calculating the matching cost between each window in the first frame image and each window in the second frame image.

In some embodiments, calculating the grayscale mean corresponding to each window includes:

according to Calculate the grayscale mean corresponding to each window;

Where, I _avg (x, y) represents the grayscale mean corresponding to the window with the coordinates of the upper left corner vertex (x, y), ω ₁ represents the length of the window, ω ₂ represents the width of the window, i represents the length variable, j represents the width variable, and I(x+i, y+j) represents the grayscale value of the pixel point with the coordinates (x+i, y+j) in the window. Where, 1≤i≤ω ₁ , 1≤j≤ω ₂ .

In essence, for each window, the embodiment of the present invention calculates the average value of the grayscale values of all pixels in the window, and determines the average value as the grayscale mean value corresponding to the window.

It should be noted that the coordinates (x, y) of the upper left corner refer to the coordinate information in the image pixel coordinate system. Correspondingly, the coordinates (x+i, y+j) also refer to the coordinate information in the image pixel coordinate system.

In some embodiments, the gradient information includes: normalized gradient magnitude. Calculating the gradient information corresponding to each window may include:

The horizontal gradient vector and the vertical gradient vector corresponding to each window are obtained respectively, and the gradient amplitude corresponding to each window is calculated based on the horizontal gradient vector and the vertical gradient vector corresponding to each window.

The gradient amplitude corresponding to each window is normalized respectively to obtain the normalized gradient amplitude corresponding to each window.

In this embodiment, filters such as Sobel, Prewitt or Scharr can be used to respectively obtain the horizontal gradient vector and the vertical gradient vector corresponding to each window.

according to The gradient amplitude corresponding to each window can be calculated.

Among them, G(x,y) represents the gradient amplitude corresponding to the window with the upper left corner vertex coordinates (x,y), I _X (x,y) represents the horizontal gradient vector corresponding to the window with the upper left corner vertex coordinates (x,y), and I _Y (x,y) represents the vertical gradient vector corresponding to the window with the upper left corner vertex coordinates (x,y).

according to The normalized gradient amplitude corresponding to each window can be obtained.

Among them, N(x,y) represents the normalized gradient amplitude corresponding to the window with the upper left corner vertex coordinate (x,y), and maxG represents the maximum value of the normalized gradient amplitudes corresponding to each window in the border image where the window with the upper left corner vertex coordinate (x,y) is located.

In some embodiments, referring to FIG. 3 , the above-mentioned calculation of the matching cost between each window in the first frame image and each window in the second frame image based on the grayscale mean and gradient information corresponding to each window may include:

Step 301 , calculating the grayscale differences between each window in the first frame image and each window in the second frame image according to the grayscale means corresponding to each window in the first frame image and the second frame image.

In some embodiments, the grayscale difference between each window in the first frame image and each window in the second frame image may be calculated according to D(x,y)=|I _lavg (x,y)-I _ravg (x,y)|;

Among them, D(x,y) represents the grayscale difference between the window with the upper left corner vertex coordinates (x,y) in the first border image and the window with the upper left corner vertex coordinates (x,y) in the second border image, I _lavg (x,y) represents the grayscale mean corresponding to the window with the upper left corner vertex coordinates (x,y) in the first border image, and I _lavg (x,y) represents the grayscale mean corresponding to the window with the upper left corner vertex coordinates (x,y) in the second border image.

The grayscale difference calculation formula exemplarily shows a method for calculating the grayscale difference between a window with a top left corner vertex coordinate of (x, y) in the first frame image and a window with a top left corner vertex coordinate of (x, y) in the second frame image. The embodiment of the present invention calculates the grayscale difference between each window in the first frame image and each window in the second frame image based on the calculation method.

Step 302 : calculating local structural information between each window in the first frame image and each window in the second frame image according to the gradient information corresponding to each window in the first frame image and the second frame image.

In some embodiments, the local structural information between each window in the first frame image and each window in the second frame image may be calculated according to S(x,y)=N _l (x,y)+N _r (x,y);

Among them, S(x,y) represents the local structural information between the window with the upper left corner vertex coordinates (x,y) in the first border image and the window with the upper left corner vertex coordinates (x,y) in the second border image, N _l (x,y) represents the gradient information corresponding to the window with the upper left corner vertex coordinates (x,y) in the first border image, and N _r (x,y) represents the gradient information corresponding to the window with the upper left corner vertex coordinates (x,y) in the second border image.

The above local structure information calculation formula exemplarily shows a method for calculating the local structure information between a window with a top left vertex coordinate of (x, y) in the first frame image and a window with a top left vertex coordinate of (x, y) in the second frame image. The embodiment of the present invention calculates the local structure information between each window in the first frame image and each window in the second frame image based on the calculation method.

Step 303: Calculate the matching cost between each window in the first frame image and each window in the second frame image according to the grayscale difference and the local structure information.

In some embodiments, the matching cost between each window in the first frame image and each window in the second frame image may be calculated according to C(x, y)=αD(x, y)+βS(x, y);

Among them, C(x,y) represents the matching cost between the window with the upper left corner vertex coordinates (x,y) in the first border image and the window with the upper left corner vertex coordinates (x,y) in the second border image, α represents the first weight, β represents the second weight, D(x,y) represents the grayscale difference between the window with the upper left corner vertex coordinates (x,y) in the first border image and the window with the upper left corner vertex coordinates (x,y) in the second border image, and S(x,y) represents the local structural information between the window with the upper left corner vertex coordinates (x,y) in the first border image and the window with the upper left corner vertex coordinates (x,y) in the second border image.

The above matching cost calculation formula exemplarily shows a method for calculating the matching cost between a window with a top left corner vertex coordinate of (x, y) in the first frame image and a window with a top left corner vertex coordinate of (x, y) in the second frame image. The embodiment of the present invention calculates the matching cost between each window in the first frame image and each window in the second frame image based on the calculation method.

Considering that the material pile has weak texture and is easily affected by light during the shooting process, the texture of the captured material pile image is poor. The embodiment of the present invention calculates the grayscale difference between each window based on the grayscale mean corresponding to each window, which helps to resist the situation of poor texture. In addition, the embodiment of the present invention calculates the matching cost based on the grayscale difference and combines the local structure information to complete the stereo matching, which solves the problem of low stereo matching accuracy caused by the poor texture of the material pile image, improves the calculation accuracy of the matching cost, and thus improves the accuracy of the material pile volume measurement.

Step 203: Perform cost aggregation, disparity calculation and disparity optimization based on the matching cost to obtain an optimal disparity map, wherein the optimal disparity map includes the optimal disparity between each window in the first frame image and the corresponding window in the second frame image.

When performing cost aggregation on the matching cost, the average filtering method can be used to reduce the impact of image noise on the matching cost and make the aggregated matching cost smoother. The calculation formula of the average filtering is:

Among them, C _smooth (x, y) represents the smoothing cost between the window with the upper left corner vertex coordinates (x, y) in the first border image and each window in the second border image after cost aggregation processing, N represents the number of windows in the neighborhood of the window with the upper left corner vertex coordinates (x, y) in the first border image, n represents the nth window in the neighborhood of the window with the upper left corner vertex coordinates (x, y) in the first border image, and C(n) represents the matching cost between the nth window in the neighborhood of the window with the upper left corner vertex coordinates (x, y) in the first border image and each window in the second border image.

The embodiment of the present invention calculates disparity based on the smoothing cost between each window in the first border image and each window in the second border image. The smoothing cost is used to reflect the similarity between two windows. The smaller the smoothing cost, the more similar the windows are. For each window in the first border image, the window with the smallest smoothing cost between the window and the second border image is determined as the same-name point of the window. The coordinate difference between each window in the first border image and its same-name point is calculated respectively, and the coordinate difference is the disparity. The disparity between each window in the first border image and its same-name point constitutes a disparity map. In the embodiment of the present invention, the coordinates of the pixel point at the center position of the window are determined as the coordinates of the window.

Based on the calculated disparity, the embodiment of the present invention further performs disparity optimization to improve the disparity accuracy and obtain the optimal disparity map. The specific process of disparity optimization is: first, the disparity map is subjected to consistency check, and the erroneous disparity and invalid disparity are eliminated and filled. Then, the disparity values in the disparity map are fitted and optimized by sub-pixel fitting technology, and finally the disparity map is smoothed by median filtering to obtain the optimal disparity map. The optimal disparity map includes the optimal disparity between each window in the first border image and the corresponding window in the second border image. Among them, the corresponding window in the second border image is the same-name point of each window in the first border image.

Step 104 , based on the optimal parallax, determine the three-dimensional point cloud data of the material pile, and triangulate the three-dimensional point cloud data to obtain the volume of the material pile.

The embodiment of the present invention obtains the camera baseline and the camera focal length, and calculates the depth information of the material pile based on the camera baseline, the camera focal length and the optimal parallax. The depth information of the material pile includes the depth information corresponding to each window.

according to The depth information corresponding to each window can be calculated.

Among them, Z represents the depth information corresponding to each window, that is, the vertical coordinate corresponding to each window in the world coordinate system, b represents the camera baseline, f represents the camera focal length, and d represents the optimal disparity corresponding to each window.

The embodiment of the present invention uses a reprojection mapping method to map the optimal disparity map to the world coordinate system, and three-dimensional point cloud data can be obtained. The reprojection mapping process involves the image pixel coordinate system, the image physical coordinate system, the camera coordinate system, and the world coordinate system. The above four coordinate systems have the following transformation relationship:

Among them, _Zc represents the vertical coordinate in the camera coordinate system, u represents the horizontal coordinate in the image pixel coordinate system, v represents the vertical coordinate in the image pixel coordinate system, _fx , _fy , _u0 and _v0 are camera parameters, R3 _×3 represents the rotation matrix of the camera, T3 _×1 represents the translation vector of the camera, _Xw represents the horizontal coordinate in the world coordinate system, _Yw represents the vertical coordinate in the world coordinate system, and _Zw represents the vertical coordinate in the world coordinate system.

The embodiment of the present invention performs an inverse transformation on the above transformation relationship, and inputs the depth information, camera parameters and coordinate information corresponding to each window in the image pixel coordinates, so as to calculate the coordinate information corresponding to each window in the world coordinate system, that is, three-dimensional point cloud data.

When triangulating three-dimensional point cloud data, the Delaunay triangulation algorithm can be used. The Delaunay triangulation algorithm projects the three-dimensional point cloud data onto the xoy reference plane and constructs a triangular mesh on the xoy reference plane, so that the vertices of each triangular mesh correspond to the three-dimensional point cloud data of the material pile, that is, each triangle on the xoy reference plane corresponds to the triangle on the surface of the material pile, and forms a spatial geometric body. Finally, the three-dimensional point cloud data is divided into multiple spatial geometric bodies. Among them, each spatial body is composed of a triangular prism and two tetrahedrons. Based on the volume calculation formula, the volume of each triangular prism and each tetrahedron can be calculated, and then the volume of each spatial geometric body can be obtained. The sum of the volumes of all spatial geometric bodies is the volume of the material pile.

When allocating space resources, the volume to be occupied of each space is obtained respectively, and the space with the same volume to be occupied as the material pile is allocated to the material pile. If there is no space with the same volume to be occupied as the material pile, all spaces with a volume to be occupied greater than the material pile volume are determined, and among all spaces with a volume to be occupied greater than the material pile volume, the space with the smallest volume to be occupied is allocated to the material pile.

Compared with the prior art, the embodiment of the present invention divides each frame image into windows, and performs stereo matching on the frame image based on the grayscale mean and gradient information corresponding to each window to obtain the optimal disparity; the volume of the material pile is then determined based on the optimal disparity. Among them, by dividing the window and performing stereo matching based on the grayscale mean and gradient information corresponding to each window, on the one hand, it is possible to capture slight changes in the frame image, and on the other hand, it is possible to improve the texture poverty caused by the influence of lighting, so as to improve the stereo matching accuracy, and then improve the measurement accuracy of the material pile volume.

It should be understood that the order of execution of the steps in the above embodiment does not necessarily mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiment of the present invention.

The following is an embodiment of the device of the present invention. For details not described in detail, reference may be made to the corresponding method embodiment described above.

FIG4 shows a schematic diagram of the structure of a material pile volume measuring device provided by an embodiment of the present invention. For ease of description, only the parts related to the embodiment of the present invention are shown, which are described in detail as follows:

As shown in FIG. 4 , the material pile volume measuring device 4 includes: an acquisition module 41 , a processing module 42 and a measurement module 43 .

An acquisition module 41 is used to acquire images of the material pile at different viewing angles;

The processing module 42 is used to perform background segmentation on each material pile image to obtain a frame image containing the material pile corresponding to each material pile image;

The processing module 42 is further used to divide each frame image into windows respectively, and perform stereo matching on the frame images based on the grayscale mean and gradient information corresponding to each window to obtain the optimal disparity of each corresponding window in each frame image;

The measurement module 43 is used to determine the three-dimensional point cloud data of the material pile based on the optimal parallax, and triangulate the three-dimensional point cloud data to obtain the volume of the material pile.

In a possible implementation, the frame image includes a first frame image and a second frame image; the processing module 42 is specifically configured to:

According to a preset window size, the first frame image and the second frame image are divided into a plurality of windows respectively;

Calculating the grayscale mean and gradient information corresponding to each window respectively, and calculating the matching cost between each window in the first frame image and each window in the second frame image based on the grayscale mean and gradient information corresponding to each window;

Cost aggregation, disparity calculation and disparity optimization are performed based on the matching cost to obtain an optimal disparity map; wherein the optimal disparity map includes an optimal disparity between each window in the first frame image and a corresponding window in the second frame image.

In a possible implementation, the processing module 42 is configured to: Calculate the grayscale mean corresponding to each window;

Where, I _avg (x, y) represents the grayscale mean corresponding to the window with the coordinates of the upper left corner vertex (x, y), ω ₁ represents the length of the window, ω ₂ represents the width of the window, i represents the length variable, j represents the width variable, and I(x+i, y+j) represents the grayscale value of the pixel point with coordinates (x+i, y+j) in the window;

The gradient information includes: normalized gradient amplitude; a processing module 42, specifically used for:

Obtaining the horizontal gradient vector and the vertical gradient vector corresponding to each window respectively, and calculating the gradient amplitude corresponding to each window based on the horizontal gradient vector and the vertical gradient vector corresponding to each window;

The gradient amplitude corresponding to each window is normalized respectively to obtain the normalized gradient amplitude corresponding to each window.

In a possible implementation, the processing module 42 is specifically configured to:

Calculating the grayscale differences between each window in the first border image and each window in the second border image according to the grayscale mean values corresponding to each window in the first border image and the second border image;

Calculating local structural information between each window in the first frame image and each window in the second frame image according to gradient information corresponding to each window in the first frame image and the second frame image;

According to the grayscale difference and the local structure information, the matching cost between each window in the first frame image and each window in the second frame image is calculated respectively.

In a possible implementation, the processing module 42 is used to calculate the grayscale difference between each window in the first frame image and each window in the second frame image according to D(x,y)=|I _lavg (x,y)-I _ravg (x,y)|;

Among them, D(x,y) represents the grayscale difference between the window with the upper left corner vertex coordinates (x,y) in the first border image and the window with the upper left corner vertex coordinates (x,y) in the second border image, I _lavg (x,y) represents the grayscale mean corresponding to the window with the upper left corner vertex coordinates (x,y) in the first border image, and I _lavg (x,y) represents the grayscale mean corresponding to the window with the upper left corner vertex coordinates (x,y) in the second border image.

In a possible implementation, the processing module 42 is used to calculate the local structural information between each window in the first frame image and each window in the second frame image according to S(x,y)=N _l (x,y)+N _r (x,y);

Among them, S(x,y) represents the local structural information between the window with the upper left corner vertex coordinates (x,y) in the first border image and the window with the upper left corner vertex coordinates (x,y) in the second border image, N _l (x,y) represents the gradient information corresponding to the window with the upper left corner vertex coordinates (x,y) in the first border image, and N _r (x,y) represents the gradient information corresponding to the window with the upper left corner vertex coordinates (x,y) in the second border image.

In a possible implementation, the processing module 42 is used to calculate the matching cost between each window in the first frame image and each window in the second frame image according to C(x,y)=αD(x,y)+βS(x,y);

Among them, C(x,y) represents the matching cost between the window with the upper left corner vertex coordinates (x,y) in the first border image and the window with the upper left corner vertex coordinates (x,y) in the second border image, α represents the first weight, β represents the second weight, D(x,y) represents the grayscale difference between the window with the upper left corner vertex coordinates (x,y) in the first border image and the window with the upper left corner vertex coordinates (x,y) in the second border image, and S(x,y) represents the local structural information between the window with the upper left corner vertex coordinates (x,y) in the first border image and the window with the upper left corner vertex coordinates (x,y) in the second border image.

The embodiment of the present invention divides each frame image into windows, and performs stereo matching on the frame image based on the grayscale mean and gradient information corresponding to each window to obtain the optimal disparity; the volume of the material pile is then determined based on the optimal disparity. Among them, the processing module 42 divides the window and performs stereo matching based on the grayscale mean and gradient information corresponding to each window. On the one hand, it can capture slight changes in the frame image, and on the other hand, it can improve the texture poverty caused by the influence of lighting, so as to improve the stereo matching accuracy, and then improve the material pile volume measurement accuracy.

FIG5 is a schematic diagram of an electronic device provided by an embodiment of the present invention. As shown in FIG5 , the electronic device 5 of this embodiment includes: a processor 50, a memory 51, and a computer program 52 stored in the memory 51 and executable on the processor 50. When the processor 50 executes the computer program 52, the steps in the above-mentioned various material pile volume measurement method embodiments are implemented, such as steps 101 to 104 shown in FIG1 . Alternatively, when the processor 50 executes the computer program 52, the functions of the modules/units in the above-mentioned various device embodiments are implemented, such as the functions of modules 41 to 43 shown in FIG4 .

Exemplarily, the computer program 52 may be divided into one or more modules/units, which are stored in the memory 51 and executed by the processor 50 to implement the present invention. The one or more modules/units may be a series of computer program instruction segments capable of implementing specific functions, which are used to describe the execution process of the computer program 52 in the electronic device 5. For example, the computer program 52 may be divided into modules 41 to 43 as shown in FIG. 4 .

The electronic device 5 may be a computing device such as a desktop computer, a notebook, a PDA, and a cloud server. The electronic device 5 may include, but is not limited to, a processor 50 and a memory 51. Those skilled in the art will appreciate that FIG5 is merely an example of the electronic device 5 and does not constitute a limitation on the electronic device 5. The electronic device 5 may include more or fewer components than shown in the figure, or may combine certain components, or different components. For example, the electronic device may also include input and output devices, network access devices, buses, etc.

The processor 50 may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor, etc.

The memory 51 may be an internal storage unit of the electronic device 5, such as a hard disk or memory of the electronic device 5. The memory 51 may also be an external storage device of the electronic device 5, such as a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, etc. equipped on the electronic device 5. Further, the memory 51 may also include both an internal storage unit of the electronic device 5 and an external storage device. The memory 51 is used to store the computer program and other programs and data required by the electronic device. The memory 51 may also be used to temporarily store data that has been output or is to be output.

The technicians in the relevant field can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional units and modules is used as an example for illustration. In practical applications, the above-mentioned function allocation can be completed by different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiment can be integrated in a processing unit, or each unit can exist physically separately, or two or more units can be integrated in one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of software functional units. In addition, the specific names of the functional units and modules are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of this application. The specific working process of the units and modules in the above-mentioned system can refer to the corresponding process in the aforementioned method embodiment, which will not be repeated here.

In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described or recorded in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention.

In the embodiments provided by the present invention, it should be understood that the disclosed devices/electronic devices and methods can be implemented in other ways. For example, the device/electronic device embodiments described above are only schematic. For example, the division of the modules or units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

If the integrated module/unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the present invention implements all or part of the processes in the above-mentioned embodiment method, and can also be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium. When the computer program is executed by the processor, the steps of the above-mentioned various material pile volume measurement method embodiments can be implemented. Among them, the computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file or some intermediate form. The computer-readable medium may include: any entity or device that can carry the computer program code, recording medium, U disk, mobile hard disk, disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electric carrier signal, telecommunication signal and software distribution medium. The embodiments described above are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features may be replaced by equivalents. Such modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention, and should be included in the protection scope of the present invention.

Claims (10)

1. A method of measuring the volume of a stack of materials, comprising:

respectively acquiring material pile images under different visual angles;

respectively carrying out background segmentation on each material pile image to obtain a frame image containing the material pile corresponding to each material pile image;

dividing windows of each frame image respectively, and carrying out three-dimensional matching on the frame images based on gray average values and gradient information corresponding to each window to obtain optimal parallax of each corresponding window in each frame image;

and determining three-dimensional point cloud data of the material pile based on the optimal parallax, and triangulating the three-dimensional point cloud data to obtain the volume of the material pile.

2. The method of claim 1, wherein the frame images comprise a first frame image and a second frame image;

the window division is performed on each frame image, and stereo matching is performed on the frame images based on gray average value and gradient information corresponding to each window, so as to obtain optimal parallax of each corresponding window in each frame image, including:

dividing the first frame image and the second frame image into a plurality of windows according to the preset window size;

Respectively calculating gray average value and gradient information corresponding to each window, and calculating matching cost between each window in the first frame image and each window in the second frame image based on the gray average value and the gradient information corresponding to each window;

performing cost aggregation, parallax calculation and parallax optimization based on the matching cost to obtain an optimal parallax map; the optimal parallax map comprises optimal parallaxes between windows in the first frame image and corresponding windows in the second frame image.

3. The method of claim 2, wherein calculating a gray scale average value for each window comprises:

according toCalculating the gray average value corresponding to each window;

wherein I is _avg (x, y) represents the gray-scale average value, ω, corresponding to the window with the upper left corner vertex coordinates of (x, y) ₁ Representing the length of the window omega ₂ Representing the width of the window, I representing the length variable, j representing the width variable, I (x+i, y+j) representing the gray value of the pixel point in the window with coordinates (x+i, y+j);

the gradient information includes: normalizing the gradient amplitude; calculating gradient information corresponding to each window, including:

respectively obtaining a horizontal gradient vector and a vertical gradient vector corresponding to each window, and calculating a gradient amplitude corresponding to each window based on the horizontal gradient vector and the vertical gradient vector corresponding to each window;

And respectively carrying out normalization processing on the gradient amplitude values corresponding to the windows to obtain normalized gradient amplitude values corresponding to the windows.

4. A method of measuring a volume of a stack of materials according to claim 2 or 3, wherein calculating a matching cost between each window in the first frame image and each window in the second frame image based on gray-scale average and gradient information corresponding to each window comprises:

respectively calculating gray level differences between each window in the first frame image and each window in the second frame image according to gray level average values corresponding to each window in the first frame image and each window in the second frame image;

according to gradient information corresponding to each window in the first frame image and the second frame image, local structure information between each window in the first frame image and each window in the second frame image is calculated respectively;

and respectively calculating the matching cost between each window in the first frame image and each window in the second frame image according to the gray level difference and the local structure information.

5. The method for measuring the volume of a material pile according to claim 4, wherein the calculating the gray scale difference between each window in the first frame image and each window in the second frame image according to the gray scale average value corresponding to each window in the first frame image and the second frame image respectively includes:

According to D (x, y) = |i _lavg (x,y)-I _ravg Respectively calculating gray scale differences between each window in the first frame image and each window in the second frame image;

wherein D (x, y) represents the gray scale difference between the window with the coordinates of the top left corner vertex (x, y) in the first frame image and the window with the coordinates of the top left corner vertex (x, y) in the second frame image, I _lavg (x, y) represents the upper left in the first frame imageGray average value corresponding to window with angular vertex coordinates of (x, y), I _lavg And (x, y) represents a gray-scale average value corresponding to a window with the coordinates of the top left corner vertex of the second frame image being (x, y).

6. The method for measuring the volume of a material pile according to claim 4, wherein the calculating the local structure information between each window in the first frame image and each window in the second frame image according to the gradient information corresponding to each window in the first frame image and the second frame image respectively includes:

according to S (x, y) =N _l (x,y)+N _r (x, y) calculating local structure information between each window in the first frame image and each window in the second frame image respectively;

wherein S (x, y) represents local structure information between a window with (x, y) coordinates of an upper left corner vertex in the first frame image and a window with (x, y) coordinates of an upper left corner vertex in the second frame image, N _l (x, y) represents gradient information corresponding to a window with (x, y) coordinates of the top left corner vertex in the first frame image, N _r And (x, y) represents gradient information corresponding to a window with the top left corner vertex coordinates of (x, y) in the second frame image.

7. The method for measuring volume of a material pile according to claim 4, wherein the calculating the matching cost between each window in the first frame image and each window in the second frame image according to the gray scale difference and the local structure information includes:

calculating matching cost between each window in the first frame image and each window in the second frame image according to C (x, y) =alpha D (x, y) +beta S (x, y);

wherein C (x, y) represents a matching cost between a window with (x, y) coordinates of an upper left corner vertex in the first frame image and a window with (x, y) coordinates of an upper left corner vertex in the second frame image, α represents a first weight, β represents a second weight, D (x, y) represents a gray scale difference between a window with (x, y) coordinates of an upper left corner vertex in the first frame image and a window with (x, y) coordinates of an upper left corner vertex in the second frame image, and S (x, y) represents local structure information between a window with (x, y) coordinates of an upper left corner vertex in the first frame image and a window with (x, y) coordinates of an upper left corner vertex in the second frame image.

8. A material stack volume measuring device, comprising:

the acquisition module is used for respectively acquiring the material pile images under different visual angles;

the processing module is used for respectively carrying out background segmentation on each material pile image to obtain a frame image which corresponds to each material pile image and contains the material pile;

the processing module is further used for dividing windows of the frame images respectively, and carrying out three-dimensional matching on the frame images based on gray average values and gradient information corresponding to the windows to obtain optimal parallax of the corresponding windows in the frame images;

and the measurement module is used for determining three-dimensional point cloud data of the material pile based on the optimal parallax, and triangulating the three-dimensional point cloud data to obtain the volume of the material pile.

9. An electronic device comprising a memory, a processor and a computer program stored in the memory and executable on the processor, characterized in that the processor implements the steps of the method for measuring the volume of a mass pile according to any one of the preceding claims 1 to 7 when the computer program is executed.

10. A computer readable storage medium storing a computer program, characterized in that the computer program when executed by a processor implements the steps of the method for measuring the volume of a pile according to any one of claims 1 to 7.