CN117974686A - Point cloud segmentation method, device and equipment of target object and storage medium - Google Patents

Abstract

The disclosure provides a point cloud segmentation method, device, equipment and storage medium of a target object. The method is used for improving the accuracy of point cloud segmentation. Comprising the following steps: in the hoisting process of the crane, acquiring point cloud data and RGB images containing a target object; extracting the RGB image contours to obtain contours; aiming at any contour, obtaining a current feature vector of the contour based on the position of each pixel point in the contour and the RGB value of each pixel point; determining a target contour of the target object through the current feature vector of each contour and the standard contour feature vector of the target object; according to the target contour, determining the three-dimensional position coordinate of the target object center point in a radar coordinate system; determining the radius of the outer sphere of the target object based on the extreme value of the horizontal position coordinate and the extreme value of the vertical position coordinate in the target contour; and dividing the point cloud data containing the target object by utilizing the three-dimensional position coordinates of the central point of the target object in the radar coordinate system and the radius of the outer sphere to obtain the point cloud data of the target object.

Description

Point cloud segmentation method, device and equipment of target object and storage medium

Technical Field

The present invention relates to the field of data processing, and in particular, to a method, an apparatus, a device, and a storage medium for point cloud segmentation of a target object.

Background

The laser radar and the computer vision technology provide accurate sensing and measuring capability, and the application of the laser radar and the computer vision technology in the field of crane hoisting can obviously improve the intelligent level of hoisting operation. Three-dimensional space point cloud information and two-dimensional image information of a hoisting operation scene can be acquired in real time through a laser radar and an RGB camera. In the complex lifting scene three-dimensional point cloud, real-time and accurate segmentation of a target object point cloud cluster is a precondition and key for realizing the positioning of lifting load (a lifted object in the lifting operation of a crane) in lifting and the real-time monitoring and tracking of the lifting position and the lifting posture in the lifting operation.

The existing real-time target detection and positioning based on laser radar and camera fusion is to acquire a two-dimensional target frame on a two-dimensional image by using a deep learning-based method, then extract corresponding point cloud data falling into the two-dimensional frame by using external parameters calibrated between the laser radar and the camera, and then perform clustering segmentation and screening on the point cloud to obtain suspended point cloud data so as to position a target. Or directly performing 3D target detection on the point cloud data by using a deep learning method to obtain a three-dimensional target frame, and then performing clustering segmentation on the point cloud in the frame to obtain suspended point cloud data. However, such a method requires sample labeling and training of the lifting load prior to the lifting operation, thereby obtaining a network model for target detection. And in actual hoisting operation, the type and shape of each hoisting may be different. Therefore, single-type sample training cannot cover all the payloads, and thus detection and segmentation of all the payloads cannot be automatically completed by means of a single network model trained in advance. The other is to complete the segmentation by setting distance or density parameters, but the method is sensitive to noise and set parameter values, and the accuracy of the point cloud segmentation is directly affected when noise exists near a target object or the parameter is set improperly.

Disclosure of Invention

The exemplary embodiments of the present disclosure provide a method, an apparatus, a device, and a storage medium for point cloud segmentation of a target object, which are used to improve accuracy of point cloud segmentation of the target object.

A first aspect of the present disclosure provides a point cloud segmentation method of a target object, the method including:

in the hoisting process of a crane, acquiring point cloud data and RGB images, which correspond to a specified duration and contain target objects, wherein the target objects are objects which need to be hoisted by the crane;

contour extraction is carried out on the RGB image, and each contour in the RGB image is obtained;

Aiming at any contour, obtaining a current feature vector of the contour based on the position of each pixel point in the contour and the RGB value of each pixel point;

Determining a target contour of the target object in each contour through the current feature vector of each contour and the standard contour feature vector of the target object;

According to the target contour of the target object, determining the three-dimensional position coordinate of the center point of the target object in a radar coordinate system;

Determining the radius of the outer sphere of the target object based on the extreme value of the horizontal position coordinate, the extreme value of the vertical position coordinate and the camera internal reference in the target contour;

and dividing the point cloud data containing the target object by utilizing the three-dimensional position coordinates of the central point of the target object in a radar coordinate system and the radius of the outer sphere of the target object to obtain the point cloud data corresponding to the target object.

In this embodiment, the standard contour feature vector of the target object is matched with each contour in the image to determine the contour of the target object in the image, and the three-dimensional position coordinate of the center point of the target object in the radar coordinate system and the sphere radius of the target object are determined based on the contour of the target object, and finally the point cloud data including the target object is segmented by using the three-dimensional position coordinate of the center point of the target object in the radar coordinate system and the sphere radius of the target object, so as to obtain the point cloud data corresponding to the target object. Therefore, in the implementation of the application, the target object is segmented through the standard contour feature vector of the target object, the shape and the color of the target object can be accurately identified based on the standard contour feature vector of the target object, the detection model of the target object does not need to be trained, the distance or the density parameter of clustering segmentation does not need to be set, and the robustness and the accuracy of the point cloud segmentation of the target object are improved.

A second aspect of the present disclosure provides a point cloud segmentation apparatus for a target object, the apparatus including:

the acquisition module is used for acquiring point cloud data and RGB images, corresponding to the appointed duration, of a target object in the hoisting process of the crane, wherein the target object is an object to be hoisted by the crane;

the contour extraction module is used for extracting the contour of the RGB image to obtain each contour in the RGB image;

the contour vector determining module is used for obtaining a current feature vector of any contour based on the position of each pixel point in the contour and the RGB value of each pixel point;

The matching module is used for determining a target contour of the target object in each contour through the current feature vector of each contour and the standard contour feature vector of the target object;

The center point position determining module is used for determining the three-dimensional position coordinates of the center point of the target object in a radar coordinate system according to the target contour of the target object;

The outer sphere radius determining module is used for determining the outer sphere radius of the target object based on the extreme value of the horizontal position coordinate, the extreme value of the vertical position coordinate and the camera internal reference in the target profile;

And the segmentation module is used for segmenting the point cloud data containing the target object by utilizing the three-dimensional position coordinates of the central point of the target object in the radar coordinate system and the radius of the outer sphere of the target object to obtain the point cloud data corresponding to the target object.

According to a third aspect of embodiments of the present disclosure, there is provided an electronic device, comprising:

at least one processor; and a memory communicatively coupled to the at least one processor; wherein the memory stores instructions for execution by the at least one processor; the instructions are executable by the at least one processor to enable the at least one processor to perform the method as described in the first aspect.

According to a fourth aspect provided by embodiments of the present disclosure, there is provided a computer storage medium storing a computer program for performing the method according to the first aspect.

Drawings

In order to more clearly illustrate the technical solutions of the embodiments of the present disclosure, the drawings that are needed in the description of the embodiments will be briefly described below, it will be apparent that the drawings in the following description are only some embodiments of the present disclosure, and that other drawings may be obtained according to these drawings without inventive effort to a person of ordinary skill in the art.

FIG. 1 is a schematic diagram of a crane in accordance with one embodiment of the disclosure;

FIG. 2 is a schematic diagram of a suitable scenario in accordance with one embodiment of the present disclosure;

FIG. 3 is one of the flow diagrams of the point cloud segmentation method of the target object according to one embodiment of the present disclosure;

FIG. 4 is a flow diagram of determining a current feature vector of a contour according to one embodiment of the present disclosure;

FIG. 5 is a flow chart of a method of determining three-dimensional position coordinates of a center point of a target object in a radar coordinate system according to one embodiment of the present disclosure;

FIG. 6 is a flow diagram of determining the height of a target object itself according to one embodiment of the present disclosure;

FIG. 7 is a flow chart of determining a depth image of a target object according to one embodiment of the present disclosure;

FIG. 8 is a flow chart of a method of determining a ball radius of an exterior of a target object according to one embodiment of the present disclosure;

FIG. 9 is a flow diagram of filtering ground point cloud data according to one embodiment of the present disclosure;

FIG. 10 is a second flow chart of a method for point cloud segmentation of a target object according to one embodiment of the disclosure;

FIG. 11 is a point cloud segmentation apparatus of a target object according to an embodiment of the present disclosure;

fig. 12 is a schematic structural view of an electronic device according to an embodiment of the present disclosure.

Detailed Description

For the purposes of making the objects, technical solutions and advantages of the embodiments of the present disclosure more apparent, the technical solutions of the embodiments of the present disclosure will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present disclosure, and it is apparent that the described embodiments are some embodiments of the present disclosure, but not all embodiments. All other embodiments, which can be made by one of ordinary skill in the art without inventive effort, based on the embodiments in this disclosure are intended to be within the scope of this disclosure.

The term "and/or" in the embodiments of the present disclosure describes an association relationship of association objects, which indicates that three relationships may exist, for example, a and/or B may indicate: a exists alone, A and B exist together, and B exists alone. The character "/" generally indicates that the context-dependent object is an "or" relationship.

The application scenario described in the embodiments of the present disclosure is for more clearly describing the technical solution of the embodiments of the present disclosure, and does not constitute a limitation on the technical solution provided by the embodiments of the present disclosure, and as a person of ordinary skill in the art can know that, with the appearance of a new application scenario, the technical solution provided by the embodiments of the present disclosure is equally applicable to similar technical problems. In the description of the present disclosure, unless otherwise indicated, the meaning of "a plurality" is two or more.

In the prior art, a two-dimensional target frame is acquired on a two-dimensional image by using a method based on deep learning, external parameters calibrated between a laser radar and a camera are used for extracting corresponding point cloud data falling into the two-dimensional frame, and then clustering segmentation and screening are carried out on the point cloud to obtain suspended point cloud data so as to realize the positioning of a target. Or directly performing 3D target detection on the point cloud data by using a deep learning method to obtain a three-dimensional target frame, and then performing clustering segmentation on the point cloud in the frame to obtain suspended point cloud data. However, the two methods require sample labeling and training of the lifting load prior to the lifting operation, so as to obtain a network model for target detection. And in actual hoisting operation, the type and shape of each hoisting may be different. Therefore, single-type sample training cannot cover all the payloads, and thus detection and segmentation of all the payloads cannot be automatically completed by means of a single network model trained in advance. The other is to complete the division among the classes by setting distance or density parameters, the method is sensitive to noise and set parameter values, and the accuracy of the point cloud division can be directly affected when noise exists near a target object or the parameter is set incorrectly.

Therefore, the present disclosure provides a point cloud segmentation method for a target object, in which a standard contour feature vector of the target object is matched with each contour in an image to determine a contour of the target object in the image, a three-dimensional position coordinate of a center point of the target object in a radar coordinate system and an outer sphere radius of the target object are determined based on the contour of the target object, and finally, the point cloud data including the target object is segmented by using the three-dimensional position coordinate of the center point of the target object in the radar coordinate system and the outer sphere radius of the target object, so as to obtain point cloud data corresponding to the target object. Therefore, in the implementation of the application, the target object is segmented through the standard contour feature vector of the target object, the shape and the color of the target object can be accurately identified based on the standard contour feature vector of the target object, the detection model of the target object does not need to be trained, the distance or the density parameter of clustering segmentation does not need to be set, and the robustness and the accuracy of the point cloud segmentation of the target object are improved.

Before describing the method for dividing the target object in detail, the structure of the crane in the present application will be described. As shown in fig. 1, the crane includes a vehicle body 11, a boom 12, a camera 13, and a radar 14. As can be seen from fig. 1, the camera and radar in the embodiment of the application are mounted on the top end of the boom. Wherein the camera 11 is for acquiring RGB images containing the target object. The radar 14 is used to acquire point cloud data containing target objects. The structure of the crane in fig. 1 is for illustration only, but is not limited to the structure of the crane. The following describes aspects of the present disclosure in detail with reference to the accompanying drawings.

As shown in fig. 2, an application scenario of a point cloud segmentation method of a target object is shown, where the application scenario includes a server 110 and a crane 120.

In one possible application scenario, during the hoisting process of the crane 220, the server 210 acquires, from the crane 220, point cloud data and RGB images including a target object corresponding to a specified duration, where the target object is an object that the crane needs to hoist. The server 210 extracts contours of the RGB images to obtain contours in the RGB images; then, the server 210 obtains a current feature vector of the contour according to the position of each pixel point in the contour and the RGB value of each pixel point for any contour; determining a target contour of the target object in each contour through the current feature vector of each contour and the standard contour feature vector of the target object; then, the server 210 determines the three-dimensional position coordinates of the center point of the target object in the radar coordinate system according to the target contour of the target object; determining the radius of the outer sphere of the target object based on the extreme value of the horizontal position coordinate, the extreme value of the vertical position coordinate and the camera internal parameter in the target contour; finally, the server 210 segments the point cloud data containing the target object by using the three-dimensional position coordinates of the center point of the target object in the radar coordinate system and the radius of the outer sphere of the target object, so as to obtain the point cloud data corresponding to the target object.

The server 210 and the crane 220 in fig. 2 may perform information interaction through a communication network, where a communication mode adopted by the communication network may be a wireless communication mode or a wired communication mode.

For example, server 210 may access the network for communication with crane 220 via cellular mobile communication technology, including, for example, fifth generation mobile communication (5th Generation Mobile Networks,5G) technology.

Alternatively, server 210 may access the network for communication with crane 220 via a short-range wireless communication means, including, for example, wireless fidelity (WIRELESS FIDELITY, wi-Fi) technology.

Also, only a single crane 220 and a single server 210 are described in detail in the description of the present application, but it should be understood by those skilled in the art that the illustrated crane 220 and server 210 are intended to represent the operations of the crane 220 and server 210 to which the present application pertains. Rather than implying a limitation on the number, type, location, etc. of cranes 220 and servers 210. It should be noted that the underlying concepts of the exemplary embodiments of this application are not altered if additional modules are added to or individual modules are removed from the illustrated environment.

It should be noted that the method for dividing the point cloud of the target object provided by the present application is not only suitable for the application scenario shown in fig. 2, but also suitable for any point cloud dividing device with a target object.

The point cloud segmentation method of the object according to the exemplary embodiment of the present application will be described below with reference to the accompanying drawings in conjunction with the above-described application scenario, and it should be noted that the above-described application scenario is only shown for the convenience of understanding the method and principle of the present application, and the embodiment of the present application is not limited in any way in this respect.

As shown in fig. 3, a flow chart of a point cloud segmentation method of a target object of the present disclosure may include the following steps:

Step 301: in the hoisting process of a crane, acquiring point cloud data and RGB images, which correspond to a specified duration and contain target objects, wherein the target objects are objects which need to be hoisted by the crane;

the point cloud data comprising the target object, which is acquired in the embodiment of the application, comprises three-dimensional position coordinates of each discrete point in a radar coordinate system. Step 301 may be performed every a specified time period in embodiments of the present application.

In one embodiment, the radar and camera of the crane are jointly calibrated to obtain the camera intrinsic and the camera extrinsic prior to performing step 301.

In the embodiment of the application, the camera and the radar can be calibrated in a combined mode through the existing method, the embodiment of the application is not limited to the combined calibration method between the camera and the radar, and the specific mode of the combined calibration in the embodiment of the application can be set according to actual conditions.

And, the specified duration in the embodiment of the application can be 0.1 second, 1 second, 10 seconds and the like. The specific specified duration may be set according to practical situations, and the embodiment of the present application is not limited to the specific value of the specified duration.

Step 302: contour extraction is carried out on the RGB image, and each contour in the RGB image is obtained;

In the embodiment of the application, the contour extraction algorithm is used for carrying out contour extraction on the RGB image, and the contour extraction algorithm in the embodiment of the application can be a hollowed-out internal method, a boundary tracking method and the like. However, the embodiment of the present application is not limited to the contour extraction algorithm, and the contour extraction algorithm in the embodiment of the present application may be set according to actual situations.

Step 303: aiming at any contour, obtaining a current feature vector of the contour based on the position of each pixel point in the contour and the RGB value of each pixel point;

As shown in fig. 4, for determining the current feature vector of the contour, the following steps may be included:

Step 401: determining Hu moment of the contour according to the position of each pixel point in the contour;

The Hu moment is a set of 7 invariant moments derived from the second and third order central moments. The manner of determining the Hu moment in the embodiment of the present application is a manner in the prior art, and the embodiment of the present application is not described herein again.

Step 402: according to the RGB values of the pixel points, an R component average value, a G component average value and a B component average value are determined;

Wherein the RGB values include R, G, and B values. In one embodiment, step 402 may be embodied as: determining an average value of R values of all pixel points in the contour as the average value of the R components; determining an average value of G values of all pixel points in the contour as the average value of the G component; and determining an average value of B values of all pixel points in the contour as the component average value.

Step 403: and obtaining the current feature vector of the contour based on the Hu moment of the contour, the R component mean value, the G component mean value and the B component mean value.

In one embodiment, step 403 may be embodied as: and determining a vector consisting of the Hu moment of the contour, the R component mean value, the G component mean value and the B component mean value as a current feature vector of the contour.

For example, the Hu moment of the contour is H _μ0、H_μ1、H_μ2、H_μ3、H_μ4、H_μ5、H_μ6. R component mean valueThe mean value of the G component is/>The mean value of the B component is/>The current feature vector of the resulting contour is

Step 304: determining a target contour of the target object in each contour through the current feature vector of each contour and the standard contour feature vector of the target object;

Before describing the manner in step 304 in detail, a description will be given of determining a standard contour feature vector of a target object in the embodiment of the present application, and in one embodiment, before executing step 301, when a boom of a crane is controlled to rise to a specified height above the target object in response to a crane control command sent by a user, the camera can nod to the full view of the target object. And controlling a camera on the crane to capture RGB images of the single-frame target object in a static state of the suspension arm. And extracting all contours including the target object on the RGB image by utilizing a contour extraction algorithm, wherein the contours are coordinate sets of continuous edge pixel points of the object in the image. And (3) superposing and displaying all the outlines and RGB images on a screen of an industrial personal computer in a crane cab, and manually clicking the screen to confirm the standard outline of the target object. And calculating Hu moment, the R component mean value, the G component mean value and the B component mean value for the standard contour of the manually selected target object. And finally, obtaining the standard contour feature vector of the target object based on the Hu moment, the R component mean value, the G component mean value and the B component mean value of the standard contour of the target object.

It should be noted that: in the embodiment of the present application, the Hu moment, the R component mean value, the G component mean value, and the B component mean value of the standard contour of the target object are determined in the same manner as in the foregoing steps 401 to 403, and the embodiment of the present application is not described herein in detail.

Next, a specific manner of determining the target contour of the target object in step 304 will be described. In one embodiment, step 304 may be embodied as: for any one contour, obtaining the similarity between the contour and the target object based on the current feature vector of the contour and the standard contour feature vector of the target object; and determining the contour with the largest similarity value among the contours with the similarity greater than the designated similarity as the target contour of the target object.

The specific similarity in the embodiment of the present application may be set according to actual situations, and the specific numerical value of the specific similarity is not limited in the embodiment of the present application, and the value range of the specific similarity in the embodiment of the present application may be a real number greater than a certain set value.

In one embodiment, the similarity between the profile and the target object is obtained by:

Based on the current feature vector of the contour and the standard contour feature vector of the target object, obtaining a chi-square distance between the current feature vector and the standard contour feature vector; and determining the similarity between the current feature vector and the standard contour feature vector according to the chi-square distance, wherein the similarity is inversely related to the chi-square distance. Wherein the chi-square distance between the current feature vector and the standard contour feature vector can be obtained by formula (1):

Wherein dist _ij is the chi-square distance between the current feature vector i of the contour and the standard contour feature vector j, hui _m is the mth parameter of the Hu moment in the current feature vector i, huj _m is the mth parameter of the Hu moment in the standard contour feature vector j, For the mean value of R component in the current feature vector i,/>Is the mean value of R component in the standard contour feature vector j,/>For the mean value of the G component in the current feature vector i,/>Is the mean value of the G component in the standard contour feature j,/>For the mean value of the B component in the current feature vector i,/>Is the mean of the B component in the standard profile feature j.

Step 305: according to the target contour of the target object, determining the three-dimensional position coordinate of the center point of the target object in a radar coordinate system;

as shown in fig. 5, for determining a three-dimensional position coordinate of a center point of a target object in a radar coordinate system, the method may include the following steps:

Step 501: obtaining 0-order moment and 1-order moment of the target contour through the positions of all pixel points in the target contour of the target object; wherein, 0 th order moment and 1 st order moment of the target contour can be obtained by the formula (2):

Where M is the maximum abscissa of the target contour, N is the maximum ordinate of the target contour, img (x, y) is the pixel value of the pixel point with x abscissa and y ordinate, the 0-order moment is p=0, the value of M ₀₀ when q=0, the 1-order moment is p=1, the value of M ₁₀ when q=0, and the value of p=0, and the value of M ₀₁ when q=1.

Step 502: according to the 0-order moment and the 1-order moment of the target contour, obtaining a two-dimensional position coordinate of a central point of the target contour in the RGB image; wherein, the two-dimensional position abscissa of the center point of the target contour in the image can be obtained by the formula (3):

Wherein C _x is the two-dimensional position abscissa of the center point of the target contour in the image, m ₁₀ is the 1 st moment of the target contour, and m ₀₀ is the 0 th moment of the target contour.

And the two-dimensional position ordinate of the center point of the target contour in the image can be obtained through the formula (4):

Wherein C _y is the ordinate of the two-dimensional position of the center point of the target contour in the image, and m ₀₁ is the other 1-order moment of the target contour of the target object.

Step 503: based on the two-dimensional position coordinates of the center point of the target contour in the RGB image and the camera internal parameters, obtaining the three-dimensional position coordinates of the center point of the target contour in a camera coordinate system;

In one embodiment, step 503 may be embodied as: determining the product of a two-dimensional position coordinate of the center point of the target contour in the RGB image, a matrix corresponding to the camera internal reference and a depth value of the center point of the target contour in a depth image containing the target object as a three-dimensional position coordinate of the center point of the target contour in a camera coordinate system; wherein, the three-dimensional position coordinates of the center point of the target contour in the camera coordinate system can be obtained by the formula (5):

Wherein x _{c_centre} is the three-dimensional position abscissa of the center point of the target contour in the camera coordinate system, y _{c_centre} is the three-dimensional position ordinate of the center point of the target contour in the camera coordinate system, z _{c_centre} is the three-dimensional position ordinate of the center point of the target contour in the camera coordinate system, K is the matrix corresponding to the camera internal reference, C _x is the two-dimensional position abscissa of the center point of the target contour in the RGB image, C _y is the two-dimensional position ordinate of the center point of the target contour in the RGB image, and d _c is the depth value of the corresponding position of the center point of the target contour in the depth image containing the target object.

Step 504: obtaining the three-dimensional position coordinate of the central point of the target object in the camera coordinate system by utilizing the three-dimensional position coordinate of the central point of the target contour in the camera coordinate system and the height of the target object;

In one embodiment, step 504 may be embodied as: determining a three-dimensional position abscissa of a central point of the target contour in three-dimensional position coordinates under a camera coordinate system as the three-dimensional position abscissa of the central point of the target object in the camera coordinate system; and determining the three-dimensional position ordinate of the central point of the target contour in the three-dimensional position coordinates of the camera coordinate system as the three-dimensional position ordinate of the central point of the target object in the camera coordinate system; and obtaining the three-dimensional position vertical coordinate of the central point of the target object in the camera coordinate system by using the three-dimensional position vertical coordinate of the central point of the target contour in the three-dimensional position coordinate of the camera coordinate system and half of the height of the target object. Wherein, the three-dimensional position coordinates of the center point of the target object in the camera coordinate system can be obtained by the formula (6):

wherein x _centre is the three-dimensional position abscissa of the center point of the target object in the camera coordinate system, y _centre is the three-dimensional position ordinate of the center point of the target object in the camera coordinate system, z _centre is the three-dimensional position ordinate of the center point of the target object in the camera coordinate system, and H is the height of the target object itself.

In the following, a description will be given of a method for determining the height of a target object according to an embodiment of the present application, as shown in fig. 6, in order to determine the height of the target object, the method may include the following steps:

Step 601: obtaining a first distance between a camera and the ground according to a width adjusting angle of a crane, the length of a suspension arm of the crane and the height of a vehicle body of the crane, wherein the width adjusting angle is an included angle between the suspension arm of the crane and the ground when the target object does not leave the ground in the lifting process of the crane; wherein the first distance is obtained by formula (7):

h₁＝L*sinα+h_car……(7)；

H ₁ is the first distance, L is the length of the suspension arm, alpha is the amplitude-adjusting angle, and h _car is the height of the crane body.

Step 602: obtaining a second distance between the center point of the target contour and the camera according to the depth value of the center point of the target contour in the depth image of the target object;

In one embodiment, step 602 may be embodied as: and determining a depth value of the central point of the target contour in the depth image of the target object as the second distance.

Step 603: and obtaining the height of the target object by the first distance and the second distance.

In one embodiment, step 603 may be implemented as: and determining the difference value between the first distance and the second distance as the height of the target object.

It should be noted that: the height of the target object in the embodiment of the application is calculated once only when the crane lifts and the target object does not leave the ground, and the height value can be directly used in the calculation of the three-dimensional position of the center point of the subsequent target object in the camera coordinate system and the radius of the outer ball.

Step 505: and obtaining the three-dimensional position coordinate of the central point of the target object in the radar coordinate system through the three-dimensional position coordinate of the central point of the target object in the camera coordinate system and the camera external parameters.

In one embodiment, step 505 may be embodied as: and multiplying the three-dimensional position coordinate of the central point of the target object in the camera coordinate system by the matrix corresponding to the camera external parameter to obtain the three-dimensional position coordinate of the central point of the target object in the radar coordinate system. Wherein, the three-dimensional position coordinates of the center point of the target object in the radar coordinate system can be obtained by the formula (8):

Wherein x _{L_centre} is the three-dimensional position abscissa of the center point of the target object in the radar coordinate system, y _{L_centre} is the three-dimensional position ordinate of the center point of the target object in the radar coordinate system, z _{L_centre} is the three-dimensional position ordinate of the center point of the target object in the radar coordinate system, and T _l-c is the matrix corresponding to the camera external parameters.

In the following, a detailed description of a manner of determining a depth image of a target object in an embodiment of the present application is provided, and as shown in fig. 7, a flowchart for determining a depth image of a target object may include the following steps:

Step 701: transforming the point cloud data containing the target object corresponding to the appointed duration by utilizing the camera external parameter to obtain camera position coordinates of each discrete point in the point cloud data in a camera coordinate system; the camera position coordinates of any one discrete point in the point cloud data in the camera coordinate system can be obtained through the formula (9):

Wherein x _l is the horizontal position coordinate of the discrete point l in the radar coordinate system in the point cloud data containing the target object, y _l is the vertical position coordinate of the discrete point l in the radar coordinate system in the point cloud data containing the target object, z _l is the vertical position coordinate of the discrete point l in the radar coordinate system in the point cloud data containing the target object, T _l-c is the matrix corresponding to the camera external reference, x _l ^′ is the horizontal position coordinate of the discrete point l in the camera coordinate system in the point cloud data containing the target object, y _l ^′ is the vertical position coordinate of the discrete point l in the camera coordinate system in the point cloud data containing the target object, z _l ^′ is the vertical position coordinate of the discrete point l in the camera coordinate system in the point cloud data containing the target object, l epsilon 0, P, and P is the total number of the discrete points in the point cloud data.

Step 702: projecting camera position coordinates of each discrete point in the point cloud data in a preset depth image by utilizing the camera internal parameters to obtain two-dimensional position coordinates of each discrete point in the point cloud data in the depth image, wherein the size of the preset depth image is the same as that of the RGB image; the two-dimensional position coordinates of each discrete point in the point cloud data in the depth image can be obtained through a formula (10):

Wherein u _l is the two-dimensional position abscissa of the discrete point l in the depth image in the point cloud data, v _l is the two-dimensional position ordinate of the discrete point l in the depth image in the point cloud data, and K is the matrix corresponding to the camera internal reference.

Step 703: setting a vertical coordinate value in a camera position coordinate corresponding to any one discrete point as a depth value of the discrete point in the preset depth image;

step 704: and obtaining the depth image of the target object according to the depth value of each discrete point in the preset depth image and the two-dimensional position coordinate of each discrete point in the depth image.

Step 306: determining the radius of the outer sphere of the target object based on the extreme value of the horizontal position coordinate, the extreme value of the vertical position coordinate and the camera internal reference in the target contour;

as shown in fig. 8, to determine the radius of the ball outside the target object, the method may include the following steps:

step 801: obtaining the length of the target contour according to the maximum value and the minimum value in the extreme value of the transverse position coordinate of the target contour;

in one embodiment, step 801 may be embodied as: and determining the difference value between the maximum value and the minimum value of the transverse position coordinates in the target contour as the length of the target contour. Wherein the length of the target profile can be obtained by equation (11):

x_size＝x_max-x_min……(11)；

Where x _size is the length of the target contour, x _max is the maximum of the abscissa in the target contour, and x _min is the minimum of the abscissa in the target contour.

Step 802: obtaining the width of the target contour according to the maximum value and the minimum value in the extreme value of the longitudinal position coordinate of the target contour;

In one embodiment, step 802 may be embodied as: and determining the difference value between the maximum value and the minimum value of the longitudinal position coordinates in the target contour as the width of the target contour. Wherein the width of the target profile can be obtained by equation (12):

y_size＝y_max-y_min……(12)；

Where y _size is the width of the target profile, y _max is the maximum value of the ordinate of the target profile, and y _min is the minimum value of the ordinate in the target profile.

Step 803: obtaining the length of the target object based on the length of the target contour, the depth value of the center point of the target contour in the depth image of the target object and the horizontal focal length in the camera internal parameter; wherein the length of the target object can be obtained by equation (13):

Wherein x _L is the length of the target object, x _size is the length of the target contour, f _x is the horizontal focal length in the camera internal reference, and d _c is the depth value of the center point of the target contour in the depth image of the target object.

Step 804: obtaining the width of the target object based on the width of the target contour, the depth value of the center point of the target contour in the depth image of the target object and the vertical focal length in the camera internal parameter; wherein the width of the target object can be obtained by equation (14):

Where y _W is the width of the target object, y _size is the width of the target outline, and f _y is the vertical focal length in the camera internal reference.

Step 805: and determining half of the maximum value of the length of the target object, the width of the target object and the height of the target object as the radius of the outer sphere of the target object. Wherein the spherical radius of the target object can be obtained by equation (15):

wherein r is the radius of the outer sphere of the target object, and H is the height of the target object.

Step 307: and dividing the point cloud data containing the target object by utilizing the three-dimensional position coordinates of the central point of the target object in a radar coordinate system and the radius of the outer sphere of the target object to obtain the point cloud data corresponding to the target object.

In one embodiment, step 307 may be embodied as: and inputting the three-dimensional position coordinates of the central point of the target object in a radar coordinate system, the radius of the outer sphere of the target object and the point cloud data containing the target object into a minimum segmentation algorithm to perform point cloud segmentation, so as to obtain the point cloud data corresponding to the target object.

The minimum segmentation algorithm in the embodiment of the present application is an algorithm in the prior art, and the embodiment of the present application will not be described herein in detail.

Because the target object is generally placed on the ground before lifting, the ground point cloud and the lifting point cloud are adhered together. To further increase the accuracy of the point cloud segmentation of the target object, in one embodiment, the ground point cloud in the acquired point cloud data needs to be filtered before step 307 is performed. Because the radar is facing the ground, the coordinate system is rotated to position the projection of the ground point cloud on the plane formed by the X-Y axes of the radar before the ground point cloud is filtered. As shown in fig. 9, which is a schematic flow chart for filtering ground point cloud data, the method may include the following steps:

step 901: rotating the point cloud data containing the target object clockwise around the Y axis by a designated angle by using a preset first rotation matrix to obtain rotating point cloud data;

The designated angle in the embodiment of the present application is 90 degrees, but the designated angle in the embodiment of the present application is not limited, and the designated angle in the embodiment of the present application may be limited according to actual situations.

In one embodiment, step 901 may be embodied as: and rotating any one discrete point in the point cloud data containing the target object clockwise around the Y axis by a designated angle through a preset first rotation matrix to obtain a rotated position coordinate corresponding to the discrete point, and determining the rotated position coordinate of each discrete point as the rotation point cloud data.

The rotated position coordinates corresponding to any one of the discrete points in the point cloud data can be obtained by the formula (16):

Wherein x _le is the abscissa after rotation of the x _le discrete point l, y _le is the ordinate after rotation of the discrete point l, z _le is the ordinate after rotation of the discrete point l, x _l is the abscissa before rotation of the discrete point l, y _l is the ordinate before rotation of the discrete point l, z _l is the ordinate before rotation of the discrete point l, For the first rotation matrix.

Step 902: performing ground point cloud filtering on the rotating point cloud data by using a filtering algorithm to obtain rotating point cloud data after ground filtering;

The filtering algorithm in the embodiment of the present application is CSF (Cloth Simulation Filter, cloth analog filtering) algorithm, but the embodiment of the present application is not limited to the filtering algorithm, and the filtering algorithm in the embodiment of the present application may be set according to actual situations.

Step 903: and rotating the rotation point cloud data subjected to ground filtration by a designated angle around the Y-axis anticlockwise direction by using a preset second rotation matrix to obtain the point cloud data subjected to ground filtration under an initial coordinate system, and determining the point cloud data subjected to ground filtration under the initial coordinate system as the point cloud data containing the target object.

In one embodiment, step 903 may be implemented as: and rotating the rotated position coordinates corresponding to the discrete points around the Y-axis anticlockwise direction by a preset second rotation matrix for a designated angle aiming at any one discrete point in the point cloud data containing the target object to obtain initial position coordinates of the discrete points, and obtaining the filtered point cloud data according to the initial position coordinates of each discrete point.

Wherein the filtered point cloud data can be obtained by formula (17):

Where x _l ^′ _e is the initial position abscissa of the discrete point l, y _l ^′ _e is the initial position ordinate of the discrete point l, z _l ^′ _e is the initial position ordinate of the discrete point l, For the second rotation matrix.

For further understanding of the technical solution of the present disclosure, the following detailed description with reference to fig. 10 may include the following steps:

step 1001: in the hoisting process of a crane, acquiring point cloud data and RGB images, which correspond to a specified duration and contain target objects, wherein the target objects are objects which need to be hoisted by the crane;

Step 1002: contour extraction is carried out on the RGB image, and each contour in the RGB image is obtained;

Step 1003: determining Hu moment of the contour according to the position of each pixel point in the contour; according to RGB values of all pixel points in the outline, an R component mean value, a G component mean value and a B component mean value are determined;

step 1004: based on the Hu moment, the R component mean, the G component mean and the B component mean of the contour, obtaining a current feature vector of the contour;

Step 1005: for any one contour, obtaining the similarity between the contour and the target object based on the current feature vector of the contour and the standard contour feature vector of the target object;

Step 1006: determining a contour with the largest similarity value among all contours with the similarity larger than the designated similarity as a target contour of the target object;

step 1007: obtaining 0-order moment and 1-order moment of the target contour through the positions of all pixel points in the target contour of the target object;

Step 1008: according to the 0-order moment and the 1-order moment of the target contour, obtaining a two-dimensional position coordinate of a central point of the target contour in the RGB image;

step 1009: based on the two-dimensional position coordinates of the center point of the target contour in the RGB image and the camera internal parameters, obtaining the three-dimensional position coordinates of the center point of the target contour in a camera coordinate system;

step 1010: obtaining the three-dimensional position coordinate of the central point of the target object in the camera coordinate system by utilizing the three-dimensional position coordinate of the central point of the target contour in the camera coordinate system and the height of the target object;

step 1011: obtaining the three-dimensional position coordinate of the center point of the target object in a radar coordinate system through the three-dimensional position coordinate of the center point of the target object in the camera coordinate system and camera external parameters;

Step 1012: obtaining the length of the target contour according to the maximum value and the minimum value in the extreme value of the transverse position coordinate of the target contour; obtaining the width of the target contour according to the maximum value and the minimum value in the extreme value of the longitudinal position coordinate of the target contour;

Step 1013: obtaining the length of the target object based on the length of the target contour, the depth value of the center point of the target contour in the depth image of the target object and the horizontal focal length in the camera internal parameter; the width of the target object is obtained based on the width of the target contour, the depth value of the center point of the target contour in the depth image of the target object and the vertical focal length in the camera internal parameter;

step 1014: determining half of the maximum value of the length of the target object, the width of the target object and the height of the target object as the radius of the outer sphere of the target object;

step 1015: rotating the point cloud data containing the target object clockwise around the Y axis by a designated angle by using a preset first rotation matrix to obtain rotating point cloud data;

step 1016: performing ground point cloud filtering on the rotating point cloud data by using a filtering algorithm to obtain rotating point cloud data after ground filtering;

Step 1017: rotating the filtered rotating point cloud data counterclockwise around a Y-axis by a designated angle by using a preset second rotating matrix to obtain ground filtered point cloud data under an initial coordinate system, and determining the filtered point cloud data as the point cloud data containing the target object;

Step 1018: and dividing the point cloud data containing the target object by utilizing the three-dimensional position coordinates of the central point of the target object in a radar coordinate system and the radius of the outer sphere of the target object to obtain the point cloud data corresponding to the target object.

Based on the same disclosure concept, the point cloud segmentation method of the target object disclosed in the disclosure can also be realized by a point cloud segmentation device of the target object. The effect of the point cloud segmentation device of the target object is similar to that of the method described above, and will not be described again here.

Fig. 11 is a schematic structural diagram of a point cloud segmentation apparatus of a target object according to an embodiment of the present disclosure.

As shown in fig. 11, the point cloud segmentation apparatus 1100 of the target object of the present disclosure may include an acquisition module 1110, a contour extraction module 1120, a contour vector determination module 1130, a matching module 1140, a center point position determination module 1150, an outer sphere radius determination module 1160, and a segmentation module 1170.

An obtaining module 1110, configured to obtain, during hoisting of a crane, point cloud data and an RGB image, which correspond to a specified duration and include a target object, where the target object is an object that needs to be hoisted by the crane;

The contour extraction module 1120 is configured to perform contour extraction on the RGB image to obtain each contour in the RGB image;

The contour vector determining module 1130 is configured to obtain, for any contour, a current feature vector of the contour based on a position of each pixel point in the contour and the RGB value of each pixel point;

A matching module 1140, configured to determine a target contour of the target object in the contours by using the current feature vector of each contour and the standard contour feature vector of the target object;

a center point position determining module 1150, configured to determine a three-dimensional position coordinate of a center point of the target object in a radar coordinate system according to a target contour of the target object;

An outer sphere radius determining module 1160, configured to determine an outer sphere radius of the target object based on the extremum of the abscissa, the extremum of the ordinate, and the camera reference in the target profile;

The segmentation module 1170 is configured to segment the point cloud data including the target object by using the three-dimensional position coordinate of the center point of the target object in the radar coordinate system and the radius of the outer sphere of the target object, so as to obtain point cloud data corresponding to the target object.

In one embodiment, the contour vector determining module 1130 is specifically configured to:

Determining Hu moment of the contour according to the position of each pixel point in the contour; and

According to the RGB values of the pixel points, an R component average value, a G component average value and a B component average value are determined;

and obtaining the current feature vector of the contour based on the Hu moment of the contour, the R component mean value, the G component mean value and the B component mean value.

In one embodiment, the matching module 1140 is specifically configured to:

for any one contour, obtaining the similarity between the contour and the target object based on the current feature vector of the contour and the standard contour feature vector of the target object;

And determining the contour with the largest similarity value among the contours with the similarity greater than the designated similarity as the target contour of the target object.

In one embodiment, the matching module 1140 executes the current feature vector based on the contour and the standard contour feature vector of the target object to obtain the similarity between the contour and the target object, which is specifically configured to:

based on the current feature vector of the contour and the standard contour feature vector of the target object, obtaining a chi-square distance between the current feature vector and the standard contour feature vector;

And determining the similarity between the current feature vector and the standard contour feature vector according to the chi-square distance, wherein the similarity is inversely related to the chi-square distance.

In one embodiment, the center point location determining module 1150 is specifically configured to:

Obtaining 0-order moment and 1-order moment of the target contour through the positions of all pixel points in the target contour of the target object;

According to the 0-order moment and the 1-order moment of the target contour, obtaining a two-dimensional position coordinate of a central point of the target contour in the RGB image;

Based on the two-dimensional position coordinates of the center point of the target contour in the RGB image and the camera internal parameters, obtaining the three-dimensional position coordinates of the center point of the target contour in a camera coordinate system;

Obtaining the three-dimensional position coordinate of the central point of the target object in the camera coordinate system by utilizing the three-dimensional position coordinate of the central point of the target contour in the camera coordinate system and the height of the target object;

And obtaining the three-dimensional position coordinate of the central point of the target object in the radar coordinate system through the three-dimensional position coordinate of the central point of the target object in the camera coordinate system and the camera external parameters.

In one embodiment, the center point position determining module 1150 performs the steps of 0 th order moment and 1 st order moment according to the target contour to obtain two-dimensional position coordinates of the center point of the target contour in the image, specifically for:

the two-dimensional position abscissa of the center point of the target contour in the image is obtained by the following formula:

Wherein C _x is the two-dimensional position abscissa of the center point of the target contour in the image, m ₁₀ is the 1-order moment of the target contour, and m ₀₀ is the 0-order moment of the target contour;

The ordinate of the two-dimensional position of the center point of the target contour in the image is obtained by the following formula:

Wherein C _y is the ordinate of the two-dimensional position of the center point of the target contour in the image, and m ₀₁ is the other 1-order moment of the target contour of the target object.

In one embodiment, the center point position determining module 1150 performs the two-dimensional position coordinates of the center point of the target contour in the RGB image and the camera internal reference, to obtain the three-dimensional position coordinates of the center point of the target contour in the camera coordinate system, specifically for:

Determining the product of a two-dimensional position coordinate of the center point of the target contour in the RGB image, a matrix corresponding to the camera internal reference and a depth value of the center point of the target contour in a depth image containing the target object as a three-dimensional position coordinate of the center point of the target contour in a camera coordinate system;

The center point position determining module 1150 performs the three-dimensional position coordinate of the center point of the target contour under the camera coordinate system and the height of the target object to obtain the three-dimensional position coordinate of the center point of the target object in the camera coordinate system, specifically for:

Determining a three-dimensional position abscissa of a central point of the target contour in three-dimensional position coordinates under a camera coordinate system as the three-dimensional position abscissa of the central point of the target object in the camera coordinate system; and

Determining the three-dimensional position ordinate of the central point of the target contour in the three-dimensional position coordinates of the camera coordinate system as the three-dimensional position ordinate of the central point of the target object in the camera coordinate system; and

A three-dimensional position vertical coordinate of a central point of the target contour in a three-dimensional position coordinate of a camera coordinate system and a half camera of the height of the target object are obtained, and the three-dimensional position vertical coordinate of the central point of the target object in the camera coordinate system is obtained;

The center point position determining module 1150 executes the three-dimensional position coordinates of the center point of the target object in the camera coordinate system and camera external parameters to obtain the three-dimensional position coordinates of the center point of the target object in the radar coordinate system, specifically for:

And multiplying the three-dimensional position coordinate of the central point of the target object in the camera coordinate system by the matrix corresponding to the camera external parameter to obtain the three-dimensional position coordinate of the central point of the target object in the radar coordinate system.

In one embodiment, the apparatus further comprises:

a target object height determination module 1180 for determining the height of the target object itself by:

Obtaining a first distance between a camera and the ground according to a width adjusting angle of a crane, the length of a suspension arm of the crane and the height of a vehicle body of the crane, wherein the width adjusting angle is an included angle between the suspension arm of the crane and the ground when the target object does not leave the ground in the lifting process of the crane; and

Obtaining a second distance between the center point of the target contour and the camera according to the depth value of the center point of the target contour in the depth image of the target object;

And obtaining the height of the target object by the first distance and the second distance.

In one embodiment, the outer ball radius determination module 1160 is specifically configured to:

obtaining the length of the target contour according to the maximum value and the minimum value in the extreme value of the transverse position coordinate of the target contour; and

Obtaining the width of the target contour according to the maximum value and the minimum value in the extreme value of the longitudinal position coordinate of the target contour;

Obtaining the length of the target object based on the length of the target contour, the depth value of the center point of the target contour in the depth image of the target object and the horizontal focal length in the camera internal parameter; the width of the target object is obtained based on the width of the target contour, the depth value of the center point of the target contour in the depth image of the target object and the vertical focal length in the camera internal parameter;

And determining half of the maximum value of the length of the target object, the width of the target object and the height of the target object as the radius of the outer sphere of the target object.

In one embodiment, the apparatus further comprises:

a depth image determination module 1190 is configured to obtain a depth image of the target object by:

Transforming the point cloud data containing the target object corresponding to the appointed duration by utilizing the camera external parameter to obtain camera position coordinates of each discrete point in the point cloud data in a camera coordinate system;

Projecting camera position coordinates of each discrete point in the point cloud data in a preset depth image by utilizing the camera internal parameters to obtain two-dimensional position coordinates of each discrete point in the point cloud data in the depth image, wherein the size of the preset depth image is the same as that of the RGB image;

setting a vertical coordinate value in a camera position coordinate corresponding to any one discrete point as a depth value of the discrete point in the preset depth image;

And obtaining the depth image of the target object according to the depth value of each discrete point in the preset depth image and the two-dimensional position coordinate of each discrete point in the depth image.

In one embodiment, the dividing module 1170 is specifically configured to:

And inputting the three-dimensional position coordinates of the central point of the target object in a radar coordinate system, the radius of the outer sphere of the target object and the point cloud data containing the target object into a minimum segmentation algorithm to perform point cloud segmentation, and obtaining the point cloud data corresponding to the target object.

Having described a method and apparatus for point cloud segmentation of a target object according to an exemplary embodiment of the present disclosure, next, an electronic device according to another exemplary embodiment of the present disclosure is described.

Those skilled in the art will appreciate that the various aspects of the present disclosure may be implemented as a system, method, or program product. Accordingly, various aspects of the disclosure may be embodied in the following forms, namely: an entirely hardware embodiment, an entirely software embodiment (including firmware, micro-code, etc.) or an embodiment combining hardware and software aspects may be referred to herein as a "circuit," module "or" system.

In some possible implementations, an electronic device according to the present disclosure may include at least one processor, and at least one computer storage medium. Wherein the computer storage medium stores program code which, when executed by a processor, causes the processor to perform the steps in the point cloud segmentation method of the target object according to various exemplary embodiments of the disclosure described above in this specification. For example, the processor may perform steps 301-307 as shown in FIG. 3.

An electronic device 1200 according to such an embodiment of the present disclosure is described below with reference to fig. 12. The electronic device 1200 shown in fig. 12 is merely an example, and should not be construed as limiting the functionality and scope of use of the disclosed embodiments.

As shown in fig. 12, the electronic device 1200 is embodied in the form of a general-purpose electronic device. Components of electronic device 1200 may include, but are not limited to: the at least one processor 1201, the at least one computer storage medium 1202, and a bus 1203 that connects the various system components, including the computer storage medium 1202 and the processor 1201.

Bus 1203 represents one or more of several types of bus structures, including a computer storage medium bus or computer storage medium controller, a peripheral bus, a processor, or a local bus using any of a variety of bus architectures.

Computer storage media 1202 may include readable media in the form of volatile computer storage media, such as random access computer storage media (RAM) 1221 and/or cache storage media 1222, and may further include read only computer storage media (ROM) 1223.

Computer storage media 1202 may also include a program/utility 1225 having a set (at least one) of program modules 1224, such program modules 1224 including, but not limited to: an operating system, one or more application programs, other program modules, and program data, each or some combination of which may include an implementation of a network environment.

The electronic device 1200 may also communicate with one or more external devices 1204 (e.g., keyboard, pointing device, etc.), with one or more devices that enable a user to interact with the electronic device 1200, and/or with any device (e.g., router, modem, etc.) that enables the electronic device 1200 to communicate with one or more other electronic devices. Such communication may occur through an input/output (I/O) interface 1205. Also, electronic device 1200 may communicate with one or more networks such as a Local Area Network (LAN), a Wide Area Network (WAN), and/or a public network, such as the Internet, through network adapter 1206. As shown, network adapter 1206 communicates with other modules for electronic device 1200 over bus 1203. It should be appreciated that although not shown, other hardware and/or software modules may be used in connection with electronic device 1200, including, but not limited to: microcode, device drivers, redundant processors, external disk drive arrays, RAID systems, tape drives, data backup storage systems, and the like.

In some possible embodiments, aspects of a method for point cloud segmentation of a target object provided by the present disclosure may also be implemented in the form of a program product comprising program code for causing a computer device to perform the steps in the method for point cloud segmentation of a target object according to various exemplary embodiments of the present disclosure as described above in the present specification, when the program product is run on the computer device.

The program product may employ any combination of one or more readable media. The readable medium may be a readable signal medium or a readable storage medium. The readable storage medium can be, for example, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or a combination of any of the foregoing. More specific examples (a non-exhaustive list) of the readable storage medium would include the following: an electrical connection having one or more wires, a portable disk, a hard disk, a random access computer storage medium (RAM), a read-only computer storage medium (ROM), an erasable programmable read-only computer storage medium (EPROM or flash memory), an optical fiber, a portable compact disc read-only computer storage medium (CD-ROM), an optical computer storage medium, a magnetic computer storage medium, or any suitable combination of the foregoing.

The program product of point cloud segmentation of a target object of embodiments of the present disclosure may employ a portable compact disc read-only computer storage medium (CD-ROM) and include program code and may be run on an electronic device. However, the program product of the present disclosure is not limited thereto, and in this document, a readable storage medium may be any tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device.

The readable signal medium may include a data signal propagated in baseband or as part of a carrier wave with readable program code embodied therein. Such a propagated data signal may take any of a variety of forms, including, but not limited to, electro-magnetic, optical, or any suitable combination of the foregoing. A readable signal medium may also be any readable medium that is not a readable storage medium and that can communicate, propagate, or transport a program for use by or in connection with an instruction execution system, apparatus, or device.

Program code embodied on a readable medium may be transmitted using any appropriate medium, including but not limited to wireless, wireline, optical fiber cable, RF, etc., or any suitable combination of the foregoing.

Program code for carrying out operations of the present disclosure may be written in any combination of one or more programming languages, including an object oriented programming language such as Java, C++ or the like and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The program code may execute entirely on the consumer electronic device, partly on the consumer electronic device, as a stand-alone software package, partly on the consumer electronic device, partly on the remote electronic device, or entirely on the remote electronic device or server. In the case of remote electronic devices, the remote electronic device may be connected to the consumer electronic device through any kind of network, including a Local Area Network (LAN) or a Wide Area Network (WAN), or may be connected to an external electronic device (e.g., connected through the internet using an internet service provider).

It should be noted that although several modules of the apparatus are mentioned in the detailed description above, this division is merely exemplary and not mandatory. Indeed, the features and functions of two or more modules described above may be embodied in one module in accordance with embodiments of the present disclosure. Conversely, the features and functions of one module described above may be further divided into a plurality of modules to be embodied.

Furthermore, although the operations of the methods of the present disclosure are depicted in the drawings in a particular order, this is not required or suggested that these operations must be performed in this particular order or that all of the illustrated operations must be performed in order to achieve desirable results. Additionally or alternatively, certain steps may be omitted, multiple steps combined into one step to perform, and/or one step decomposed into multiple steps to perform.

It will be apparent to those skilled in the art that embodiments of the present disclosure may be provided as a method, system, or computer program product. Accordingly, the present disclosure may take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present disclosure may take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, magnetic disk computer storage media, CD-ROM, optical computer storage media, and the like) having computer-usable program code embodied therein.

The present disclosure is described with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems) and computer program products according to the disclosure. It will be understood that each flow and/or block of the flowchart illustrations and/or block diagrams, and combinations of flows and/or blocks in the flowchart illustrations and/or block diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to a processor of a general purpose computer, special purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.

These computer program instructions may also be stored in a computer-readable computer storage medium that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable computer storage medium produce an article of manufacture including instruction means which implement the function specified in the flowchart flow or flows and/or block diagram block or blocks.

These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.

It will be apparent to those skilled in the art that various modifications and variations can be made to the present disclosure without departing from the spirit or scope of the disclosure. Thus, the present disclosure is intended to include such modifications and alterations insofar as they come within the scope of the appended claims or the equivalents thereof.

Claims (14)

1. A method for point cloud segmentation of a target object, the method comprising:

in the hoisting process of a crane, acquiring point cloud data and RGB images, which correspond to a specified duration and contain target objects, wherein the target objects are objects which need to be hoisted by the crane;

contour extraction is carried out on the RGB image, and each contour in the RGB image is obtained;

Aiming at any contour, obtaining a current feature vector of the contour based on the position of each pixel point in the contour and the RGB value of each pixel point;

Determining a target contour of the target object in each contour through the current feature vector of each contour and the standard contour feature vector of the target object;

According to the target contour of the target object, determining the three-dimensional position coordinate of the center point of the target object in a radar coordinate system;

Determining the radius of the outer sphere of the target object based on the extreme value of the horizontal position coordinate, the extreme value of the vertical position coordinate and the camera internal reference in the target contour;

and dividing the point cloud data containing the target object by utilizing the three-dimensional position coordinates of the central point of the target object in a radar coordinate system and the radius of the outer sphere of the target object to obtain the point cloud data corresponding to the target object.

2. The method of claim 1, wherein the obtaining the current feature vector of the contour based on the position of each pixel in the contour and the RGB values of each pixel comprises:

Determining Hu moment of the contour according to the position of each pixel point in the contour; and

According to the RGB values of the pixel points, an R component average value, a G component average value and a B component average value are determined;

and obtaining the current feature vector of the contour based on the Hu moment of the contour, the R component mean value, the G component mean value and the B component mean value.

3. The method of claim 1, wherein said determining a target contour of said target object from said current feature vector of each contour and said standard contour feature vector of said target object comprises:

for any one contour, obtaining the similarity between the contour and the target object based on the current feature vector of the contour and the standard contour feature vector of the target object;

And determining the contour with the largest similarity value among the contours with the similarity greater than the designated similarity as the target contour of the target object.

4. A method according to claim 3, wherein the obtaining the similarity between the contour and the target object based on the current feature vector of the contour and the standard contour feature vector of the target object comprises:

based on the current feature vector of the contour and the standard contour feature vector of the target object, obtaining a chi-square distance between the current feature vector and the standard contour feature vector;

And determining the similarity between the current feature vector and the standard contour feature vector according to the chi-square distance, wherein the similarity is inversely related to the chi-square distance.

5. The method of claim 1, wherein determining three-dimensional position coordinates of a center point of the target object in a radar coordinate system according to a target contour of the target object comprises:

Obtaining 0-order moment and 1-order moment of the target contour through the positions of all pixel points in the target contour of the target object;

According to the 0-order moment and the 1-order moment of the target contour, obtaining a two-dimensional position coordinate of a central point of the target contour in the RGB image;

Based on the two-dimensional position coordinates of the center point of the target contour in the RGB image and the camera internal parameters, obtaining the three-dimensional position coordinates of the center point of the target contour in a camera coordinate system;

Obtaining the three-dimensional position coordinate of the central point of the target object in the camera coordinate system by utilizing the three-dimensional position coordinate of the central point of the target contour in the camera coordinate system and the height of the target object;

And obtaining the three-dimensional position coordinate of the central point of the target object in the radar coordinate system through the three-dimensional position coordinate of the central point of the target object in the camera coordinate system and the camera external parameters.

6. The method according to claim 5, wherein the obtaining the two-dimensional position coordinates of the center point of the target contour in the image according to the 0 th order moment and the 1 st order moment of the target contour includes:

the two-dimensional position abscissa of the center point of the target contour in the image is obtained by the following formula:

Wherein C _x is the two-dimensional position abscissa of the center point of the target contour in the image, m ₁₀ is the 1-order moment of the target contour, and m ₀₀ is the 0-order moment of the target contour;

The ordinate of the two-dimensional position of the center point of the target contour in the image is obtained by the following formula:

Wherein C _y is the ordinate of the two-dimensional position of the center point of the target contour in the image, and m ₀₁ is the other 1-order moment of the target contour of the target object.

7. The method of claim 5, wherein the obtaining the three-dimensional position coordinates of the center point of the target contour in the camera coordinate system based on the two-dimensional position coordinates of the center point of the target contour in the RGB image and the camera internal parameters comprises:

Determining the product of a two-dimensional position coordinate of the center point of the target contour in the RGB image, a matrix corresponding to the camera internal reference and a depth value of the center point of the target contour in a depth image containing the target object as a three-dimensional position coordinate of the center point of the target contour in a camera coordinate system;

The obtaining the three-dimensional position coordinate of the center point of the target object in the camera coordinate system by using the three-dimensional position coordinate of the center point of the target contour in the camera coordinate system and the height of the target object comprises the following steps:

Determining a three-dimensional position abscissa of a central point of the target contour in three-dimensional position coordinates under a camera coordinate system as the three-dimensional position abscissa of the central point of the target object in the camera coordinate system; and

Determining the three-dimensional position ordinate of the central point of the target contour in the three-dimensional position coordinates of the camera coordinate system as the three-dimensional position ordinate of the central point of the target object in the camera coordinate system; and

A three-dimensional position vertical coordinate of a central point of the target contour in a three-dimensional position coordinate of a camera coordinate system and a half camera of the height of the target object are obtained, and the three-dimensional position vertical coordinate of the central point of the target object in the camera coordinate system is obtained;

The three-dimensional position coordinate of the center point of the target object in the radar coordinate system is obtained through the three-dimensional position coordinate of the center point of the target object in the camera coordinate system and camera external parameters, and the three-dimensional position coordinate of the center point of the target object in the radar coordinate system comprises the following steps:

And multiplying the three-dimensional position coordinate of the central point of the target object in the camera coordinate system by the matrix corresponding to the camera external parameter to obtain the three-dimensional position coordinate of the central point of the target object in the radar coordinate system.

8. Method according to claim 5 or 7, characterized in that the height of the target object itself is determined by:

Obtaining a first distance between a camera and the ground according to a width adjusting angle of a crane, the length of a suspension arm of the crane and the height of a vehicle body of the crane, wherein the width adjusting angle is an included angle between the suspension arm of the crane and the ground when the target object does not leave the ground in the lifting process of the crane; and

Obtaining a second distance between the center point of the target contour and the camera according to the depth value of the center point of the target contour in the depth image of the target object;

And obtaining the height of the target object by the first distance and the second distance.

9. The method of claim 1, wherein determining the spherical aberration radius of the target object based on the extremum of the abscissa, the extremum of the ordinate, and the camera reference in the target profile comprises:

obtaining the length of the target contour according to the maximum value and the minimum value in the extreme value of the transverse position coordinate of the target contour; and

Obtaining the width of the target contour according to the maximum value and the minimum value in the extreme value of the longitudinal position coordinate of the target contour;

Obtaining the length of the target object based on the length of the target contour, the depth value of the center point of the target contour in the depth image of the target object and the horizontal focal length in the camera internal parameter; the width of the target object is obtained based on the width of the target contour, the depth value of the center point of the target contour in the depth image of the target object and the vertical focal length in the camera internal parameter;

And determining half of the maximum value of the length of the target object, the width of the target object and the height of the target object as the radius of the outer sphere of the target object.

10. Method according to claim 7 or 9, characterized in that the depth image of the target object is obtained by:

Transforming the point cloud data containing the target object corresponding to the appointed duration by utilizing the camera external parameter to obtain camera position coordinates of each discrete point in the point cloud data in a camera coordinate system;

Projecting camera position coordinates of each discrete point in the point cloud data in a preset depth image by utilizing the camera internal parameters to obtain two-dimensional position coordinates of each discrete point in the point cloud data in the depth image, wherein the size of the preset depth image is the same as that of the RGB image;

setting a vertical coordinate value in a camera position coordinate corresponding to any one discrete point as a depth value of the discrete point in the preset depth image;

And obtaining the depth image of the target object according to the depth value of each discrete point in the preset depth image and the two-dimensional position coordinate of each discrete point in the depth image.

11. The method according to claim 1, wherein the dividing the point cloud data including the target object by using the three-dimensional position coordinates of the center point of the target object in the radar coordinate system and the radius of the outer sphere of the target object to obtain the point cloud data corresponding to the target object includes:

And inputting the three-dimensional position coordinates of the central point of the target object in a radar coordinate system, the radius of the outer sphere of the target object and the point cloud data containing the target object into a minimum segmentation algorithm to perform point cloud segmentation, and obtaining the point cloud data corresponding to the target object.

12. A point cloud segmentation apparatus for a target object, the apparatus comprising:

the acquisition module is used for acquiring point cloud data and RGB images, corresponding to the appointed duration, of a target object in the hoisting process of the crane, wherein the target object is an object to be hoisted by the crane;

the contour extraction module is used for extracting the contour of the RGB image to obtain each contour in the RGB image;

the contour vector determining module is used for obtaining a current feature vector of any contour based on the position of each pixel point in the contour and the RGB value of each pixel point;

The matching module is used for determining a target contour of the target object in each contour through the current feature vector of each contour and the standard contour feature vector of the target object;

The center point position determining module is used for determining the three-dimensional position coordinates of the center point of the target object in a radar coordinate system according to the target contour of the target object;

The outer sphere radius determining module is used for determining the outer sphere radius of the target object based on the extreme value of the horizontal position coordinate, the extreme value of the vertical position coordinate and the camera internal reference in the target profile;

And the segmentation module is used for segmenting the point cloud data containing the target object by utilizing the three-dimensional position coordinates of the central point of the target object in the radar coordinate system and the radius of the outer sphere of the target object to obtain the point cloud data corresponding to the target object.

13. An electronic device comprising at least one processor; and a memory communicatively coupled to the at least one processor; wherein the memory stores instructions for execution by the at least one processor; the instructions being executable by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1-11.

14. A computer storage medium, characterized in that the computer storage medium stores a computer program for executing the method according to any one of claims 1-11.