CN111695402B - Tools and methods for annotating human poses in 3D point cloud data - Google Patents

Abstract

Methods and apparatus for annotating point cloud data. The device may be configured to cause the point cloud data to be displayed; marking points in the point cloud data with a plurality of marking points, the plurality of marking points corresponding to points on a human body; moving one or more of the annotation points in response to user input to define a human pose and create annotated point cloud data; and outputting the marked point cloud data.

Description

Tools and methods for annotating human poses in 3D point cloud data

priority

This application claims priority from U.S. Application No. 16/692,901 filed on November 22, 2019 and U.S. Provisional Application No. 62/817,400 filed on March 12, 2019, the entire contents of each of which are incorporated herein by reference.

Technical field

This application relates to pose estimation and pose annotation in computer vision-based graphics detection technology.

Background technique

Pose estimation is a computer vision technique for detecting human figures from image or video data. In addition to detecting the presence of a human figure, computer vision technology can also determine the position and orientation (i.e., posture) of a human figure's limbs. Pose estimation can be useful in many areas, including autonomous driving. For example, a person's posture can be used to determine the attention and intentions of a person (eg, pedestrian, traffic police, etc.). Autonomous driving applications for automobiles can use the person's intentions and concerns predicted or inferred from the estimated posture to determine driving behavior.

Contents of the invention

In the example described below, this application describes a method for estimating one or more people based on a point cloud generated by a LiDAR (Light Detection and Ranging, also known as lidar) sensor or other similar sensor. gesture of technology and equipment. In some examples, the estimated pose of one or more people can be used to make driving decisions for autonomous vehicles. However, the techniques of the present disclosure are not limited to autonomous driving applications and may be used to estimate human posture for any number of applications for which posture estimation may be useful. By using the output of a LiDAR sensor, for example, as opposed to a camera sensor, pose estimation can be performed quickly in difficult environments including low-light environments.

The computing system can be configured to receive point cloud data from LiDAR sensors or other similar sensors. The computing system may also be configured to convert the point cloud data into a structured data format, such as frames of voxels (volume pixels). The computing system can then use a deep neural network to process the voxelized frames. Deep neural networks can be configured with models that determine the presence of a human being. Deep neural networks can also perform regression to estimate the pose of each of the detected person or persons. In some examples, the computing system makes determinations and pose estimates of the person sequentially. That is, in some examples, first, the computing system uses a deep neural network to detect a person, and then the computing system uses the deep neural network to estimate the pose of the person. In other examples, the computing system performs human determination and pose estimation in parallel. That is, in some examples, the computing system simultaneously determines the presence of the person and the corresponding pose of the person for each voxel. If the deep neural network determines that a person is not present in the voxel, the computational system discards the estimated pose.

A deep neural network may be configured to process voxelized frames using one or more three-dimensional (3D) convolutional layers followed by one or more 2D convolutional layers. 3D convolutional layers generally provide more accurate determinations of people and pose estimates, while 2D convolutional layers generally provide faster determinations of people and pose estimates. By using a combination of 3D and 2D convolutional layers in a deep neural network, human detection and pose estimation can be performed with the desired accuracy while also maintaining a speed useful for autonomous driving applications.

In another example, this disclosure describes techniques for annotating point cloud data. To train a deep neural network to estimate the pose of a person in point cloud data, the deep neural network can be configured and modified by processing a training set of point cloud data. The training set of point cloud data was previously labeled (e.g., by manual labeling) with the accurate positions and poses of people within the point cloud. This previous labeling of the pose in the point cloud data may be called an annotation. Techniques exist for annotating human poses in two-dimensional images. However, annotating point cloud data is very different. First, point cloud data is three-dimensional. Furthermore, point cloud data is sparse relative to 2D image data.

This disclosure describes methods, devices, and software for annotating point cloud data. Users can annotate point clouds using the techniques of the present disclosure to mark one or more poses found in the point cloud data. The annotated point cloud data can then be used to train neural networks to more accurately identify and label poses in point cloud data in real time.

In one example, the present disclosure describes a method that includes: causing point cloud data to be displayed; marking points in the point cloud data with a plurality of annotation points, the plurality of annotation points corresponding to points on a human body; responding Moving one or more of the annotated points in response to user input to define a human body pose and creating annotated point cloud data; and outputting the annotated point cloud data.

In another example, the present disclosure describes an apparatus that includes a memory configured to store point cloud data, and one or more processors in communication with the memory, the one or more processors configured to cause the point cloud data to be displayed; to mark points in the point cloud data with a plurality of annotation points, the plurality of annotation points corresponding to points on a human body; and to move the annotation points in response to user input. One or more annotated points are used to define the human body posture and create annotated point cloud data; and output the annotated point cloud data.

In another example, the present disclosure describes an apparatus that includes means for causing point cloud data to be displayed; means for marking points in the point cloud data with a plurality of annotation points, the plurality of annotation points being annotation points corresponding to points on the human body; means for moving one or more of the annotation points in response to user input to define a pose of the human body and create annotated point cloud data, and for outputting the annotated points Cloud data installation.

In another example, the present disclosure describes a non-transitory computer-readable storage medium storing instructions that, when executed, cause one or more processors to: display point cloud data; label points with a plurality of Marking points in the point cloud data, the plurality of annotated points corresponding to points on a human body; moving one or more of the annotated points in response to user input to define a human body pose and create the labeled points cloud data and output the labeled point cloud data.

The details of one or more embodiments of the invention are set forth in the accompanying drawings and the description below. Other features, objects and advantages of the invention will be apparent from the description and drawings, and from the claims.

Description of the drawings

FIG. 1 is a conceptual diagram showing an exemplary operating environment of the technology of the present disclosure.

2 is a block diagram showing an exemplary device configured to perform the techniques of the present disclosure.

3 is a block diagram showing a processing flow of one example of the present disclosure.

4 is a conceptual diagram showing a parallel processing flow using a deep neural network according to one example of the present disclosure.

5 is a conceptual diagram showing a sequential processing flow using a deep neural network according to one example of the present disclosure.

Figure 6 is a conceptual diagram illustrating an exemplary anchor skeleton.

Figure 7 is a conceptual diagram illustrating an exemplary point cloud including multiple classified skeletons with estimated poses.

8 is a flowchart showing exemplary operations of a device configured to perform pose estimation according to one example of the present disclosure.

9 is a block diagram showing an exemplary device configured to perform point cloud annotation in accordance with the techniques of this disclosure.

Figure 10 is a conceptual user interface diagram illustrating an input point cloud for annotation.

Figure 11 is a conceptual user interface diagram illustrating a cropped point cloud for annotation.

Figure 12 is a conceptual diagram illustrating an exemplary skeleton for annotation.

Figure 13 is a conceptual user interface diagram illustrating estimated annotation of a point cloud.

Figure 14 is a conceptual user interface diagram showing an annotated point cloud.

Figure 15 is a flowchart illustrating exemplary operations of an annotation tool according to one example of the present disclosure.

Detailed ways

Pose estimation is a computer vision technique for detecting human figures from images or videos. In addition to detecting the presence of a human figure, computer vision technology can also determine the position and orientation (i.e., pose) of a human figure's limbs. Pose estimation is useful in many fields, including autonomous driving. For example, the person's gesture can be used to determine the attention and intention of the person (eg, a pedestrian, a traffic policeman, etc.) or the needs of the person (eg, a pedestrian raising his hand to stop a taxi). Autonomous driving applications in cars can use the predicted human intentions and concerns based on the estimated posture to determine driving behavior.

In some examples, pose estimation is performed on image data received from a camera sensor. Such data has several disadvantages. For example, if the output from a camera sensor does not include depth information, it may be difficult to discern the relative positions of people in the image. Even if the output from the camera sensor does include depth information, performing pose estimation in dark environments may be difficult or impossible.

This disclosure describes techniques for performing pose estimation using point cloud data, such as that produced by a LiDAR sensor. The point cloud output from the LiDAR sensor provides a 3D map of objects in the vicinity of the sensor. Therefore, depth information can be obtained. Additionally, as opposed to camera sensors, LiDAR sensors can generate point clouds in dark environments. Technology of the present disclosure includes using deep neural networks to process point clouds from LiDAR sensors to detect the presence of a person near the sensor and estimate the person's posture in order to make autonomous driving decisions.

FIG. 1 is a conceptual diagram showing an exemplary operating environment of the technology of the present disclosure. In one example of the present disclosure, the car 2 may include components configured to perform attitude estimation. In this example, car 2 may include LiDAR sensor 10 , computing system 14 , and optionally, camera 16 .

The technology of the present disclosure is described with reference to automotive applications, including autonomous driving applications. However, it should be understood that the techniques of this disclosure for person detection and pose estimation may be used in other contexts.

Car 2 can be any type of passenger car. LiDAR sensor 10 may be mounted to car 2 using bracket 12 . In other examples, the LiDAR sensor 10 may be mounted to the vehicle 2 in other configurations, or integrated into or carried by vehicle structures such as bumpers, sides, windshield, etc. . Additionally, car 2 may be configured to use multiple LiDAR sensors. As will be explained in more detail below, computing system 14 may be configured to receive point cloud data from LiDAR sensor 10 and determine the position and posture of a person in the field of view of LiDAR sensor 10 .

LiDAR sensor 10 includes a laser configured to emit laser pulses. LiDAR sensor 10 further includes a receiver to receive laser light reflected from objects near LiDAR sensor 10 . The LiDAR sensor 10 measures the distance to an object by illuminating the object with pulsed laser light and measuring the reflected pulses. The difference in return time and wavelength of the reflected pulses is used to determine a 3D representation of one or more objects (eg, people).

LiDAR sensor 10 may also include a global positioning sensor (GPS) or similar sensor to determine the exact physical location of the sensor and the object sensed from the reflected laser light. LiDAR sensor 10 may also be configured to detect additional information, such as intensity. The intensity of points in the point cloud may be indicative of the reflectivity of the object detected by LiDAR sensor 10 . Typically, the 3D representation captured by the LiDAR sensor 10 is stored in the form of a point cloud. A point cloud is a collection of points that represent a 3D shape or feature. Each point has its own set of x, y, and z coordinates, and in some cases additional attributes (eg, GPS location and intensity). The resulting point cloud from the LiDAR collection method may be saved and/or transferred to the computing system 14 .

Although LiDAR sensors are described in this disclosure, the techniques for pose estimation described herein can be used with the output of any sensor that operates in low light and/or outputs point cloud data. Additional sensor types that may be used with the techniques of the present disclosure may include, for example, radar, ultrasonic, camera/imaging sensors, and/or sonar sensors.

Computing system 14 may be connected to the LiDAR sensor via wired or wireless communication technology. The computing system may include one or more processors configured to receive point clouds from LiDAR sensor 10 . As will be explained in greater detail below, computing system 14 may be configured to perform pose estimation. For example, computing system 14 may be configured to: receive a point cloud from LiDAR sensor 10 that includes a plurality of points representing the position of an object relative to the LiDAR sensor; and process the point cloud to generate a point cloud that includes a plurality of voxels. voxelizing the frame; processing the voxelized frame using a deep neural network to determine one or more persons and a pose of each of the one or more persons relative to the LiDAR sensor; and outputting the determined one or more persons The position and posture of each of the identified person or persons. The technology of the present disclosure is not limited to detection and posture estimation of people (eg, pedestrians, cyclists, etc.), but may also be used for posture detection of animals (eg, dogs, cats, etc.).

Mount 12 may include one or more cameras 16 . The use of brackets is just an example. The camera 16 can be positioned at any suitable location on the car 2 . The car 2 may further comprise additional cameras not shown in Figure 1 . Computing system 14 may be connected to camera 16 to receive image data. In one example of the present disclosure, computing system 14 may be further configured to perform pose estimation using camera-based techniques. In such examples, computing system 14 may be configured to estimate the pose of one or more persons using both camera-based technology and the LiDAR-based technology described in this disclosure. Computing system 14 may be configured to assign a weight to each pose determined by the camera-based technology and the LiDAR-based technology and determine the final pose of the person based on a weighted average of the determined poses. Computing system 14 may be configured to determine weights based on the confidence of each technology. For example, LiDAR-based technology may have higher-confidence accuracy in low-light environments than camera-based technology.

2 is a block diagram showing an exemplary device configured to perform the techniques of the present disclosure. In particular, FIG. 2 shows an example of computing system 14 of FIG. 1 in greater detail. Again, in some examples, computing system 14 may be part of automobile 2 . However, in other examples, computing system 14 may be a stand-alone system or may be integrated into other devices for other applications that may benefit from pose estimation.

Computing system 14 includes microprocessor 22 in communication with memory 24 . In some examples, computing system 14 may include multiple microprocessors. Microprocessor 22 may be implemented as fixed-function processing circuitry, programmable processing circuitry, or a combination thereof. A fixed-function circuit is a circuit that provides a specific function and is preset to perform an operation. Programmable circuits are circuits that can be programmed to perform a variety of tasks and provide flexible functionality in the operations that can be performed. For example, a programmable circuit may execute software or firmware that causes the programmable circuit to operate in a manner defined by the instructions of the software or firmware. Fixed-function circuits may execute software instructions (eg, to receive parameters or output parameters), but the types of operations performed by fixed-function processing circuits are generally immutable. In some examples, one or more units may be distinct circuit blocks (fixed function or programmable), and in some examples, one or more units may be integrated circuits.

In the example of FIG. 2 , microprocessor 22 may be configured to execute one or more sets of instructions in LiDAR-based pose estimation module 40 to perform pose estimation in accordance with the techniques of the present disclosure. Instructions defining LiDAR-based pose estimation module 40 may be stored in memory 24 . In some examples, instructions defining LiDAR-based attitude estimation module 40 may be downloaded to memory 24 over a wired or wireless network.

In some examples, memory 24 may be temporary storage, meaning that the primary purpose of memory 24 is not long-term storage. Memory 24 may be configured as volatile memory for short-term storage of information and therefore does not retain stored contents if power is lost. Examples of volatile memory include random access memory (RAM), dynamic random access memory (DRAM), static random access memory (SRAM), and other forms of volatile memory known in the art.

Memory 24 may include one or more non-transitory computer-readable storage media. Memory 24 may be configured to store greater amounts of information than is typically stored by volatile memory. Memory 24 may also be configured as a non-volatile storage space for long-term storage of information and retention of information after power on/off cycles. Examples of non-volatile memory include magnetic hard disks, optical disks, flash memory, or forms of electrically programmable memory (EPROM) or electrically erasable and programmable memory (EEPROM). Memory 24 may store program instructions (eg, LiDAR-based pose estimation module 40) and/or information (eg, point cloud 30 and detected pose and position of person 32) that, when executed, cause the microprocessor 22 Perform the techniques of the present disclosure.

The following techniques of the present disclosure will be described with reference to microprocessor 22 executing various software modules. However, it should be understood that each software module described herein may also be implemented in dedicated hardware, firmware, software, or any combination of hardware, software, and firmware.

LiDAR-based pose estimation module 40 may include a pre-processing unit 42, a deep neural network (DNN) 44, and a post-processing unit 46. LiDAR-based pose estimation module 40 is configured to receive point cloud 30 from a LiDAR sensor (eg, LiDAR sensor 10 of FIG. 1 ). Preprocessing unit 42 is configured to convert unstructured raw input (ie, point cloud 30) into structured frames (eg, matrix data) so that deep neural network 44 can process the input data.

Preprocessing unit 42 may be configured to process point cloud 30 into structured frames in a number of ways. In one example, preprocessing unit 42 may be configured to convert point clouds into voxels (volume pixels). Preprocessing unit 42 may be configured to perform such voxelization according to a predetermined data structure for the voxels. For example, each voxel may be defined by the dimensions of a three-dimensional (3D) bin (eg, represented by X, Y, and Z coordinates) and the data type used for 3D bin storage. For example, each 3D bin (ie, voxel) may include data indicating the number of points from point cloud 30 located in the bin, the location of the points from point cloud 30 in the bin, and the intensity of those points. Other examples of data that can be stored in voxels include the mean and variance of the height, width, length (x, y, z coordinates) of the point cloud within said voxel or even near the voxel ;Mean and variance of intensity/reflectance; and other statistics. In some examples, a voxel may include a zero point from point cloud 30 , one point from point cloud 30 , or multiple points from point cloud 30 . Using predetermined bins can be called manual voxelization. In other examples, preprocessing unit 42 may be configured to voxelize point cloud 30 in an adaptive manner, such as by using a neural network that takes raw point cloud 30 as input and outputs structured (voxelized) frames.

Deep neural network 44 receives voxelized frames from pre-processing unit 42 . Deep neural network is a machine learning algorithm. Deep neural network 44 may be configured with multiple processing layers, each layer configured to determine and/or extract features from input data (in this case voxelized frames of point cloud 30). Each successive layer of deep neural network 44 may be configured to use the output from the previous layer as input.

In some examples, deep neural network 44 may be configured as a convolutional deep neural network. A convolutional deep neural network is a deep, feedforward neural network. Each layer of a convolutional deep neural network can be called a convolutional layer. A convolutional layer applies a convolutional operation to an input (e.g., the voxels of a voxelized frame) and then passes the result to the next layer. The deep neural network 44 can be configured with 3D and 2D convolutional layers. 3D convolutional layers provide more accurate feature extraction (e.g., more accurate recognition of people and corresponding poses), while 2D convolutional layers provide faster feature extraction compared to 3D convolutional layers. The deep neural network 44 may be configured to first process the voxelized frames using one or more 3D convolutional layers and then continue to process the voxelized frames using one or more 2D convolutional layers. The 2D convolutional layer can be configured to process data from voxelized frames only in the X and Y directions (i.e. not in the Z direction). The number of 3D and 2D convolutional layers, as well as the split points between layers, determine the trade-off between speed and accuracy of pose estimation. By using a combination of 3D and 2D convolutional layers in a deep neural network 44, human detection and pose estimation can be performed with the desired accuracy while also maintaining a speed useful for autonomous driving applications.

The deep neural network 44 is configured to analyze the voxelized frames and produce two outputs for each voxel. An output can be called a classification. Classification indicates whether a person is present in the voxel being analyzed. The other output may be called a pose estimate resulting from regression. If the person is present in the voxel, regression determines the person's pose (or the person's keypoints). As will be explained in more detail below, deep neural network 44 may be configured to perform classification and regression techniques in a serial or parallel manner.

Deep neural network 44 may be configured to process each voxel through DNN model 48. The DNN model 48 defines the number of 3D and 2D convolutional layers and the functions performed for each layer. The DNN model 48 can be trained with a large number of data-label pairs. In a data-label pair, the data are voxelized point cloud data and the labels are possible 3D poses. The DNN model 48 is trained by manually annotating (eg, labeling) point cloud data and then using the labeled data to train a deep neural network 44 . The output of the deep neural network 44 is compared to the expected output given the labeled data. The technician can then tune the DNN model 48 to find the optimal set of weights for the layers of the deep neural network 44 such that, given the pre-labeled point cloud, the desired labeling is processed by the deep neural network 44 time is predicted. The DNN model 38 can be scheduled and updated periodically.

Deep neural network 44 may be configured to produce classification and regression results for each anchor position. In one example, the deep neural network can be configured to consider the center of the voxel as the anchor location. For each anchor location, the deep neural network 44 may be configured to compare the data stored in the voxel to one or more predetermined anchor skeletons (also referred to as standard or canonical skeletons). An anchor skeleton can be defined by multiple keypoints. In one example, the anchor skeleton is defined by fourteen joints and/or keypoints: head, neck, left shoulder, right shoulder, left elbow, right elbow, left hand, right hand, left waist, right waist, left knee, right Knee, left foot and right foot. Typically, keypoints may correspond to features or structures of human anatomy (e.g., points on the human body).

During processing by the deep neural network 44, if the overlapping area between the bounding box of the anchor skeleton and the bounding box of any ground truth skeleton (i.e., the data present in the voxel) satisfies a threshold condition, the anchor skeleton is activated (i.e., is classified as positive for the person's existence). For example, if the overlapping area of the anchor skeleton and the voxel's bounding box is above a certain threshold (e.g., 0.5), the anchor skeleton is activated for that voxel, and the presence of a person is detected. The threshold may be a measurement of the amount of overlap (eg, intersection-over-union (IOU)). The deep neural network 44 may perform classification based on comparison to one or more multiple different anchor skeletons. The deep neural network 44 may also be implemented to perform regression that encodes the difference between the anchor skeleton and the true skeleton (ie, the data in the actual voxels). Deep neural network 44 may be configured to encode this difference for each of the plurality of keypoints defined by the anchor skeleton. The difference between the keypoints of the anchor skeleton and the data in the voxels represents the actual pose of the person detected during classification. The deep neural network may then be configured to provide the classification (eg, the determined location of the person or persons) and the determined pose of each of the one or more persons to post-processing unit 46 . When multiple people are detected from the point cloud, multiple anchor skeletons are activated, enabling multi-person pose estimation.

Post-processing unit 46 may be configured to convert the output of deep neural network 44 into a final output. For example, post-processing unit 46 may be configured to perform non-maximum suppression on the classified and estimated poses produced by deep neural network 44 and produce a final position and pose of the detected person. Non-maximum suppression is an edge thinning technique. In some cases, a deep neural network 44 will classify a person and estimate the pose of many dense groups of voxels where only one person actually exists. That is, in some cases, deep neural networks will detect overlapping duplicates of the same person. Post-processing unit 46 may use non-maximum suppression techniques to remove duplicate skeletons. The post-processing unit 46 outputs the detected posture and position of the person data 32 . The detected pose and position of the person data 32 may include the position of the person detected by the LiDAR-based pose estimation module 40 (eg, represented by GPS coordinates) as well as the skeletal pose that defines the person (eg, the location of key points). The detected pose and position of person data 32 may be stored in memory 24 , sent to autonomous driving application 52 , other applications 54 , camera-based pose estimation application 56 , or transmitted from computing system 14 to another computing system .

In one example, the autonomous driving application 52 may be configured to receive the posture and location of the detected person data 32 and predict or determine the identified person's intentions and/or concerns, or other behavioral cues, to make autonomous decisions. Driving decisions.

In other examples, camera-based pose estimation application 56 may receive the detected pose and location of person data 32 . Camera-based pose estimation application 56 may be configured to determine the pose of one or more persons using image data produced by camera 16 (FIG. 1). The camera-based pose estimation application 56 may also be configured to assign a weight to each pose determined by the camera-based technology and the Li-DAR-based technology and determine the final pose of the person based on a weighted average of the determined poses. . The camera-based pose estimation application 56 may be configured to determine weights based on the confidence of each technique. For example, LiDAR-based technology may have higher-confidence accuracy in low-light environments than camera-based technology.

Other applications 54 represent various other contexts in which the detected posture and location of person data 32 may be used. For example, the pose and position output by the LiDAR-based pose estimation module 40 can be used in various applications: for body language recognition, motion understanding (e.g., traffic, police personnel, emergency service personnel, or other personnel signaling/directing traffic). ), attention and intent detection (e.g., pedestrians waiting/crossing the street), film, animation, games, robotics, human-computer interaction, machine learning, virtual reality, alternative reality, surveillance, abnormal behavior detection, and public safety .

3 is a block diagram showing a processing flow of one example of the present disclosure. As shown in FIG. 3 , the LiDAR sensor 10 may be configured to capture a point cloud 30 that is the raw input to the LiDAR-based pose estimation module 40 . The LiDAR-based pose estimation module 40 processes the point cloud 30 using a pre-processing unit 42 (voxelization) to produce voxelized frames. The deep neural network 44 then processes the voxelized frames to produce a classification of the one or more persons (eg, the location of the one or more persons) and one or more poses for the classified one or more persons. A person's pose is defined by the positions of multiple key points of the skeleton. The output of the deep neural network 44 is a preliminary 3D pose. Post-processing unit 46 processes the preliminary 3D pose using a non-maximum suppression algorithm to produce an output 3D pose.

4 is a conceptual diagram showing a parallel processing flow using a deep neural network according to an example of the present invention. As shown in Figure 4, the point cloud 30 is first converted into a voxelized frame that includes a plurality of voxels. In this example, the deep neural network 44 processes each voxel 70 of the voxelized frame. Deep neural network 44 processes voxels 70 using one or more 3D convolutional layers 72. 3D convolutional layer 74 represents the final layer that operates on 3D voxel data. Following the 3D convolutional layer 74, the deep neural network 44 processes the voxels 70 with one or more 2D convolutional layers 76. The 2D convolutional layers 76 operate only on voxel data in two dimensions (eg, XY data). 2D convolutional layer 78 represents the final 2D convolutional layer that outputs both classification and pose estimation. In the example of Figure 4, the layers of deep neural network 44 are configured to classify and estimate pose in parallel for each voxel. That is, layers of deep neural network 44 may be configured to classify and estimate poses for more than one voxel simultaneously. If the deep neural network 44 determines that the voxel is not classified as a person, any estimated pose may be discarded.

5 is a conceptual diagram showing a sequential processing flow using a deep neural network according to one example of the present disclosure. In the example of Figure 5, 3D convolutional layers 72 and 74 and 2D convolutional layers 76 and 78 are configured to classify input voxels as human or non-human. If the 2D convolutional layer 78 does not classify as a person, the process ends. If 2D convolutional layer 78 does classify a person, deep neural network 44 will process the input voxels using 3D convolutional layers 80 and 82 and 2D convolutional layers 84 and 86 to estimate the pose of the classified person. That is, deep neural network 44 may be configured to use separate neural networks for classification and pose estimation. In this example, the classification and pose estimation processes are performed sequentially.

Figure 6 is a conceptual diagram showing an exemplary skeleton. Skeleton 100 may represent a predetermined anchor point skeleton or a pose of a real skeleton estimated using the disclosed techniques described above. In one example of this disclosure, skeleton 100 may be defined by a plurality of key points and/or joints. In the example of Figure 6, skeleton 100 includes 14 key points. As shown in Figure 6, the skeleton 100 consists of a head key point 102, a neck key point 104, a left shoulder key point 108, a right shoulder key point 106, a left elbow 112, a right elbow key point 110, a left hand key point 116, a right hand key point 114, The left waist point 120, the right waist point 118, the left knee point 124, the right knee point 122, the left foot point 128, and the right foot point 126 are limited. To determine the pose, microprocessor 22 (see Figure 2) may be configured to determine the position (eg, position in 3D space) of each key point of skeleton 100. That is, the position of each keypoint of the skeleton 100 relative to each other defines the pose of the skeleton 100 and therefore the pose of the person detected from the point cloud.

For other applications, more or fewer keypoints can be used. The more keypoints used to define the skeleton 100, the more unique poses can be estimated. However, more keypoints may also result in longer processing time to estimate the pose.

Figure 7 is a conceptual diagram illustrating an exemplary point cloud 30 with multiple classified skeletons with estimated poses. As shown in FIG. 7 , point cloud 30 is shown with visualization of three detected skeletons 140 , 142 , and 144 . The skeleton is shown with different poses that can be obtained by different positions of the 14 key points from Figure 6. Note that skeleton 140 shows one example of a skeleton that is not processed by the non-maximum suppression algorithm. Skeleton 140 is actually multiple overlapping skeletons rather than showing a single skeleton. In some examples of the present disclosure, computing system 14 may be further configured to generate a visualization of the detected skeleton, such as the visualization shown in FIG. 7 .

8 is a flowchart showing exemplary operations of a device configured to perform pose estimation according to one example of the present disclosure. One or more processors may be configured to perform the techniques illustrated in FIG. 8 , including microprocessor 22 of computing system 14 . As mentioned above, in some examples, computing system 14 may be part of automobile 2 . In this example, car 2 may be configured to use attitude estimates generated by computing system 14 to make autonomous driving decisions. However, the technology of the present disclosure is not limited thereto. Any processor or processing circuit may be configured to perform the techniques of Figure 8 for pose estimation for any number of applications, including AR/VR, gaming, HCI, surveillance and monitoring, and the like.

In one example of the present disclosure, computing system 14 may include memory 24 configured to receive point cloud 30 (see FIG. 2 ) from LiDAR sensor 10 (see FIG. 1 ). Computing system 14 may further include one or more processors (eg, microprocessor 22 of FIG. 2) implemented as circuitry in communication with memory. Microprocessor 22 may be configured to receive the point cloud from LiDAR sensor 10 (800). The point cloud includes a plurality of points representing the position of the object relative to the LiDAR sensor 10 . Microprocessor 22 may be further configured to process the point cloud to generate a voxelized frame that includes a plurality of voxels (802). In one example of the present disclosure, each voxel of the voxelized frame includes a data structure indicating the presence or absence of a point from the point cloud in the voxel.

Microprocessor 22 may also be configured to process voxelized frames using one or more 3D convolutional layers of a deep neural network (804) and to process volumes using one or more 2D convolutional layers of a deep neural network. Pixelated frames (806). Microprocessor 22 processes the voxelized frames using 3D and 2D convolutional layers to determine the pose of the one or more persons and each of the one or more persons relative to the LiDAR sensor. Microprocessor 22 may then output the determined location of the one or more persons and the determined posture of each of the one or more persons (808).

In one example, microprocessor 22 may be configured to determine whether a person is present for a first voxel of the voxelized frame and activate an anchor skeleton for the first voxel based on the determination, wherein the first voxel The data represented in voxels is defined as the real skeleton. Microprocessor 22 may be configured to determine the presence of persons and the gestures of such persons either sequentially or in parallel. In one example, microprocessor 22 may be configured to determine a difference between the real skeleton and the anchor skeleton in parallel with determining whether a person is present, estimate the pose of the real skeleton based on the difference, and, with the anchor skeleton activated, Output the gesture. In another example, the microprocessor 22 may be configured to determine a difference between the real skeleton and the anchor skeleton when the anchor skeleton is activated, and estimate a pose of the real skeleton based on the difference, and output the pose .

The anchor skeleton is defined by multiple keypoints. To determine the difference between the real skeleton and the anchor skeleton, the microprocessor 22 may be configured to determine the difference between each key point of the real skeleton and the anchor skeleton.

In another example of the present disclosure, microprocessor 22 may also be configured to use non-maximum suppression techniques to process the determined one or more persons and the pose of each of the one or more persons relative to the LiDAR sensor. , to remove duplicates of the person or persons.

In other examples of the present disclosure, the pose estimation techniques of the present disclosure can be extended to a series of frames to detect a series of gestures that can constitute a certain action (eg, waving, walking, running, etc.). Such action recognition can use temporal information (e.g., LiDAR point cloud data from multiple time phases) to perform action recognition. Thus, in one example, DNN 44 may be configured to process a plurality of voxelized frames to determine at least one person and a series of poses of the at least one person relative to the LiDAR sensor. DNN 44 may then determine an action for at least one person based on the sequence of gestures. Two example ways of implementing action recognition are described below.

In a first example, DNN 44 may be configured to stack and/or concatenate a fixed number of outputs for each frame of point cloud 30 into a single data sample. DNN 44 can feed the single data sample into a classifier to classify action categories. At frame index t, DNN 44 can be configured to use a time window size of w to produce a single sample that is the combined w output from frames t-w+1 to t. DNN 44 may be configured to include a classifier that is a (multi-class) deep neural network or any type of machine learning model, such as a support vector machine (SVM).

In another example, DNN 44 may be configured to use each frame output in a sequential manner. For example, DNN 44 may be configured to feed each frame output to a recurrent neural network and determine a prediction of an action at each frame or after a certain number of frames.

Therefore, instead of or in addition to per-frame pose estimation (e.g., skeleton output), the DNN 44 may be configured to stitch the output into batches or feed the output sequentially to obtain higher-level Action recognition. Some possible categories of actions to recognize include standing, walking, running, biking, skateboarding, waving, etc.

Other examples and combinations of the techniques of the present disclosure are described below.

Embodiment 1. A method for attitude estimation, the method comprising: receiving a point cloud from a LiDAR sensor, the point cloud including a plurality of points representing the position of an object relative to the LiDAR sensor; processing the point cloud to generate voxelized frames including a plurality of voxels; processing the voxelized frames using a deep neural network to determine one or more persons and a pose of the one or more persons relative to the LiDAR sensor; and outputting the determined The location of one or more persons and the determined posture of each of the one or more persons.

Embodiment 2. The method of embodiment 1, wherein each voxel of the voxelized frame includes a data structure indicating the presence or absence of a point from the point cloud in the voxel.

Option 3. The method of Option 1 or 2, wherein using a deep neural network to process the voxelized frame includes using a convolutional deep neural network to process the voxelized frame, wherein the convolution A convolutional deep neural network consists of one or more three-dimensional convolutional layers, followed by one or more two-dimensional convolutional layers.

Embodiment 4. The method according to any combination of Embodiments 1 to 3, wherein processing the voxelized frame using a deep neural network includes: for a first voxel of the voxelized frame, determining whether a person is present; and Based on the determination, an anchor skeleton is activated for the first voxel, wherein the data represented in the first voxel is defined as a ground truth skeleton.

Option 5. The method according to any combination of Option 1 to 4, further comprising: determining a difference between the real skeleton and the anchor point skeleton in parallel with determining whether the person is present; estimating the difference based on the difference the pose of the real skeleton; and outputting the pose if the anchor skeleton is activated.

Option 6. The method according to any combination of Option 1 to 4, further comprising: when the anchor point skeleton is activated, determining the difference between the real skeleton and the anchor point skeleton; estimating based on the difference the pose of the real skeleton; and outputting the pose.

Option 7. The method according to any combination of Option 1 to 6, wherein the anchor point skeleton is defined by a plurality of key points.

Option 8. The method according to any combination of aspects 1 to 7, wherein determining the difference between the real skeleton and the anchor point skeleton includes: determining the difference between each key point of the real skeleton and the anchor point skeleton. difference.

Embodiment 9. The method according to any combination of embodiments 1 to 8, further comprising: using a non-maximum suppression technique to process the determined pose of the one or more persons relative to the LiDAR sensor and each of the one or more persons. , to remove duplicates of the person or persons.

Embodiment 10. A device configured to perform pose estimation, the device comprising: a memory configured to receive a point cloud from a LiDAR sensor; and one or more processors implemented in circuitry, the one or more A processor is in communication with the memory and is configured to: receive a point cloud from the LiDAR sensor, the point cloud including a plurality of points representing the position of the object relative to the LiDAR sensor; process the point cloud to generate a point cloud including a plurality of voxels voxelizing the frame; processing the voxelized frame using a deep neural network to determine one or more persons relative to the LiDAR sensor and a pose of each of the one or more persons; and outputting the determined one The location of the person or persons and the determined posture of each of the one or more persons.

Embodiment 11. The apparatus of embodiment 10, wherein each voxel of the voxelized frame includes a data structure indicating the presence or absence of a point from a point cloud in the voxel.

Embodiment 12. The device according to any combination of embodiments 10 to 11, wherein to process the voxelized frames using a deep neural network, the one or more processors are further configured to: use a convolutional deep neural network To process the voxelized frames, the convolutional deep neural network includes one or more three-dimensional convolutional layers, followed by one or more two-dimensional convolutional layers.

Embodiment 13. The device according to any combination of embodiments 10 to 12, wherein, in order to process the voxelized frames using a deep neural network, the one or more processors are further configured to: for the voxelized a first voxel of the frame, determining whether a person is present; and activating an anchor skeleton for the first voxel based on the determination, wherein the data represented in the first voxel is defined as the true skeleton .

Option 14. The device according to any combination of aspects 10 to 13, wherein the one or more processors are further configured to: determine between the real skeleton and the anchor point skeleton in parallel with determining whether the person is present. the difference between; estimating the pose of the real skeleton based on the difference; and outputting the pose with the anchor point skeleton activated.

Option 15. The device according to any combination of solutions 10 to 13, wherein the one or more processors are further configured to: determine the real skeleton and the anchor point skeleton when the anchor point skeleton is activated the difference between; estimating the pose of the real skeleton based on the difference; and outputting the pose.

Embodiment 16. The device according to any combination of embodiments 10-15, wherein the anchor point skeleton is defined by a plurality of key points.

Item 17. The device according to any combination of items 10-11, wherein, in order to determine the difference between the real skeleton and the anchor point skeleton, the one or more processors are further configured to: determine the real skeleton. The difference between each of the keypoints of the skeleton and the anchor point skeleton.

Embodiment 18. The apparatus according to any combination of embodiments 10 to 17, wherein the one or more processors are further configured to: use non-maximum suppression techniques to process the determined one or more persons relative to the LiDAR sensor and The gesture of each of the one or more persons to remove duplicates of the one or more persons.

Embodiment 19. The device according to any combination of aspects 10 to 18, wherein the device includes a car having a LiDAR sensor.

Embodiment 20. An apparatus configured to perform pose estimation, the apparatus comprising: means for receiving a point cloud from a LiDAR sensor, the point cloud including a plurality of points representing a position of an object relative to the LiDAR sensor; means for processing the point cloud to produce a voxelized frame comprising a plurality of voxels; for processing the voxelized frame using a deep neural network to determine one or more persons relative to a LiDAR sensor and the one or means for a posture of each of a plurality of persons; and means for outputting the determined position of one or more persons and the determined posture of each of the one or more persons.

Embodiment 21. An apparatus configured to perform pose estimation, the apparatus comprising means for performing any combination of the steps of the method of embodiments 1 to 9.

Embodiment 22. A non-transitory computer-readable medium that may be configured to store instructions that, when executed, cause one or more processors to receive a point cloud from a LiDAR sensor, the point cloud including a plurality of points representing a position of an object relative to a LiDAR sensor; processing the point cloud to generate a voxelized frame that includes a plurality of voxels; processing the voxelized frame using a deep neural network to determine a position relative to the LiDAR sensor one or more people and the posture of each of the one or more people; and output the determined position of the one or more people and the determined posture of each of the one or more people.

In another example, this disclosure describes techniques for annotating point cloud data. In order to train a deep neural network to estimate the pose of a person in point cloud data, the deep neural network can be configured and modified by processing a training set of point cloud data. The training set of point cloud data is pre-labeled with the exact position and pose of the person within the point cloud (eg, by manual labeling). This prior marking of poses in point cloud data may be called annotation. Techniques exist for annotating the pose of a person in two-dimensional images. However, labeling point cloud data is very different. First, point cloud data is three-dimensional. Furthermore, point cloud data is sparse relative to 2D image data.

The present disclosure describes a method, apparatus, and software tool for annotating point cloud data. Users can annotate point clouds using the techniques of the present disclosure to mark one or more gestures found in the point cloud data. The annotated point cloud data can then be used to train neural networks to more accurately identify and label poses in point cloud data in real time.

Figure 9 is a block diagram showing an exemplary computing system 214 configured to perform the point cloud annotation techniques of the present disclosure. Computing system 214 may be implemented using, for example, a desktop computer, notebook computer, tablet computer, or any type of computing device. Computing system 214 includes a processor 222, memory 224, and one or more input devices 218. In some examples, computing system 214 may include multiple processors 222 .

Processor 222 may be implemented as fixed-function processing circuitry, programmable processing circuitry, or a combination thereof. Fixed-function circuits are circuits that provide a specific function and are designed to perform operations. Programmable circuits are circuits that can be programmed to perform a variety of tasks and provide flexible functionality in the operations that can be performed. For example, a programmable circuit may execute software or firmware that causes the programmable circuit to operate in a manner defined by instructions of the software or firmware. Fixed-function circuits may execute software instructions (eg, to receive parameters or output parameters), but the types of operations performed by fixed-function processing circuits are generally immutable. In some examples, one or more units may be distinct circuit blocks (fixed function or programmable), and in some examples, one or more units may be integrated circuits.

Computing system 214 may be configured to generate information for display on display device 216 . For example, as will be described in greater detail below, computing system 214 may generate graphical user interface (GUI) 250 and cause GUI 250 to be displayed on display device 216 . A user may interact with GUI 250 to annotate point cloud data, such as through input device 218. In some examples, display device 216 is part of computing system 214 , but in other examples, display device 216 may be separate from computing system 214 . Display device 216 may be implemented with any electronic display, such as a liquid crystal display (LCD), a light emitting diode (LED) display, or an organic light emitting diode (OLED) display.

Input device 218 is a device configured to receive user commands or other information. In some examples, input device 218 is part of computing system 214 , but in other examples, input device 218 may be separate from computing system 214 . Input device 218 may include any device for inputting information or commands, such as a keyboard, microphone, cursor control device, or touch screen.

In accordance with the techniques of the present disclosure, the processor 222 may be configured to execute a set of instructions of the annotation tool 242 to perform point cloud annotation in accordance with the techniques of the present disclosure. Instructions defining annotation tool 242 may be stored in memory 224 . In some examples, instructions defining annotation tool 242 may be downloaded to memory 224 over a wired or wireless network.

In some examples, memory 224 may be temporary storage, meaning that the primary purpose of memory 224 is not long-term storage. Memory 224 may be configured as volatile memory for short-term storage of information, and therefore does not retain stored content if power is lost. Examples of volatile memory include random access memory (RAM), dynamic random access memory (DRAM), static random access memory (SRAM), and other forms of volatile memory known in the art.

Memory 224 may include one or more non-transitory computer-readable storage media. Memory 224 may be configured to store greater amounts of information than is typically stored by volatile memory. Memory 224 may also be configured as a non-volatile memory space for long-term storage of information and to retain information after power on/off cycles. Examples of non-volatile memory include magnetic hard disks, optical disks, flash memory, or forms of electrically programmable memory (EPROM) or electrically erasable programmable memory (EEPROM). Memory 224 may store program instructions (eg, annotation tool 242) and/or information (eg, point cloud training data 230 and annotated point cloud 232) that, when executed, cause processor 222 to perform the techniques of the present disclosure. .

The following techniques of the present disclosure will be described with reference to processor 222 executing various software modules. However, it should be understood that each software module described herein may also be implemented in dedicated hardware, firmware, software, or any combination of hardware, software, and firmware.

In accordance with techniques of the present disclosure, processor 222 may be configured to load point cloud training data 230 by executing annotation tool 242 . Point cloud training data 230 may include one or more frames of point cloud data, such as point cloud data captured by a LiDAR sensor or any other type of sensor that captures point cloud data. The annotation tool 242 may be configured to generate a GUI 250 including one or more frames of point cloud training data 230 and cause the display device 216 to display the GUI 250 . The user may then interact with the GUI 250 to annotate human poses onto a frame of point cloud training data 230 to define the human poses that may be present in the point cloud data. After annotation, annotation tool 242 may be configured to output annotated point cloud 232 . Annotated point cloud 232 may be stored in memory 224 and/or downloaded external to computing system 214 . The annotated point cloud 232 may then be used to train a deep neural network configured to estimate human pose from the point cloud data (eg, deep neural network 44 of Figure 2). Deep neural network 244 of Figure 9 represents a version of deep neural network 44 before and/or during the training process. 10-14 illustrate various examples of GUI 250 that may be generated by annotation tool 242. The operation of the annotation tool 242 with respect to the generated GUI 250 and user input is discussed in greater detail below.

Figure 10 is a conceptual user interface diagram illustrating an input point cloud for annotation. Annotation tool 242 may cause display device 216 to display GUI 250 including point cloud frame 252 . For example, annotation tool 242 may generate GUI 250 in response to user input to load point cloud frames 252 of point cloud training data 230 (see Figure 9). Annotation tool 242 may be configured to load and display point cloud frames 252 in response to user interaction with import data controls 254 . As shown in Figure 10, point cloud frame 252 may include points 264, which may correspond to one or more people captured in the point cloud data.

Import data controls 254 include a load match button, a load cloud(s) button, and a load Skel button. When the user selects the Load Matches button, the annotation tool 242 opens a file browser dialog box and reads the user-selected file containing the matching pairs of images and point clouds from the corresponding directory. . In this example, annotation tool 242 can load files from a directory that includes both images and matching point clouds. This way, paired images and point clouds can be viewed simultaneously. In other examples, annotation tool 242 may load only one of the image or point cloud.

When the user selects the Load Cloud button, the annotation tool 242 opens a file browser dialog and populates a list with available point cloud files in the directory selected by the user. The user can then select the point cloud to view from the populated drop-down list.

When the user selects the load skeleton, the annotation tool 242 opens a file browser dialog and loads any previously annotated skeleton from the file selected by the user. The user can then edit any previously annotated skeleton.

In some examples, annotation tool 242 may be configured to load multiple frames of point cloud training data 230 in response to user input. The annotation tool may initially display a single point cloud frame 252 among multiple frames. The annotation tool 242 may further generate one or more video controls 256 that cause the annotation tool 242 to display each of the plurality of frames of the point cloud data sequentially (eg, like a video). In the example of Figure 10, video control buttons 256 include play and stop buttons, but in other examples more controls may be provided (eg, pause, rewind, fast forward, etc.).

Annotation tool 242 may also include edit data controls 262 in GUI 250 . The user can interact with edit data control 262 to change the amount of data or area of point cloud frame 252 displayed in GUI 250 . The user can specify the area of the point cloud frame 252 to be displayed by specifying the minimum (min) and maximum (max) dimensions in the horizontal (x-lims), vertical (y-lims), and depth (z-lims) directions. . Annotation tool 242 may be configured to crop point cloud frame 252 in response to user input in edit data control 262 and display the cropped region of point cloud frame 252 . Edit data control 262 may further include a rot button that changes the angle at which point cloud frame 252 is viewed. The size buttons in the edit data control 262 change the size of each point in the point cloud.

In the above example, the user can use the annotation tool 242 to manually crop the area of the point cloud frame 252 to be displayed by manipulating the edit data control 262 . Cropping the point cloud frame 252 , such as around one or more potential people in the data, may make it easier to label the point cloud frame 252 . In other examples, rather than having the user manually crop the point cloud frame 252 , the annotation tool 242 is configured to automatically identify regions of interest in the point cloud frame 252 and automatically crop the point cloud frame 252 to display only the identified areas of interest. area. In one example, annotation tool 242 may be configured to identify regions of interest by detecting which areas of point cloud frame 252 include data indicative of persons on whom gestures may be annotated. In one example, annotation tool 242 may provide point cloud frames 242 to deep neural network 244 (FIG. 9) in order to identify regions of interest. Deep neural network 244 may be a deep neural network that recognizes people and estimates poses from point cloud data in the same manner as deep neural network 44 described above (see FIG. 2 ). The deep neural network 244 may be a complete deep neural network configured for pose estimation, or a deep neural network trained using the annotated point cloud 232 (see Figure 9) generated by the annotation tool 242. The deep neural network 244 may provide an indication of the region of interest to the annotation tool 242 , and the indication of the region of interest by the annotation tool 242 may crop the indicated region of interest or the point cloud frame 252 surrounding it.

Figure 11 is a conceptual user interface diagram illustrating a cropped point cloud for annotation. In FIG. 11 , annotation tool 242 displays cropped regions of point cloud frame 252 through GUI 250 in top view window 266 , side view window 268 , and front view window 270 . In top view window 266 , annotation tool 242 displays a cropped region of point cloud frame 252 from directly overhead. In the side view window 268, the annotation tool 242 displays the cropped region of the point cloud frame 252 from an angle (eg, 90 degrees) to a predetermined front angle. In the front view window 270, the annotation tool 242 displays a cropped region of the point cloud frame 252 from a predetermined front angle. In other examples, annotation tool 242 may display cropped regions of point cloud frame 242 from larger or smaller angles and/or perspectives, including isometric perspectives. Additionally, annotation tool 242 is not limited to displaying cropped regions of point cloud frame 252 at different angles. The annotation tool 242 can also display the entire cropped frame 252 at various angles.

Returning to Figure 10, the annotation tool 242 may further include an annotation control 260 (eg, a button labeled "Skeleton"). When the user selects the "Skeleton" button of annotation control 262, annotation tool 242 can mark points in point cloud frame 252 with a plurality of annotation points. That is, the annotation tool 242 may overlay a skeleton defined by a plurality of annotation points, where the plurality of annotation points correspond to points on the human body.

Figure 12 is a conceptual diagram illustrating an exemplary skeleton for annotation. Skeleton 400 represents an exemplary skeleton that annotation tool 242 may use to label points of point cloud frame 252 . In the example of Figure 12, the skeleton 400 is defined by 14 annotation points, each annotation point corresponding to a human joint or other human anatomy. In other examples, skeleton 400 may include more or fewer annotation points. As will be explained below, the user can manipulate and move one or more annotated points of the skeleton 400 to define the pose of the person represented by the points in the point cloud frame 242 .

In Figure 12, skeleton 400 faces the viewer. Thus, the "right" limb is shown on the left side of Figure 12 . The skeleton 400 consists of the top of the head mark point 402 (1), the center of the neck mark point 404 (2), the left shoulder mark point 408 (10), the right shoulder mark point 406 (5), and the left elbow mark point 412 (11). , right elbow mark point 410 (6), left hand mark point 416 (12), right hand mark point 414 (7), left hip mark point 420 (4), right hip mark point 418 (3), left knee mark point 424 ( 13), right knee marking point 422 (8), left foot marking point 428 (14) and right foot marking point 426 (9). The numbers in parentheses next to the reference numbers of the annotation points refer to the selection buttons of the annotation tool 242 as will be shown in Figures 13 and 14.

In addition to displaying various annotation points of the skeleton 400 , the annotation tool 242 may also display lines between the annotation points to define limbs and/or major body parts of the skeleton 400 . For example, the annotation tool 242 may display a line between the top of the head annotation point 402 and the center of the neck annotation point 404 to define the head. Annotation tool 242 may display a line extending from the center of neck annotation point 404 through right shoulder annotation point 406 and right elbow annotation point 410 and ending at right hand annotation point 414 to define the right arm. Annotation tool 242 may display a line extending from the center of neck annotation point 404 through left shoulder annotation point 408 and left elbow annotation point 412 and ending at left hand annotation point 416 to define the left arm. Annotation tool 242 may display a line from the center of neck annotation point 404 to left hip annotation point 420, to right hip annotation point 418, and back to the center of neck annotation point 404 to define the body. Annotation tool 242 may display a line from right hip annotation point 418 to right knee annotation point 422 and ending at right foot annotation point 426 to define the right leg. Annotation tool 242 may display a line from left hip annotation point 420 to left knee annotation point 424 and ending at left foot annotation point 428 to define the right leg.

As shown in Figure 12, annotation tool 242 may use different line widths and/or dash line types for different limbs. In this manner, the user can more easily distinguish the limbs of the skeleton 400 from each other in order to select appropriate annotation points for manipulation. In other examples, annotation tool 242 may use different colors to distinguish different limbs rather than using line weight or dashed line type. For example, the line between the label points defining the head could be blue, the line between the label points defining the right arm could be green, the line between the label points defining the left arm could be red, and the line between the label points defining the torso could be red. The line between the label points can be yellow, the line between the label points defining the right leg can be magenta, and the line between the label points defining the left leg can be cyan. Of course, other colors can be used.

Figure 13 is a conceptual user interface diagram illustrating estimated annotation of a point cloud. In FIG. 13 , annotation tool 242 labels points in point cloud frame 252 with annotation points of skeleton 400 . Annotation tool 242 shows skeleton 400 in each of top view 266 , side view 268 , and front view 270 in GUI 250 . Initially, annotation tool 242 may display skeleton 400 in a default position and default pose. As can be seen in Figure 13, the default position of skeleton 400 does not match the actual pose of the person depicted in the point cloud data of point cloud frame 252. The user may manipulate and/or move one or more of the annotated points of the skeleton 400 to match the actual pose of one or more persons present in the data of the point cloud frame 252 to produce an annotated point cloud.

In the example of Figure 13, annotation tool 242 displays a single skeleton 400. In other examples, if there are multiple poses to be annotated in point cloud frame 242, annotation tool 242 may generate and display multiple skeletons. In one example, annotation tool 242 may position skeleton 400 in a default location, such as in the center of each view. In other examples, the user may manually position the skeleton 400 (eg, by moving all annotation points together) into a location. In other examples, annotation tool 242 may automatically determine the location of one or more people in point cloud frame 252 and position skeleton 400 at the automatically determined location. In one example, annotation tool 242 may provide point cloud frames 242 to deep neural network 244 . Deep neural network 244 may be a deep neural network configured to estimate pose in the same manner as deep neural network 44 (see Figure 2) described above. Deep neural network 244 may be a full deep neural network configured for pose estimation, or may be a deep neural network trained on annotated point cloud 232 (see Figure 9) generated by annotation tool 242. Deep neural network 244 may determine the location of people in point cloud frame 252 and indicate the location of such people to annotation tool 242 . Annotation tool 242 may then display default skeleton 400 at the location indicated by deep neural network 244 .

The annotation tool 242 may also display the skeleton 400 in a default pose. That is, the annotation tool 242 may display the annotation points of the skeleton in a default orientation relative to each other. FIG. 13 shows an example of the default posture of the skeleton 400. Of course, the annotation tool 242 can generate other default poses. In other examples, rather than using the default pose, annotation tool 242 may estimate the position and pose of the person in point cloud frame 252 . In one example, annotation tool 242 may provide point cloud frames 242 to deep neural network 244 . Deep neural network 244 may be a deep neural network configured to estimate pose in the same manner as deep neural network 44 (see FIG. 2) described above. The deep neural network 244 may be a complete deep neural network configured for pose estimation, or a deep neural network trained on the annotated point cloud 232 (see Figure 9) generated by the annotation tool 242. Deep neural network 244 may determine estimated positions and poses of people found in point cloud frames 252 and indicate the positions and poses of these people to annotation tool 242 .

The annotation tool 242 may then display the default skeleton 400 at the position and pose indicated by the deep neural network 244 . Such estimated poses are usually not completely accurate. However, the estimated pose produced by the deep neural network 244 may be closer to the actual pose found in the point cloud data than the default pose. Therefore, the user of annotation tool 244 may start with a gesture that is closer to the actual gesture to be annotated, thereby making the manual process for annotating the gesture easier and faster.

Annotation tools 242 may provide the user with several different tools to manipulate the skeleton 400 into positions that represent the actual pose of the person found in the point cloud frame 252 . Annotation tool 242 may generate select area button 272 . The user can activate the select area button 272 and then select from the top view 266 , the side view 268 , or the front view 270 which view the user will interact with. Depending on the position and orientation of the person captured in point cloud frame 252, manipulation of skeleton 400 may be easier in different views. Label point control 274 allows the user to select specific label points for control. The numbers 1-14 of the check boxes in the label point control 274 correspond to the numbers in the label point brackets shown in Figure 12. The user can select one or more label points by selecting the corresponding checkbox. Annotation tool 242 may then move one or more of the selected annotation points in response to user input to define the human pose and create annotated point cloud data 232 (see FIG. 9 ).

In one example, the annotation tool 242 may move the selected annotation point in response to user interaction with the mouse (eg, click and drag). In other examples, annotation tool 242 may move one or more annotation points in response to user interaction with rotation control 280 and/or position control 282 . In the example of Figure 13, rotation control 280 is a slider control that causes annotation tool 242 to rotate all annotation points of skeleton 400 around the top of the head annotation point. Position control 282 includes respective sliders for the horizontal (X), vertical (Y), and depth (Z) dimensions of point cloud frame 252 . In response to the user moving the slider, the annotation tool 242 moves the selected annotation points to each specified dimension within the point cloud frame 252 . Although the example of FIG. 13 shows a slider control, other control types may be used, including text input of specific coordinates within point cloud frame 252 (eg, (X, Y, Z) coordinates).

In one example of the present disclosure, annotation tool 242 may assign a unique identifier to each annotation point in the skeleton. In this way, the pose and annotation points of a specific skeleton can be tracked across multiple frames. Additionally, in conjunction with the action recognition techniques discussed above, the annotation tool 242 may also include action category tags (eg, human gestures) for each frame and each skeleton.

In one example of the present disclosure, to increase the accuracy of locating annotation points, annotation tool 242 may be configured to only allow selection of a single annotation point at a time. Once a single label point is selected, the user may cause the label tool 242 to move only the single selected label point. For example, once one of the checkbox annotation point controls 274 is selected, the annotation tool 242 can render the other checkboxes unavailable for selection. In order to move other label points, the user can first deselect the selected label point. While moving a single selected annotation point, annotation tool 242 holds all other annotation points stationary. In other examples, annotation tool 242 may allow selection of multiple annotation points. In this example, the annotation tool 242 will move only the selected annotation points (eg, one or more selected annotation points) in response to the user input. Unselected label points will remain stationary.

That is, in one example, the annotation tool 242 receives a selection of a single annotation point among a plurality of annotation points and, in response to user input, moves only the single selected annotation point to define a portion of the human pose. In other examples, the annotation tool 242 receives a selection of two or more of the plurality of annotation points and, in response to the user input, moves only the two or more selected annotation points to define the human pose. part. The user can type comments about the annotated point cloud frame 252 in the annotation window 276 . After labeling is complete, the user can activate save control 278 to save the final location of each label point. The annotated point cloud 232 (see Figure 9) can then be saved in memory (eg, in a .json file) and then used to train a neural network (eg, neural network 244).

Figure 14 is a conceptual user interface diagram showing a labeled point cloud. Figure 14 shows the pose of skeleton 400 from Figure 13 after user manipulation. As can be seen in FIG. 14 , the annotated points of the skeleton 400 have been moved to a position (ie, pose) that better matches the actual pose captured by the person in the point cloud 252 .

Figure 15 is a flowchart showing example operations of an annotation tool according to one example of the present disclosure. The techniques in Figure 15 may be performed by processor 222 executing instructions for annotation tool 242 (see Figure 9). The technique of Figure 15 depicts an exemplary process for annotating a single frame of point cloud data. The process of Figure 15 can be repeated for multiple frames of point cloud data.

Annotation tool 242 can load and display point cloud data (900). For example, annotation tool 242 may load point cloud data from a point cloud file (eg, point cloud training data 230 of FIG. 9 ) and may cause display device 216 to display GUI 250 , where GUI 250 includes the loaded point cloud data. Annotation tool 242 may then determine whether to crop the point cloud automatically or through user input (902). If so, the annotation tool 902 displays the cropped point cloud in one or more views (904). The annotation tool 904 may label points in the displayed point cloud (eg, cropped point cloud) with annotation points (906). If the point cloud data is not clipped (902), the labeling tool 904 also labels points in the displayed point cloud (eg, the entire point cloud) with label points (906).

The annotation tool 242 then waits until an annotation point is selected (908). Once an annotation point is selected, annotation tool 242 will then move the selected annotation point in response to user input (910). Annotation tool 242 will then check for user input indicating annotation is complete (912). If so, the annotation tool 242 outputs the annotated point cloud (914). If not, then the annotation tool 242 will wait for another annotation point to be selected (908) and then repeat the movement process in response to user input (910).

It will be appreciated that, by way of example, certain actions or events of any technology described herein may be performed in a different order, may be added, combined, or eliminated altogether (e.g., not all described actions or events may be required to implement the technology). required). Furthermore, in some examples, actions or events may be performed concurrently rather than sequentially, such as through multi-threading, interrupt handling, or multiple processors.

In one or more examples, the described functionality may be implemented in hardware, software, firmware, or any combination thereof. If implemented in software, the functions may be stored or transmitted on a computer-readable medium as one or more instructions or code and executed by a hardware-based processing unit. Computer-readable media may include computer-readable storage media, which corresponds to tangible media such as data storage media, or communication media including any medium that facilitates transfer of a computer program from one place to another, for example, according to a communications protocol. In this manner, computer-readable media generally may correspond to (1) non-transitory, tangible computer-readable storage media, or (2) communications media such as signals or carrier waves. Data storage media may be any available media that can be accessed by one or more computers or one or more processors to retrieve instructions, code, and/or data structures to implement the techniques described in this disclosure. A computer program product may include computer-readable media.

By way of example and not limitation, such computer-readable data storage media may include RAM, ROM, EEPROM, CD-ROM or other optical disk storage, magnetic disk storage, or other magnetic storage devices, flash memory, or may be used to store instructions or data. Any other medium in a structured form that stores the required program code and can be accessed by a computer. Also, any connection is properly termed a computer-readable medium. For example, the definition of medium if coaxial cable, fiber optic cable, twisted pair, digital subscriber line (DSL), or wireless technologies (such as infrared, radio, and microwave) are used to send instructions from a website, server, or other remote source This includes coaxial cable, fiber optic cable, twisted pair, DSL or wireless technologies such as infrared, radio and microwave. However, it should be understood that computer-readable storage media and data storage media do not include connections, carrier waves, signals, or other transitory media, but are directed to non-transitory tangible storage media. Combinations of the above should also be included within the scope of computer-readable media.

Instructions may be executed by one or more processors, such as one or more digital signal processors (DSPs), general purpose microprocessors, application specific integrated circuits (ASICs), field programmable gate arrays (FPGAs), complex programmable logic device (CPLD), or other equivalent integrated or discrete logic circuit. Accordingly, the term "processor" as used herein may refer to any of the foregoing structures or any other structure suitable for implementing the techniques described herein. Also, the technology may be fully implemented in one or more circuit or logic elements.

The techniques of the present disclosure may be implemented in a variety of devices or devices including an integrated circuit (IC) or a group of ICs (eg, a chipset). Various components, modules, or units are described in this disclosure to emphasize functional aspects of devices configured to perform the disclosed techniques, but do not necessarily require implementation by different hardware units.

Any changes and/or modifications to the methods and apparatus of the disclosed technology known to those of ordinary skill in the art are within the scope of the invention. Various examples of the invention have been described. These and other embodiments are within the scope of the following claims.

Claims (20)

1. A method for labeling human posture in 3D point cloud data, including: causing, by one or more processors, to display at least one of the plurality of frames of point cloud data; Mark points in the at least one frame of the point cloud data with a plurality of annotation points by the one or more processors, the plurality of annotation points corresponding to points on the human body; causing a video display of the plurality of frames of the point cloud data by the one or more processors, wherein the video display can be controlled to forward, fast forward, pause, rewind; moving, by the one or more processors and in response to user input, one or more of the annotation points to define human poses and create annotated point cloud data; and The annotated point cloud data is output by the one or more processors for training by the neural network. 2. The method of claim 1, wherein marking points in the at least one frame of the point cloud data with a plurality of annotation points by the one or more processors includes: Estimating, by the one or more processors, the position of a potential human pose in the at least one frame of the point cloud data; and The annotation points are marked by the one or more processors to correspond to estimated locations of the potential human poses. 3. The method of claim 1, wherein one or more of the annotated points are moved by the one or more processors and in response to user input to define human poses and create annotated point cloud data include: receiving, by the one or more processors, a selection of a single annotation point from the plurality of annotation points; Only the single annotation point is moved by the one or more processors and in response to user input to define a portion of the human pose. 4. The method of claim 1, wherein one or more of the annotated points are moved by the one or more processors and in response to user input to define human poses and create annotated point cloud data include: receiving, by the one or more processors, selections of two or more annotation points from the plurality of annotation points; By the one or more processors and in response to user input, only the two or more annotation points are moved to define a portion of the human pose. 5. The method of claim 1, further comprising: displaying an image corresponding to the at least one frame of the point cloud data while displaying the point cloud data; Cropping, by the one or more processors, the point cloud data and the image around potential human pose regions in the point cloud data; and The cropped area is displayed by the one or more processors. 6. The method of claim 5, further comprising: The cropped area is displayed in multiple viewing angles by the one or more processors. 7. The method of claim 1, wherein the plurality of marking points include the top of the head, the center of the neck, the right hip, the left hip, the right shoulder, the right elbow, the right hand, the right knee, and the right foot. , left shoulder, left elbow, left hand, left knee, and left foot corresponding marking points, and wherein the group of said marking points corresponds to human limbs, the method further includes: Displaying, by the one or more processors, lines between the annotation points to define limbs, including displaying different limbs using different colors. 8. The method of claim 1, further comprising: An action tag is added to each frame of point cloud data and human posture in the plurality of frames of the point cloud data by the one or more processors. 9. The method of claim 1, further comprising: The neural network is trained with the annotated point cloud data by the one or more processors, wherein the neural network is configured to estimate a human pose from the LiDAR point cloud data. 10. A device for labeling human posture in 3D point cloud data, including: a memory configured to store point cloud data; and One or more processors in communication with the memory, the one or more processors being configured to: causing at least one frame among the plurality of frames of the point cloud data to be displayed; Mark points in the at least one frame of the point cloud data with a plurality of annotation points, the plurality of annotation points corresponding to points on the human body; causing a video to display the plurality of frames of the point cloud data, wherein the video display can be controlled to forward, fast forward, pause, rewind; moving one or more of the annotated points to define human poses and create annotated point cloud data in response to user input; and The output labeled point cloud data is used for training by the neural network. 11. The apparatus of claim 10, wherein to label points in the at least one frame of the point cloud data with a plurality of annotation points, the one or more processors are further configured to: estimating the location of a potential human pose in the at least one frame of the point cloud data; and The annotation points are marked to correspond to the estimated positions of the potential human poses. 12. The device of claim 10, wherein the one or more processors are further configured to move one or more of the annotated points to define human poses and create annotated point cloud data in response to user input. Configured as: receiving a selection of a single annotation point among the plurality of annotation points; In response to user input, only the single annotation point is moved to define a portion of the human pose. 13. The device of claim 10, wherein the one or more processors are further configured to move one or more of the annotated points to define human poses and create annotated point cloud data in response to user input. Configured as: receiving selection of two or more annotation points from the plurality of annotation points; In response to user input, only the two or more annotation points are moved to define a portion of the human pose. 14. The device of claim 10, wherein the one or more processors are further configured to: displaying an image corresponding to the at least one frame of the point cloud data while displaying the point cloud data; cropping the point cloud data and the image around potential human pose regions in the point cloud data; and Makes the cropped area appear. 15. The device of claim 14, wherein the one or more processors are further configured to: Makes the cropped area visible from multiple perspectives. 16. The device according to claim 10, wherein the plurality of marking points include the top of the head, the center of the neck, the right hip, the left hip, the right shoulder, the right elbow, the right hand, the right knee, the right foot. , left shoulder, left elbow, left hand, left knee, and left foot corresponding annotation points, and wherein the group of annotation points corresponds to human limbs, and wherein the one or more processors are further configured to : Causes display of lines between the annotation points to define limbs, including causing display of different limbs using different colors. 17. The device of claim 10, wherein the one or more processors are further configured to: Add an action tag to each frame of point cloud data and human body posture in the plurality of frames of the point cloud data. 18. The device of claim 10, wherein the one or more processors are further configured to: The neural network is trained with the annotated point cloud data, wherein the neural network is configured to estimate human pose from the LiDAR point cloud data. 19. A device for labeling human posture in 3D point cloud data, including: means for causing at least one frame of a plurality of frames of point cloud data to be displayed; A device for marking points in the at least one frame of the point cloud data with a plurality of annotation points, the plurality of annotation points corresponding to points on the human body; means for causing a video display of the plurality of frames of the point cloud data, wherein the video display is controllable to forward, fast forward, pause, rewind; means for moving one or more of the annotated points to define a human pose and create annotated point cloud data in response to user input; and A device for outputting labeled point cloud data for training by a neural network. 20. A non-transitory computer-readable storage medium storing instructions that, when executed, cause one or more processors to: causing at least one frame of the plurality of frames of point cloud data to be displayed; Mark points in the at least one frame of the point cloud data with a plurality of annotation points, the plurality of annotation points corresponding to points on the human body; causing a video to display the plurality of frames of the point cloud data, wherein the video display can be controlled to forward, fast forward, pause, rewind; moving one or more of the annotated points to define a human pose and create annotated point cloud data in response to user input; and The output labeled point cloud data is used for training by the neural network.