TWI855330B - Method for detecting three-dimensional object, electronic device and storage medium - Google Patents

Abstract

The present application provides a method for detecting three-dimensional (3D) object, an electronic device and a storage medium. The method includes: acquiring a training image; constructing and training a semantic segmentation model and obtaining a trained semantic segmentation model; inputting detected images into the trained semantic segmentation model and obtaining object categories and object positions; determining an object model from a 3D object model library; obtaining point cloud data of the object and a distance between a depth camera and the object model according to the depth image; determining a rotation angle of the object model according to the point cloud data and the object model. According to the distance from the depth camera to the object model, the rotation angle, and the object positions, the position of the object model in a 3D space is determined, thereby determining the position of the object in the 3D space. This application can quickly determine the position of the object in 3D space.

Description

Three-dimensional target detection method, electronic device and computer-readable storage medium

This application relates to computer vision and deep learning technology, and in particular to a three-dimensional target detection method, electronic equipment, and computer-readable storage medium.

In the field of autonomous driving, the autonomous driving system uses different types of sensors to detect objects in front of or near the vehicle and make corresponding decisions. For example, the autonomous driving system needs to quickly and accurately detect the type and three-dimensional position of the object, and then make corresponding decisions to ensure driving safety. However, the three-dimensional target detection algorithm is currently used to detect the type and position of the object, and the three-dimensional position of the object is obtained by regression operation, but the regression operation takes a long time to predict. In addition, when performing three-dimensional target detection, the existing autonomous driving system uses lidar or radar to obtain depth information to detect the distance between the vehicle and the object in front, but the current cost of using lidar or radar is high and the field of view is relatively small.

In view of the above, it is necessary to provide a three-dimensional target detection method, electronic equipment and computer-readable storage medium to solve the problems of slow object detection efficiency and high cost.

The present application embodiment provides a three-dimensional target detection method, which includes: obtaining a training image; constructing a semantic segmentation model based on a fully convolutional network; inputting the training image into the semantic segmentation model, performing multiple convolutions and pooling using the convolution layer and the pooling layer in the semantic segmentation model, and obtaining multiple feature maps of different sizes; upsampling the multiple feature maps of different sizes to obtain a first image of the same size as the training image, performing pixel classification on the first image and optimizing classification loss, outputting the object category of the object in the training image and the position of the object, and obtaining a trained semantic segmentation model; obtaining a detection image and the detection image; A depth image corresponding to the image is obtained by a depth camera; the detection image is input into the trained semantic segmentation model to obtain the object category and object position of the object in the detection image; according to the object category, the object model corresponding to the object is determined from the three-dimensional object model library; according to the depth image, the point cloud data of the object and the distance from the depth camera to the object model are obtained; according to the point cloud data and the object model, the rotation angle of the object model is determined; according to the distance from the depth camera to the object model, the rotation angle and the position of the object, the position of the object model in three-dimensional space is determined.

In an optional implementation, upsampling the feature maps of different sizes to obtain a first image of the same size as the training image includes: upsampling the feature maps of different sizes, performing a deconvolution operation to obtain an operation result; and summing the operation results to obtain a first image of the same size as the training image.

In an optional implementation, the step of determining the object model corresponding to the object from the three-dimensional object model library according to the object category includes: searching the three-dimensional object model library according to the object category to determine the object model of the object, wherein the three-dimensional object model library includes the object category and the object model corresponding to the object category.

In an optional implementation, obtaining the point cloud data of the object and the distance from the depth camera to the object model according to the depth image includes: obtaining the depth value and coordinates of the object according to the depth image, determining the distance from the depth camera to the object model according to the depth value; and obtaining the point cloud data according to the coordinates and the internal and external parameter matrix transformation formula of the depth camera.

In an optional implementation, determining the rotation angle of the object model according to the point cloud data and the object model includes: obtaining first point cloud data of the object outline according to the point cloud data; converting the object model into second point cloud data; performing point cloud matching on the first point cloud data and the second point cloud data, merging the points of the object outline in the first point cloud data into a first plane and calculating the curvature of the first plane, merging the points of the second point cloud data into a second plane and calculating the curvature of the second plane; calculating the difference between the curvature of the first plane and the curvature of the second plane to obtain a curvature deviation value, and determining the rotation angle of the object model of the object according to the curvature deviation value.

In an optional implementation, converting the object model into the second point cloud data includes: using multiple functions in the point cloud library to process the object model of the object and generate the second point cloud data.

In an optional implementation, determining the position of the object model in three-dimensional space according to the distance from the depth camera to the object model, the rotation angle, and the position of the object includes: determining the direction of the object model in three-dimensional space according to the rotation angle; determining the position of the object model in three-dimensional space according to the direction of the object model in the three-dimensional space, the distance from the depth camera to the object model, and the position of the object.

In an optional implementation, the method further includes: using the position of the object model of the object in the three-dimensional space as the position of the object in the three-dimensional space, and outputting the object category and the position of the object in the three-dimensional space.

This application embodiment also provides an electronic device, which includes a processor and a memory, and the processor is used to implement the three-dimensional target detection method when executing a computer program stored in the memory.

This application embodiment also provides a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, the three-dimensional target detection method is implemented.

The technical solution of this application is low-cost and does not require complex calculations, and can quickly obtain the three-dimensional position of an object.

4: Electronic equipment

401: Memory

402:Processor

403:Computer Program

404: Communication bus

101-107: Steps

Figure 1 is a flow chart of a three-dimensional target detection method provided in an embodiment of this application.

Figure 2 is an image segmented by the trained semantic segmentation model provided in the embodiment of this application.

Figure 3 is a schematic diagram of the three-dimensional object model library provided in this application embodiment.

Figure 4 is a schematic diagram of the structure of an electronic device provided in an embodiment of this application.

In order to more clearly understand the above-mentioned purpose, features and advantages of this application, the following is a detailed description of this application in conjunction with the attached drawings and specific embodiments. It should be noted that the specific embodiments described here are only used to explain this application and are not intended to limit this application.

In the following description, many specific details are explained to facilitate a full understanding of this application. The embodiments described are only part of the embodiments of this application, not all of them. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without creative labor are within the scope of protection of this application.

Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by technicians in the technical field of this application. The terms used in this specification of this application are only for the purpose of describing specific embodiments and are not intended to limit this application.

Refer to FIG. 1, which is a flow chart of a three-dimensional target detection method provided in an embodiment of the present application. The method is applied to an electronic device (for example, the electronic device 4 shown in FIG. 4), and the electronic device can be any electronic product that can interact with a user, such as a personal computer, a tablet computer, a smart phone, a personal digital assistant (PDA), a game console, an interactive network television (IPTV), a smart wearable device, etc.

The electronic device is a device that can automatically perform numerical calculations and/or information processing according to pre-set or stored instructions, and its hardware includes, but is not limited to: microprocessors, application specific integrated circuits (ASIC), field-programmable gate arrays (FPGA), digital signal processors (DSP), embedded devices, etc.

The electronic device may also include network equipment and/or user equipment. The network equipment includes, but is not limited to, a single network server, a server group consisting of multiple network servers, or a cloud consisting of a large number of hosts or network servers based on cloud computing.

The network where the electronic device is located includes but is not limited to the Internet, wide area network, metropolitan area network, local area network, virtual private network (VPN), etc.

The method specifically includes the following:

101, get the training image.

In at least one embodiment of the present application, the training images include, but are not limited to, scene images of various types of roads such as cities and rural areas at different time periods. In this embodiment, the training images include images in the Pascal VOC dataset and the Cityscapes dataset.

In at least one embodiment of the present application, the method further includes: performing a data enhancement operation on the training image to expand different training images. The data enhancement operation includes, but is not limited to, flipping the image, rotating the image, scaling the image, and cropping the image. It should be noted that by performing the data enhancement operation, more images of the front of the vehicle in different scenes can be obtained as training samples, and the semantic segmentation model learning is trained and optimized to make the semantic segmentation model more robust.

102, construct a semantic segmentation model and use the training images to train the semantic segmentation model to obtain a trained semantic segmentation model.

In at least one embodiment of the present application, constructing a semantic segmentation model includes: constructing a semantic segmentation model based on a fully convolutional network.

Based on the fully convolutional network, multiple targets in the image input to the fully convolutional network can be segmented and identified. For example, an image is input to the semantic segmentation model, and the image includes multiple targets. For example, a car, a person, a puppy, etc. After the semantic segmentation model segments and identifies the multiple targets in the image, it outputs the object category of the object in the training image and the position of the object.

In at least one embodiment of the present application, the method for training a semantic segmentation model to obtain a trained semantic segmentation model includes: inputting the training image into the semantic segmentation model, performing multiple convolutions and poolings using the convolution layer and the pooling layer in the semantic segmentation model to obtain multiple feature maps of different sizes; upsampling the multiple feature maps of different sizes, and then performing deconvolution operations to obtain operation results; adding the operation results to obtain a first image of the same size as the training image; performing pixel classification on the first image, and calculating and optimizing the classification loss, and outputting the object category of the object in the training image and the position of the object to obtain the trained semantic segmentation model. In this embodiment, the softmax cross entropy function is used as the loss function to train the semantic segmentation model. For example, as shown in FIG2, FIG2 is an image segmented by the semantic segmentation model after the training provided in the embodiment of the present application.

103, obtaining a detection image and a depth image corresponding to the detection image.

In at least one embodiment of the present application, a camera installed inside or outside a vehicle is used to take pictures, and the image in front of the vehicle is used as a detection image.

In at least one embodiment of the present application, obtaining a depth image corresponding to the detection image includes: obtaining a depth image using a depth camera, and using an image in front of the vehicle taken by the depth camera installed on the vehicle as the depth image. It should be noted that when a camera installed inside or outside the vehicle is used to take an image in front of the vehicle as a detection image, the depth camera simultaneously takes an image in front of the vehicle as a depth image, and the depth image corresponds to the detection image. For example, different types of cameras are used to take pictures of the same object in front of the vehicle to obtain a detection image and a depth image.

It should be noted that the depth camera is an existing depth camera, and the method further includes: obtaining depth information of the depth image from the depth camera.

104, according to the object category, determine the object model corresponding to the object from the three-dimensional object model library.

In at least one embodiment of the present application, the method of determining the object model corresponding to the object from the three-dimensional object model library according to the object category includes: establishing a three-dimensional object model library corresponding to the object category and the object model, wherein the three-dimensional object model library includes the object category and the object model corresponding to the object category; and determining the object model of the object by searching the three-dimensional object model library according to the object category. Referring to FIG3 , a schematic diagram of the three-dimensional object model library provided in the embodiment of the present application is shown. When the object type is a car, the object model of the car is searched based on the three-dimensional object model library; when the object type is a small truck, the object model of the small truck is searched based on the three-dimensional object model library; when the object type is an electric car, the object model of the electric car is searched based on the three-dimensional object model library; when the object type is a large bus, the object model of the large bus is searched based on the three-dimensional object model library.

In this application embodiment, the object model of the object includes a three-dimensional model of the object.

105, obtaining the point cloud data and the distance from the depth camera to the object model according to the depth image.

In at least one embodiment of the present application, determining the distance from the depth camera to the object model includes: obtaining a depth value of the object according to the depth image; and determining the distance from the depth camera to the object model according to the depth value. In this embodiment, the depth value is obtained from the depth camera. Specifically, when the depth image is obtained by shooting with the depth camera, the depth camera displays a depth value, and the depth value is the distance from the depth camera to the object. In this embodiment, the distance from the depth camera to the object is used as the distance from the depth camera to the object model of the object.

In at least one embodiment of the present application, the method for obtaining the point cloud data includes: obtaining the coordinate set of the object according to the depth image; obtaining the point cloud data of the object according to the coordinate set and the intrinsic and extrinsic parameter matrix transformation formula of the depth camera. In this embodiment, the coordinate set of the object is the pixel coordinate set of the object; the point cloud data of the object is the world coordinates corresponding to the coordinates in the coordinate set of the object. The point cloud data of the object is used to characterize the contour of the object. The coordinates in the coordinate set of the object are converted into corresponding world coordinates through the intrinsic and extrinsic parameter matrix transformation formula, and the intrinsic and extrinsic parameter matrix transformation formula is:

Where (x, y, z) is the world coordinate, used to represent a point cloud of pixel coordinates, f is the focal length, D is the depth value, (x1 _, y1 ₎ is the pixel coordinate of any pixel point in the coordinate set of the object in the two-dimensional boundary frame. All coordinates in the coordinate set are converted into world coordinates one by one using the above formula to obtain the point cloud data of the object.

106, determining the rotation angle of the object model according to the point cloud data and the object model.

In at least one embodiment of the present application, determining the rotation angle of the object model according to the point cloud data and the object model includes: obtaining first point cloud data of the object outline according to the point cloud data; converting the object model into second point cloud data; performing point cloud matching on the first point cloud data and the second point cloud data, merging the points of the object outline in the first point cloud data into a first plane and calculating the curvature of the first plane, merging the points of the second point cloud data into a second plane and calculating the curvature of the second plane; calculating the difference between the curvature of the first plane and the curvature of the second plane to obtain a curvature deviation value, and determining the rotation angle of the object model of the object according to the curvature deviation value.

In at least one embodiment of the present application, the converting the object model of the object into the second point cloud data includes: using multiple functions in the Point Cloud Library (PCL) to process the object model of the object and generate the point cloud data of the object model of the object as the second point cloud data.

107, determine the position of the object in three-dimensional space.

In at least one embodiment of the present application, the direction of the object model in the three-dimensional space is determined according to the rotation angle, and the position of the object model in the three-dimensional space is determined according to the direction of the object model in the three-dimensional space, the distance from the depth camera to the object model, and the position of the object. Specifically, the position of the object model in the three-dimensional space is used as the position of the object in the three-dimensional space, and the object category and the position of the object in the three-dimensional space are output. For example, the object category and the position of the object in the three-dimensional space are displayed on a display screen.

The above is only the specific implementation method of this application, but the protection scope of this application is not limited to this. For ordinary technical personnel in this field, improvements can be made without departing from the creative concept of this application, but these are all within the protection scope of this application.

As shown in FIG. 4 , FIG. 4 is a schematic diagram of an electronic device structure provided in an embodiment of the present application. The electronic device 4 includes a memory 401, at least one processor 402, a computer program 403 stored in the memory 401 and executable on the at least one processor 402, and at least one communication bus 404.

Those skilled in the art can understand that the schematic diagram shown in FIG4 is only an example of the electronic device 4 and does not constitute a limitation on the electronic device 4. The electronic device 4 may include more or fewer components than shown in the diagram, or may combine certain components, or different components. For example, the electronic device 4 may also include input and output devices, network access devices, etc.

The at least one processor 402 may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA), or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components, etc. The at least one processor 402 may be a microprocessor or any conventional processor, etc. The at least one processor 402 is the control center of the electronic device 4, and uses various interfaces and lines to connect the various parts of the entire electronic device 4.

The memory 401 can be used to store the computer program 403. The at least one processor 402 realizes various functions of the electronic device 4 by running or executing the computer program 403 stored in the memory 401 and calling the data stored in the memory 401. The memory 401 can mainly include a program storage area and a data storage area, wherein the program storage area can store an operating system, an application required for at least one function (such as a sound playback function, an image playback function, etc.), etc.; the data storage area can store data created according to the use of the electronic device 4 (such as audio data), etc. In addition, the memory 401 may include non-volatile memory, such as a hard disk, memory, a plug-in hard disk, a Smart Media Card (SMC), a Secure Digital (SD) card, a Flash Card, at least one disk memory device, a flash memory device, or other non-volatile solid-state memory devices.

If the module/unit integrated in the electronic device 4 is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the present application implements all or part of the processes in the above-mentioned embodiment method, and can also be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium. When the computer program is executed by the processor, the steps of the above-mentioned method embodiments can be implemented. Among them, the computer program includes computer program code, and the computer program code can be in the form of source code, object code, executable file or some intermediate form. The computer-readable medium may include: any entity or device capable of carrying the computer program code, recording media, a flash drive, a removable hard drive, a magnetic disk, an optical disk, a computer memory, and a read-only memory (ROM).

It is obvious to those skilled in the art that the present application is not limited to the details of the above exemplary embodiments and can be implemented in other specific forms without departing from the spirit or basic features of the present application. Therefore, no matter from which point of view, the embodiments should be regarded as exemplary and non-restrictive, and the scope of the present application is defined by the attached claims rather than the above description, so it is intended to include all changes that fall within the meaning and scope of the equivalent elements of the claims. Any attached figure mark in the claims should not be regarded as limiting the claims involved.

Finally, it should be noted that the above embodiments are only used to illustrate the technical solution of this application and are not limiting. Although this application is described in detail with reference to the preferred embodiments, ordinary technicians in this field should understand that the technical solution of this application can be modified or replaced by equivalents without departing from the spirit and scope of the technical solution of this application.

101-107: Steps

Claims (9)

A three-dimensional target detection method is applied to electronic equipment, wherein the three-dimensional target detection method comprises: obtaining a training image; constructing a semantic segmentation model based on a fully convolutional network; inputting the training image into the semantic segmentation model, performing multiple convolutions and pooling using the convolutional layer and the pooling layer in the semantic segmentation model to obtain multiple feature maps of different sizes; upsampling the multiple feature maps of different sizes to obtain the same feature map as the A first image of the same size as the training image is obtained, pixel classification is performed on the first image and classification loss is optimized, the object category of the object in the training image and the position of the object are output, and a trained semantic segmentation model is obtained, wherein the upsampling of the feature maps of the plurality of different sizes to obtain the first image of the same size as the training image comprises: after upsampling the feature maps of the plurality of different sizes, inverse A convolution operation is performed to obtain an operation result; an addition operation is performed on the operation result to obtain a first image of the same size as the training image; a detection image and a depth image corresponding to the detection image are obtained, wherein the depth image is obtained by a depth camera; the detection image is input into the trained semantic segmentation model to obtain the object category and object position of the object in the detection image; according to the object category, an object is obtained from the three Determine the object model corresponding to the object in the three-dimensional object model library; obtain the point cloud data of the object and the distance from the depth camera to the object model according to the depth image; determine the rotation angle of the object model according to the point cloud data and the object model; Determine the position of the object model in three-dimensional space according to the distance from the depth camera to the object model, the rotation angle and the position of the object. The three-dimensional target detection method as described in claim 1, wherein the determining the object model corresponding to the object from the three-dimensional object model library according to the object category comprises: searching the three-dimensional object model library according to the object category to determine the object model of the object, wherein the three-dimensional object model library comprises the object category and the object model corresponding to the object category. The three-dimensional target detection method as described in claim 1, wherein the step of obtaining the point cloud data of the object and the distance from the depth camera to the object model according to the depth image comprises: obtaining the depth value and coordinates of the object according to the depth image, determining the distance from the depth camera to the object model according to the depth value; and obtaining the point cloud data according to the coordinates and the internal and external parameter matrix transformation formula of the depth camera. The three-dimensional target detection method as described in claim 1, wherein the determining the rotation angle of the object model according to the point cloud data and the object model comprises: obtaining first point cloud data of the object outline according to the point cloud data; converting the object model into second point cloud data; performing point cloud matching on the first point cloud data and the second point cloud data, merging the points of the object outline in the first point cloud data into a first plane and calculating the curvature of the first plane, merging the points of the second point cloud data into a second plane and calculating the curvature of the second plane; calculating the difference between the curvature of the first plane and the curvature of the second plane to obtain a curvature deviation value, and determining the rotation angle of the object model of the object according to the curvature deviation value. The three-dimensional target detection method as described in claim 4, wherein the converting the object model into the second point cloud data includes: using multiple functions in the point cloud library to process the object model of the object and generate the second point cloud data. A three-dimensional target detection method as described in any one of claims 1 to 5, wherein determining the position of the object model in three-dimensional space according to the distance from the depth camera to the object model, the rotation angle, and the position of the object comprises: determining the direction of the object model in three-dimensional space according to the rotation angle; determining the position of the object model in three-dimensional space according to the direction of the object model in the three-dimensional space, the distance from the depth camera to the object model, and the position of the object. The three-dimensional target detection method as described in claim 6, wherein the method further comprises: taking the position of the object model of the object in the three-dimensional space as the position of the object in the three-dimensional space, and outputting the object category and the position of the object in the three-dimensional space. An electronic device, wherein the electronic device includes a processor and a memory, and the processor is used to execute a computer program stored in the memory to implement a three-dimensional target detection method as described in any one of claims 1 to 7. A computer-readable storage medium, wherein the computer-readable storage medium stores at least one instruction, and when the at least one instruction is executed by a processor, the three-dimensional target detection method as described in any one of claim items 1 to 7 is implemented.