CN115774844A

CN115774844A - Category determination method, device, equipment and storage medium

Info

Publication number: CN115774844A
Application number: CN202211335652.1A
Authority: CN
Inventors: 黄梓航; 周航; 李宜恒; 伍小军; 刘妮妮; 陈炫翰; 李玉鑫
Original assignee: Huizhou Desay SV Intelligent Transport Technology Research Institute Co Ltd
Current assignee: Huizhou Desay SV Intelligent Transport Technology Research Institute Co Ltd
Priority date: 2022-10-28
Filing date: 2022-10-28
Publication date: 2023-03-10

Abstract

The invention discloses a category determination method, a category determination device, category determination equipment and a storage medium. The method comprises the following steps: acquiring a projection matrix between at least one camera and at least one laser radar; controlling each camera to acquire image data of a target object; controlling each laser radar to obtain point cloud data corresponding to a target object; and determining the target category information of the target object according to the projection matrix, the image data of the target object and the point cloud data corresponding to the target object. According to the embodiment of the invention, the problem that the detection result of the camera and the detection result of the laser radar in the prior art cannot be effectively fused is solved by acquiring the projection matrix between the at least one camera and the at least one laser radar, controlling each camera to acquire the image data of the target object, controlling each laser radar to acquire the point cloud data corresponding to the target object, and determining the target category information of the target object according to the projection matrix, the image data of the target object and the point cloud data corresponding to the target object.

Description

Method, device, equipment and storage medium for determining category

Technical Field

The invention relates to the technical field of multi-sensor data fusion, in particular to a category determination method, a category determination device, category determination equipment and a storage medium.

Background

Multi-sensor data fusion is an emerging field, and is a research on data processing developed for the specific problem that a system uses multiple sensors. In the last decade, with the rapid development of unmanned and mobile robots, a precise and efficient sensing system can ensure the safety of the robot and other surrounding moving objects, and the fusion of multiple sensors plays an important role in the sensing system. In recent years, the application of lidar to unmanned sensing systems has been heavily developed, and camera-based two-dimensional object detection has been rapidly developed in unmanned systems, but still more improvements are required in fusing camera-detected objects and lidar-detected objects with each other.

The task of this multi-sensor fusion is usually related to sensors such as laser radar, millimeter wave radar and camera. Lidar sensors are commonly used in object segmentation and object detection algorithms based on 3D world coordinate systems. Images produced by cameras generally have richer semantic information and are therefore often used in object segmentation and detection algorithms based on image planes. However, the two sensors detect based on different reference coordinate systems, and for the laser radar, the detection accuracy is lower than the accurate accuracy of the camera, so that detection omission often occurs, but the distance accuracy is better than the distance accuracy provided by the camera. For a camera detection algorithm, the accuracy of target object detection is higher than that of laser radar, but the distance accuracy is far lower than that provided by laser radar. Therefore, if the detection results of the two sensors can be effectively fused, safety can be provided for the automatic driving system.

The existing multi-sensor fusion algorithm has many problems of practical application. For example: 1) A unified fusion theory and an effective generalized fusion model are not established; 2) The research on the specific method of data fusion is still in the preliminary stage; 3) The existing algorithm fails to well solve the problems of fault tolerance and robustness in a fusion system; 4) The ambiguity of the association is a major obstacle in data fusion.

Disclosure of Invention

The invention provides a category determination method, a category determination device, category determination equipment and a storage medium, and aims to solve the problem that the detection result of a camera and the detection result of a laser radar cannot be effectively fused.

According to an aspect of the present invention, there is provided a category determination method, including:

acquiring a projection matrix between at least one camera and at least one laser radar;

controlling each camera to acquire image data of a target object;

controlling each laser radar to obtain point cloud data corresponding to the target object;

and determining the target category information of the target object according to the projection matrix, the image data of the target object and the point cloud data corresponding to the target object.

According to another aspect of the present invention, there is provided a category determination apparatus including:

the first acquisition module is used for acquiring a projection matrix between at least one camera and at least one laser radar;

the first control module is used for controlling each camera to acquire image data of a target object;

the second control module is used for controlling each laser radar to acquire point cloud data corresponding to the target object;

and the determining module is used for determining the target category information of the target object according to the projection matrix, the image data of the target object and the point cloud data corresponding to the target object.

According to another aspect of the present invention, there is provided an electronic apparatus including:

at least one processor; and a memory communicatively coupled to the at least one processor; wherein the memory stores a computer program executable by the at least one processor, the computer program being executable by the at least one processor to enable the at least one processor to perform the method of class determination according to any of the embodiments of the invention.

According to another aspect of the present invention, there is provided a computer-readable storage medium storing computer instructions for causing a processor to implement the category determination method according to any one of the embodiments of the present invention when the computer instructions are executed.

According to the technical scheme, the method and the device for detecting the target object of the laser radar system solve the problem that in the prior art, the detection result of the camera and the detection result of the laser radar cannot be effectively fused, by acquiring the projection matrix between the at least one camera and the at least one laser radar, controlling each camera to acquire the image data of the target object, controlling each laser radar to acquire the point cloud data corresponding to the target object, and determining the target category information of the target object according to the projection matrix, the image data of the target object and the point cloud data corresponding to the target object.

It should be understood that the statements in this section do not necessarily identify key or critical features of the embodiments of the present invention, nor do they necessarily limit the scope of the invention. Other features of the present invention will become apparent from the following description.

Drawings

In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings needed to be used in the description of the embodiments will be briefly introduced below, and it is obvious that the drawings in the following description are only some embodiments of the present invention, and it is obvious for those skilled in the art to obtain other drawings based on these drawings without creative efforts.

Fig. 1 is a flowchart of a category determining method according to an embodiment of the present invention;

fig. 2 is a schematic structural diagram of a category determining apparatus according to a second embodiment of the present invention;

fig. 3 is a schematic structural diagram of an electronic device implementing the category determination method according to the embodiment of the present invention.

Detailed Description

In order to make those skilled in the art better understand the technical solutions of the present invention, the technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments. All other embodiments, which can be obtained by a person skilled in the art without making any creative effort based on the embodiments in the present invention, shall fall within the protection scope of the present invention.

It should be noted that the terms "first," "target," and the like in the description and claims of the present invention and in the above-described drawings are used for distinguishing between similar elements and not necessarily for describing a particular sequential or chronological order. It is to be understood that the data so used is interchangeable under appropriate circumstances such that the embodiments of the invention described herein are capable of operation in other sequences than those illustrated or described herein. Furthermore, the terms "comprises," "comprising," and "having," and any variations thereof, are intended to cover a non-exclusive inclusion, such that a process, method, system, article, or apparatus that comprises a list of steps or elements is not necessarily limited to those steps or elements expressly listed, but may include other steps or elements not expressly listed or inherent to such process, method, article, or apparatus.

Example one

Fig. 1 is a flowchart of a category determining method provided in an embodiment of the present invention, which is applicable to a category determining situation, and the method may be implemented by a category determining apparatus, which may be implemented in a form of hardware and/or software, and the category determining apparatus may be integrated in any electronic device providing a category determining function. As shown in fig. 1, the method includes:

s101, acquiring a projection matrix between at least one camera and at least one laser radar.

In this embodiment, the camera may be at least one camera for acquiring an image of a person or an object in the environment around the vehicle body, and the type and the like of the camera are not limited in this embodiment. The camera may be mounted on the vehicle body, and the mounting position of the camera in the present embodiment is not limited, and for example, one camera may be mounted on each of the front, rear, left, right, and roof of the vehicle body, or one camera may be mounted on each of the four corners of the front, right, left, and right sides of the vehicle body, and the roof of the vehicle body.

In this embodiment, the lidar may be at least one lidar configured to acquire point cloud data of a person or an object in an environment around a vehicle body, and the present embodiment does not limit a type and a category of the lidar. The laser radar may be mounted on the vehicle body, and the mounting position of the laser radar is not limited in this embodiment, and for example, three laser radars may be mounted on the roof of the vehicle body.

It should be explained that the projection matrix can be understood as a conversion matrix of the positional relationship between each camera and each lidar. Illustratively, the projection matrix may be, for example, a 4 × 4 matrix. In this embodiment, the projection matrix may include a rotation matrix and a translation matrix.

Specifically, external reference calibration is carried out on at least one camera, the position coordinate of each camera relative to the center of the rear wheel of the vehicle is calculated, external reference calibration is carried out on at least one laser radar, the position coordinate of each laser radar relative to the center of the rear wheel of the vehicle is calculated, and a projection matrix between each camera and each laser radar is determined according to the position coordinate of each camera relative to the center of the rear wheel of the vehicle and the position coordinate of each laser radar relative to the center of the rear wheel of the vehicle.

And S102, controlling each camera to acquire image data of the target object.

For example, the target object may be a person or an object in the current vehicle body surroundings, and the current vehicle may be a vehicle driven by a user and equipped with at least one camera and at least one lidar.

It should be noted that each camera corresponds to an image coordinate system, and the image data may be data information of an image of the target object captured by the camera in the image coordinate system corresponding to the camera.

Specifically, each camera is controlled to acquire an image of a target object around the vehicle body in real time, and image data of the image of the target object captured by each camera in an image coordinate system corresponding to the camera is acquired.

S103, controlling each laser radar to obtain point cloud data corresponding to the target object.

It is known that point cloud data refers to a set of vectors in a three-dimensional coordinate system. In this embodiment, the point cloud data may be point cloud data corresponding to a target object obtained by scanning the target object around the vehicle body with the laser radar.

Specifically, each laser radar is controlled to acquire point cloud data corresponding to target objects around the vehicle body in real time.

And S104, determining target category information of the target object according to the projection matrix, the image data of the target object and the point cloud data corresponding to the target object.

It should be noted that the target category information may be information of which category the target object specifically belongs to. In this embodiment, the target category information may include 6 types: pedestrians, trucks, cars, vans, riders (motorcycles and bicycles), and buses.

Specifically, the target category information to which the target object specifically belongs is determined according to the projection matrix, the image data of the target object and the point cloud data corresponding to the target object.

Optionally, the point cloud data corresponding to the target object includes: the first point cloud data and the second point cloud data.

In this embodiment, the first point cloud data may be point cloud data acquired by the laser radar when the target object around the vehicle body is an irregular polygonal divided object. Specifically, first point cloud data based on a world coordinate system is generated according to point cloud data of target objects around a vehicle body, which are acquired by a laser radar, based on a laser radar segmentation algorithm.

In this embodiment, the second point cloud data may be detected objects, which are obtained by the laser radar and whose target objects around the vehicle body are 3D bounding boxes, that is, point cloud data corresponding to the detected objects when the target objects are regular objects. Specifically, second point cloud data based on a world coordinate system is generated according to point cloud data of target objects around a vehicle body, which are acquired by a laser radar, based on a laser radar target detection algorithm.

Correspondingly, determining the target category information of the target object according to the projection matrix, the image data of the target object and the point cloud data corresponding to the target object, includes:

and determining first category information of the target object and a first confidence coefficient corresponding to the first category information according to the image data of the target object.

The first category information may be information of which category the target object specifically belongs to, which is determined from image data of the target object acquired by the camera. The first confidence may be an accuracy confidence of information of which category the target object specifically belongs to, which is determined according to the image data of the target object acquired by the camera.

Specifically, the first category information of the target object and the first confidence corresponding to the first category information may be determined according to the image data of the target object based on a target detection algorithm.

And determining first contour information of the target object according to the first point cloud data, the projection matrix and the image data of the target object.

It should be explained that the first contour information may be contour information of the target object when the first point cloud data corresponding to the target object around the vehicle body is a polygonal irregular divided object is projected onto a plane where the image data of the target object is located, and contour information of the target object after data association with the image data of the target object acquired by the camera.

Specifically, the polygonal irregular segmented objects are projected to an image coordinate system corresponding to each camera, that is, first point cloud data corresponding to each polygonal irregular segmented object is multiplied by a projection matrix corresponding to each camera, and the obtained result is subjected to data association with image data of the target object obtained by the cameras, so that first contour information of the target object is finally obtained.

In the actual operation process, after the first contour information of the target object is obtained, fitting is carried out on the contour of the polygonal irregular segmentation object based on an L-Fitting algorithm, namely, a 3D boundary box of the polygonal irregular segmentation object is fitted.

And determining first result type information of the target object according to the first contour information of the target object and the first type information of the target object.

The first result type information may be type information of the target object when the target object around the vehicle body is an irregular divided object of a polygon.

Specifically, when the target object is a polygonal irregular divided object, the first class information of the target object is given to the first contour information of the target object, that is, when the target object around the vehicle body is a polygonal irregular divided object, the first class information of the target object is used as the first result class information of the target object.

And determining second category information of the target object and a second confidence coefficient corresponding to the second category information according to the second point cloud data.

The second category information may be information of which category the target object specifically belongs to, which is determined according to the second point cloud data corresponding to the detected object, which is obtained by the laser radar and around the vehicle body and is the 3D bounding box, that is, when the target object is a regular object. The second confidence may be an accuracy confidence of information of which category the target object specifically belongs to, which is determined by the second point cloud data corresponding to the target object when the target object around the vehicle body is the detected object of the 3D bounding box, that is, the target object is the regular object, acquired by the laser radar.

Specifically, second point cloud data based on a world coordinate system can be generated according to point cloud data of target objects around a vehicle body, which are acquired by a laser radar, based on a laser radar target detection algorithm, and second category information of the target objects and second confidence degrees corresponding to the second category information are determined according to the second point cloud data.

And determining second contour information of the target object according to the second point cloud data, the projection matrix and the image data of the target object.

It should be explained that the second contour information may be contour information of the target object when the target object around the vehicle body is a detection object of the 3D bounding box, that is, when the target object is a regular object, the corresponding second point cloud data is projected onto the plane where the image data of the target object is located, and the contour information of the target object obtained by data association with the image data of the target object obtained by the camera.

Specifically, the detected objects of the 3D bounding box are projected to the image coordinate system corresponding to each camera, that is, the second point cloud data corresponding to the detected objects of each 3D bounding box are multiplied by the projection matrix corresponding to each camera, and the obtained result is subjected to data association with the image data of the target object obtained by the camera, so as to finally obtain the second contour information of the target object.

And determining second result category information of the target object according to the second contour information of the target object, the first category information of the target object, the first confidence degree corresponding to the first category information, the second category information of the target object and the second confidence degree corresponding to the second category information.

The second result type information may be a detected object whose target object is a 3D bounding box around the vehicle body, that is, the type information of the target object when the target object is a regular object.

Specifically, when the first category information of the target object determined according to the image data of the target object is the same as the second category information of the target object determined according to the second point cloud data, the target object is a detected object of the 3D bounding box, that is, when the target object is a regular object, the second result category information of the target object may be the first category information of the target object or the second category information of the target object; and when the first category information of the target object determined according to the image data of the target object is different from the second category information of the target object determined according to the second point cloud data, judging the magnitude of a first confidence coefficient corresponding to the first category information of the target object and a second confidence coefficient corresponding to the second category information of the target object, and taking the category information corresponding to the maximum confidence coefficient as second result category information of the target object.

And determining the target class information of the target object according to the first result class information of the target object and the second result class information of the target object.

Specifically, 3D IOU (Intersection over Union) fitting is performed on first result category information when the target object is an irregular segmented object and second result category information when the target object is a 3D bounding box detection object, that is, the target object is a regular object, and target category information of the target object, that is, a final category information result of the target object is determined, so that the problem of normalization of the target category information of the repeated target object is solved.

Optionally, determining first contour information of the target object according to the first point cloud data, the projection matrix, and the image data of the target object, includes:

first contour data of the target object is determined according to the first point cloud data and the projection matrix.

The first contour data may be contour information data of the target object when the first point cloud data corresponding to the target object around the vehicle body being an irregular divided object having a polygonal shape is projected onto the plane where the image data of the target object is located.

Specifically, the polygonal irregular segmented objects are projected to an image coordinate system corresponding to each camera, that is, the first point cloud data corresponding to each polygonal irregular segmented object is multiplied by the projection matrix corresponding to each camera, and the obtained result is the first contour data of the target object.

In the actual operation process, after the first contour data of the target object is obtained, the values of the data of the contour boundary in the obtained first contour data, which exceed the length and width of the image collected by the camera, are modified, and the values of the data of the length and width of the image collected by the camera are modified, namely the boundary values of the first contour data are limited. For example, the resolution size of the image captured by the camera may be 1920 (width) × 1208 (height), and the width of the resolution size of the irregular segmented object of the polygon in the obtained first contour data may be 1980, which exceeds 1920 which is the width of the resolution of the image captured by the camera, and 1980 is forcibly limited to 1920.

And performing data association on the first contour data of the target object and the image data of the target object to determine first contour information of the target object.

In this embodiment, the data association may be to generate a correspondence between the first contour data of the target object and the image data of the target object.

Specifically, data association is performed on first contour data of each target object when the target object is an irregular segmented object and image data of the target object based on a Hungarian algorithm, and first contour information of the target object is determined.

Optionally, determining second contour information of the target object according to the second point cloud data, the projection matrix, and the image data of the target object, includes:

and determining second contour data of the target object according to the second point cloud data and the projection matrix.

It should be noted that the second contour data may be contour information data of the target object when the target object around the vehicle body is a 3D bounding box detection object, that is, when the target object is a regular object, the corresponding second point cloud data is projected onto the plane where the image data of the target object is located.

Specifically, the 3D bounding box detection objects are projected to the image coordinate system corresponding to each camera, that is, the second point cloud data corresponding to each 3D bounding box detection object is multiplied by the projection matrix corresponding to each camera, and the obtained result is the second contour data of the target object.

In the actual operation process, after the second contour data of the target object is obtained, the value of the data of the contour boundary in the obtained second contour data, which exceeds the length and width of the image collected by the camera, is modified into the value of the data of the length and width of the image collected by the camera, namely the boundary value of the second contour data is limited. For example, the resolution of the image captured by the camera may be 1920 (wide) × 1208 (high), and the resolution of the 3D bounding box detection object in the obtained second contour data may be 1980, which exceeds 1920, i.e., 1920 which is the width of the resolution of the image captured by the camera, so that 1980 is forcibly limited to 1920.

And performing data association on the second contour data of the target object and the image data of the target object to determine second contour information of the target object.

In this embodiment, the data association may be to generate a correspondence between the second contour data of the target object and the image data of the target object.

Specifically, based on the Hungarian algorithm, data association is performed between the second contour data of each target object and the image data of the target object when the target object is a 3D bounding box detection object, that is, the target object is a regular object, and the second contour information of the target object is determined.

Optionally, determining second result category information of the target object according to the second contour information of the target object, the first category information of the target object, the first confidence degree corresponding to the first category information, the second category information of the target object, and the second confidence degree corresponding to the second category information includes:

and if the first category information of the target object is different from the second category information of the target object, determining the category information corresponding to the confidence coefficient with the maximum value in the first confidence coefficient corresponding to the first category information and the second confidence coefficient corresponding to the second category information as the second result category information of the target object.

Specifically, if the first category information of the target object is different from the second category information of the target object, a first confidence corresponding to the first category information is compared with a second confidence corresponding to the second category information, and the category information corresponding to the confidence with the maximum value in the first confidence corresponding to the first category information and the second confidence corresponding to the second category information is determined as the second result category information of the target object.

Optionally, obtaining a projection matrix between the at least one camera and the at least one lidar includes:

coordinate information of each camera relative to the vehicle target position is obtained.

In the present embodiment, the vehicle target position may be any position of the vehicle, and preferably, the vehicle target position may be a center of a rear wheel of the vehicle.

Specifically, coordinate information of all cameras mounted on the vehicle body with respect to the target position of the vehicle is acquired.

Coordinate information of each laser radar relative to the vehicle target position is acquired.

Specifically, coordinate information of all the lidar mounted on the vehicle body relative to the target position of the vehicle is acquired.

And determining a projection matrix between each camera and each laser radar according to the coordinate information of each camera relative to the vehicle target position and the coordinate information of each laser radar relative to the vehicle target position.

Specifically, a conversion matrix of the position relationship between each camera and each lidar, that is, a projection matrix between each camera and each lidar, is determined according to the coordinate information of each camera with respect to the vehicle target position and the coordinate information of each lidar with respect to the vehicle target position.

Optionally, before determining the target category information of the target object according to the projection matrix, the image data of the target object, and the point cloud data corresponding to the target object, the method further includes:

a first timestamp of the image data is acquired.

In this embodiment, the first timestamp may be understood as the time when each camera acquires the image data of the target object, for example, each camera acquires image data 1 time every 1 second, and each camera may acquire image data 10 times within 10 seconds from the current time.

Specifically, a first time stamp of image data of a target object is acquired.

A second timestamp of the point cloud data is obtained.

In this embodiment, the second timestamp may be understood as a time when each lidar acquires point cloud data corresponding to the target object, for example, each lidar may acquire the point cloud data 1 time every 1 second, and each lidar may acquire the point cloud data 10 times within 10 seconds from the current time.

Specifically, a second time stamp of the point cloud data corresponding to the target object is obtained.

Time synchronization is performed according to the first time stamp and the second time stamp.

Specifically, before each camera is controlled to acquire image data of a target object and each lidar is controlled to acquire point cloud data corresponding to the target object, time synchronization of a Global Navigation Satellite System (GNSS) is performed on each camera and each lidar, and then time synchronization is performed according to a first timestamp for acquiring the image data by each camera and a second timestamp for acquiring the point cloud data by each lidar, namely, the image data and the point cloud data acquired at the same time are ensured to be fused.

In the actual operation process, spatial synchronization is required to be performed according to the coordinate information of each camera relative to the vehicle target position and the coordinate information of each laser radar relative to the vehicle target position.

According to the technical scheme of the embodiment of the invention, each camera is controlled to obtain the image data of the target object by obtaining the projection matrix between at least one camera and at least one laser radar, each laser radar is controlled to obtain the point cloud data corresponding to the target object, and the target category information of the target object is determined according to the projection matrix, the image data of the target object and the point cloud data corresponding to the target object, so that the problem that the detection result of the camera and the detection result of the laser radar cannot be effectively fused in the prior art is solved, the detection results of the two sensors, namely the camera and the laser radar, can be effectively fused, and safety guarantee is provided for automatic driving.

Example two

Fig. 2 is a schematic structural diagram of a category determining apparatus according to a second embodiment of the present invention. As shown in fig. 2, the apparatus includes: a first acquisition module 201, a first control module 202, a second control module 203, and a determination module 204.

The first obtaining module 201 is configured to obtain a projection matrix between at least one camera and at least one laser radar;

the first control module 202 is configured to control each camera to acquire image data of a target object;

the second control module 203 is configured to control each laser radar to acquire point cloud data corresponding to the target object;

a determining module 204, configured to determine target category information of the target object according to the projection matrix, the image data of the target object, and the point cloud data corresponding to the target object.

Optionally, the point cloud data corresponding to the target object includes: first point cloud data and second point cloud data;

accordingly, the determining module 204 includes:

the first determining unit is used for determining first category information of the target object and a first confidence coefficient corresponding to the first category information according to image data of the target object;

a second determination unit, configured to determine first contour information of the target object according to the first point cloud data, the projection matrix, and image data of the target object;

a third determining unit, configured to determine first result category information of the target object according to the first contour information of the target object and the first category information of the target object;

a fourth determining unit, configured to determine, according to the second point cloud data, second category information of the target object and a second confidence corresponding to the second category information;

a fifth determining unit, configured to determine second contour information of the target object according to the second point cloud data, the projection matrix, and the image data of the target object;

a sixth determining unit, configured to determine second result category information of the target object according to second contour information of the target object, first category information of the target object, a first confidence degree corresponding to the first category information, second category information of the target object, and a second confidence degree corresponding to the second category information;

a seventh determining unit, configured to determine the target category information of the target object according to the first result category information of the target object and the second result category information of the target object.

Optionally, the second determining unit includes:

a first determining subunit, configured to determine first contour data of the target object according to the first point cloud data and the projection matrix;

and the second determining subunit is used for performing data association on the first contour data of the target object and the image data of the target object to determine the first contour information of the target object.

Optionally, the fifth determining unit includes:

a third determining subunit, configured to determine second contour data of the target object according to the second point cloud data and the projection matrix;

and the fourth determining subunit is configured to perform data association on the second contour data of the target object and the image data of the target object, and determine second contour information of the target object.

Optionally, the sixth determining unit includes:

a fifth determining subunit, configured to determine, if the first category information of the target object is different from the second category information of the target object, the category information corresponding to the confidence level with the largest value among the first confidence level corresponding to the first category information and the second confidence level corresponding to the second category information as the second result category information of the target object.

Optionally, the first obtaining module 201 includes:

the first acquisition unit is used for acquiring coordinate information of each camera relative to a vehicle target position;

the second acquisition unit is used for acquiring coordinate information of each laser radar relative to the vehicle target position;

and the eighth determining unit is used for determining a projection matrix between each camera and each laser radar according to the coordinate information of each camera relative to the vehicle target position and the coordinate information of each laser radar relative to the vehicle target position.

Optionally, the category determining apparatus further includes:

the second acquisition module is used for acquiring a first time stamp of image data before determining the target category information of the target object according to the projection matrix, the image data of the target object and the point cloud data corresponding to the target object;

the third acquisition module is used for acquiring a second time stamp of the point cloud data before determining the target category information of the target object according to the projection matrix, the image data of the target object and the point cloud data corresponding to the target object;

and the time synchronization module is used for performing time synchronization according to the first time stamp and the second time stamp before determining the target category information of the target object according to the projection matrix, the image data of the target object and the point cloud data corresponding to the target object.

The category determining device provided by the embodiment of the invention can execute the category determining method provided by any embodiment of the invention, and has corresponding functional modules and beneficial effects of the executing method.

EXAMPLE III

FIG. 3 shows a schematic block diagram of an electronic device 30 that may be used to implement an embodiment of the invention. Electronic devices are intended to represent various forms of digital computers, such as laptops, desktops, workstations, personal digital assistants, servers, blade servers, mainframes, and other appropriate computers. The electronic device may also represent various forms of mobile devices, such as personal digital assistants, cellular phones, smart phones, wearable devices (e.g., helmets, glasses, watches, etc.), and other similar computing devices. The components shown herein, their connections and relationships, and their functions, are meant to be exemplary only, and are not meant to limit implementations of the inventions described and/or claimed herein.

As shown in fig. 3, the electronic device 30 includes at least one processor 31, and a memory communicatively connected to the at least one processor 31, such as a Read Only Memory (ROM) 32, a Random Access Memory (RAM) 33, and the like, wherein the memory stores a computer program executable by the at least one processor, and the processor 31 may perform various suitable actions and processes according to the computer program stored in the Read Only Memory (ROM) 32 or the computer program loaded from the storage unit 38 into the Random Access Memory (RAM) 33. In the RAM 33, various programs and data necessary for the operation of the electronic apparatus 30 can also be stored. The processor 31, the ROM 32, and the RAM 33 are connected to each other via a bus 34. An input/output (I/O) interface 35 is also connected to bus 34.

A plurality of components in the electronic device 30 are connected to the I/O interface 35, including: an input unit 36 such as a keyboard, a mouse, etc.; an output unit 37 such as various types of displays, speakers, and the like; a storage unit 38 such as a magnetic disk, an optical disk, or the like; and a communication unit 39 such as a network card, modem, wireless communication transceiver, etc. The communication unit 39 allows the electronic device 30 to exchange information/data with other devices via a computer network such as the internet and/or various telecommunication networks.

The processor 31 may be a variety of general and/or special purpose processing components having processing and computing capabilities. Some examples of processor 31 include, but are not limited to, a Central Processing Unit (CPU), a Graphics Processing Unit (GPU), various specialized Artificial Intelligence (AI) computing chips, various processors running machine learning model algorithms, a Digital Signal Processor (DSP), and any suitable processor, controller, microcontroller, or the like. The processor 31 performs the various methods and processes described above, such as the category determination method:

controlling each camera to acquire image data of a target object;

In some embodiments, the category determination method may be implemented as a computer program tangibly embodied in a computer-readable storage medium, such as storage unit 38. In some embodiments, part or all of the computer program may be loaded and/or installed onto the electronic device 30 via the ROM 32 and/or the communication unit 39. When the computer program is loaded into the RAM 33 and executed by the processor 31, one or more steps of the category determination method described above may be performed. Alternatively, in other embodiments, the processor 31 may be configured to perform the category determination method by any other suitable means (e.g., by means of firmware).

Various implementations of the systems and techniques described here above may be implemented in digital electronic circuitry, integrated circuitry, field Programmable Gate Arrays (FPGAs), application Specific Integrated Circuits (ASICs), application Specific Standard Products (ASSPs), system on a chip (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and/or combinations thereof. These various embodiments may include: implemented in one or more computer programs that are executable and/or interpretable on a programmable system including at least one programmable processor, which may be special or general purpose, receiving data and instructions from, and transmitting data and instructions to, a storage system, at least one input device, and at least one output device.

Computer programs for implementing the methods of the present invention can be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general purpose computer, special purpose computer, or other programmable data processing apparatus, such that the computer programs, when executed by the processor, cause the functions/acts specified in the flowchart and/or block diagram block or blocks to be performed. A computer program can execute entirely on a machine, partly on a machine, as a stand-alone software package partly on a machine and partly on a remote machine or entirely on a remote machine or server.

In the context of the present invention, a computer-readable storage medium may be a tangible medium that can contain, or store a computer program for use by or in connection with an instruction execution system, apparatus, or device. A computer readable storage medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. Alternatively, the computer readable storage medium may be a machine readable signal medium. More specific examples of a machine-readable storage medium would include an electrical connection based on one or more wires, a portable computer diskette, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

To provide for interaction with a user, the systems and techniques described here can be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to a user; and a keyboard and a pointing device (e.g., a mouse or a trackball) by which a user can provide input to the electronic device. Other kinds of devices may also be used to provide for interaction with a user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user may be received in any form, including acoustic, speech, or tactile input.

The systems and techniques described here can be implemented in a computing system that includes a back-end component (e.g., as a data server), or that includes a middleware component (e.g., an application server), or that includes a front-end component (e.g., a user computer having a graphical user interface or a web browser through which a user can interact with an implementation of the systems and techniques described here), or any combination of such back-end, middleware, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: local Area Networks (LANs), wide Area Networks (WANs), blockchain networks, and the Internet.

The computing system may include clients and servers. A client and server are generally remote from each other and typically interact through a communication network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other. The server can be a cloud server, also called a cloud computing server or a cloud host, and is a host product in a cloud computing service system, so that the defects of high management difficulty and weak service expansibility in the traditional physical host and VPS service are overcome.

It should be understood that various forms of the flows shown above may be used, with steps reordered, added, or deleted. For example, the steps described in the present invention may be executed in parallel, sequentially, or in different orders, and are not limited herein as long as the desired results of the technical solution of the present invention can be achieved.

The above-described embodiments should not be construed as limiting the scope of the invention. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions may be made in accordance with design requirements and other factors. Any modification, equivalent replacement, and improvement made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A method for class determination, comprising:

controlling each camera to acquire image data of a target object;

2. The method of claim 1, wherein the point cloud data corresponding to the target object comprises: first point cloud data and second point cloud data;

correspondingly, determining the target category information of the target object according to the projection matrix, the image data of the target object and the point cloud data corresponding to the target object includes:

determining first category information of the target object and a first confidence coefficient corresponding to the first category information according to the image data of the target object;

determining first contour information of the target object according to the first point cloud data, the projection matrix and the image data of the target object;

determining first result type information of the target object according to the first outline information of the target object and the first type information of the target object;

determining second category information of the target object and a second confidence coefficient corresponding to the second category information according to the second point cloud data;

determining second contour information of the target object according to the second point cloud data, the projection matrix and the image data of the target object;

determining second result category information of the target object according to second contour information of the target object, first category information of the target object, a first confidence degree corresponding to the first category information, second category information of the target object and a second confidence degree corresponding to the second category information;

3. The method of claim 2, wherein determining first contour information of the target object from the first point cloud data, the projection matrix, and image data of the target object comprises:

determining first contour data of the target object according to the first point cloud data and the projection matrix;

4. The method of claim 2, wherein determining second contour information for the target object from the second point cloud data, the projection matrix, and the image data of the target object comprises:

determining second contour data of the target object according to the second point cloud data and the projection matrix;

5. The method of claim 2, wherein determining the second result category information of the target object according to the second contour information of the target object, the first category information of the target object, the first confidence degree corresponding to the first category information, the second category information of the target object, and the second confidence degree corresponding to the second category information comprises:

if the first category information of the target object is different from the second category information of the target object, determining the category information corresponding to the confidence coefficient with the maximum value in the first confidence coefficient corresponding to the first category information and the second confidence coefficient corresponding to the second category information as the second result category information of the target object.

6. The method of claim 1, wherein obtaining a projection matrix between at least one camera and at least one lidar comprises:

acquiring coordinate information of each camera relative to a vehicle target position;

acquiring coordinate information of each laser radar relative to a vehicle target position;

7. The method of claim 1, further comprising, prior to determining target category information for the target object from the projection matrix, image data for the target object, and point cloud data corresponding to the target object:

acquiring a first time stamp of image data;

acquiring a second time stamp of the point cloud data;

and performing time synchronization according to the first time stamp and the second time stamp.

8. A category determination device, comprising:

the system comprises a first acquisition module, a second acquisition module and a third acquisition module, wherein the first acquisition module is used for acquiring a projection matrix between at least one camera and at least one laser radar;

9. An electronic device, characterized in that the electronic device comprises:

at least one processor; and

a memory communicatively coupled to the at least one processor; wherein the content of the first and second substances,

the memory stores a computer program executable by the at least one processor, the computer program being executable by the at least one processor to enable the at least one processor to perform the category determination method of any one of claims 1-7.

10. A computer-readable storage medium having stored thereon computer instructions for causing a processor to perform the method of determining a category according to any one of claims 1-7 when executed.