CN113362383A

CN113362383A - Point cloud and image fusion method and device

Info

Publication number: CN113362383A
Application number: CN202010136592.5A
Authority: CN
Inventors: 高文; 崔金林; 王博; 徐峣; 郝志成
Original assignee: Huawei Technologies Co Ltd; Changchun Institute of Optics Fine Mechanics and Physics of CAS
Current assignee: Huawei Technologies Co Ltd; Changchun Institute of Optics Fine Mechanics and Physics of CAS
Priority date: 2020-03-02
Filing date: 2020-03-02
Publication date: 2021-09-07

Abstract

The application discloses a method and a device for fusing point cloud and an image, which are beneficial to improving the accuracy of the obtained fused point cloud. The method for fusing the point cloud and the image is applied to computer equipment and comprises the following steps: a first point cloud is obtained. Wherein the first point cloud is a point cloud that is used to describe a first scene in the autonomous driving environment. The first point cloud comprises a first sub-point cloud and a second sub-point cloud, and the first sub-point cloud and the second sub-point cloud are respectively used for describing a first sub-scene and a second sub-scene in the first scene. Acquiring a first image; wherein the first image is an image for describing a first scene. The first image is divided into a plurality of sub-images. The plurality of sub-images comprise a first sub-image and a second sub-image, and the first sub-image and the second sub-image are used for describing a first sub-scene and a second sub-scene respectively. Dividing each sub-image of the plurality of sub-images into a plurality of image blocks, and fusing the first point cloud and the first image based on the plurality of image blocks.

Description

Point cloud and image fusion method and device

Technical Field

The application relates to the technical field of data fusion, in particular to a method and a device for fusing point cloud and an image.

Background

In order to meet the requirement of sensing the environment for automatic driving of automobiles, environment detection by combining a lidar (laser light detection and ranging) and a camera has become one of the main forms of detecting the environment at present. As shown in fig. 1, an environment detection system including a camera, a laser radar, and a computer device obtains an image and a point cloud (point cloud) from the camera for a same scene (e.g., any scene in a natural environment, including a tree, a power pole, a street lamp, a traffic sign, a lane line, etc.), where the point cloud is composed of a plurality of scanning points. A computer device receives the image and the point cloud and divides the image into a plurality of image blocks of the same size. And the computer equipment fuses the scanning points with the corresponding relation with the image blocks to obtain fused point cloud. The fused point cloud is used for navigation and obstacle avoidance of an automatic driving automobile.

At present, a computer device divides an image into a plurality of image blocks with the same size based on the whole image, so that matching errors and/or ghosts of fused point clouds are easily caused, even matching errors occur, the accuracy of the fused point clouds obtained by the computer device is poor, and the navigation and obstacle avoidance of an automatic driving automobile are not facilitated.

Disclosure of Invention

The embodiment of the application provides a method and a device for fusing point clouds and images, which can reduce matching errors and/or ghost images generated by fused point clouds, thereby improving the accuracy of the obtained fused point clouds.

In order to achieve the above purpose, the embodiment of the present application adopts the following technical solutions:

in a first aspect, a method for fusing a point cloud and an image is provided, which is applied to a computer device, and includes: a first point cloud is obtained. Wherein the first point cloud is a point cloud that is used to describe a first scene in the autonomous driving environment. The first point cloud comprises a first sub-point cloud and a second sub-point cloud, the first sub-point cloud and the second sub-point cloud are respectively used for describing a first sub-scene and a second sub-scene in a first scene, and the first sub-scene and the second sub-scene are respectively sub-scenes under a first view field partition and a second view field partition of the point cloud acquisition equipment. A first image is acquired. Wherein the first image is an image for describing a first scene. The first image is divided into a plurality of sub-images. The plurality of sub-images comprise a first sub-image and a second sub-image, and the first sub-image and the second sub-image are used for describing a first sub-scene and a second sub-scene respectively. Each of the plurality of sub-images is divided into a plurality of image blocks. And fusing the first point cloud and the first image based on the plurality of image blocks. Therefore, the computer device divides the first point cloud area into a plurality of sub-point clouds based on the view field area of the point cloud acquisition device, divides the first image into a plurality of sub-images based on the view field area, divides each sub-image into a plurality of image blocks, and does not include pixel points in two sub-images in one of the divided image blocks, so that the problem of fusion data ghost caused by the fact that one image block includes the pixel points in the two sub-images is avoided, the accuracy of the obtained fusion point clouds is improved, and the navigation and obstacle avoidance of an automatic driving automobile are facilitated.

In one possible implementation, dividing each of the plurality of sub-images into a plurality of image blocks includes: the first sub-image is divided into a plurality of first image blocks. Wherein, the sizes of the different first image blocks are the same. The second sub-image is divided into a plurality of second image blocks. Wherein, the sizes of the different second image blocks are the same.

In another possible implementation, dividing the first sub-image into a plurality of first image blocks includes: and acquiring a target interval, wherein the target interval is an interval where the density of the registration points of the first sub-image is located. The registration points are pixel points in the first image that have corresponding scan points in the first point cloud. The first pixel point in the first image corresponds to the first scanning point in the first point cloud, which means that the two-dimensional coordinate of the first scanning point in the coordinate system of the first image is the same as the two-dimensional coordinate of the first pixel point in the coordinate system of the first image. And acquiring the size of an image block corresponding to the target interval according to the corresponding relation between each interval in the plurality of intervals and the size of the image block, and taking the acquired size of the image block as the size of the first image block. The first sub-image is divided into a plurality of first image blocks based on the first image block size. Thus, for two sub-images, the target intervals in which the registration point densities of the two sub-images acquired by the computer device are different, that is, the registration point densities of the two sub-images are different, the sizes of the image blocks acquired from the correspondence between each of the intervals and the sizes of the image blocks according to the target intervals are different, the sizes of the image blocks in the two sub-images obtained by dividing the sub-images based on the acquired sizes of the image blocks are different, and the sizes of the image blocks in the same sub-image are the same. And the corresponding relation between each interval in the plurality of intervals and the size of the image block can be preset in advance according to experience, so that the computer equipment can select the proper size of the image block according to the density of the registration points so as to solve the problem of poor accuracy of the fused point cloud.

In another possible implementation, dividing the first sub-image into a plurality of first image blocks includes: a first graph is acquired, the first graph being determined based on the registration points in the first sub-image. And determining the number of the first image blocks based on the similarity degree of the first image and the target image. The first sub-image is divided into a plurality of first image blocks based on the number of the first image blocks. In this way, the computer device acquires the degree of similarity between the first pattern and the target pattern, and acquires the number of divided image blocks according to the acquired degree of similarity. The lower the similarity degree is, the more uneven the distribution of the registration points is, if the distribution of the registration points is uneven, the density of the registration points in some areas in the first sub-image is high, and the density of the registration points in some areas is low, so that the number of the first image blocks divided by the first sub-image needs to be increased to ensure that the number of the registration points in each first image block in the divided first image blocks is small. Therefore, the computer equipment determines the number of the first image blocks divided in the first sub-image according to the similarity degree of the first image and the target image, so that the size of the first image blocks is determined, the computer equipment selects the proper size of the image blocks, and the problem of poor accuracy of the fused point cloud is solved.

In another possible implementation, dividing each of the plurality of sub-images into a plurality of image blocks includes: a target object in the first sub-image is extracted, and the first sub-image is divided into a plurality of first image blocks based on the extracted target object. For example, the target object in the first sub-image includes an image of a pedestrian and an image of a vehicle, if the first sub-image is not divided according to the target object, there may exist pixel points in the same image block that includes both the image of the pedestrian and the image of the vehicle in the image block obtained by dividing the first sub-image, which is very likely to cause ghost images or matching errors of the acquired fusion point cloud, and if the first sub-image is divided based on the target object, it is avoided that the pixel points in different target objects belong to the same image block, so that the accuracy of the acquired fusion point cloud is improved.

In another possible implementation, dividing each of the plurality of sub-images into a plurality of image blocks includes: and taking each registration point in the first sub-image as a pixel point in one first image block. The non-registration points are assigned to the first image block to which one of the registration points belongs based on a distance between the non-registration point and each registration point in the first sub-image. In this way, when the sub-image is divided, based on one registration point in the sub-image, a non-registration point closest to the registration point may be allocated to the registration point, and a minimum outer-wrapped rectangular region of a region formed by the non-registration point and the registration point is used as an image block, so as to limit the number of registration points included in each image block, and when the number of registration points included in the image block determines different registration point densities, the image blocks are different in size. For example: when the density of the registration points is larger, a smaller image block can comprise 5 registration points, and when the density of the registration points is smaller, a larger image block can comprise 5 registration points, so that the subimage is divided into image blocks with proper sizes, and the accuracy of the acquired fusion point cloud is improved.

In another possible implementation manner, the method further includes: acquiring a plurality of groups of first partition parameters based on the target point cloud; wherein the target point cloud is a point cloud that is used to describe a second scene in the autonomous driving environment. The second scene is one scene before the first scene. Different sets of first partition parameters are obtained based on different partition modes. And acquiring the weight of each group of first partition parameters in the plurality of groups of first partition parameters. And acquiring second partition parameters based on the multiple groups of first partition parameters and the weight of each group of first partition parameters. And dividing the field of view of the point cloud acquisition equipment into a plurality of field of view zones by using the second zone parameters. Wherein the plurality of field of view partitions includes a first field of view partition and a second field of view partition. Therefore, the division of the view field of the point cloud acquisition equipment integrates multiple partition modes, so that the division of the view field of the point cloud acquisition equipment is more reasonable. The total dotting quantity is certain when the point cloud acquisition equipment acquires a frame of point cloud. The point cloud acquisition equipment can scan in different field of view subareas with different scanning densities, and when the scanning density in one field of view subarea is increased, the scanning densities of the rest field of view subareas are necessarily reduced to a certain extent. Therefore, the reasonable field of view partition of the point cloud acquisition equipment is beneficial to improving the information quantity of the scanning points acquired by the point cloud acquisition equipment. For example, if many points are hit on the same object under a field of view, the information of the multiple scanning points in the acquired point cloud is indicative of the object, which results in the waste of the scanning points. If different points are hit on different objects as much as possible under one field of view zone, the information of a plurality of scanning points in the acquired point cloud represents different objects, thereby increasing the information amount of the acquired scanning points.

In another implementation, the method further comprises: a resolution of the first image is acquired. And if the resolution of the first image is greater than the first threshold, adjusting at least one of the scanning density of the first field of view partition and the scanning density of the second field of view partition based on any one of the fused point cloud, the first point cloud or the first image. The fused point cloud is obtained by fusing the first image and the first point cloud. And if the resolution of the first image is less than or equal to the first threshold, adjusting at least one of the scanning density of the first field of view partition and the scanning density of the second field of view partition based on the first point cloud or the fused point cloud. Therefore, the problem that the judgment basis is inaccurate due to the fact that the first image cannot accurately describe the first scene and the scanning density of each field of view partition in the laser radar field of view is adjusted based on the first image can be avoided, and the scanning density of each field of view partition is adjusted.

In a second aspect, a method of partitioning a field of view of a point cloud acquisition device is provided, the method comprising: and acquiring the multiple groups of first partition parameters and the weight of each group of first partition parameters in the multiple groups of first partition parameters. The first partition parameter is used for partitioning the field of view of the point cloud acquisition equipment. Different sets of first partition parameters are obtained based on different partition modes. And acquiring second partition parameters based on the multiple groups of first partition parameters and the weight of each group of first partition parameters. And dividing the field of view of the point cloud acquisition equipment into a plurality of field of view zones by using the second zone parameters. Therefore, the division of the view field of the point cloud acquisition equipment integrates multiple partition modes, so that the division of the view field of the point cloud acquisition equipment is more reasonable. The total dotting quantity is certain when the point cloud acquisition equipment acquires a frame of point cloud. The point cloud acquisition equipment can scan in different field of view subareas with different scanning densities, and when the scanning density in one field of view subarea is increased, the scanning densities of the rest field of view subareas are necessarily reduced to a certain extent. Therefore, the reasonable field of view partition of the point cloud acquisition equipment is beneficial to improving the information quantity of the scanning points acquired by the point cloud acquisition equipment. For example, if many points are hit on the same object under a field of view, the information of the multiple scanning points in the acquired point cloud is indicative of the object, which results in the waste of the scanning points. If different points are hit on different objects as much as possible under one field of view zone, the information of a plurality of scanning points in the acquired point cloud represents different objects, thereby increasing the information amount of the acquired scanning points.

In one possible implementation, the partition mode includes any one of the following: a zoning mode of zoning based on view field safety proportion; a partition mode based on scanning point density partition; and partitioning mode based on the similarity of the target graph.

In a third aspect, a method for adjusting scan density is provided, the method comprising: a resolution of the first image is acquired. And if the resolution of the first image is greater than the first threshold, adjusting at least one of the scanning density of the first sub-point cloud and the scanning density of the second sub-point cloud based on any one of the fused point cloud, the first image or the first point cloud. The fused point cloud is obtained by fusing the first image and the first point cloud. And if the resolution of the first image is less than or equal to the first threshold, adjusting at least one of the scanning density of the first sub-point cloud and the scanning density of the second sub-point cloud based on the first point cloud or the fused point cloud. Therefore, the problem that the judgment basis is inaccurate due to the fact that the first image cannot accurately describe the first scene and the scanning density of each field of view partition in the laser radar field of view is adjusted based on the first image can be avoided, and the scanning density of each field of view partition is adjusted.

In one possible implementation, adjusting the scan density of the first sub-point cloud includes: based on the sub-fused point clouds, the scan density of the first sub-point cloud is adjusted. The sub-fusion point cloud is obtained by fusing the first sub-point cloud and the first sub-image.

In a fourth aspect, a device for fusing a point cloud and an image is provided, which may be used to perform any of the methods provided in any of the possible implementations of the first to third aspects. For example, the fusion device may be a computer device (such as a terminal device, a server, or a cloud server), a chip, or the like.

In a first possible implementation manner of the ninth aspect, the fusion device may be divided into functional modules according to any one of the methods provided in the first to third aspects. For example, each functional unit may be divided for each function, or two or more functions may be integrated into one processing unit.

In a second possible implementation manner of the ninth aspect, the apparatus may include a processor configured to perform any one of the methods provided in the first to third aspects.

In a fifth aspect, a computer-readable storage medium, such as a computer-non-transitory readable storage medium, is provided. Having stored thereon a computer program (or instructions) which, when run on a computer, causes the computer to perform any of the methods provided by any of the possible implementations of the first to third aspects described above.

A sixth aspect provides a computer program product enabling any of the methods provided in any of the possible implementations of the first to third aspects to be performed when the computer program product runs on a computer.

In a seventh aspect, a chip is provided, which includes: and the processor is used for calling and running the computer program stored in the memory from the memory and executing any method provided by any possible implementation manner of the first aspect to the third aspect.

In an eighth aspect, there is provided a chip comprising: and the processor is used for calling and running the computer program stored in the memory from the memory and executing any method provided by any possible implementation manner of the first aspect to the third aspect.

It is understood that any of the fusion apparatus, the computer storage medium, the computer program product or the chip provided above can be applied to the corresponding method provided above, and therefore, the beneficial effects achieved by the fusion apparatus can refer to the beneficial effects in the corresponding method, and are not described herein again.

Drawings

FIG. 1 is a schematic diagram of a currently acquired fused point cloud;

FIG. 2 is a schematic diagram of an environment detection system suitable for use in embodiments of the present application;

fig. 3 is a schematic structural diagram of a computer device to which the technical solution provided by the embodiment of the present application is applied;

fig. 4 is a schematic flowchart of a method for fusing a point cloud and an image according to an embodiment of the present disclosure;

FIG. 5 is an exemplary diagram of a first point cloud;

FIG. 6 is an exemplary diagram of a first image;

fig. 7 is a schematic flowchart of a method for dividing a lidar field of view according to an embodiment of the present disclosure;

FIG. 8 is a diagram illustrating the result of the computer device partitioning the target point cloud according to the first partition parameter;

FIG. 9 is a schematic flowchart of a method for adjusting scan densities of different field sections in a lidar field of view according to an embodiment of the present disclosure;

fig. 10 is a schematic structural diagram of a device for fusing a point cloud and an image according to an embodiment of the present application.

Detailed Description

Fig. 2 is a schematic structural diagram of an environment detection system applicable to the embodiment of the present application, and the environment detection system includes a computer device 10, a point cloud collection device 11, and a camera 12. It should be noted that, in the embodiment of the present application, the number of the computer devices 10, the point cloud acquisition devices 11, and the cameras 12 in the environment detection system is not limited, and fig. 2 illustrates that the environment detection system includes one computer device 10, one point cloud acquisition device 11, and one camera 12. In the environment detection system, a computer device 10 is connected to a point cloud collection device 11 and a camera 12, respectively.

The computer device 10 is configured to send an instruction to the point cloud collecting device 11, so that the point cloud collecting device 11 partitions the point cloud collecting device field of view according to the instruction, and the point cloud collecting device 11 scans in different field partitions at different scanning densities to obtain a point cloud. The computer device 10 is also configured to send instructions to the camera 12 to instruct the camera 12 to acquire images. The computer device 10 receives the point cloud sent by the point cloud acquisition device 11 and the image sent by the camera 12, and fuses the point cloud and the image to obtain a fused point cloud.

The point cloud acquisition equipment 11 is used for scanning scenes in a field of view of the point cloud acquisition equipment to acquire point clouds.

A camera 12 for capturing a scene within the camera field of view to acquire an image.

Fig. 3 is a schematic structural diagram of a computer device to which the technical solution provided in the embodiment of the present application is applied. In one example, computer device 10 in FIG. 2 may be computer device 10 in FIG. 3 from a hardware configuration perspective. The computer device 10 shown in fig. 3 may include at least one processor 101, a communication line 102, a memory 103, and at least one communication interface 104.

The processor 101 may be a general-purpose Central Processing Unit (CPU), a microprocessor, an application-specific integrated circuit (ASIC), or one or more ics for controlling the execution of programs in accordance with the present disclosure.

The communication link 102 may include at least one path, such as a data bus, and/or a control bus, for communicating information between the aforementioned components (e.g., the at least one processor 101, the communication link 102, the memory 103, and the at least one communication interface 104).

The communication interface 104 may be any device, such as a transceiver, for communicating with other devices or communication networks, such as Wide Area Networks (WAN), Local Area Networks (LAN), and the like.

The memory 103 may be, but is not limited to, a read-only memory (ROM) or other type of static storage device that can store static information and instructions, a Random Access Memory (RAM) or other type of dynamic storage device that can store information and instructions, an electrically erasable programmable read-only memory (EEPROM), a compact disc read-only memory (CD-ROM) or other optical disk storage, optical disk storage (including compact disc, laser disc, optical disc, digital versatile disc, blu-ray disc, etc.), magnetic disk storage media or other magnetic storage devices, or any other medium that can be used to carry or store desired program code in the form of instructions or data structures and that can be accessed by a computer. The memory 103 may be separate and coupled to the processor 101 via a communication link 102. The memory 103 may also be integrated with the processor 101. The memory 103 provided by the embodiments of the present application generally includes a nonvolatile memory. The memory 103 is used for storing computer instructions for executing the scheme of the application, and is controlled by the processor 101 to execute. The processor 101 is configured to execute computer instructions stored in the memory 103, thereby implementing the methods provided by the embodiments described below in the present application.

The storage 103 includes a memory and a hard disk.

Optionally, the computer instructions in the embodiments of the present application may also be referred to as application program code or system, which is not specifically limited in the embodiments of the present application.

In one embodiment, the computer device 10 may include a plurality of processors, and each of the processors may be a single-core (single-CPU) processor or a multi-core (multi-CPU) processor. A processor herein may refer to one or more devices, circuits, and/or processing cores for processing data (e.g., computer program instructions).

In particular implementations, computer device 10 may also include an output device 105 and/or an input device 106, as one embodiment. The output device 105 is in communication with the processor 101 and may display information in a variety of ways. For example, the output device 105 may be a Liquid Crystal Display (LCD), a Light Emitting Diode (LED) display device, a Cathode Ray Tube (CRT) display device, a projector (projector), or the like. The input device 106 is in communication with the processor 101 and may receive user input in a variety of ways. For example, the input device 106 may be a mouse, a keyboard, a touch screen device, or a sensing device, among others.

It should be noted that the computer device shown in fig. 3 is only an example, and does not limit the computer device to which the embodiments of the present application are applicable. In actual implementation, the computer device may include more or fewer devices or components than those shown in FIG. 3.

In the following, some terms referred to in the present application are explained:

1) scene, scene

A scene refers to an environment within a particular temporal and spatial range. The environment includes: road conditions, areas, weather, lighting, traffic participants, and the like. Wherein, the road conditions include: intersections (such as crossroads, T-shaped intersections or Y-shaped intersections) and road types (such as expressways, ordinary roads or mountain roads). The area includes: urban areas, villages and towns, mountainous areas and the like. The weather includes: rain, snow, fog, and the like. The illumination includes: day and night, etc. The traffic participant comprises: moving objects (such as vehicles, people, animals and the like) and stationary objects (such as trees, telegraph poles, street lamps, traffic signs, lane lines, stones, trash cans, buildings, mountains and the like).

2) Point cloud, point cloud collection device, laser radar, scanning point, scanning density, scanning point density

The point cloud is a collection of scanning points obtained by a point cloud acquisition device after acquiring a plurality of scanning points on the surface of an object, and one scanning point has a three-dimensional coordinate, an angle, laser reflection intensity and the like.

The point cloud collection device includes: three-dimensional laser scanners (e.g., lidar such as point-scan lidar, line-scan lidar, or area-array lidar, etc.), photographic scanners, and the like. For convenience of explanation, the point cloud collecting device is exemplified as a lidar hereinafter.

Lidar is a range finding system. The distance measuring system comprises a light source, a detector, a timer, a processor and the like. The method for acquiring a scanning point by the laser radar comprises the following steps: the light source (e.g., laser) emits a beam of light (e.g., laser light) toward the object to be measured, and the detector receives the light reflected by the object to be measured. The timer (such as a system clock) calculates the difference value between the time point of the light emitted by the light source and the time point of the reflected light received by the detector, and the processor determines the distance between the target to be measured and the device where the distance measuring system is located according to the difference value and the light speed, so as to obtain a scanning point. The attribute of the scanning point comprises a three-dimensional coordinate of the scanning point in the field of view of the laser radar (the three-dimensional coordinate of the scanning point is for short), the three-dimensional coordinate of the scanning point comprises the depth of the scanning point, and the depth of the scanning point is the distance between the target to be detected and the device where the ranging system is located. The laser radar scans the field of view of the laser radar by the method to obtain a plurality of scanning points in the field of view of the laser radar, and a set formed by the plurality of scanning points is a point cloud.

The scanning density is the number of scanning points acquired in a unit area under an ideal state of the laser radar. The ideal state refers to that the number of scanning points acquired by the laser radar in a unit area is the same as the number of dotting points in the unit area.

And the scanning point density is the number of scanning points in a unit area in the point cloud.

3) Camera and image display device

The camera may be any camera used to capture images. The cameras in the embodiments of the present application include a Red Green Blue (RGB) camera, a grayscale camera, and the like.

The color camera uses a three-primary color mode, which is a color standard in the industry, and obtains color modes of multiple colors by changing three color channels of red, green and blue and superimposing the color channels on each other. RGB represents three channel colors of red, green and blue, where R represents red, G represents green and B represents blue. The color mode includes all colors that human vision can perceive, and is one of the most widely used color modes at present.

4) Visual field, laser radar visual field and camera visual field

The field of view, usually expressed in degrees, is used to indicate range. Generally, the larger the field of view, the larger the range. The field of view may be 50 degrees, 58 degrees, 52 degrees, 180 degrees, 360 degrees, or the like.

The field of view of the lidar refers to the range that the lidar can detect. The embodiment of the application does not limit the size of the field of view of the laser radar.

The camera field of view refers to the range that the camera can capture. The size of the camera field of view is not limited in the embodiments of the present application.

5) Object point, image point, same name point and characteristic point

One point in the scene is called an object point.

A point in the image is called an image point. An image point in the image of the scene corresponds to an object point in the scene.

The corresponding point is an image point (e.g., a scanning point in the first point cloud and a pixel point in the first image) of an object point in the scene in different images (e.g., the first point cloud and the first image).

The characteristic point is a stable pixel point or scanning point in the image, which can not change in rotation and overcome gray inversion.

6) Target object

The target object is an image in the image that characterizes a person or object in the scene. Such as images of a person, an automobile, a road sign, a bicycle, a dog, etc.

7) Other terms

In the embodiments of the present application, words such as "exemplary" or "for example" are used to mean serving as an example, instance, or illustration. Any embodiment or design described herein as "exemplary" or "e.g.," is not necessarily to be construed as preferred or advantageous over other embodiments or designs. Rather, use of the word "exemplary" or "such as" is intended to present concepts related in a concrete fashion.

In the embodiments of the present application, "at least one" means one or more. "plurality" means two or more.

In the embodiment of the present application, "and/or" is only one kind of association relationship describing an association object, and indicates that three relationships may exist, for example, a and/or B may indicate: a exists alone, A and B exist simultaneously, and B exists alone. In addition, the character "/" herein generally indicates that the former and latter related objects are in an "or" relationship.

The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application, and it is obvious that the described embodiments are only a part of the embodiments of the present application, and not all of the embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present application.

Fig. 4 is a schematic flow chart of a method for fusing a point cloud and an image according to an embodiment of the present disclosure. For example, the present embodiment may be applied to the environment detection system shown in fig. 2, and the point cloud collection device in fig. 2 is taken as a laser radar for illustration. The method shown in fig. 4 may comprise the steps of:

s100: the laser radar acquires a first point cloud. The first point cloud is a point cloud of a first scene in the autonomous driving environment. Wherein the first point cloud comprises a first sub-point cloud and a second sub-point cloud. The first sub-point cloud is used for describing a first sub-scene in the first scene. The second sub-point cloud is used for describing a second sub-scene in the first scene.

Specifically, a laser radar scans a first scene in an automatic driving environment to obtain a first point cloud. The first scene is divided into a plurality of different sub-scenes according to different view field partitions, and the view field partitions correspond to the sub-scenes one to one. The plurality of different sub-scenes includes a first sub-scene and a second sub-scene. And scanning the first sub-scene by the laser radar to obtain a first sub-point cloud, and scanning the second sub-scene to obtain a second sub-point cloud. The method for dividing the laser radar into the view fields is not limited in the embodiment of the application.

Illustratively, a first point cloud is shown in FIG. 5.

S101: the laser radar sends the first point cloud to the computer device.

S102: the camera acquires a first image, which is an image of a first scene, i.e., an image of the first scene captured by the camera.

Illustratively, the first image is shown in FIG. 6.

S103: the camera sends a first image to the computer device.

In the embodiment of the present application, the execution order of S100 to S101 and S102 to S103 is not limited. Illustratively, S102-S103 are executed, and S100-S101 are executed.

S104: the computer device divides the first image into a plurality of sub-images. The plurality of sub-images includes a first sub-image and a second sub-image. The first sub-picture is used to describe the first sub-scene and the second sub-picture is used to describe the second sub-scene.

Specifically, the computer device obtains, according to a mapping relationship between a three-dimensional coordinate of a scanning point in a coordinate system where the first point cloud is located and a two-dimensional coordinate of a pixel point in the coordinate system where the first image is located, a two-dimensional coordinate in the coordinate system where the first image is located, the two-dimensional coordinate corresponding to a three-dimensional coordinate of a boundary of each field zone of the laser radar field. Then, the computer device divides the first image into a plurality of sub-images based on the two-dimensional coordinates in the coordinate system where the first image is located, which correspond to the three-dimensional coordinates of the boundary of each field partition of the acquired laser radar field. The subimages correspond to the field partitions one to one.

It should be noted that, in the embodiment of the present application, a method for obtaining a mapping relationship between a three-dimensional coordinate in a coordinate system where the first point cloud is located and a two-dimensional coordinate in a coordinate system where the first image is located is not limited.

In one implementation, the computer device obtains a mapping relationship between a three-dimensional coordinate of a scanning point in a coordinate system where the first point cloud is located and a two-dimensional coordinate of a pixel point in a coordinate system where the first image is located through Direct Linear Transformation (DLT).

In this implementation manner, the mapping relationship between the three-dimensional coordinates of the scanning point in the coordinate system where the first point cloud is located and the two-dimensional coordinates of the pixel point in the coordinate system where the first image is located can be embodied by the following formula:

wherein (x)_w,y_w,z_w) Is the coordinate of the w-th scanning point in the first point cloud, wherein w is a positive integer. (u, v) is the first imageAnd the coordinates of the pixel point corresponding to the w-th scanning point in the coordinate system. l1 to l11 are 11 undetermined parameters of the direct linear transformation method, and the 11 undetermined parameters represent the mapping relation between the three-dimensional coordinates of the scanning point in the coordinate system where the first point cloud is located and the two-dimensional coordinates of the pixel point in the coordinate system where the first image is located.

In another implementation manner, the computer device obtains a correspondence between a three-dimensional coordinate in a coordinate system where the first point cloud is located and a two-dimensional coordinate in a coordinate system where the first image is located through the light beam adjustment.

(u,v)＝P(x_w,y_w,z_w) Where P is the transformation matrix. The transformation matrix P represents the mapping relation between the three-dimensional coordinates of the scanning points in the coordinate system where the first point cloud is located and the two-dimensional coordinates of the pixel points in the coordinate system where the first image is located.

In both implementations, the direct linear transformation requires the computer device to solve for l₁To l₁₁The values of the 11 undetermined parameters, the beam adjustment, require computer equipment to solve for the transformation matrix P. The computer equipment obtains the first point cloud and the same-name point in the first image, and substitutes the coordinate of the scanning point in the first point cloud in the same-name point into (x)_w,y_w,z_w) And substituting the coordinates of the pixel points in the first image in the same-name point into (u, v) to solve the 11 undetermined parameters in the direct linear transformation or the transformation matrix in the beam luminous adjustment difference.

The method for acquiring the first point cloud and the homonymous point in the first image by the computer device can comprise the following steps:

the method comprises the following steps: the computer device establishes a planar coordinate system of the first point cloud, the planar coordinate system being the same as a coordinate system in which the first image is located.

Step two: the computer device projects the first point cloud into a first mapping image within the established planar coordinate system.

Specifically, the computer device may map each scanning point in the first point cloud to a coordinate system where the first image is located by using any one of perspective projection, parallel projection (a mode of projecting each point on the first point cloud vertically onto the plane of the first mapping image with the plane of the first image as the plane of the first mapping image) and scanning projection (a mode of projecting according to the point cloud scanning process according to the principle of point cloud acquisition), so as to obtain the first mapping image. And one scanning point in the first point cloud corresponds to one pixel point in the first mapping image.

Step three: and the computer equipment acquires the feature points in the first mapping image and the feature points in the first image by adopting a feature point extraction algorithm based on the first mapping image and the first image. The characteristic point is a stable and rotation-invariant effective characteristic capable of overcoming gray inversion. The feature point extraction algorithm may be a Moravec operator, a Forstner operator, a harris operator, or the like.

Step four: the computer device establishes a corresponding relationship between the feature points in the first mapping image and the feature points in the first image by adopting a feature point matching algorithm. The feature point matching algorithm may be a correlation coefficient method, a relaxation method, a least square method, or the like.

Step five: the computer equipment acquires the relation between the scanning point in the first point cloud and the pixel point in the first image based on the corresponding relation between the characteristic point in the first mapping image and the characteristic point in the first image and the corresponding relation between the pixel point in the first mapping image and the scanning point in the first point cloud. The scanning point in the first point cloud with the corresponding relation and the pixel point in the first image are a pair of homonymous points.

The computer equipment obtains 11 undetermined parameters in direct linear transformation or a conversion matrix in the light beam emission adjustment based on the obtained multiple pairs of homonymous points, so that the corresponding relation between the three-dimensional coordinate in the coordinate system where the first point cloud is located and the two-dimensional coordinate in the coordinate system where the first image is located is obtained.

In this embodiment, a point cloud obtained by scanning a laser radar by a computer device and an image obtained by shooting with a camera are taken as examples, and a method for fusing the point cloud and the image is described. Optionally, the computer device may first process the point cloud obtained by scanning the laser radar and/or the image obtained by shooting with the camera, and then fuse the processed point cloud and/or image.

S105: the computer device divides each sub-image in the first image into a plurality of image blocks.

Specifically, the method for dividing the first sub-image by the computer device may include the following method, where the first sub-image may be any one of the sub-images in the first image:

the method comprises the following steps: the computer device divides the first sub-image into a plurality of first image blocks. Wherein, the sizes of the different first image blocks are the same.

In one implementation of the first method:

first, a computer device acquires a target interval. And the target interval is the interval where the density of the registration points of the first sub-image is located. The registration points are pixel points in the first image and have corresponding scanning points in the first point cloud; the first pixel point in the first image corresponds to the first scanning point in the first point cloud, which means that the two-dimensional coordinate of the first scanning point in the coordinate system of the first image is the same as the two-dimensional coordinate of the first pixel point in the coordinate system of the first image.

Then, the computer device obtains an image block size corresponding to the target interval according to a corresponding relationship between each of the plurality of intervals and the image block size, and takes the obtained image block size as a first image block size. The correspondence of each of the plurality of intervals to the image block size may be predefined in the computer device. The correspondence relationship between each of the plurality of sections and the size of the image block may be set empirically.

Next, the computer device divides the first sub-image into a plurality of first image blocks based on the first image block size.

In this way, the target intervals acquired by the computer device are different, that is, the density of the registration points of the first sub-image is different, and the image blocks of the divided first sub-image are different in size. Therefore, the computer equipment can select the proper image block size according to the registration point density so as to solve the problem of poor accuracy of the fused point cloud.

In another implementation of the first method:

first, a computer device obtains a first graphic. Wherein the first graph is determined based on the registration points in the first sub-image. For example: the first pattern is a pattern of registration points in the first sub-image.

Second, the computer device determines the number of first image blocks based on the degree of similarity of the first graphic to the target graphic. As an example, the target graphic includes: any one of a rectangle, a parallelogram, or a trapezoid.

Next, the computer device divides the first sub-image into a plurality of first image blocks based on the number of the first image blocks.

In this way, the computer device obtains the similarity between the first graph and the target graph, and obtains the number of divided image blocks according to the obtained similarity. The lower the similarity degree is, the more uneven the distribution of the registration points is, if the distribution of the registration points is uneven, the density of the registration points in some areas in the first sub-image is high, and the density of the registration points in some areas is low, so that the number of the first image blocks into which the first sub-image is divided needs to be increased to ensure that the number of the registration points in each first image block in the divided first image blocks is smaller than the first threshold. Therefore, the computer equipment determines the number of the first image blocks divided in the first sub-image according to the similarity degree of the first image and the target image, so that the size of the first image blocks is determined, the computer equipment selects the proper size of the image blocks, and the problem of poor accuracy of the fused point cloud is solved.

The second method comprises the following steps: the computer device extracts a target object in the first sub-image and divides the first sub-image into a plurality of first image blocks based on the extracted target object.

In one implementation of the second method, the computer device extracts the target objects in the first sub-image, and uses the minimum bounding rectangle of each target object as a first image block. Based on this, the computer device may divide the first sub-image into a plurality of first image blocks. The target object comprises at least one target object in an image database. The image database may include: pascal visual object classes, contextual common data sets (micro common objects in context), or city scapes data sets (city scapes), etc.

And thirdly, the computer device takes the minimum outsourcing rectangular area of the continuous area with the same texture features in the first sub-image as the first image block based on the texture features of the first sub-image. Based on this, the computer device may divide the first sub-image into a plurality of first image blocks.

The method four comprises the following steps: the computer device takes each registration point in the first sub-image as a pixel point in one first image block. The computer device then assigns a non-registration point in the first sub-image to the first image block to which one of the registration points belongs based on the distance between the non-registration point and each registration point.

In one implementation of the method four, the computer device assigns a pixel point of a non-registration point in the first sub-image to a registration point in the first sub-image closest to the non-registration point. Based on this, the computer device obtains a minimum outsourced rectangular area of the pattern of each registration point and the non-registration points assigned to the registration point, and treats the minimum outsourced rectangular area as one first image block.

In the embodiment of the present application, the method for dividing the sub-image into the image blocks by the computer device may also be other methods in the prior art. Such as: any one or a combination of plural kinds of a threshold-based segmentation method, a sub-image-based segmentation method, an edge-based segmentation method, an image segmentation method based on cluster analysis, a segmentation method based on a fuzzy set theory, a segmentation method based on gene coding, and the like.

S106: the computer device fuses the first point cloud and the first image based on the plurality of image blocks.

Specifically, the computer device determines a scanning point corresponding to the registration point in the first point cloud according to the registration point in each image block of the plurality of image blocks, and fuses the scanning point and the image block where the registration point corresponding to the scanning point is located.

It should be noted that, for image blocks that do not include registration points, the computer device may combine image blocks that do not include registration points and image blocks that include registration points into one image block according to the distance between the image blocks, and fuse the image block and a scan point in a point cloud corresponding to the registration point in the image block.

The method for fusing the scanning points and the image blocks is not limited in the embodiments of the present application, and for example, the computer device generates the fused point cloud by using the color attributes (such as RGB values) of the image blocks and the attributes (such as three-dimensional coordinates, angles, laser reflection intensity, and the like) of the scanning points corresponding to the image blocks, where the fused point cloud has both the attributes of the scanning points and the attributes of the image blocks.

Subsequently, the computer device can perform navigation and obstacle avoidance of the automatic driving vehicle according to the fusion point cloud obtained by fusion.

In the embodiment of the application, the first point cloud is divided into a plurality of sub-point clouds by the view field partition of the laser radar, and the first image is divided into a plurality of different sub-images by the computer device based on the view field partition of the laser radar. The computer device divides each sub-image into a plurality of image blocks, and fuses the first point cloud and the first image based on the plurality of image blocks. Therefore, one image block of the plurality of image blocks obtained by dividing the image block by the computer equipment cannot comprise pixel points in two sub-images, so that the problem of data ghost fusion caused by the fact that one image block comprises the pixel points in the two sub-images is avoided, the accuracy of the obtained fusion point cloud is improved, and the navigation and obstacle avoidance of an automatic driving automobile are facilitated.

Fig. 7 is a schematic flowchart of a method for dividing a lidar field of view according to an embodiment of the present disclosure. For example, the present embodiment may be applied to the environment detection system shown in fig. 2, and the point cloud collection device in fig. 2 is taken as a laser radar for illustration. The method shown in fig. 7 may include the steps of:

s200: and scanning the current scene in the automatic driving environment in the field of view of the laser radar by the laser radar to obtain target point cloud. The field of view of the laser radar is divided into a plurality of different field of view zones according to the second zone parameter 1. The second partition parameter 1 includes indication information of a plurality of field-of-view partitions that partition the laser radar field of view. Wherein the indication information of the field of view partition can be identified by the boundary coordinates of the field of view partition; alternatively, the indication information of the field-of-view partition may be identified by a division ratio of the lidar field-of-view, for example, the transverse division ratio of the lidar field-of-view is 1:2:1, and the longitudinal division ratio is 1:8: 1.

S201: and the laser radar sends the target point cloud to the computer equipment.

S202: the computer equipment obtains a plurality of groups of first partition parameters based on the target point cloud. Wherein, a group of first partition parameters is used for partitioning the target point cloud. Different sets of first partition parameters are obtained based on different partition modes. The computer equipment obtains different weights of the first partition parameters based on different partition modes. The sum of the weights of the plurality of sets of first partition parameters is 1.

Specifically, the computer device may obtain multiple sets of first partition parameters based on the target point cloud as follows.

In a first mode, the computer device obtains a set of first partition parameters based on the field-of-view security weight.

Specifically, the computer device obtains the view field safety specific gravity corresponding to the scene of the target point cloud from the corresponding relationship between each scene and the view field safety specific gravity in the plurality of scenes according to the scene of the target point cloud. Wherein the field-of-view safety weighting is embodied by a set of first partition parameters. The correspondence of each of the plurality of scenes to the field of view safety weight may be preset empirically. Illustratively, the scene of the target point cloud is a high-speed scene, and the computer device obtains a view field safety proportion corresponding to the high-speed scene from the corresponding relationship between each scene in the plurality of scenes and the view field safety proportion. The safety proportion of the field of view is that the longitudinal division ratio of the field of view of the laser radar is 1:8: 1. Because the safety proportion right in front is higher when the vehicle runs at a high speed, the safety proportion of the field of view acquired by the computer equipment is that the longitudinal division ratio of the field of view of the laser radar is 1:8: 1.

In a second mode, the computer device obtains a set of first partition parameters based on the density of the scanning points of the target point cloud.

In one implementation, the computer device determines that any scanning point in the target point cloud grows a same area in different directions as a growing point to obtain a plurality of areas; if the number of the scanning points in each of the plurality of areas is approximately equal, the density of the scanning points in the plurality of areas is approximately the same. Based on the method, the computer equipment grows the continuous range with approximately the same scanning point density into rectangular areas, and obtains the boundary coordinates of each rectangular area to obtain a group of first partition parameters.

Illustratively, the set of first partition parameters obtained by the computer device is: { P1, (25 × 5); p2, (25 x 15); p3, (25 × 5) }, wherein P1 includes coordinates of a vertex of the first region, 25 is a length of the divided first region, and 5 is a width of the divided first region. P2 includes the coordinates of the vertex of the second region, 25 is the length of the second region being demarcated and 15 is the width of the second region being demarcated. P3 includes the coordinates of the vertex of the third region, 25 is the length of the third region being divided, and 5 is the width of the third region being divided. The first region, the second region, and the third region each represent a regular rectangular partition, and P1 may be a set of four vertex coordinates of the rectangular partition. The result of the computer device partitioning the target point cloud according to the set of first partition parameters is shown in fig. 8.

And thirdly, the computer equipment acquires a group of first partition parameters based on the similarity of the target graph.

Specifically, the method comprises the following steps: first, a computer device projects a target point cloud into a mapping image within a planar coordinate system perpendicular to a horizontal plane. Then, the computer equipment obtains the similarity degree between a geometric figure formed by the scanning points in the target point cloud in the mapped image and the target figure based on the mapped image, takes the boundary pixel points of the geometric figure with the similarity degree larger than a threshold value as growing points, and obtains a plurality of regular rectangular areas by using an area generation method. The target pattern includes: any one of a rectangle, a parallelogram, or a trapezoid. Finally, the computer device obtains the boundary coordinates of the regular rectangular areas to obtain a group of first partition parameters.

Of course, the computer device may also obtain the set of first partition parameters by other methods known in the art. This is not limited in the embodiments of the present application.

S203: the computer device obtains a second partition parameter 2 based on the obtained plurality of sets of first partition parameters and the weight of each set of first partition parameters in the plurality of sets of first partition parameters. And the second partition parameter 2 is used for dividing the field of view of the laser radar into a plurality of field of view partitions.

Specifically, the computer device calculates a second partition parameter 2 based on the obtained multiple sets of first partition parameters and the weight of each set of first partition parameters.

Illustratively, the computer device obtains three sets of first partition parameters. The first group of first partition parameters is that the transverse division ratio of the laser radar field of view is 1:2:1, and the weight of the first group of first partition parameters is 20%. The second group of first partition parameters are that the transverse division ratio of the laser radar field of view is 1:1: 1:1, the second set of first partition parameters is weighted 50%. And the transverse division ratio of the laser radar field of view of the third group of first partition parameters is 1:1.5:1, the weight of the first partition parameters of the third group is 30%, the longitudinal division ratio of the laser radar field of view is 1:8:1, and the weight is 100%. And the computer equipment calculates to obtain a group of second partition parameters according to the three groups of first partition parameters and the weight of each group of first partition parameters, wherein the group of second partition parameters comprises a transverse division ratio of the field of view of the laser radar and a longitudinal division ratio of the field of view of the laser radar. Wherein, the transverse division proportion of the laser radar field of view is: (1 × 20% +1 × 50% +1 × 30%): (2 × 20% +1 × 50% +1.5 × 30%): (1 × 20% +1 × 50% +1 × 30%): and (0+1 × 50% +0) ═ 1:1.35:1:0.5, and the longitudinal division ratio of the field of view of the laser radar is 1:8:1 according to the same calculation method.

It is understood that the environment detection system may acquire a plurality of sets of second partition parameters through the above-described S200 to S203 during the running of the autonomous vehicle. Illustratively, the computer device acquires a point cloud of a scene every 5ms, and acquires and stores a set of second partition parameters through the above S200 to S203. The computer device may perform the following steps to obtain the second partition parameters 2 after each set of second partition parameters is obtained.

The method comprises the following steps: the computer device obtains a stored set number of second partition parameters.

Illustratively, the computer device stores the obtained set of second partition parameters in a data structure (e.g., a table, an array, a linked list, a queue, or a stack), and the computer device obtains the number of sets of second partition parameters stored in the data structure.

Step two: the computer device determines whether the number of sets of the first partition parameter is greater than a second threshold. The second threshold value can be determined according to the number of point cloud frames required by the computer equipment for judging that the scene of the point cloud acquired by the laser radar tends to be stable.

If yes, the number of the groups of the second partition parameters reaches the minimum standard, and the third step is executed. If not, the number of the groups of the second partition parameters does not reach the minimum standard, and the process is finished.

Step three: the computer device obtains stored fluctuation values of the plurality of sets of second partition parameters.

For example, the computer device calculates at least one of a variance, a standard deviation, or a covariance of the plurality of sets of second partition parameters, and takes the calculated variance, standard deviation, or covariance as a fluctuation value of the plurality of sets of second partition parameters.

Step four: and the computer equipment judges whether the fluctuation values of the multiple groups of second partition parameters are larger than a third threshold value, if not, the division of the laser radar field tends to be stable, and the step five is executed. If yes, indicating that the division of the laser radar field of view is unstable, and ending. Wherein the third threshold may be determined based on the accuracy required to change the field of view of the lidar.

And step five, the computer equipment acquires a second partition parameter 2. The second partition parameter 2 is any one of the stored sets of second partition parameters.

The computer device preferentially selects the newly stored second partition parameter as the second partition parameter 2.

Step six, the computer equipment determines that the second partition parameter 1 is different from the second partition parameter 2.

If the computer device determines that the second partition parameter 1 is the same as the second partition parameter 2, the process ends.

The computer device may delete the stored sets of second partition parameters after performing step six.

S204: and the computer equipment sends indication information to the laser radar, wherein the indication information is used for indicating the laser radar to divide the field of view of the laser radar according to the second partition parameter 2 in the indication information.

It should be noted that this embodiment may be performed before the first embodiment, where the target point cloud is a frame of point cloud acquired before the first point cloud, and is used to partition the laser radar field of view for acquiring the first point cloud in the first embodiment. The embodiment may also be performed after the first embodiment, at this time, the target point cloud may be the first point cloud, and is used to partition the laser radar view field for acquiring the second point cloud. The second point cloud is a frame of point cloud obtained after the first point cloud.

S205: and the laser radar divides the field of view of the laser radar according to the indication information.

In this embodiment, the computer device obtains a plurality of sets of first partition parameters according to the target point cloud, and obtains a weight of each set of partition parameters in the plurality of sets of first partition parameters. The partition modes of any two different sets of first partition parameters are different. Then, the computer device calculates a second partition parameter 2 according to the multiple groups of first partition parameters and the weight of each group of first partition parameters. And finally, dividing the field of view of the laser radar by using a second partition parameter 2. Therefore, the division of the laser radar field of view integrates multiple partition modes, so that the division of the laser radar field of view is more reasonable. The total dotting quantity is certain when the laser radar acquires a frame of point cloud. The laser radar can scan in different field partitions with different scanning densities, and when the scanning density in one field partition is increased, the scanning density of the rest field partitions is necessarily reduced to a certain extent. Therefore, reasonable laser radar view field division is beneficial to improving the information quantity of the scanning points acquired by the laser radar. For example, if many points are hit on the same object under a field of view, the information of the multiple scanning points in the acquired point cloud is indicative of the object, which results in the waste of the scanning points. If different points are hit on different objects as much as possible under one field of view zone, the information of a plurality of scanning points in the acquired point cloud represents different objects, thereby increasing the information amount of the acquired scanning points.

Fig. 9 is a schematic flowchart of a method for adjusting scan densities of different field sections in a lidar field of view according to an embodiment of the present disclosure. Illustratively, the present embodiment may be applied to the computer device shown in fig. 3. The method shown in fig. 9 may include the steps of:

s300: a computer device obtains a first fused point cloud. The first fused point cloud is a fused point cloud of the first point cloud and the first image. The first point cloud is a point cloud of a first scene in the automatic driving environment, and the first image is an image of the first scene. The first point cloud includes a first sub-point cloud. The first sub-point cloud and the first sub-image are used to describe a first sub-scene in the first scene.

Specifically, the first fused point cloud is obtained by referring to the method in the first embodiment, which is not described again. The lidar field of view includes a first field of view zone. And the laser radar scans the first sub-scene in the first view field subarea at the first scanning density to obtain a first sub-point cloud.

S301, the computer device judges whether the resolution of the first image is larger than a fourth threshold value. If yes, it is shown that the first image can describe the first scene more accurately, then S302 is performed. If not, it indicates that the first image cannot accurately describe the first scene, then S303 is performed.

S302: the computer device adjusts a scan density of each of a plurality of field of view partitions in the current lidar field of view based on any of the first fused point cloud, the first image, or the first point cloud.

The embodiment of the application does not limit the implementation mode of adjusting the scanning density of each field partition in the field of view of the laser radar.

In one implementation, the computer device adjusts the first scan density based on features of the first sub-fused point cloud. The first sub-fusion point cloud is a fusion point cloud obtained by fusing the first sub-point cloud and the first sub-image.

In a first example, the computer device obtains an area of a region of the textural features of the first sub-fused point cloud having the largest area of the region of the same textural features. If the ratio of the obtained area of the region to the area of the first sub-fusion point cloud is greater than the fifth threshold, it is indicated that the first sub-scene can be described only by a small number of scanning points, and the first scanning density is reduced. And if the ratio of the obtained area of the region to the area of the first sub-fusion point cloud is smaller than or equal to a fifth threshold, which indicates that more scanning points are needed to describe the first sub-scene, increasing the first scanning density.

In a second example, the computer device obtains a plurality of feature points in the first sub-fused point cloud using a feature point selection algorithm. The computer device calculates difference values (such as variance, standard deviation or covariance) of the depths of the acquired plurality of feature points. The depth of one feature point represents the distance between a traffic participant and the laser radar in the real scene described by the feature point. If the depth difference value calculated by the computer equipment is larger than a sixth threshold value, which indicates that more scanning points are needed to describe the first sub-scene, increasing the first scanning density; if the depth difference value calculated by the computer device is less than or equal to the sixth threshold, which indicates that the current scanning point can describe the first sub-scene, the first scanning density may be maintained unchanged.

In a third example, the computer device determines whether the number of scanning points included in the first sub-fusion point cloud area in the first view field partition is smaller than a seventh threshold, and if the number of scanning points included is smaller than the seventh threshold and the depth of the scanning point with the smallest depth is larger than an eighth threshold, it indicates that no object exists in a certain distance range from the laser radar in the first view field partition (for example, most of an actual scene in the first view field partition is sky and a small part is a street lamp in a distance), and at this time, the computer device takes a preset scanning density with the lowest security requirement as the first scanning density.

In a fourth example, the computer device obtains a number of target objects in the first sub-fused point cloud. Wherein the target object comprises at least one target in an image database. If the number of the target objects acquired by the computer equipment is larger than the ninth number, increasing the first scanning density; and if the number of the targets acquired by the computer equipment is less than or equal to the ninth threshold, maintaining the current first scanning density unchanged.

It should be noted that, in the above implementation, if the information determined by the computer device can be acquired from the point cloud, the scan density of each field partition may also be adjusted based on the acquired point cloud alone (as in the above second, third, and fourth examples). If the information determined by the computer device can be obtained from the image, the computer device may also adjust the scan density of each field of view partition based on the image alone (as in the first and fourth examples described above).

And S303, the computer equipment adjusts the scanning density of each field of view subarea in the plurality of field of view subareas in the current laser radar field of view based on the first point cloud or the first fusion point cloud.

Exemplarily, reference is made to the second example, the third example, and the fourth example in step S302.

The computer device may adjust the scan density of the field of view partition by sending indication information to the lidar. The laser radar can realize the adjustment of the scanning density by controlling the dotting time interval of the laser.

In this embodiment, the computer device may determine whether to adjust the scan density of each field of view partition in the lidar field of view based on the first image according to the resolution of the first image. If the resolution of the first image is not judged, the first image cannot accurately describe the first scene, and when the scanning density of each field partition in the laser radar field of view is adjusted based on the first image, the first image cannot accurately describe the first scene, so that the judgment basis for adjusting the scanning density of each field partition in the laser radar field of view, which is acquired by the computer device, is not accurate. The embodiment of the application can avoid the problem of inaccurate judgment basis, thereby enabling the computer equipment to better control the scanning density of each field of view partition.

The scheme provided by the embodiment of the application is mainly introduced from the perspective of a method. To implement the above functions, it includes hardware structures and/or software modules for performing the respective functions. Those of skill in the art will readily appreciate that the present application is capable of hardware or a combination of hardware and computer software implementing the exemplary method steps described in connection with the embodiments disclosed herein. Whether a function is performed as hardware or computer software drives hardware depends upon the particular application and design constraints imposed on the solution. Skilled artisans may implement the described functionality in varying ways for each particular application, but such implementation decisions should not be interpreted as causing a departure from the scope of the present application.

In the embodiment of the present application, the computer device may be divided into the functional modules according to the method example, for example, each functional module may be divided corresponding to each function, or two or more functions may be integrated into one processing module. The integrated module can be realized in a hardware mode, and can also be realized in a software functional module mode. It should be noted that, in the embodiment of the present application, the division of the module is schematic, and is only one logic function division, and there may be another division manner in actual implementation.

Fig. 10 is a schematic structural diagram of a device for fusing a point cloud and an image according to an embodiment of the present disclosure. The point cloud and image fusion apparatus 20 may be used to perform the functions performed by the computer device in any of the above embodiments (such as the embodiments shown in fig. 4, fig. 7, or fig. 9). The point cloud and image fusion device 20 includes: the device comprises an acquisition module 201, a dividing module 202 and a fusion module 203, and optionally, the device 20 for fusing point cloud and image further comprises an adjustment module 204. The obtaining module 201 is configured to obtain a first point cloud, where the first point cloud is a point cloud describing a first scene in an automatic driving environment. The first point cloud comprises a first sub-point cloud and a second sub-point cloud, and the first sub-point cloud and the second sub-point cloud are respectively used for describing a first sub-scene and a second sub-scene in the first scene. The first sub-scene and the second sub-scene are respectively sub-scenes under a first view field partition and a second view field partition of the point cloud acquisition equipment. A first image is acquired. Wherein the first image is an image for describing a first scene. A dividing module 202 for dividing the first image into a plurality of sub-images. Wherein the plurality of sub-images includes a first sub-image and a second sub-image. The first sub-image and the second sub-image are used to describe a first sub-scene and a second sub-scene, respectively. Each of the plurality of sub-images is divided into a plurality of image blocks. And the fusion module 203 is configured to fuse the first point cloud and the first image based on the plurality of image blocks. For example, in conjunction with fig. 4, the obtaining module 201 may be configured to perform the receiving step in S103. The partitioning module 202 may be configured to perform S104-S105. The fusion module 203 may be configured to perform S106. With reference to fig. 7, the obtaining module 201 may be configured to perform the receiving steps in S201, S202 to S203. With reference to fig. 9, the obtaining module 201 may be configured to perform S300, and the adjusting module 204 may be configured to perform S301 to S303.

Optionally, the dividing module 202 is specifically configured to: the first sub-image is divided into a plurality of first image blocks. Wherein, the sizes of the different first image blocks are the same. The second sub-image is divided into a plurality of second image blocks. Wherein, the sizes of the different second image blocks are the same.

Optionally, the obtaining module 201 is specifically configured to: and acquiring a target interval, wherein the target interval is an interval where the density of the registration points of the first sub-image is located. The registration points are pixel points in the first image that have corresponding scan points in the first point cloud. The first pixel point in the first image corresponds to the first scanning point in the first point cloud, which means that the two-dimensional coordinate of the first scanning point in the coordinate system of the first image is the same as the two-dimensional coordinate of the first pixel point in the coordinate system of the first image. And acquiring the size of an image block corresponding to the target interval according to the corresponding relation between each interval in the plurality of intervals and the size of the image block, and taking the acquired size of the image block as the size of the first image block. The dividing module 202 is specifically configured to: the first sub-image is divided into a plurality of first image blocks based on the first image block size.

Optionally, the obtaining module 201 is specifically configured to: a first graph is acquired, the first graph being determined based on the registration points in the first sub-image. And determining the number of the first image blocks based on the similarity degree of the first image and the target image. The dividing module 202 is specifically configured to: the first sub-image is divided into a plurality of first image blocks based on the number of the first image blocks.

Optionally, the dividing module 202 is specifically configured to: a target object in the first sub-image is extracted, and the first sub-image is divided into a plurality of first image blocks based on the extracted target object.

Optionally, the dividing module 202 is specifically configured to: and taking each registration point in the first sub-image as a pixel point in one first image block. The non-registration points are assigned to the first image block to which one of the registration points belongs based on a distance between the non-registration point and each registration point in the first sub-image.

Optionally, the obtaining module 201 is further configured to: a plurality of sets of first partition parameters are obtained based on the target point cloud. Wherein the target point cloud is a point cloud that is used to describe a second scene in the autonomous driving environment. The second scene is a scene preceding the first scene; different sets of first partition parameters are obtained based on different partition modes. And acquiring the weight of each group of first partition parameters in the plurality of groups of first partition parameters. And acquiring second partition parameters based on the multiple groups of first partition parameters and the weight of each group of first partition parameters. The dividing module 202 is further configured to divide the field of view of the point cloud acquisition device into a plurality of field of view partitions using the second partition parameter. Wherein the plurality of field of view partitions includes a first field of view partition and a second field of view partition.

Optionally, the obtaining module 201 is further configured to: a resolution of the first image is acquired. The adjustment module 204 is configured to: and if the resolution of the first image is greater than the first threshold, adjusting at least one of the scanning density of the first field of view partition and the scanning density of the second field of view partition based on any one of the fused point cloud, the first point cloud or the first image. The fused point cloud is obtained by fusing the first image and the first point cloud. And if the resolution of the first image is less than or equal to the first threshold, adjusting at least one of the scanning density of the first field of view partition and the scanning density of the second field of view partition based on the first point cloud or the fused point cloud.

In one example, referring to fig. 3, the receiving function in the obtaining module 201 may be implemented by the communication interface 104 in fig. 3, and the dividing module 202, the fusing module 203, and the adjusting module 204 may all be implemented by the processor 101 in fig. 3 calling a computer program stored in the memory 103.

For the detailed description of the above alternative modes, reference is made to the foregoing method embodiments, which are not described herein again. In addition, for the explanation and the description of the beneficial effects of any point cloud and image fusion device 20 provided above, reference may be made to the corresponding method embodiment described above, and details are not repeated.

It should be noted that the actions performed by the modules are only specific examples, and the actions actually performed by the units refer to the actions or steps mentioned in the description of the embodiments based on fig. 4, fig. 7, or fig. 9.

An apparatus (e.g., a computer device or a chip) is also provided in an embodiment of the present application, including: a memory and a processor; the memory is for storing a computer program, and the processor is for invoking the computer program to perform the actions or steps mentioned in any of the embodiments provided above.

Embodiments of the present application also provide a computer-readable storage medium, which stores a computer program, and when the computer program runs on a computer, the computer program causes the computer to execute the actions or steps mentioned in any of the embodiments provided above.

The embodiment of the application also provides a chip. Integrated in this chip are the circuits and one or more interfaces for implementing the functions of the above-mentioned fusion means 20 of point clouds and images. Optionally, the functions supported by the chip may include processing actions in the embodiments described based on fig. 4, fig. 7, or fig. 9, which are not described herein again. Those skilled in the art will appreciate that all or part of the steps for implementing the above embodiments may be implemented by a program instructing the associated hardware to perform the steps. The program may be stored in a computer-readable storage medium. The above-mentioned storage medium may be a read-only memory, a random access memory, or the like. The processing unit or processor may be a central processing unit, a general purpose processor, an Application Specific Integrated Circuit (ASIC), a microprocessor (DSP), a Field Programmable Gate Array (FPGA) or other programmable logic device, a transistor logic device, a hardware component, or any combination thereof.

The embodiments of the present application also provide a computer program product containing instructions, which when executed on a computer, cause the computer to execute any one of the methods in the above embodiments. The computer program product includes one or more computer instructions. The procedures or functions described in accordance with the embodiments of the present application are all or partially generated upon loading and execution of computer program instructions on a computer. The computer may be a general purpose computer, a special purpose computer, a network of computers, or other programmable device. The computer instructions may be stored on a computer readable storage medium or transmitted from one computer readable storage medium to another computer readable storage medium, for example, the computer instructions may be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wire (e.g., coaxial cable, fiber optic, Digital Subscriber Line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.). Computer-readable storage media can be any available media that can be accessed by a computer or can comprise one or more data storage devices, such as servers, data centers, and the like, that can be integrated with the media. The usable medium may be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., Solid State Disk (SSD)), among others.

It should be noted that the above devices for storing computer instructions or computer programs provided in the embodiments of the present application, such as, but not limited to, the above memories, computer readable storage media, communication chips, and the like, are all nonvolatile (non-volatile).

Other variations to the disclosed embodiments can be understood and effected by those skilled in the art in practicing the claimed application, from a review of the drawings, the disclosure, and the appended claims. In the claims, the word "comprising" does not exclude other elements or steps, and the word "a" or "an" does not exclude a plurality. A single processor or other unit may fulfill the functions of several items recited in the claims. The mere fact that certain measures are recited in mutually different dependent claims does not indicate that a combination of these measures cannot be used to advantage.

Although the present application has been described in conjunction with specific features and embodiments thereof, various modifications and combinations can be made thereto without departing from the spirit and scope of the application. Accordingly, the specification and figures are merely exemplary of the present application as defined in the appended claims and are intended to cover any and all modifications, variations, combinations, or equivalents within the scope of the present application.

Claims

1. A method for fusing point cloud and image is applied to a computer device, and comprises the following steps:

acquiring a first point cloud; wherein the first point cloud is a point cloud describing a first scene; the first point cloud comprises a first sub-point cloud and a second sub-point cloud, the first sub-point cloud and the second sub-point cloud are respectively used for describing a first sub-scene and a second sub-scene in the first scene, and the first sub-scene and the second sub-scene are respectively sub-scenes under a first view field partition and a second view field partition of the point cloud acquisition equipment;

acquiring a first image; wherein the first image is an image depicting the first scene;

dividing the first image into a plurality of sub-images; wherein the plurality of sub-images comprises a first sub-image and a second sub-image, the first sub-image and the second sub-image being used to describe the first sub-scene and the second sub-scene, respectively;

dividing each of the plurality of sub-images into a plurality of image blocks;

fusing the first point cloud and the first image based on the plurality of image blocks.

2. The method of claim 1, wherein dividing each of the plurality of sub-images into a plurality of image blocks comprises:

dividing the first sub-image into a plurality of first image blocks; wherein the sizes of the different first image blocks are the same;

dividing the second sub-image into a plurality of second image blocks; and the sizes of the second image blocks are the same.

3. The method of claim 2, wherein the dividing the first sub-image into a plurality of first image blocks comprises:

acquiring a target interval, wherein the target interval is an interval where the density of the registration points of the first sub-image is located; the registration points are pixel points in the first image, which have corresponding scanning points in the first point cloud; the first pixel point in the first image corresponds to the first scanning point in the first point cloud, which means that the two-dimensional coordinate of the first scanning point in the coordinate system of the first image is the same as the two-dimensional coordinate of the first pixel point in the coordinate system of the first image;

acquiring the size of an image block corresponding to the target interval according to the corresponding relation between each interval in the multiple intervals and the size of the image block, and taking the acquired size of the image block as the size of a first image block;

dividing the first sub-image into a plurality of first image blocks based on the first image block size.

4. The method of claim 2, wherein the dividing the first sub-image into a plurality of first image blocks comprises:

acquiring a first graph, the first graph being determined based on registration points in the first sub-image;

determining the number of the first image blocks based on the similarity degree of the first image and a target image;

the first sub-image is divided into a plurality of first image blocks based on the number of the first image blocks.

5. The method of claim 1, wherein dividing each of the plurality of sub-images into a plurality of image blocks comprises:

extracting a target object in the first sub-image, and dividing the first sub-image into a plurality of first image blocks based on the extracted target object.

6. The method of claim 1, wherein dividing each of the plurality of sub-images into a plurality of image blocks comprises:

taking each registration point in the first sub-image as a pixel point in one first image block;

assigning a non-registration point in the first sub-image to a first image block to which one registration point in the each registration point belongs based on a distance between the non-registration point and the each registration point.

7. The method according to any one of claims 1 to 6, further comprising:

acquiring a plurality of groups of first partition parameters based on the target point cloud; wherein the target point cloud is a point cloud describing a second scene; the second scene is a scene prior to the first scene; different groups of first partition parameters are obtained based on different partition modes;

acquiring the weight of each group of first partition parameters in the multiple groups of first partition parameters;

acquiring a second partition parameter based on the multiple groups of first partition parameters and the weight of each group of first partition parameters;

dividing a field of view of the point cloud acquisition device into a plurality of field of view zones using the second zone parameters; wherein the plurality of field of view partitions includes the first field of view partition and the second field of view partition.

8. The method according to any one of claims 1 to 7, further comprising:

acquiring the resolution of the first image;

if the resolution of the first image is greater than a first threshold, adjusting at least one of the scan density of the first field of view partition and the scan density of the second field of view partition based on any one of the fused point cloud, the first point cloud, or the first image; the fused point cloud is obtained after the first image and the first point cloud are fused;

if the resolution of the first image is less than or equal to the first threshold, adjusting at least one of the scan density of the first field of view partition and the scan density of the second field of view partition based on the first point cloud or the fused point cloud.

9. A point cloud and image fusion device, characterized in that the fusion device comprises:

the acquisition module is used for acquiring a first point cloud; wherein the first point cloud is a point cloud describing a first scene; the first point cloud comprises a first sub-point cloud and a second sub-point cloud, the first sub-point cloud and the second sub-point cloud are respectively used for describing a first sub-scene and a second sub-scene in the first scene, and the first sub-scene and the second sub-scene are respectively sub-scenes under a first view field partition and a second view field partition of the point cloud acquisition equipment; acquiring a first image; wherein the first image is an image depicting the first scene;

a dividing module for dividing the first image into a plurality of sub-images; wherein the plurality of sub-images comprises a first sub-image and a second sub-image, the first sub-image and the second sub-image being used to describe the first sub-scene and the second sub-scene, respectively; dividing each of the plurality of sub-images into a plurality of image blocks;

and the fusion module is used for fusing the first point cloud and the first image based on the image blocks.

10. The fusion device according to claim 9, wherein the partitioning module is specifically configured to:

11. The fusion device according to claim 10, wherein the obtaining module is specifically configured to:

acquiring a target interval, wherein the target interval is an interval where the density of the registration points of the first sub-image is located; the registration points are pixel points in the first image, which have corresponding scanning points in the first point cloud; the first pixel point in the first image corresponds to the first scanning point in the first point cloud, which means that the two-dimensional coordinate of the first scanning point in the coordinate system of the first image is the same as the two-dimensional coordinate of the first pixel point in the coordinate system of the first image; acquiring the size of an image block corresponding to the target interval according to the corresponding relation between each interval in the multiple intervals and the size of the image block, and taking the acquired size of the image block as the size of a first image block;

the dividing module is specifically configured to: dividing the first sub-image into a plurality of first image blocks based on the first image block size.

12. The fusion device according to claim 10, wherein the obtaining module is specifically configured to:

acquiring a first graph, the first graph being determined based on registration points in the first sub-image; determining the number of the first image blocks based on the similarity degree of the first image and a target image;

the dividing module is specifically configured to: the first sub-image is divided into a plurality of first image blocks based on the number of the first image blocks.

13. The fusion device according to claim 9, wherein the partitioning module is specifically configured to:

14. The fusion device according to claim 9, wherein the partitioning module is specifically configured to:

15. The fusion device of any one of claims 9 to 14, wherein the obtaining module is further configured to:

the dividing module is further used for dividing the field of view of the point cloud acquisition equipment into a plurality of field of view partitions by using the second partition parameters; wherein the plurality of field of view partitions includes the first field of view partition and the second field of view partition.

16. The fusion device of any one of claims 9 to 15, wherein the obtaining module is further configured to:

acquiring the resolution of the first image;

the fusion device further comprises: an adjustment module;

the adjustment module is used for: if the resolution of the first image is greater than a first threshold, adjusting at least one of the scan density of the first field of view partition and the scan density of the second field of view partition based on any one of the fused point cloud, the first point cloud, or the first image; the fused point cloud is obtained after the first image and the first point cloud are fused;

17. A computer-readable storage medium, having stored thereon a computer program which, when run on a computer, causes the computer to perform the method of any one of claims 1 to 8.

18. A computer device comprising a memory, a processor and a computer program stored on the memory and executable on the processor, characterized in that the steps of the method according to any of claims 1-8 are implemented when the program is executed by the processor.