CN113506305B - Image enhancement method, semantic segmentation method and device for three-dimensional point cloud data - Google Patents
Image enhancement method, semantic segmentation method and device for three-dimensional point cloud data Download PDFInfo
- Publication number
- CN113506305B CN113506305B CN202110642257.7A CN202110642257A CN113506305B CN 113506305 B CN113506305 B CN 113506305B CN 202110642257 A CN202110642257 A CN 202110642257A CN 113506305 B CN113506305 B CN 113506305B
- Authority
- CN
- China
- Prior art keywords
- image
- dimensional
- enhanced
- point cloud
- cloud data
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 238000000034 method Methods 0.000 title claims abstract description 93
- 230000011218 segmentation Effects 0.000 title claims abstract description 61
- 238000013527 convolutional neural network Methods 0.000 claims description 14
- 238000009826 distribution Methods 0.000 claims description 14
- 238000004590 computer program Methods 0.000 claims description 11
- 230000008569 process Effects 0.000 claims description 10
- 238000006243 chemical reaction Methods 0.000 claims description 7
- 238000012545 processing Methods 0.000 abstract description 21
- 238000004422 calculation algorithm Methods 0.000 abstract description 5
- 238000010586 diagram Methods 0.000 description 4
- 238000013528 artificial neural network Methods 0.000 description 3
- 230000000694 effects Effects 0.000 description 3
- 238000013459 approach Methods 0.000 description 2
- 238000013473 artificial intelligence Methods 0.000 description 2
- 230000008901 benefit Effects 0.000 description 2
- 238000013135 deep learning Methods 0.000 description 2
- 230000002708 enhancing effect Effects 0.000 description 2
- 239000004973 liquid crystal related substance Substances 0.000 description 2
- 238000009877 rendering Methods 0.000 description 2
- 238000012360 testing method Methods 0.000 description 2
- 238000004364 calculation method Methods 0.000 description 1
- 230000008859 change Effects 0.000 description 1
- 230000007547 defect Effects 0.000 description 1
- 238000005315 distribution function Methods 0.000 description 1
- 230000006870 function Effects 0.000 description 1
- 238000010801 machine learning Methods 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 238000003672 processing method Methods 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/10—Segmentation; Edge detection
- G06T7/11—Region-based segmentation
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
- G06N3/045—Combinations of networks
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/08—Learning methods
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T5/00—Image enhancement or restoration
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/10—Image acquisition modality
- G06T2207/10024—Color image
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/10—Image acquisition modality
- G06T2207/10028—Range image; Depth image; 3D point clouds
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/20—Special algorithmic details
- G06T2207/20081—Training; Learning
Abstract
The application relates to an image enhancement method, a semantic segmentation method and a device for three-dimensional point cloud data, wherein the image enhancement method comprises the following steps: projecting the obtained three-dimensional point cloud data into a two-dimensional image; dividing the two-dimensional image into a plurality of subareas; converting the image of each subarea into a gray level histogram, and carrying out equalization treatment on each gray level histogram to realize image enhancement of each subarea, so as to obtain an enhanced two-dimensional image, wherein the enhanced two-dimensional image is used for extracting space geometric features. According to the application, the image enhancement processing is carried out on each subarea to obtain the enhanced two-dimensional image, so that the local features are more obvious, and the problem that the features of the two-dimensional image obtained by the existing algorithm are fuzzy and the space geometric features cannot be extracted effectively is solved.
Description
Technical Field
The application relates to an image enhancement method, a semantic segmentation method and a semantic segmentation device for three-dimensional point cloud data, and belongs to the technical field of image processing of the three-dimensional point cloud data.
Background
The point cloud is a set of point data representing the surface of the object under test acquired by a measuring instrument. The basic data of the point cloud includes three-dimensional coordinates (i.e., XYZ) and reflection intensity. In some cases, the point cloud may also contain RGB color information. This requires capturing a color image of the surface of the object under test by using a camera and assigning it to each data point. In the processing of point cloud data, one basic task is semantic segmentation. At present, semantic segmentation of point cloud data is mostly performed based on a deep learning technology. According to the input data type of the deep learning neural network, the existing segmentation method can be roughly classified into three types: projection-based methods, discrete representation-based methods, and three-dimensional data point-based methods.
The projection-based method is that three-dimensional point cloud data are converted into a multi-view two-dimensional image or a panoramic image through projection, then semantic segmentation is carried out on the image by utilizing a two-dimensional convolutional neural network, and then the segmentation result is mapped to the three-dimensional point cloud data. However, existing algorithms of such methods rely on accurate color information contained in the point cloud data for semantic segmentation, resulting in that they cannot be effectively applied to point cloud data that does not contain color information or color information distortion. This limits the segmentation performance and application scenarios of such methods to some extent.
The three-dimensional point cloud data is projected into a multichannel panoramic photograph according to one of the commonly used projection methods (i.e., conversion of a three-dimensional rectangular coordinate system to a spherical coordinate system). The panoramic photo corresponding to each data channel obtained through projection comprises: an original RGB map, a reflected intensity map, an elevation map (normalized elevation Z coordinate values), and a depth map (normalized point-to-origin euclidean distance). The elevation map and the depth map are space coordinate information based on point cloud data, and are therefore paths for extracting space geometric features of objects in theory. However, most objects in the elevation and depth maps, which are converted from the coordinate information of the point cloud data, appear very blurred compared to the RGB map and the reflection intensity map. Thus, existing image-based semantic segmentation algorithms cannot efficiently extract useful geometric features from two-dimensional images.
Disclosure of Invention
The application provides an image enhancement method, a semantic segmentation method and a semantic segmentation device for three-dimensional point cloud data, which are used for solving the problem that space geometric features cannot be extracted effectively due to the fact that most objects are blurred in a two-dimensional image based on three-dimensional point cloud data conversion.
In order to solve the technical problems, the application provides the following technical scheme:
in a first aspect of an embodiment of the present application, there is provided a method for enhancing a point cloud image, the method including:
projecting the obtained three-dimensional point cloud data into a two-dimensional image;
dividing the two-dimensional image into a plurality of subareas;
converting the image of each subarea into a gray level histogram, and carrying out equalization treatment on each gray level histogram to realize image enhancement of each subarea, so as to obtain an enhanced two-dimensional image, wherein the enhanced two-dimensional image is used for extracting space geometric features.
Optionally, in an implementation manner of the first aspect of the present application, each two adjacent sub-regions partially overlap in space, and after the adjusting the gray level histogram corresponding to the image in each sub-region, the method further includes:
the pixel values of the overlapping portions of adjacent sub-regions are averaged.
Optionally, in an embodiment of the first aspect of the application, the size of the overlapping portion of adjacent sub-regions is 1/4 to 1/64 of the size of the sub-region.
Alternatively, in an embodiment of the first aspect of the present application, each of the sub-regions is a square sub-region of the same size.
Optionally, in an implementation manner of the first aspect of the present application, a side length of the square sub-region ranges from 1/25 to 1/100 of a long side of the two-dimensional image.
A second aspect of the present application provides a point cloud image enhancement apparatus, the apparatus comprising:
the projection module is used for projecting the three-dimensional point cloud data into a two-dimensional image;
the region dividing module is used for dividing the two-dimensional image into a plurality of subareas;
the region enhancement module is used for converting the image of each subarea into a gray level histogram, carrying out equalization processing on each gray level histogram, and realizing the image enhancement of each subarea so as to obtain an enhanced two-dimensional image, wherein the enhanced two-dimensional image is used for extracting space geometric features.
The third aspect of the application provides a semantic segmentation method of three-dimensional point cloud data, comprising the following steps:
projecting the three-dimensional point cloud data into a two-dimensional multichannel panoramic image, wherein the images corresponding to the multichannel comprise a reflection intensity image, an elevation image and a depth image;
adopting the image enhancement method of any embodiment of the first aspect to perform image enhancement on the elevation map and the depth map, so as to obtain an enhanced elevation map and enhanced depth map;
obtaining an enhanced multichannel panoramic image based on the enhanced elevation map, the enhanced depth map and the corresponding reflection intensity map;
and carrying out semantic segmentation on the enhanced multichannel panoramic image by adopting a pre-trained convolutional neural network.
A fourth aspect of the present application provides a semantic segmentation apparatus for three-dimensional point cloud data, including:
the image conversion module is used for projecting the three-dimensional point cloud data into a two-dimensional multichannel panoramic image, wherein the images corresponding to the multichannel comprise a reflection intensity image, an elevation image and a depth image;
the image enhancement module is used for carrying out image enhancement on the elevation map and the depth map by adopting the image enhancement method in any one of the first aspect, so as to obtain an enhanced elevation map and a depth map;
the channel combination module is used for obtaining an enhanced multi-channel panoramic image based on the enhanced elevation map, the enhanced depth map and the corresponding reflection intensity map;
the semantic segmentation module is used for carrying out semantic segmentation on the enhanced multi-channel panoramic image by adopting a pre-trained convolutional neural network.
A fifth aspect of the application provides an electronic device comprising a processor and a memory storing a computer program for carrying out the steps of the method according to the embodiments of the first and third aspects of the application when the computer program is executed by the processor.
A sixth aspect of the application provides a computer readable storage medium storing a computer program for implementing the steps of the method according to embodiments of the first and third aspects of the application when the computer program is executed by a processor.
According to the image enhancement method provided by the application, the two-dimensional image obtained by three-dimensional point cloud data projection is divided into a plurality of subareas, adjacent subareas are partially overlapped, each subarea is adjusted to be in line with Rayleigh distribution, and the pixel value of the overlapped part is further subjected to average pixel value, so that the image of each subarea is enhanced, and finally the enhanced two-dimensional image is obtained. Therefore, the image enhancement method can enable local geometric features in the three-dimensional point cloud data to be clearly displayed through the two-dimensional image, and when the enhanced two-dimensional image is used for semantic segmentation, the enhanced two-dimensional image can be used for replacing RGB images and has a better semantic segmentation effect.
The foregoing description is only an overview of the present application, and is intended to provide a better understanding of the present application, as it is embodied in the following description, with reference to the preferred embodiments of the present application and the accompanying drawings.
Drawings
FIG. 1 is a flow chart of an image enhancement method provided by one embodiment of the present application;
FIG. 2 is an original RGB map provided by one embodiment of the present application;
FIG. 3 is a raw elevation view provided by one embodiment of the present application;
FIG. 4 is a partial elevation view of an enhanced process provided by one embodiment of the present application;
FIG. 5 is a partial RGB image provided by one embodiment of the present application;
FIG. 6 is an overall elevation view of an enhancement process provided by one embodiment of the present application;
FIG. 7 is an overall elevation view of an enhancement process provided by another embodiment of the present application;
FIG. 8 is a flow chart of a semantic segmentation method provided by one embodiment of the present application;
fig. 9 is a schematic structural view of an image enhancement device according to an embodiment of the present application;
FIG. 10 is a schematic diagram of a semantic segmentation device according to one embodiment of the present application;
FIG. 11 is a block diagram of an electronic device according to one embodiment of the present application;
FIG. 12 is a raw gray level histogram provided by one embodiment of the present application;
fig. 13 is a gray level histogram after the equalization process according to an embodiment of the present application.
Detailed Description
The following examples are illustrative of the application and are not intended to limit the scope of the application.
In order to solve the problem that the useful geometric features cannot be effectively extracted from the two-dimensional image, two technical schemes are adopted at present:
three-dimensional point cloud data is typically discretized using spatially regularly arranged three-dimensional voxels based on discrete representation methods, followed by semantic segmentation using three-dimensional convolutional neural networks. Color information is not necessary in this approach, as three-dimensional voxels can naturally express geometric features in the point cloud data. However, due to the high memory consumption and high computational intensity of three-dimensional convolutional neural networks, the voxel resolution that can be used in such methods is typically low, resulting in that it inevitably introduces large discretization artifacts and information loss during discretization.
The method based on the three-dimensional data points directly uses unprocessed data points as the input of the neural network, and uses the self-defined deep neural network to carry out semantic segmentation. Such methods typically extract corresponding features based on point cloud data within each local neighborhood of points. The data point-based method does not need any discretized intermediate representation form in the processing process, so that segmentation errors possibly caused by the discretization process are avoided from the source. However, such methods can employ a large number of three-dimensional or even higher-dimensional nearest neighbor searches or other operations, resulting in their processing speed often being very slow.
The two methods adopted at present have the defect of calculation efficiency in practical application, and the requirements of rapid data processing and efficient prediction in application cannot be met.
While projection-based approaches have significant advantages in terms of computational efficiency, they too rely on RGB color information. However, in many cases, the color information of the point cloud data is missing or locally distorted, and thus, it is difficult for the present projection-based method to extract a clear geometric feature of the object from the three-dimensional point cloud data.
In order to solve the problem that the projection method is difficult to extract the clear geometric characteristics of the target object in the three-dimensional point cloud data, the embodiment of the application provides a point cloud image enhancement method and device.
The present application is described by taking an execution body of each embodiment as an electronic device, where the electronic device may be a terminal or a server with data processing capability, and the terminal may be a mobile phone, a computer, a tablet computer, a wearable device, etc., and the embodiment does not limit types of the terminal and the electronic device.
Fig. 1 is a flowchart of a point cloud image enhancement method according to an embodiment of the present application, where an enhanced image obtained by an image enhancement algorithm according to the present embodiment is used to extract spatial geometric features of corresponding point cloud data from a two-dimensional image obtained by three-dimensional point cloud data projection.
The image enhancement method according to the embodiment of the present application is described in detail below. As shown in fig. 1, the image enhancement method of this embodiment includes:
s101: and projecting the acquired three-dimensional point cloud data into a two-dimensional image.
Specifically, for example, in an application scenario of semantic segmentation, three-dimensional point cloud data may be projected as a two-dimensional panorama of multiple channels, and images corresponding to each image channel include an RGB map, a reflection intensity map, an elevation map, and a depth map.
In order to avoid over-reliance on RGB color information, the present embodiment performs image enhancement processing on the elevation map and the depth map therein to achieve semantic segmentation.
In this embodiment, enhancement of a elevation chart is taken as an example, fig. 2 is an RGB chart obtained by three-dimensional point cloud data projection provided by an embodiment of the present application, and fig. 3 is a corresponding elevation chart.
The conversion of three-dimensional point cloud data projection into two-dimensional images belongs to a well-known technology in the art, and is not described in detail herein.
S102: the two-dimensional image is divided into a number of sub-regions.
Specifically, in order to solve the problem that the existing algorithm excessively depends on RGB color information, but objects in the elevation map and the depth map are very blurred and geometric features cannot be extracted effectively, in this embodiment, the elevation map and/or the depth map of a corresponding image channel are divided into a plurality of subareas by using a grid division method, and an image of each subarea is respectively enhanced.
Alternatively, the sub-regions of the present embodiment are square sub-regions of the same size. By dividing into square subregions of equal size, image enhancement processing is facilitated.
Alternatively, if the present embodiment is divided into square sub-regions (e.g., the boxes shown in fig. 2 and 3), the size of each sub-region may be 1/25 to 1/100 of the two-dimensional image.
Of course, the division into any shape of sub-regions can be performed according to actual requirements, for example, into rectangular sub-regions with the same size.
The specific shape of the sub-region is not limited in this embodiment.
S103: converting the image of each subarea into a gray level histogram, and carrying out equalization processing on each gray level histogram so as to realize image enhancement of each subarea and obtain an enhanced two-dimensional image.
Specifically, the enhanced two-dimensional image of the present embodiment is used to extract spatial geometric features.
Taking the elevation chart as an example, as shown in fig. 12, the gray level histogram of the sub-region corresponding to the box in fig. 3 is given, and the elevation value variation in the sub-region is usually very small compared with the elevation value span range of the whole point cloud data. Therefore, the elevation values in the sub-regions are highly concentrated around a specific value, and the local features cannot be represented.
In view of the above problem, and considering the fact that the convolutional neural network is good at learning local features rather than global features, the present embodiment performs equalization processing on the gray level histogram of the image in each sub-region to implement image enhancement on each sub-region, so as to obtain an enhanced elevation map and/or an enhanced depth map.
Fig. 4 shows a partial elevation view (enhanced for the sub-region corresponding to the box in fig. 3) of an enhanced embodiment of the present application, in which many distinct geometric detail features are included, fig. 5 is a partial RGB view corresponding to the box in fig. 2, and in the partial elevation view after enhancement in fig. 4, many detail features are more abundant and clear than the partial RGB image shown in fig. 5.
Optionally, the scheme adopted by the equalization processing in the embodiment of the application is as follows: and adjusting each gray level histogram to conform to the Rayleigh distribution.
The Rayleigh distribution refers to the fact that when two components of a random two-dimensional vector are in independent normal distribution with the mean value of 0 and the variance of the two components being the same, the modulus of the vector is in Rayleigh distribution. The expression of the Rayleigh distribution is:
as a possible implementation, the present example takes the σ value in the rayleigh distribution as 0.4, so that the expected gray value of the changed image is 0.5.
In this embodiment, the above rayleigh distribution function is adopted, and the gray histogram of each sub-region is adjusted so that the gray histogram conforms to the rayleigh distribution, and equalization processing is implemented. Fig. 13 is a result of equalizing the gradation histogram shown in fig. 12, and as can be seen from fig. 13, the gradation values are distributed over a wide span, and equalization is achieved.
Of course, the gray histogram may be adjusted to other specific distributions in the present embodiment, as long as the gray value can be equalized, and the specific equalization processing method is not limited in the embodiment of the present application.
In the embodiment, the obtained enhanced elevation map and/or enhanced depth map is used for replacing an RGB image, and a two-dimensional image based on the enhanced elevation map and/or enhanced depth map and reflection intensity map image channel combination is used as input data of a convolutional neural network for realizing semantic segmentation, so that a better semantic segmentation effect is achieved.
In addition to the application scenario of semantic segmentation, the image enhancement method in the above embodiment may be used for other application scenarios where spatial geometric feature information needs to be extracted. Of course, in other application scenarios, the three-dimensional point cloud data may be selected to be projected as different types of two-dimensional images as desired.
The conversion of a two-dimensional image into a gray-level histogram belongs to a well-known technique in the art and is not described in detail herein.
Fig. 6 is an overall elevation view of the present application after a localized enhancement process. The enhanced elevation map is obtained by dividing the original elevation map into mutually non-overlapping subareas for local enhancement according to the embodiment.
As can be seen from fig. 6, the gray values of the adjacent sub-regions at the boundary are abrupt. This is because point cloud data in adjacent sub-areas typically has a general trend of change.
In order to solve the above technical problem, optionally, when dividing the subareas, each two adjacent subareas partially overlap in space.
The embodiment further includes, after adjusting the gray level histogram corresponding to the image in each of the sub-regions so that the gray level histogram conforms to the rayleigh distribution:
the pixel values of the overlapping portions of adjacent sub-regions are averaged.
After image enhancement by dividing overlapping sub-regions, the resulting image is shown in fig. 7. With the benefit of the employed rayleigh distribution, only a small number of pixel values in each sub-region are in the vicinity of the extremum (0 or 1).
Alternatively, for each sub-region divided into the same size, the size of the overlapping portion of the adjacent sub-regions in this embodiment may be set to 1/4 to 1/64 of the size of the sub-region. This eliminates abrupt and discontinuous gray value changes at the boundary between adjacent sub-regions.
The image enhancement method according to the embodiment of the present application will be described below by taking the application of the image enhancement method according to the above embodiment to a semantic segmentation scene as an example.
First, the acquired 15 point cloud data are projected into 15 original high resolution two-dimensional multi-channel panoramas 7200 x 3600. The multiple channels include a reflected intensity map, an RGB map, an elevation map, and a depth map. Then, through S102 to S103 of the image enhancement method according to the above embodiment of the present application, the image enhancement is performed on the elevation map and the depth map (the divided sub-areas have a size of 7200/50=144, and the images of the overlapping portions of the adjacent sub-areas have 144/8=18 pixels), so as to obtain two newly added image channels: enhanced elevation maps and enhanced depth maps.
The present embodiment semantically segments a two-dimensional multi-channel panorama based on an image channel combination of a reflection intensity map and an RGB map (channel combination 1) and a data channel combination of a reflection intensity map, an enhanced elevation map and an enhanced depth map (channel combination 2), respectively, using convolutional neural networks of 5 different encoder-decoder structures.
The decoders of the 5 convolutional neural networks all adopt the structure of DeepLab v < 3+ >, and the encoders adopt 5 different backbone networks, namely ResNet18, resNet50, mobileNet V2 and Xception, inception-ResNet-v2. Table 1 gives the results of semantic segmentation accuracy of 5 different convolutional neural networks on two-dimensional images under two image channel combinations, respectively.
TABLE 1
As can be seen from table 1, the two-dimensional panorama of the image channel combination (channel combination 2) based on the reflection intensity map, the enhanced elevation map and the enhanced depth map is semantically segmented, and the accuracy of the obtained semantic segmentation is superior to that of the two-dimensional panorama of the data channel combination (channel combination 1) based on the reflection intensity map and the RGB map.
In summary, according to the image enhancement method provided by the application, the two-dimensional image obtained by projecting the three-dimensional point cloud data is divided into a plurality of subareas, adjacent subareas are partially overlapped, each subarea is adjusted to be in line with Rayleigh distribution, and the pixel value of the overlapped part is further averaged, so that the image of each subarea is enhanced, and finally the enhanced two-dimensional image is obtained. Therefore, the image enhancement method can enable local geometric features in the three-dimensional point cloud data to be clearly displayed through the two-dimensional image, and when the enhanced two-dimensional image is used for semantic segmentation, the enhanced two-dimensional image can be used for replacing RGB images and has a better semantic segmentation effect.
The embodiment of the application also provides a semantic segmentation method based on the three-dimensional point cloud data, as shown in fig. 8, the semantic segmentation method of the embodiment comprises the following steps:
s401: projecting the three-dimensional point cloud data into a two-dimensional multichannel panoramic image, wherein the images corresponding to the multichannel comprise a reflection intensity image, an elevation image and a depth image;
s402: image enhancement is carried out on the elevation map and the depth map, and the enhanced elevation map and the enhanced depth map are obtained;
and performing image enhancement on the elevation map and the depth map, wherein the image enhancement method described in the embodiment of the image enhancement method is adopted.
S403: obtaining an enhanced multichannel panoramic image based on the enhanced elevation map, the enhanced depth map and the corresponding reflection intensity map;
s404: and carrying out semantic segmentation on the enhanced multichannel panoramic image by adopting a pre-trained convolutional neural network.
In this embodiment, steps S401 to S403 refer to the related description of the above image enhancement method embodiment, and are not repeated here.
In step S404, the semantic segmentation may use an existing semantic segmentation method, which belongs to a well-known technology in the art, and will not be described herein.
Fig. 9 is a schematic structural view of an image enhancement device according to another embodiment of the present application, and as shown in fig. 9, the image enhancement device includes:
the projection module is used for projecting the three-dimensional point cloud data into a two-dimensional image;
the region dividing module is used for dividing the two-dimensional image into a plurality of subareas;
the region enhancement module is used for adjusting the gray level histogram corresponding to the image in each sub-region so that the gray level histogram accords with Rayleigh distribution, and enhancing the image of each sub-region, thereby obtaining an enhanced two-dimensional image.
The image enhancement device provided in the above embodiment and the corresponding image enhancement method embodiment belong to the same concept, and the specific implementation process of the image enhancement device is detailed in the method embodiment and will not be described herein.
Fig. 10 is a schematic structural diagram of a semantic segmentation device according to another embodiment of the present application, as shown in fig. 10, where the semantic segmentation device includes:
the image conversion module is used for projecting the three-dimensional point cloud data into a two-dimensional multichannel panoramic image, wherein the images corresponding to the multichannel comprise a reflection intensity image, an elevation image and a depth image;
the image enhancement module is used for carrying out image enhancement on the elevation map and the depth map by adopting the image enhancement method described in the embodiment of the image enhancement method, so as to obtain an enhanced elevation map and an enhanced depth map;
the channel combination module is used for obtaining an enhanced multi-channel panoramic image based on the enhanced elevation map, the enhanced depth map and the corresponding reflection intensity map;
the semantic segmentation module is used for carrying out semantic segmentation on the enhanced multi-channel panoramic image by adopting a pre-trained convolutional neural network.
The semantic segmentation device provided in the above embodiment and the corresponding semantic segmentation method embodiment belong to the same concept, and the specific implementation process of the semantic segmentation device is detailed in the method embodiment and will not be described herein.
It should be noted that: the image enhancement device and the semantic segmentation device provided in the above embodiments are only exemplified by the division of the functional modules in the corresponding method embodiments, and in practical application, the above functional allocation may be performed by different functional modules according to needs, that is, the internal structure of the image enhancement device/the semantic segmentation device is divided into different functional modules, so as to complete all or part of the functions described above.
Fig. 11 is a block diagram of an electronic device according to an embodiment of the present application, where the electronic device may be a computing device such as a desktop computer, a notebook computer, a palm computer, and a cloud server, and the computer device may include, but is not limited to, a processor and a memory. Wherein, the liquid crystal display device comprises a liquid crystal display device,
the processor may include one or more processing cores, such as: 4 core processor, 6 core processor, etc. The processor may be implemented in at least one hardware form of DSP (Digital Signal Processing ), FPGA (Field-Programmable Gate Array, field programmable gate array), PLA (Programmable Logic Array ). The processor may also include a main processor, which is a processor for processing data in an awake state, also called a CPU (Central Processing Unit ), and a coprocessor; a coprocessor is a low-power processor for processing data in a standby state. In some embodiments, the processor may incorporate a GPU (Graphics Processing Unit, image processor) for rendering and rendering of content required to be displayed by the display screen. In some embodiments, the processor may also include an AI (Artificial Intelligence ) processor for processing computing operations related to machine learning. The processor is the control center of the computer device and connects the various parts of the entire computer device using various interfaces and lines.
The memory may include high-speed random access memory, and may also include non-volatile memory, such as a hard disk, memory, plug-in hard disk, smart Media Card (SMC), secure Digital (SD) Card, flash Card (Flash Card), at least one disk storage device, memory device, or other volatile solid-state storage device.
The memory has stored thereon a computer program executable on the processor which, when executed, performs all or part of the steps of the image enhancement method or semantic segmentation method related embodiments of the present application, and/or other content described in the text.
It will be appreciated by those skilled in the art that fig. 11 is merely one possible implementation of an embodiment of the present application, and in other implementations, more or fewer components may be included, or some components may be combined, or different components may be combined, and the embodiment is not limited in this regard.
Optionally, the present application further provides a computer readable storage medium storing a computer program for implementing the steps of the image enhancement method or the semantic segmentation method in any of the above embodiments or implementations when executed by a processor.
Optionally, the present application further provides a computer program product comprising a computer readable storage medium having a program stored therein, the program being loaded and executed by a processor to implement the steps of the image enhancement method or the semantic segmentation method in any of the embodiments or implementations described above.
The technical features of the above-described embodiments may be arbitrarily combined, and all possible combinations of the technical features in the above-described embodiments are not described for brevity of description, however, as long as there is no contradiction between the combinations of the technical features, they should be considered as the scope of the description.
The above examples illustrate only a few embodiments of the application, which are described in detail and are not to be construed as limiting the scope of the application. It should be noted that it will be apparent to those skilled in the art that several variations and modifications can be made without departing from the spirit of the application, which are all within the scope of the application. Accordingly, the scope of protection of the present application is to be determined by the appended claims.
Claims (9)
1. The semantic segmentation method of the three-dimensional point cloud data is characterized by comprising the following steps of:
projecting the three-dimensional point cloud data into a two-dimensional multichannel panoramic image, wherein the images corresponding to the multichannel comprise a reflection intensity image, an elevation image and a depth image;
adopting an image enhancement method of three-dimensional point cloud data to carry out image enhancement on the elevation map and the depth map, and obtaining the enhanced elevation map and depth map; the image enhancement method of the three-dimensional point cloud data comprises the following steps: projecting the obtained three-dimensional point cloud data into a two-dimensional image; dividing the two-dimensional image into a plurality of subareas; converting the image of each subarea into a gray level histogram, and carrying out equalization treatment on each gray level histogram to realize image enhancement of each subarea, so as to obtain an enhanced two-dimensional image, wherein the enhanced two-dimensional image is used for extracting space geometric features;
obtaining an enhanced multichannel panoramic image based on the enhanced elevation map, the enhanced depth map and the corresponding reflection intensity map;
and carrying out semantic segmentation on the enhanced multichannel panoramic image by adopting a pre-trained convolutional neural network.
2. The method of claim 1, wherein said equalizing each of said gray level histograms comprises:
and adjusting each gray level histogram to conform to the Rayleigh distribution.
3. The method according to claim 1, wherein each two adjacent sub-regions among the sub-regions divided by the two-dimensional image partially overlap in space, and further comprising, after converting the image of each sub-region into a gray-level histogram and performing an equalization process on each of the gray-level histograms:
the pixel values of the overlapping portions of adjacent sub-regions are averaged.
4. A method according to claim 3, wherein the overlap of adjacent sub-regions has a size of 1/4 to 1/64 of the size of the sub-region.
5. The method of any one of claims 1 to 4, wherein each of the subregions is a square subregion of equal size.
6. The method according to claim 5, wherein the side length of the square sub-region is 1/25-1/100 of the long side of the two-dimensional image.
7. A semantic segmentation device for three-dimensional point cloud data, comprising:
the image conversion module is used for projecting the three-dimensional point cloud data into a two-dimensional multichannel panoramic image, wherein the images corresponding to the multichannel comprise a reflection intensity image, an elevation image and a depth image;
the image enhancement module is used for carrying out image enhancement on the elevation map and the depth map by adopting an image enhancement method of three-dimensional point cloud data to obtain an enhanced elevation map and an enhanced depth map; the image enhancement method of the three-dimensional point cloud data comprises the following steps: projecting the obtained three-dimensional point cloud data into a two-dimensional image; dividing the two-dimensional image into a plurality of subareas; converting the image of each subarea into a gray level histogram, and carrying out equalization treatment on each gray level histogram to realize image enhancement of each subarea, so as to obtain an enhanced two-dimensional image, wherein the enhanced two-dimensional image is used for extracting space geometric features;
the channel combination module is used for obtaining an enhanced multi-channel panoramic image based on the enhanced elevation map, the enhanced depth map and the corresponding reflection intensity map;
the semantic segmentation module is used for carrying out semantic segmentation on the enhanced multi-channel panoramic image by adopting a pre-trained convolutional neural network.
8. An electronic device comprising a processor and a memory, the memory storing a computer program, characterized in that the computer program is adapted to implement the steps of the semantic segmentation method according to any one of claims 1 to 6 when being executed by the processor.
9. A computer readable storage medium storing a computer program, characterized in that the computer program is for implementing the steps of the semantic segmentation method according to any one of claims 1 to 6 when being executed by a processor.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110642257.7A CN113506305B (en) | 2021-06-09 | 2021-06-09 | Image enhancement method, semantic segmentation method and device for three-dimensional point cloud data |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110642257.7A CN113506305B (en) | 2021-06-09 | 2021-06-09 | Image enhancement method, semantic segmentation method and device for three-dimensional point cloud data |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN113506305A CN113506305A (en) | 2021-10-15 |
| CN113506305B true CN113506305B (en) | 2023-10-24 |
Family
ID=78009567
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202110642257.7A Active CN113506305B (en) | 2021-06-09 | 2021-06-09 | Image enhancement method, semantic segmentation method and device for three-dimensional point cloud data |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN113506305B (en) |
Families Citing this family (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN116091571A (en) * | 2021-11-05 | 2023-05-09 | 罗伯特·博世有限公司 | Method and equipment for carrying out instance segmentation on picture |
| CN117392038B (en) * | 2023-12-05 | 2024-03-08 | 北京智源人工智能研究院 | Medical image histogram equalization method and device, electronic equipment and storage medium |
Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN105654438A (en) * | 2015-12-27 | 2016-06-08 | 西南技术物理研究所 | Gray scale image fitting enhancement method based on local histogram equalization |
| CN107527329A (en) * | 2017-09-19 | 2017-12-29 | 重庆邮电大学 | A kind of image defogging method based on dark channel prior and Markov random field |
| CN110570429A (en) * | 2019-08-30 | 2019-12-13 | 华南理工大学 | Lightweight real-time semantic segmentation method based on three-dimensional point cloud |
| US10650278B1 (en) * | 2017-07-21 | 2020-05-12 | Apple Inc. | Semantic labeling of point clouds using images |
| CN111652085A (en) * | 2020-05-14 | 2020-09-11 | 东莞理工学院 | Object identification method based on combination of 2D and 3D features |
| CN112818925A (en) * | 2021-02-26 | 2021-05-18 | 清华大学 | Urban building and crown identification method |
-
2021
- 2021-06-09 CN CN202110642257.7A patent/CN113506305B/en active Active
Patent Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN105654438A (en) * | 2015-12-27 | 2016-06-08 | 西南技术物理研究所 | Gray scale image fitting enhancement method based on local histogram equalization |
| US10650278B1 (en) * | 2017-07-21 | 2020-05-12 | Apple Inc. | Semantic labeling of point clouds using images |
| CN107527329A (en) * | 2017-09-19 | 2017-12-29 | 重庆邮电大学 | A kind of image defogging method based on dark channel prior and Markov random field |
| CN110570429A (en) * | 2019-08-30 | 2019-12-13 | 华南理工大学 | Lightweight real-time semantic segmentation method based on three-dimensional point cloud |
| CN111652085A (en) * | 2020-05-14 | 2020-09-11 | 东莞理工学院 | Object identification method based on combination of 2D and 3D features |
| CN112818925A (en) * | 2021-02-26 | 2021-05-18 | 清华大学 | Urban building and crown identification method |
Also Published As
| Publication number | Publication date |
|---|---|
| CN113506305A (en) | 2021-10-15 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US10861133B1 (en) | Super-resolution video reconstruction method, device, apparatus and computer-readable storage medium | |
| GB2544590B (en) | Image synthesis utilizing an active mask | |
| Possa et al. | A multi-resolution FPGA-based architecture for real-time edge and corner detection | |
| KR100924689B1 (en) | Apparatus and method for transforming an image in a mobile device | |
| CN113506305B (en) | Image enhancement method, semantic segmentation method and device for three-dimensional point cloud data | |
| US8498500B2 (en) | Image processing apparatus and image processing method | |
| CN110163831B (en) | Method and device for dynamically displaying object of three-dimensional virtual sand table and terminal equipment | |
| CN111612882B (en) | Image processing method, image processing device, computer storage medium and electronic equipment | |
| CN110675334A (en) | Image enhancement method and device | |
| CN111415300A (en) | Splicing method and system for panoramic image | |
| CN104360847A (en) | Method and equipment for processing image | |
| CN111292335B (en) | Method and device for determining foreground mask feature map and electronic equipment | |
| CN111275824A (en) | Surface reconstruction for interactive augmented reality | |
| CN111353955A (en) | Image processing method, device, equipment and storage medium | |
| CN110363837B (en) | Method and device for processing texture image in game, electronic equipment and storage medium | |
| CN114359048A (en) | Image data enhancement method and device, terminal equipment and storage medium | |
| CN115330803B (en) | Surface defect data enhancement method and device, electronic equipment and storage medium | |
| CN115601616A (en) | Sample data generation method and device, electronic equipment and storage medium | |
| Pan et al. | An automatic 2D to 3D video conversion approach based on RGB-D images | |
| CN115082345A (en) | Image shadow removing method and device, computer equipment and storage medium | |
| CN111191580B (en) | Synthetic rendering method, apparatus, electronic device and medium | |
| Hu et al. | 3D map reconstruction using a monocular camera for smart cities | |
| Hariyama et al. | Architecture of a stereo matching VLSI processor based on hierarchically parallel memory access | |
| Ahmed et al. | Digital image inpainting techniques for cultural heritage preservation and restoration | |
| CN113570518B (en) | Image correction method, system, computer equipment and storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |