CN113506305A - Image enhancement method, semantic segmentation method and device for three-dimensional point cloud data - Google Patents
Image enhancement method, semantic segmentation method and device for three-dimensional point cloud data Download PDFInfo
- Publication number
- CN113506305A CN113506305A CN202110642257.7A CN202110642257A CN113506305A CN 113506305 A CN113506305 A CN 113506305A CN 202110642257 A CN202110642257 A CN 202110642257A CN 113506305 A CN113506305 A CN 113506305A
- Authority
- CN
- China
- Prior art keywords
- image
- dimensional
- enhanced
- point cloud
- sub
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
- 238000000034 method Methods 0.000 title claims abstract description 98
- 230000011218 segmentation Effects 0.000 title claims abstract description 64
- 238000013527 convolutional neural network Methods 0.000 claims description 13
- 238000004590 computer program Methods 0.000 claims description 11
- 230000008569 process Effects 0.000 claims description 9
- 238000006243 chemical reaction Methods 0.000 claims description 5
- 230000002708 enhancing effect Effects 0.000 claims description 4
- 238000010586 diagram Methods 0.000 description 6
- 238000013528 artificial neural network Methods 0.000 description 4
- 230000000694 effects Effects 0.000 description 3
- 238000013473 artificial intelligence Methods 0.000 description 2
- 230000008901 benefit Effects 0.000 description 2
- 230000008859 change Effects 0.000 description 2
- 238000013135 deep learning Methods 0.000 description 2
- 230000006870 function Effects 0.000 description 2
- 230000007547 defect Effects 0.000 description 1
- 238000005315 distribution function Methods 0.000 description 1
- 238000010801 machine learning Methods 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 238000009877 rendering Methods 0.000 description 1
- 239000007787 solid Substances 0.000 description 1
- 239000000126 substance Substances 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/10—Segmentation; Edge detection
- G06T7/11—Region-based segmentation
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
- G06N3/045—Combinations of networks
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/08—Learning methods
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T5/00—Image enhancement or restoration
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/10—Image acquisition modality
- G06T2207/10024—Color image
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/10—Image acquisition modality
- G06T2207/10028—Range image; Depth image; 3D point clouds
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/20—Special algorithmic details
- G06T2207/20081—Training; Learning
Abstract
The application relates to an image enhancement method, a semantic segmentation method and a semantic segmentation device for three-dimensional point cloud data, wherein the image enhancement method comprises the following steps: projecting the acquired three-dimensional point cloud data into a two-dimensional image; dividing the two-dimensional image into a plurality of sub-regions; and converting the image of each subregion into a gray level histogram, and carrying out equalization processing on each gray level histogram to realize image enhancement on each subregion so as to obtain an enhanced two-dimensional image, wherein the enhanced two-dimensional image is used for extracting space geometric features. According to the method and the device, the two-dimensional image after enhancement is obtained by carrying out image enhancement processing on each subregion, and the local features are more obvious, so that the problem that the spatial geometric features cannot be effectively extracted due to the fuzzy features of the two-dimensional image obtained by the conventional algorithm is solved.
Description
Technical Field
The application relates to an image enhancement method, a semantic segmentation method and a semantic segmentation device for three-dimensional point cloud data, and belongs to the technical field of image processing of three-dimensional point cloud data.
Background
The point cloud is a point data set representing the surface of the object to be measured acquired by the measuring instrument. The base data of the point cloud includes three-dimensional coordinates (i.e., XYZ) and reflection intensity. In some cases, the point cloud may also contain RGB color information. This requires that a color image of the surface of the measured object be acquired by using a camera and assigned to each data point. In the processing of point cloud data, one basic task is semantic segmentation. Currently, semantic segmentation of point cloud data is mostly performed based on a deep learning technology. According to the input data type of the deep learning neural network, the existing segmentation methods can be roughly classified into three types: projection-based methods, discrete representation-based methods, and three-dimensional data point-based methods.
The projection-based method is to convert the three-dimensional point cloud data into a multi-view two-dimensional image or a panoramic image through projection, then perform semantic segmentation on the image by using a two-dimensional convolution neural network, and then map the segmentation result to the three-dimensional point cloud data. However, the existing algorithms of this kind of method all rely on accurate color information contained in the point cloud data for semantic segmentation, so that it cannot be effectively applied to point cloud data that does not contain color information or has color information distortion. This limits the segmentation performance and application scenarios of this type of method to some extent.
The three-dimensional point cloud data is projected as a multi-channel panoramic photograph according to one of the commonly used projection methods (i.e., conversion of a three-dimensional rectangular coordinate system to a spherical coordinate system). The panoramic photo corresponding to each data channel obtained through projection comprises: raw RGB maps, reflection intensity maps, elevation maps (normalized elevation Z coordinate values), and depth maps (normalized point-to-origin euclidean distance). The elevation map and the depth map are space coordinate information based on point cloud data, and therefore, the method is theoretically a way for extracting the space geometric features of the object. However, most objects in the elevation map and the depth map converted from the coordinate information of the point cloud data appear blurred compared to the RGB map and the reflection intensity map. Therefore, the existing semantic segmentation algorithm based on the image cannot effectively extract useful geometric features from the two-dimensional image.
Disclosure of Invention
The application provides an image enhancement method, a semantic segmentation method and a semantic segmentation device for three-dimensional point cloud data, which are used for solving the problem that spatial geometric features cannot be effectively extracted due to the fact that most objects are fuzzy in a two-dimensional image based on three-dimensional point cloud data conversion.
In order to solve the technical problem, the application provides the following technical scheme:
in a first aspect of the embodiments of the present application, a method for enhancing a point cloud image is provided, where the method includes:
projecting the acquired three-dimensional point cloud data into a two-dimensional image;
dividing the two-dimensional image into a plurality of sub-regions;
and converting the image of each subregion into a gray level histogram, and carrying out equalization processing on each gray level histogram to realize image enhancement on each subregion so as to obtain an enhanced two-dimensional image, wherein the enhanced two-dimensional image is used for extracting space geometric features.
Optionally, in an embodiment of the first aspect of the present application, after that, each two adjacent sub-regions are partially overlapped in space, and the adjusting a corresponding gray histogram of an image in each sub-region to make the gray histogram conform to a rayleigh distribution, the method further includes:
the pixel values of the overlapping portions of the adjacent sub-areas are taken as the pixel average value.
Optionally, in an embodiment of the first aspect of the present application, the overlapping portion of adjacent sub-areas is 1/4-1/64 of the size of the sub-area.
Optionally, in an embodiment of the first aspect of the present application, each of the sub-regions is a square sub-region with the same size.
Optionally, in an embodiment of the first aspect of the present application, a side length of the square subregion ranges from 1/25 to 1/100 of a long side of the two-dimensional image.
A second aspect of the present application provides a point cloud image enhancement apparatus, the apparatus comprising:
the projection module is used for projecting the three-dimensional point cloud data into a two-dimensional image;
the area dividing module is used for dividing the two-dimensional image into a plurality of sub-areas;
and the region enhancement module is used for converting the image of each subregion into a gray level histogram and carrying out equalization processing on each gray level histogram to realize the image enhancement of each subregion so as to obtain an enhanced two-dimensional image, and the enhanced two-dimensional image is used for extracting the space geometric features.
The third aspect of the present application provides a semantic segmentation method for three-dimensional point cloud data, including:
projecting the three-dimensional point cloud data into a two-dimensional multi-channel panoramic image, wherein the multi-channel corresponding image comprises a reflection intensity image, an elevation image and a depth image;
performing image enhancement on the elevation map and the depth map by using the image enhancement method of any embodiment of the first aspect to obtain an enhanced elevation map and a enhanced depth map;
obtaining an enhanced multi-channel panoramic image based on the enhanced high-level image, the enhanced depth image and the corresponding reflection intensity image;
and performing semantic segmentation on the enhanced multi-channel panoramic image by adopting a pre-trained convolutional neural network.
The fourth aspect of the present application provides a semantic segmentation apparatus for three-dimensional point cloud data, including:
the image conversion module is used for projecting the three-dimensional point cloud data into a two-dimensional multi-channel panoramic image, wherein the images corresponding to multiple channels comprise a reflection intensity map, an elevation map and a depth map;
an image enhancement module, configured to perform image enhancement on the elevation map and the depth map by using the image enhancement method according to any one of the first aspect, so as to obtain an enhanced elevation map and enhanced depth map;
the channel combination module is used for obtaining an enhanced multi-channel panoramic image based on the enhanced high-level image, the enhanced depth image and the corresponding reflection intensity image;
and the semantic segmentation module is used for performing semantic segmentation on the enhanced multi-channel panoramic image by adopting a pre-trained convolutional neural network.
A fifth aspect of the present application provides an electronic device comprising a processor and a memory, said memory storing a computer program for implementing the steps of the method according to the embodiments of the first and third aspects of the present application when executed by the processor.
A sixth aspect of the present application provides a computer readable storage medium having stored thereon a computer program for implementing the steps of the method as embodied in the first and third aspects of the present application when the computer program is executed by a processor.
According to the image enhancement method, a two-dimensional image obtained by projecting three-dimensional point cloud data is divided into a plurality of sub-areas, the adjacent sub-areas are partially overlapped, each sub-area is adjusted to be in accordance with Rayleigh distribution, and the pixel values of the overlapped part are further averaged, so that the image of each sub-area is enhanced, and the enhanced two-dimensional image is finally obtained. Therefore, the image enhancement method can enable local geometric features in the three-dimensional point cloud data to be clearly shown through the two-dimensional image, and the enhanced two-dimensional image can be used for replacing an RGB image and has a better semantic segmentation effect when being used for semantic segmentation.
The foregoing description is only an overview of the technical solutions of the present application, and in order to make the technical solutions of the present application more clear and clear, and to implement the technical solutions according to the content of the description, the following detailed description is made with reference to the preferred embodiments of the present application and the accompanying drawings.
Drawings
FIG. 1 is a flow chart of an image enhancement method provided by an embodiment of the present application;
FIG. 2 is a raw RGB map provided by one embodiment of the present application;
FIG. 3 is a raw elevation map provided by an embodiment of the present application;
FIG. 4 is a local elevation map after enhancement provided by an embodiment of the present application;
FIG. 5 is a partial RGB image provided by one embodiment of the present application;
FIG. 6 is an overall elevation map of enhancement processing provided by one embodiment of the present application;
FIG. 7 is an overall elevation map of enhancement processing provided by another embodiment of the present application;
FIG. 8 is a flow chart of a semantic segmentation method provided by one embodiment of the present application;
FIG. 9 is a schematic structural diagram of an image enhancement apparatus according to an embodiment of the present application;
FIG. 10 is a schematic structural diagram of a semantic segmentation apparatus according to an embodiment of the present application;
FIG. 11 is a block diagram of an electronic device provided in an embodiment of the present application;
FIG. 12 is an original gray level histogram provided by an embodiment of the present application;
fig. 13 is a gray level histogram after the equalization process according to an embodiment of the present application.
Detailed Description
The following examples are intended to illustrate the present application but are not intended to limit the scope of the present application.
In order to solve the above problem that useful geometric features cannot be effectively extracted from a two-dimensional image, two technical solutions are currently adopted:
discrete representation-based methods typically use spatially regularly arranged voxels to discretize the three-dimensional point cloud data, followed by semantic segmentation using three-dimensional convolutional neural networks. Color information is not necessary in this approach, as the three-dimensional voxels can naturally represent geometric features in the point cloud data. However, due to the high memory consumption and high computational intensity of the three-dimensional convolutional neural network, the voxel resolution that can be used in such methods is generally low, which inevitably introduces large discretization artifacts and information loss during the discretization process.
The method based on three-dimensional data points directly uses unprocessed data points as the input of a neural network, and uses a self-defined deep neural network for semantic segmentation. Such methods typically extract corresponding features based on point cloud data within a local neighborhood of each point. The data point-based method does not need any discretization intermediate representation form in the processing process, so that the segmentation error possibly caused by the discretization process is avoided from the source. However, such methods use a large number of three-dimensional and even higher-dimensional nearest neighbor searches or other operations, resulting in processing speeds that are typically very slow.
The two methods adopted at present have the defects of computational efficiency in practical application, and cannot meet the requirements of rapid data processing and efficient prediction in application.
Although projection-based methods have clear advantages in terms of computational efficiency, they rely too much on RGB color information. However, in many cases, color information of the point cloud data is missing or locally distorted, and therefore, it is difficult for the current projection-based method to extract clear geometric features of the target object from the three-dimensional point cloud data.
In order to solve the problem that the projection method is difficult to extract clear geometric features of a target object in three-dimensional point cloud data, the embodiment of the application provides a point cloud image enhancement method and device.
The present application takes the execution subject of each embodiment as an example of an electronic device, where the electronic device may be a terminal or a server with data processing capability, and the terminal may be a mobile phone, a computer, a tablet computer, a wearable device, and the like.
Fig. 1 is a flowchart of a point cloud image enhancement method according to an embodiment of the present application, where an enhanced image obtained by the image enhancement algorithm according to the embodiment is used to extract a spatial geometric feature of corresponding point cloud data from a two-dimensional image obtained by three-dimensional point cloud data projection.
The image enhancement method according to the embodiment of the present application will be described in detail below. As shown in fig. 1, the image enhancement method of the present embodiment includes:
s101: and projecting the acquired three-dimensional point cloud data into a two-dimensional image.
Specifically, for example, in a semantic segmentation application scenario, the three-dimensional point cloud data may be projected as a multi-channel two-dimensional panorama, and the image corresponding to each image channel includes an RGB map, a reflection intensity map, an elevation map, and a depth map.
In order to avoid over-dependence on the RGB color information, the present embodiment performs image enhancement processing on the elevation map and the depth map thereof to implement semantic segmentation.
In this embodiment, an enhancement of an elevation map is taken as an example for explanation, fig. 2 is an RGB map obtained by three-dimensional point cloud data projection provided in an embodiment of the present application, and fig. 3 is a corresponding elevation map.
The projection of the three-dimensional point cloud data is converted into a two-dimensional image, which belongs to the well-known technology in the field and is not described herein again.
S102: the two-dimensional image is divided into a number of sub-regions.
Specifically, in order to solve the problem that the existing algorithm excessively depends on RGB color information, and objects in the elevation map and the depth map appear very fuzzy, and geometric features cannot be effectively extracted, the embodiment divides the elevation map and/or the depth map of the corresponding image channel into a plurality of sub-regions by using a grid division method, and performs enhancement processing on the image of each sub-region respectively.
Optionally, the sub-regions of this embodiment are square sub-regions with the same size. By dividing into square sub-regions of equal size, the image enhancement process is facilitated.
Optionally, if the present embodiment is divided into square sub-regions (for example, the boxes shown in fig. 2 and fig. 3), the size of each sub-region may be 1/25-1/100 of the two-dimensional image.
Of course, the sub-area can be divided into sub-areas with any shapes according to actual requirements, for example, the sub-areas can be divided into rectangular sub-areas with the same size.
The present embodiment does not limit the specific shape of the sub-region.
S103: and converting the image of each subregion into a gray level histogram, and carrying out equalization processing on each gray level histogram to realize image enhancement of each subregion so as to obtain an enhanced two-dimensional image.
Specifically, the enhanced two-dimensional image of the present embodiment is used for extracting spatial geometric features.
Taking an elevation map as an example, as shown in fig. 12, a gray level histogram of a sub-area corresponding to the box in fig. 3 is given, because the variation of the elevation value in the sub-area is usually very small compared to the span range of the elevation value of the whole point cloud data. Therefore, the elevation values in the sub-area are highly concentrated around a specific value, and local features cannot be represented.
In view of the above problems and considering the fact that the convolutional neural network is good at learning local features rather than global features, the present embodiment performs equalization processing on the gray level histogram of the image in each sub-region to enhance the image in each sub-region, so as to obtain an enhanced elevation map and/or an enhanced depth map.
FIG. 4 is a partial elevation map (obtained by enhancing sub-regions corresponding to the boxes in FIG. 3) after enhancement processing according to an embodiment of the present application, where the partial elevation map includes many sharp geometric detail features, FIG. 5 is a partial RGB map corresponding to the boxes in FIG. 2, and many detail features in the partial elevation map after enhancement processing in FIG. 4 are richer and clearer than those in the partial RGB image shown in FIG. 5.
Optionally, the scheme adopted for performing equalization processing in the embodiment of the present application is as follows: and adjusting each gray level histogram to conform to Rayleigh distribution.
The rayleigh distribution means that when two components of a random two-dimensional vector are in a normal distribution with an independent mean value of 0 and the same variance, the mode of the vector is rayleigh distribution. The expression of the rayleigh distribution is:
as a possible implementation, the embodiment takes σ in the rayleigh distribution to 0.4, so that the expectation of the changed image gray value is 0.5.
In this embodiment, the above rayleigh distribution function is used to adjust the gray level histogram of each sub-region so that the gray level histogram conforms to rayleigh distribution, thereby implementing equalization processing. Fig. 13 shows the result of the equalization process performed on the gradation histogram shown in fig. 12, and it can be seen from fig. 13 that the gradation values are distributed over a wide span range, and the equalization is performed.
Of course, in this embodiment, the gray histogram may be adjusted to another specific distribution as long as the gray value can be equalized, and the specific equalization processing manner is not limited in this embodiment.
In this embodiment, the obtained enhanced elevation map and/or enhanced depth map replaces an RGB image, and a two-dimensional image based on an image channel combination of the enhanced elevation map and/or enhanced depth map and a reflection intensity map is used as input data of a convolutional neural network for implementing semantic segmentation, so that a better semantic segmentation effect is achieved.
Besides the application scene of semantic segmentation, the image enhancement method in the above embodiment may also be used in other application scenes that need to extract spatial geometric feature information. Of course, in other application scenarios, the three-dimensional point cloud data can be projected into different types of two-dimensional images according to needs.
Converting the two-dimensional image into a gray histogram is well known in the art and will not be described herein.
FIG. 6 is an overall elevation map after local enhancement processing according to the present application. The enhanced elevation map is obtained by dividing the original elevation map into sub-regions that do not overlap with each other and performing local enhancement according to the above embodiment.
As can be seen from fig. 6, the gray values of the neighboring subregions at the boundary abruptly change. This is because the point cloud data in neighboring sub-regions usually have an overall trend of change.
In order to solve the above technical problem, optionally, when the sub-regions are divided in the embodiment of the present application, every two adjacent sub-regions are partially overlapped in space.
In this embodiment, after adjusting the gray level histogram corresponding to the image in each of the sub-regions to make the gray level histogram conform to the rayleigh distribution, the method further includes:
the pixel values of the overlapping portions of the adjacent sub-areas are taken as the pixel average value.
The resulting image after image enhancement by dividing the overlapping sub-regions is shown in fig. 7. With the benefit of the rayleigh distribution employed, only a small number of pixel values in each sub-region are in the vicinity of the extremum (0 or 1).
Optionally, for each sub-region with the same size, the size of the overlapping portion of the adjacent sub-regions in the embodiment can be set to be 1/4-1/64 of the size of the sub-region. This eliminates abrupt and discontinuous gray scale values at the boundary between adjacent sub-regions.
The following describes an image enhancement method according to an embodiment of the present application, taking an example in which the image enhancement method according to the above embodiment is applied to a semantic segmentation scene.
First, the acquired 15 point cloud data are projected into 15 original high-resolution two-dimensional multi-channel panoramas of 7200 × 3600. The multiple channels include a reflection intensity map, an RGB map, an elevation map and a depth map. Then, through S102 to S103 of the image enhancement method in the above embodiment of the present application, image enhancement is performed on the elevation map and the depth map (the size of the divided sub-region is 7200/50-144, and the image of the overlapping portion of the adjacent sub-regions is 144/8-18 pixels), so as to obtain two new image channels: an enhanced elevation map and an enhanced depth map.
This embodiment utilizes 5 convolutional neural networks of different encoder-decoder structures to perform semantic segmentation on a two-dimensional multi-channel panorama based on an image channel combination (channel combination 1) of a reflection intensity map and an RGB map, and a two-dimensional multi-channel panorama based on a data channel combination (channel combination 2) of a reflection intensity map, an enhanced elevation map, and an enhanced depth map, respectively.
The decoders of the 5 convolutional neural networks all adopt a DeepLab v3+ structure, and the encoders adopt 5 different backbone networks, namely ResNet18, ResNet50, MobileNet V2, Xceptation and inclusion-ResNet-v 2. Table 1 shows the result of semantic segmentation accuracy of two-dimensional images under the combination of two image channels by 5 different convolutional neural networks.
TABLE 1
As can be seen from table 1, the accuracy of semantic segmentation obtained by performing semantic segmentation on the two-dimensional panorama based on the image channel combination (channel combination 2) of the reflection intensity map, the enhanced elevation map, and the enhanced depth map is better than that obtained by performing semantic segmentation on the two-dimensional panorama based on the data channel combination (channel combination 1) of the reflection intensity map and the RGB map.
To sum up, the image enhancement method provided by the application divides a two-dimensional image obtained by projecting three-dimensional point cloud data into a plurality of sub-areas, partially overlaps adjacent sub-areas, adjusts each sub-area to conform to rayleigh distribution, and further averages pixel values of the overlapped part, so that the image of each sub-area is enhanced, and finally the enhanced two-dimensional image is obtained. Therefore, the image enhancement method can enable local geometric features in the three-dimensional point cloud data to be clearly shown through the two-dimensional image, and the enhanced two-dimensional image can be used for replacing an RGB image and has a better semantic segmentation effect when being used for semantic segmentation.
The embodiment of the present application further provides a semantic segmentation method based on three-dimensional point cloud data, as shown in fig. 8, the semantic segmentation method of the embodiment includes:
s401: projecting the three-dimensional point cloud data into a two-dimensional multi-channel panoramic image, wherein the multi-channel corresponding image comprises a reflection intensity image, an elevation image and a depth image;
s402: performing image enhancement on the elevation map and the depth map to obtain an enhanced elevation map and a enhanced depth map;
and performing image enhancement on the elevation map and the depth map by adopting the image enhancement method in the embodiment of the image enhancement method.
S403: obtaining an enhanced multi-channel panoramic image based on the enhanced high-level image, the enhanced depth image and the corresponding reflection intensity image;
s404: and performing semantic segmentation on the enhanced multi-channel panoramic image by adopting a pre-trained convolutional neural network.
Please refer to the related description of the above embodiments of the image enhancement method in steps S401 to S403 of this embodiment, which is not described herein again.
In step S404, the semantic segmentation may adopt an existing semantic segmentation method, which belongs to the well-known technology in the art and is not described herein again.
Fig. 9 is a schematic structural diagram of an image enhancement apparatus according to another embodiment of the present application, as shown in fig. 9, the image enhancement apparatus includes:
the projection module is used for projecting the three-dimensional point cloud data into a two-dimensional image;
the area dividing module is used for dividing the two-dimensional image into a plurality of sub-areas;
and the region enhancement module is used for adjusting the gray level histogram corresponding to the image in each sub-region so as to enable the gray level histogram to conform to Rayleigh distribution, so that the image enhancement of each sub-region is realized, and the enhanced two-dimensional image is obtained.
The image enhancement device provided by the above embodiment and the corresponding image enhancement method embodiment belong to the same concept, and the specific implementation process thereof is described in the method embodiment, and is not described herein again.
Fig. 10 is a schematic structural diagram of a semantic segmentation apparatus according to another embodiment of the present application, and as shown in fig. 10, the semantic segmentation apparatus includes:
the image conversion module is used for projecting the three-dimensional point cloud data into a two-dimensional multi-channel panoramic image, wherein the images corresponding to multiple channels comprise a reflection intensity map, an elevation map and a depth map;
the image enhancement module is used for carrying out image enhancement on the elevation map and the depth map by adopting the image enhancement method in the embodiment of the image enhancement method to obtain an enhanced elevation map and an enhanced depth map;
the channel combination module is used for obtaining an enhanced multi-channel panoramic image based on the enhanced high-level image, the enhanced depth image and the corresponding reflection intensity image;
and the semantic segmentation module is used for performing semantic segmentation on the enhanced multi-channel panoramic image by adopting a pre-trained convolutional neural network.
The semantic segmentation device provided by the above embodiment and the corresponding semantic segmentation method embodiment belong to the same concept, and the specific implementation process thereof is described in the method embodiment, and will not be described herein again.
It should be noted that: the image enhancement device and the semantic segmentation device provided in the above embodiments are only exemplified by the division of each functional module in the above corresponding method embodiments, and in practical applications, the above functions may be distributed by different functional modules according to needs, that is, the internal structure of the image enhancement device/the semantic segmentation device is divided into different functional modules to complete all or part of the above described functions.
Fig. 11 is a block diagram of an electronic device according to an embodiment of the present application, where the electronic device may be a computing device such as a desktop computer, a notebook computer, a palm computer, and a cloud server, and the computing device may include, but is not limited to, a processor and a memory. Wherein the content of the first and second substances,
the processor may include one or more processing cores, such as: 4 core processors, 6 core processors, etc. The processor may be implemented in at least one hardware form of a DSP (Digital Signal Processing), an FPGA (Field-Programmable Gate Array), and a PLA (Programmable Logic Array). The processor may also include a main processor and a coprocessor, where the main processor is a processor for Processing data in an awake state, and is also called a Central Processing Unit (CPU); a coprocessor is a low power processor for processing data in a standby state. In some embodiments, the processor may be integrated with a GPU (Graphics Processing Unit), which is responsible for rendering and drawing the content that the display screen needs to display. In some embodiments, the processor may further include an AI (Artificial Intelligence) processor for processing computing operations related to machine learning. The processor is the control center of the computer equipment and is connected with all parts of the whole computer equipment by various interfaces and lines.
The memory may include high speed random access memory, and may also include non-volatile memory, such as a hard disk, a memory, a plug-in hard disk, a Smart Media Card (SMC), a Secure Digital (SD) Card, a Flash memory Card (Flash Card), at least one magnetic disk storage device, a memory device, or other volatile solid state storage device.
The memory has stored thereon a computer program which is executable on the processor, and the processor, when executing the computer program, implements all or part of the implementation steps in the embodiments related to the image enhancement method or semantic segmentation method of the present application, and/or other contents described in the text.
Those skilled in the art will appreciate that fig. 11 is only one possible implementation manner of the embodiments of the present application, and other embodiments may include more or less components, or combine some components, or different components, and the present embodiment is not limited thereto.
Optionally, the present application further provides a computer-readable storage medium, which stores a computer program, and the computer program is used for implementing the steps of the image enhancement method or the semantic segmentation method in any one of the above embodiments or implementations when being executed by a processor.
Optionally, the present application further provides a computer program product, which includes a computer readable storage medium, in which a program is stored, and the program is loaded and executed by a processor to implement the steps of the image enhancement method or the semantic segmentation method in any of the above embodiments or implementations.
The technical features of the embodiments described above may be arbitrarily combined, and for the sake of brevity, all possible combinations of the technical features in the embodiments described above are not described, but should be considered as being within the scope of the present specification as long as there is no contradiction between the combinations of the technical features.
The above-mentioned embodiments only express several embodiments of the present application, and the description thereof is more specific and detailed, but not construed as limiting the scope of the invention. It should be noted that, for a person skilled in the art, several variations and modifications can be made without departing from the concept of the present application, which falls within the scope of protection of the present application. Therefore, the protection scope of the present patent shall be subject to the appended claims.
Claims (11)
1. A method for enhancing an image of three-dimensional point cloud data, the method comprising:
projecting the acquired three-dimensional point cloud data into a two-dimensional image;
dividing the two-dimensional image into a plurality of sub-regions;
and converting the image of each subregion into a gray level histogram, and carrying out equalization processing on each gray level histogram to realize image enhancement on each subregion so as to obtain an enhanced two-dimensional image, wherein the enhanced two-dimensional image is used for extracting space geometric features.
2. The method according to claim 1, wherein the equalizing each of the gray level histograms includes:
and adjusting each gray level histogram to conform to Rayleigh distribution.
3. The method according to claim 1, wherein the two-dimensional image is divided into a plurality of sub-regions, each two adjacent sub-regions are partially overlapped in space, and after the converting the image of each sub-region into the gray histogram and performing the equalization process on each gray histogram, the method further comprises:
the pixel values of the overlapping portions of the adjacent sub-areas are taken as the pixel average value.
4. The method of claim 3, wherein the overlapping portions of adjacent sub-areas are 1/4-1/64 of the size of the sub-areas.
5. A method according to any one of claims 1 to 4, wherein each of said sub-regions is a square sub-region of the same size.
6. The method of claim 5, wherein the side length of the sub-region of the square is 1/25-1/100 of the long side of the two-dimensional image.
7. A semantic segmentation method for three-dimensional point cloud data is characterized by comprising the following steps:
projecting the three-dimensional point cloud data into a two-dimensional multi-channel panoramic image, wherein the multi-channel corresponding image comprises a reflection intensity image, an elevation image and a depth image;
performing image enhancement on the elevation map and the depth map by using the image enhancement method according to any one of claims 1 to 6 to obtain an enhanced elevation map and a enhanced depth map;
obtaining an enhanced multi-channel panoramic image based on the enhanced high-level image, the enhanced depth image and the corresponding reflection intensity image;
and performing semantic segmentation on the enhanced multi-channel panoramic image by adopting a pre-trained convolutional neural network.
8. An apparatus for enhancing an image of three-dimensional point cloud data, the apparatus comprising:
the projection module is used for projecting the three-dimensional point cloud data into a two-dimensional image;
the area dividing module is used for dividing the two-dimensional image into a plurality of sub-areas;
and the region enhancement module is used for converting the image of each subregion into a gray level histogram and carrying out equalization processing on each gray level histogram to realize the image enhancement of each subregion so as to obtain an enhanced two-dimensional image, and the enhanced two-dimensional image is used for extracting the space geometric features.
9. A semantic segmentation device for three-dimensional point cloud data is characterized by comprising:
the image conversion module is used for projecting the three-dimensional point cloud data into a two-dimensional multi-channel panoramic image, wherein the images corresponding to multiple channels comprise a reflection intensity map, an elevation map and a depth map;
an image enhancement module, configured to perform image enhancement on the elevation map and the depth map by using the image enhancement method according to any one of claims 1 to 6, so as to obtain an enhanced elevation map and enhanced depth map;
the channel combination module is used for obtaining an enhanced multi-channel panoramic image based on the enhanced high-level image, the enhanced depth image and the corresponding reflection intensity image;
and the semantic segmentation module is used for performing semantic segmentation on the enhanced multi-channel panoramic image by adopting a pre-trained convolutional neural network.
10. An electronic device comprising a processor and a memory, said memory storing a computer program, wherein said computer program, when executed by the processor, is adapted to carry out the steps of the image enhancement method according to any one of claims 1 to 6 and the semantic segmentation method according to claim 7.
11. A computer-readable storage medium, in which a computer program is stored which, when being executed by a processor, is adapted to carry out the steps of the image enhancement method according to any one of claims 1 to 6 and the semantic segmentation method according to claim 7.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110642257.7A CN113506305B (en) | 2021-06-09 | 2021-06-09 | Image enhancement method, semantic segmentation method and device for three-dimensional point cloud data |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110642257.7A CN113506305B (en) | 2021-06-09 | 2021-06-09 | Image enhancement method, semantic segmentation method and device for three-dimensional point cloud data |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN113506305A true CN113506305A (en) | 2021-10-15 |
| CN113506305B CN113506305B (en) | 2023-10-24 |
Family
ID=78009567
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202110642257.7A Active CN113506305B (en) | 2021-06-09 | 2021-06-09 | Image enhancement method, semantic segmentation method and device for three-dimensional point cloud data |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN113506305B (en) |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2023078686A1 (en) * | 2021-11-05 | 2023-05-11 | Robert Bosch Gmbh | Method and device for performing instance segmentation of picture |
| CN117392038A (en) * | 2023-12-05 | 2024-01-12 | 北京智源人工智能研究院 | Medical image histogram equalization method and device, electronic equipment and storage medium |
Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN105654438A (en) * | 2015-12-27 | 2016-06-08 | 西南技术物理研究所 | Gray scale image fitting enhancement method based on local histogram equalization |
| CN107527329A (en) * | 2017-09-19 | 2017-12-29 | 重庆邮电大学 | A kind of image defogging method based on dark channel prior and Markov random field |
| CN110570429A (en) * | 2019-08-30 | 2019-12-13 | 华南理工大学 | Lightweight real-time semantic segmentation method based on three-dimensional point cloud |
| US10650278B1 (en) * | 2017-07-21 | 2020-05-12 | Apple Inc. | Semantic labeling of point clouds using images |
| CN111652085A (en) * | 2020-05-14 | 2020-09-11 | 东莞理工学院 | Object identification method based on combination of 2D and 3D features |
| CN112818925A (en) * | 2021-02-26 | 2021-05-18 | 清华大学 | Urban building and crown identification method |
-
2021
- 2021-06-09 CN CN202110642257.7A patent/CN113506305B/en active Active
Patent Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN105654438A (en) * | 2015-12-27 | 2016-06-08 | 西南技术物理研究所 | Gray scale image fitting enhancement method based on local histogram equalization |
| US10650278B1 (en) * | 2017-07-21 | 2020-05-12 | Apple Inc. | Semantic labeling of point clouds using images |
| CN107527329A (en) * | 2017-09-19 | 2017-12-29 | 重庆邮电大学 | A kind of image defogging method based on dark channel prior and Markov random field |
| CN110570429A (en) * | 2019-08-30 | 2019-12-13 | 华南理工大学 | Lightweight real-time semantic segmentation method based on three-dimensional point cloud |
| CN111652085A (en) * | 2020-05-14 | 2020-09-11 | 东莞理工学院 | Object identification method based on combination of 2D and 3D features |
| CN112818925A (en) * | 2021-02-26 | 2021-05-18 | 清华大学 | Urban building and crown identification method |
Cited By (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2023078686A1 (en) * | 2021-11-05 | 2023-05-11 | Robert Bosch Gmbh | Method and device for performing instance segmentation of picture |
| CN117392038A (en) * | 2023-12-05 | 2024-01-12 | 北京智源人工智能研究院 | Medical image histogram equalization method and device, electronic equipment and storage medium |
| CN117392038B (en) * | 2023-12-05 | 2024-03-08 | 北京智源人工智能研究院 | Medical image histogram equalization method and device, electronic equipment and storage medium |
Also Published As
| Publication number | Publication date |
|---|---|
| CN113506305B (en) | 2023-10-24 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US10861133B1 (en) | Super-resolution video reconstruction method, device, apparatus and computer-readable storage medium | |
| GB2544590B (en) | Image synthesis utilizing an active mask | |
| US8494297B2 (en) | Automatic detection and mapping of symmetries in an image | |
| US7149355B2 (en) | Image processing apparatus, image processing method, image processing program, and computer-readable record medium storing image processing program | |
| CN106485720A (en) | Image processing method and device | |
| CN113506305B (en) | Image enhancement method, semantic segmentation method and device for three-dimensional point cloud data | |
| CN107564063B (en) | Virtual object display method and device based on convolutional neural network | |
| CN113643414B (en) | Three-dimensional image generation method and device, electronic equipment and storage medium | |
| CN111598777A (en) | Sky cloud image processing method, computer device and readable storage medium | |
| CN111353955A (en) | Image processing method, device, equipment and storage medium | |
| Patel et al. | Accelerated seam carving for image retargeting | |
| CN111563517A (en) | Image processing method, image processing device, electronic equipment and storage medium | |
| CN114359048A (en) | Image data enhancement method and device, terminal equipment and storage medium | |
| CN108734712B (en) | Background segmentation method and device and computer storage medium | |
| Hsin | Combination of saliency histogram equalisation and seam carving for image resizing | |
| CN106056575B (en) | A kind of image matching method based on like physical property proposed algorithm | |
| CN113538502A (en) | Picture clipping method and device, electronic equipment and storage medium | |
| CN113256484A (en) | Method and device for stylizing image | |
| CN114821216A (en) | Method for modeling and using picture descreening neural network model and related equipment | |
| CN111667499A (en) | Image segmentation method, device and equipment for traffic signal lamp and storage medium | |
| Ahmed et al. | Digital image inpainting techniques for cultural heritage preservation and restoration | |
| CN113570518B (en) | Image correction method, system, computer equipment and storage medium | |
| CN112237002A (en) | Image processing method and apparatus | |
| CN115330803B (en) | Surface defect data enhancement method and device, electronic equipment and storage medium | |
| Hsin | Compressed domain image resizing based on saliency histogram equalisation and seam carving |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |