CN113744256A - Depth map hole filling method and device, server and readable storage medium - Google Patents

Depth map hole filling method and device, server and readable storage medium Download PDF

Info

Publication number
CN113744256A
CN113744256A CN202111053968.7A CN202111053968A CN113744256A CN 113744256 A CN113744256 A CN 113744256A CN 202111053968 A CN202111053968 A CN 202111053968A CN 113744256 A CN113744256 A CN 113744256A
Authority
CN
China
Prior art keywords
image
processed
depth
hole
depth image
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Pending
Application number
CN202111053968.7A
Other languages
Chinese (zh)
Inventor
崔岩
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
China Germany Zhuhai Artificial Intelligence Institute Co ltd
4Dage Co Ltd
Original Assignee
China Germany Zhuhai Artificial Intelligence Institute Co ltd
4Dage Co Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by China Germany Zhuhai Artificial Intelligence Institute Co ltd, 4Dage Co Ltd filed Critical China Germany Zhuhai Artificial Intelligence Institute Co ltd
Priority to CN202111053968.7A priority Critical patent/CN113744256A/en
Publication of CN113744256A publication Critical patent/CN113744256A/en
Pending legal-status Critical Current

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/0002—Inspection of images, e.g. flaw detection
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/30—Determination of transform parameters for the alignment of images, i.e. image registration
    • G06T7/33—Determination of transform parameters for the alignment of images, i.e. image registration using feature-based methods
    • G06T7/337—Determination of transform parameters for the alignment of images, i.e. image registration using feature-based methods involving reference images or patches
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/90—Determination of colour characteristics
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/10—Image acquisition modality
    • G06T2207/10024—Color image

Abstract

The application is applicable to the technical field of spatial data processing, and provides a depth map hole filling method, a depth map hole filling device, a server and a readable storage medium, wherein the method comprises the following steps: acquiring an RGB-D image, wherein the RGB-D image comprises a depth image to be processed and a color image corresponding to the depth image to be processed; identifying the type of a cavity region of the depth image to be processed; and based on the color image, performing hole filling processing on the depth image to be processed according to a hole filling algorithm corresponding to the type of the hole area to obtain a target depth image. Therefore, the color image corresponding to the depth image is used as the guide image, the hole of the depth image is filled according to the pixel information of the color image, the hole filling accuracy is improved, and the depth information of the depth image is prevented from being lost.

Description

Depth map hole filling method and device, server and readable storage medium
Technical Field
The application belongs to the technical field of spatial data processing, and particularly relates to a depth map hole filling method and device, a server and a readable storage medium.
Background
The depth map is a three-dimensional scene information expression mode, values of the depth map represent the distance from a depth detector to an object, and the larger the value, the farther the distance is. However, in the acquisition process of the depth map, due to the limitation of hardware conditions of the device (such as low acquisition accuracy and limited acquisition distance) and the interference of external environmental factors, the acquired depth map has depth information deficiency of different degrees. The locations where such depth information is missing usually appear as image holes. Therefore, a depth map hole filling method is provided.
Disclosure of Invention
The embodiment of the application provides a depth map hole filling method, a depth map hole filling device, a server and a readable storage medium, and can solve the problem that depth information of an obtained depth map is missing in the prior art.
In a first aspect, an embodiment of the present application provides a depth map hole filling method, including:
acquiring an RGB-D image, wherein the RGB-D image comprises a depth image to be processed and a color image corresponding to the depth image to be processed;
identifying the type of a cavity region of the depth image to be processed;
and based on the color image, carrying out hole filling processing on the depth image to be processed according to a hole filling algorithm corresponding to the type of the hole area to obtain a target depth image.
In one possible implementation manner of the first aspect, acquiring an RGB-D image includes:
acquiring a color image;
acquiring a depth image to be processed;
and registering the color image and the depth image to be processed to obtain an RGB-D image.
In a possible implementation manner of the first aspect, acquiring a depth image to be processed includes:
acquiring a to-be-processed three-dimensional panoramic image;
and inputting the three-dimensional panoramic image to be processed into a pre-trained depth estimation model, and outputting the depth image to be processed.
In a possible implementation manner of the first aspect, the hole region type includes an edge hole sub-region and a non-edge hole sub-region;
identifying the type of the hole area of the depth image to be processed, including:
inputting the depth image to be processed into a preset classifier, and determining a cavity region in the depth image to be processed;
and determining an edge hole sub-region and a non-edge hole sub-region in the hole region by adopting an edge detection algorithm.
In a possible implementation manner of the first aspect, the hole filling algorithm includes a first hole filling algorithm and a second hole filling algorithm;
based on the color image, performing hole filling processing on the depth image to be processed according to a hole filling algorithm corresponding to the type of the hole area to obtain a target depth image, including:
segmenting an edge area and a non-edge area in the color image;
calling a first hole filling algorithm corresponding to a non-edge hole sub-region, and performing first hole filling on the non-edge hole sub-region based on the non-edge region in the color image;
calling a second hole filling algorithm corresponding to the edge hole sub-region, and performing second hole filling on the edge hole sub-region based on a non-edge region in the color image;
and taking the depth image to be processed filled by the first hole and the second hole as a target depth image.
In a second aspect, an embodiment of the present application provides a depth map hole filling apparatus, including:
the system comprises an acquisition module, a processing module and a display module, wherein the acquisition module is used for acquiring an RGB-D image, and the RGB-D image comprises a depth image to be processed and a color image corresponding to the depth image to be processed;
the identification module is used for identifying the type of the cavity region of the depth image to be processed;
and the filling processing module is used for carrying out cavity filling processing on the depth image to be processed according to a cavity filling algorithm corresponding to the type of the cavity region based on the color image to obtain a target depth image.
In one possible implementation manner, the obtaining module includes:
the first acquisition sub-module is used for acquiring a color image;
the second acquisition submodule is used for acquiring a depth image to be processed;
and the registration submodule is used for registering the color image and the depth image to be processed to obtain an RGB-D image.
In one possible implementation manner, the second obtaining sub-module includes:
the device comprises an acquisition unit, a processing unit and a processing unit, wherein the acquisition unit is used for acquiring a to-be-processed three-dimensional panoramic image;
and the depth estimation unit is used for inputting the three-dimensional panoramic image to be processed into a depth estimation model trained in advance and outputting the depth image to be processed.
In one possible implementation manner, the hole region type includes an edge hole sub-region and a non-edge hole sub-region;
the identification module comprises:
inputting the depth image to be processed into a preset classifier, and determining a cavity region in the depth image to be processed;
and determining an edge hole sub-region and a non-edge hole sub-region in the hole region by adopting an edge detection algorithm.
In one possible implementation, the hole filling algorithm includes a first hole filling algorithm and a second hole filling algorithm;
the filling processing module comprises:
the segmentation submodule is used for segmenting an edge area and a non-edge area in the color image;
the first filling sub-module is used for calling a first hole filling algorithm corresponding to a non-edge hole sub-region and carrying out first hole filling on the non-edge hole sub-region based on the non-edge region in the color image;
the second filling submodule is used for calling a second cavity filling algorithm corresponding to the edge cavity sub-area and carrying out second cavity filling on the edge cavity sub-area based on the edge area in the color image;
and the determining submodule is used for taking the depth image to be processed which is filled by the first cavity and the second cavity as a target depth image.
In a third aspect, an embodiment of the present application provides a server, including a memory, a processor, and a computer program stored in the memory and executable on the processor, where the processor, when executing the computer program, implements the method according to the first aspect.
In a fourth aspect, the present application provides a readable storage medium, in which a computer program is stored,
it is understood that the beneficial effects of the second aspect to the fifth aspect can be referred to the related description of the first aspect, and are not described herein again.
Compared with the prior art, the embodiment of the application has the advantages that:
in the embodiment of the application, an RGB-D image is obtained, wherein the RGB-D image comprises a depth image to be processed and a color image corresponding to the depth image to be processed; identifying the type of a cavity region of the depth image to be processed; and based on the color image, carrying out hole filling processing on the depth image to be processed according to a hole filling algorithm corresponding to the type of the hole area to obtain a target depth image. Therefore, the color image corresponding to the depth image is used as the guide image, the hole of the depth image is filled according to the pixel information of the color image, the hole filling accuracy is improved, and the depth information of the depth image is prevented from being lost.
Drawings
In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings needed to be used in the embodiments or the prior art descriptions will be briefly described below, and it is obvious that the drawings in the following description are only some embodiments of the present application, and it is obvious for those skilled in the art to obtain other drawings based on these drawings without inventive exercise.
Fig. 1 is a schematic flowchart of a depth map hole filling method according to an embodiment of the present disclosure;
fig. 2 is a schematic flowchart illustrating a specific process of step S102 in fig. 1 of a depth map hole filling method according to an embodiment of the present disclosure;
fig. 3 is a schematic flowchart illustrating a specific process of step S204 in fig. 2 of a depth map hole filling method according to an embodiment of the present application;
fig. 4 is a schematic flowchart illustrating a specific process of step S104 in fig. 1 of a depth map hole filling method according to an embodiment of the present application;
fig. 5 is a schematic flowchart illustrating a specific process of step S106 in fig. 1 of a depth map hole filling method according to an embodiment of the present application;
FIG. 6 is a schematic structural diagram of a depth map cavity filling device according to an embodiment of the present disclosure;
fig. 7 is a schematic structural diagram of a server according to an embodiment of the present application.
Detailed Description
In the following description, for purposes of explanation and not limitation, specific details are set forth, such as particular system structures, techniques, etc. in order to provide a thorough understanding of the embodiments of the present application. It will be apparent, however, to one skilled in the art that the present application may be practiced in other embodiments that depart from these specific details. In other instances, detailed descriptions of well-known systems, devices, circuits, and methods are omitted so as not to obscure the description of the present application with unnecessary detail.
It will be understood that the terms "comprises" and/or "comprising," when used in this specification and the appended claims, specify the presence of stated features, integers, steps, operations, elements, and/or components, but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and/or groups thereof.
It should also be understood that the term "and/or" as used in this specification and the appended claims refers to and includes any and all possible combinations of one or more of the associated listed items.
As used in this specification and the appended claims, the term "if" may be interpreted contextually as "when", "upon" or "in response to" determining "or" in response to detecting ". Similarly, the phrase "if it is determined" or "if a [ described condition or event ] is detected" may be interpreted contextually to mean "upon determining" or "in response to determining" or "upon detecting [ described condition or event ]" or "in response to detecting [ described condition or event ]".
Furthermore, in the description of the present application and the appended claims, the terms "first," "second," "third," and the like are used for distinguishing between descriptions and not necessarily for describing or implying relative importance.
Reference throughout this specification to "one embodiment" or "some embodiments," or the like, means that a particular feature, structure, or characteristic described in connection with the embodiment is included in one or more embodiments of the present application. Thus, appearances of the phrases "in one embodiment," "in some embodiments," "in other embodiments," or the like, in various places throughout this specification are not necessarily all referring to the same embodiment, but rather "one or more but not all embodiments" unless specifically stated otherwise. The terms "comprising," "including," "having," and variations thereof mean "including, but not limited to," unless expressly specified otherwise.
Referring to fig. 1, a flowchart of a depth map hole filling method provided in this embodiment of the present application is schematically illustrated, by way of example and not limitation, the method may be applied to a server including, but not limited to, a computing device such as a cloud server, the server being connected to a depth camera and a dome camera in a calibration relationship, respectively, where the depth map hole filling method may include the following steps:
and step S102, acquiring an RGB-D image.
The RGB-D image comprises a depth image to be processed and a color image corresponding to the depth image to be processed. It is understood that an RGB image refers to an image having three color channels of red, green, and blue to describe the appearance, color, and texture of scene objects. The grayscale image, each pixel value of which is the actual distance of the sensor from the scene object, i.e., the depth value, to describe the shape, dimension, and geometric space of the scene object.
In a specific application, as shown in fig. 2, for a specific flowchart schematic diagram of step S102 in fig. 1 of the depth map hole filling method provided in the embodiment of the present application, acquiring an RGB-D image includes:
step S202, color images are acquired.
In a specific application, a color image (i.e., an RGB image) shot by a dome camera is collected.
And step S204, acquiring a depth image to be processed.
Exemplarily, as shown in fig. 3, a specific flowchart of step S204 in fig. 2 of the depth map hole filling method provided in the embodiment of the present application is schematically illustrated, and acquiring a depth image to be processed includes:
and S302, acquiring a to-be-processed three-dimensional panoramic image.
In specific application, a to-be-processed three-dimensional panoramic image shot by a depth camera is collected. The depth camera can be an eight-eye camera which consists of an upper group of four fisheye lenses and a lower group of four fisheye lenses, and the four lenses collect four groups of lens images respectively and are spliced into a 360-degree panoramic image.
And S304, inputting the three-dimensional panoramic image to be processed into a depth estimation model trained in advance, and outputting the depth image to be processed.
The pre-trained depth estimation model is a full convolution neural network and is obtained by training an open source data set.
And S206, registering the color image and the depth image to be processed to obtain an RGB-D image.
It can be understood that the purpose of registration is to combine the color image and the depth image to be processed, that is, to convert the image coordinate system of the color image into the image coordinate system of the depth image to be processed, so as to obtain an image containing depth information and color information, that is, an RGB-D image. The depth points in the color image correspond to the depth points in the depth image to be processed one by one.
And step S104, identifying the type of the cavity area of the depth image to be processed.
The hole area type comprises an edge hole sub-area and a non-edge hole sub-area.
In a specific application, as shown in fig. 4, for a specific flowchart of step S104 in fig. 1 of the depth map hole filling method provided in the embodiment of the present application, identifying a hole region type of a depth image to be processed includes:
and S402, inputting the depth image to be processed into a preset classifier, and determining a cavity area in the depth image to be processed.
The preset classifier can be naive Bayes, logistic regression, K-neighborhood or decision tree algorithm.
And S404, determining an edge hole sub-region and a non-edge hole sub-region in the hole region by adopting an edge detection algorithm.
Wherein, the edge detection algorithm is canny algorithm, sobel algorithm or Marr-Hildreth algorithm.
And S106, based on the color image, performing hole filling processing on the depth image to be processed according to a hole filling algorithm corresponding to the type of the hole area to obtain a target depth image.
The hole filling algorithm comprises a first hole filling algorithm and a second hole filling algorithm.
In a specific application, as shown in fig. 5, for a specific flowchart of step S106 in fig. 1 of the depth map hole filling method provided in the embodiment of the present application, based on a color image, performing hole filling processing on a depth image to be processed according to a hole filling algorithm corresponding to a hole region type to obtain a target depth image, the method includes:
and step S502, dividing the edge area and the non-edge area in the color image.
In the specific application, the SLIC superpixel algorithm is adopted to carry out rough segmentation on the color image according to the pixel information of the color image, and an edge area and a non-edge area in the color image are obtained.
Step S504, a first hole filling algorithm corresponding to the non-edge hole sub-region is called, and first hole filling is carried out on the non-edge hole sub-region based on the non-edge region in the color image.
The first hole filling algorithm is a region direction type combined bilateral filtering algorithm. It can be understood that the non-edge hole sub-area is filled according to the pixel information of the color image by using the corresponding relation between the color image and the depth image to be processed.
And S506, calling a second cavity filling algorithm corresponding to the edge cavity sub-region, and filling a second cavity in the edge cavity sub-region based on the edge region in the color image.
And the second hole filling algorithm is a directional combined bilateral filtering algorithm. It can be understood that, by using the correspondence between the color image and the depth image to be processed, the second hole filling is performed on the edge hole sub-area based on the pixel information of the edge area in the color image.
And step S508, taking the depth image to be processed filled by the first cavity and the second cavity as a target depth image.
It can be understood that different hole filling algorithms are adopted to fill hole areas with different types, so that the effects of saving image edge characteristics and improving filling accuracy are achieved.
In the embodiment of the application, an RGB-D image is obtained, wherein the RGB-D image comprises a depth image to be processed and a color image corresponding to the depth image to be processed; identifying the type of a cavity region of the depth image to be processed; and based on the color image, carrying out hole filling processing on the depth image to be processed according to a hole filling algorithm corresponding to the type of the hole area to obtain a target depth image. Therefore, the color image corresponding to the depth image is used as the guide image, the hole of the depth image is filled according to the pixel information of the color image, the hole filling accuracy is improved, and the depth information of the depth image is prevented from being lost.
It should be understood that, the sequence numbers of the steps in the foregoing embodiments do not imply an execution sequence, and the execution sequence of each process should be determined by its function and inherent logic, and should not constitute any limitation to the implementation process of the embodiments of the present application.
Fig. 6 shows a block diagram of a depth map hole filling apparatus provided in an embodiment of the present application, and only shows portions related to the embodiment of the present application for convenience of description.
Referring to fig. 6, the apparatus includes:
the acquisition module 61 is configured to acquire an RGB-D image, where the RGB-D image includes a depth image to be processed and a color image corresponding to the depth image to be processed;
the identification module 62 is configured to identify a type of a cavity region of the depth image to be processed;
and the filling processing module 63 is configured to perform, based on the color image, a hole filling process on the depth image to be processed according to a hole filling algorithm corresponding to the type of the hole region, so as to obtain a target depth image.
In one possible implementation manner, the obtaining module includes:
the first acquisition sub-module is used for acquiring a color image;
the second acquisition submodule is used for acquiring a depth image to be processed;
and the registration submodule is used for registering the color image and the depth image to be processed to obtain an RGB-D image.
In one possible implementation manner, the second obtaining sub-module includes:
the device comprises an acquisition unit, a processing unit and a processing unit, wherein the acquisition unit is used for acquiring a to-be-processed three-dimensional panoramic image;
and the depth estimation unit is used for inputting the three-dimensional panoramic image to be processed into a depth estimation model trained in advance and outputting the depth image to be processed.
In one possible implementation manner, the hole region type includes an edge hole sub-region and a non-edge hole sub-region;
the identification module comprises:
inputting the depth image to be processed into a preset classifier, and determining a cavity region in the depth image to be processed;
and determining an edge hole sub-region and a non-edge hole sub-region in the hole region by adopting an edge detection algorithm.
In one possible implementation, the hole filling algorithm includes a first hole filling algorithm and a second hole filling algorithm;
the filling processing module comprises:
the segmentation submodule is used for segmenting an edge area and a non-edge area in the color image;
the first filling sub-module is used for calling a first hole filling algorithm corresponding to a non-edge hole sub-region and carrying out first hole filling on the non-edge hole sub-region based on the non-edge region in the color image;
the second filling submodule is used for calling a second cavity filling algorithm corresponding to the edge cavity sub-area and carrying out second cavity filling on the edge cavity sub-area based on the edge area in the color image;
and the determining submodule is used for taking the depth image to be processed which is filled by the first cavity and the second cavity as a target depth image.
It should be noted that, for the information interaction, execution process, and other contents between the above-mentioned devices/units, the specific functions and technical effects thereof are based on the same concept as those of the embodiment of the method of the present application, and specific reference may be made to the part of the embodiment of the method, which is not described herein again.
Fig. 7 is a schematic structural diagram of a server according to an embodiment of the present application. As shown in fig. 7, the server 7 of this embodiment includes: at least one processor 70, a memory 71, and a computer program 72 stored in the memory 71 and executable on the at least one processor 70, the processor 70 implementing the steps in any of the various method embodiments described above when executing the computer program 72.
The server 7 may be a computing device such as a cloud server. The server may include, but is not limited to, a processor 70, a memory 71. Those skilled in the art will appreciate that fig. 7 is merely an example of the server 7, and does not constitute a limitation of the server 7, and may include more or less components than those shown, or combine certain components, or different components, such as input output devices, network access devices, etc.
The Processor 70 may be a Central Processing Unit (CPU), and the Processor 70 may be other general purpose Processor, a Digital Signal Processor (DSP), an Application Specific Integrated Circuit (ASIC), an off-the-shelf Programmable Gate Array (FPGA) or other Programmable logic device, discrete Gate or transistor logic, discrete hardware components, etc. A general purpose processor may be a microprocessor or the processor may be any conventional processor or the like.
The memory 71 may in some embodiments be an internal storage unit of the server 7, such as a hard disk or a memory of the server 7. The memory 71 may also be an external storage device of the server 7 in other embodiments, such as a plug-in hard disk, a Smart Media Card (SMC), a Secure Digital (SD) Card, a Flash memory Card (Flash Card), and the like, which are provided on the server 7. Further, the memory 71 may also include both an internal storage unit and an external storage device of the server 7. The memory 71 is used for storing an operating system, an application program, a BootLoader (BootLoader), data, and other programs, such as program codes of the computer program. The memory 71 may also be used to temporarily store data that has been output or is to be output.
It will be apparent to those skilled in the art that, for convenience and brevity of description, only the above-mentioned division of the functional units and modules is illustrated, and in practical applications, the above-mentioned function distribution may be performed by different functional units and modules according to needs, that is, the internal structure of the apparatus is divided into different functional units or modules to perform all or part of the above-mentioned functions. Each functional unit and module in the embodiments may be integrated in one processing unit, or each unit may exist alone physically, or two or more units are integrated in one unit, and the integrated unit may be implemented in a form of hardware, or in a form of software functional unit. In addition, specific names of the functional units and modules are only for convenience of distinguishing from each other, and are not used for limiting the protection scope of the present application. The specific working processes of the units and modules in the system may refer to the corresponding processes in the foregoing method embodiments, and are not described herein again.
The present application further provides a readable storage medium, which is preferably a computer readable storage medium, and the computer readable storage medium stores a computer program, and the computer program is implemented to implement the steps in the above method embodiments when executed by a processor.
The integrated unit, if implemented in the form of a software functional unit and sold or used as a stand-alone product, may be stored in a computer readable storage medium. Based on such understanding, all or part of the processes in the methods of the embodiments described above can be implemented by a computer program, which can be stored in a computer-readable storage medium and can implement the steps of the embodiments of the methods described above when the computer program is executed by a processor. Wherein the computer program comprises computer program code, which may be in the form of source code, object code, an executable file or some intermediate form, etc. The computer readable medium may include at least: any entity or device capable of carrying computer program code to a server, recording medium, computer Memory, Read-Only Memory (ROM), Random-Access Memory (RAM), electrical carrier wave signals, telecommunications signals, and software distribution medium. Such as a usb-disk, a removable hard disk, a magnetic or optical disk, etc. In certain jurisdictions, computer-readable media may not be an electrical carrier signal or a telecommunications signal in accordance with legislative and patent practice.
In the above embodiments, the descriptions of the respective embodiments have respective emphasis, and reference may be made to the related descriptions of other embodiments for parts that are not described or illustrated in a certain embodiment.
Those of ordinary skill in the art will appreciate that the various illustrative elements and algorithm steps described in connection with the embodiments disclosed herein may be implemented as electronic hardware or combinations of computer software and electronic hardware. Whether such functionality is implemented as hardware or software depends upon the particular application and design constraints imposed on the implementation. Skilled artisans may implement the described functionality in varying ways for each particular application, but such implementation decisions should not be interpreted as causing a departure from the scope of the present application.
In the embodiments provided in the present application, it should be understood that the disclosed apparatus/network device and method may be implemented in other ways. For example, the above-described apparatus/network device embodiments are merely illustrative, and for example, the division of the modules or units is only one logical division, and there may be other divisions when actually implementing, for example, a plurality of units or components may be combined or integrated into another system, or some features may be omitted, or not implemented. In addition, the shown or discussed mutual coupling or direct coupling or communication connection may be an indirect coupling or communication connection through some interfaces, devices or units, and may be in an electrical, mechanical or other form.
The units described as separate parts may or may not be physically separate, and parts displayed as units may or may not be physical units, may be located in one place, or may be distributed on a plurality of network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the solution of the embodiment.
The above-mentioned embodiments are only used for illustrating the technical solutions of the present application, and not for limiting the same; although the present application has been described in detail with reference to the foregoing embodiments, it should be understood by those of ordinary skill in the art that: the technical solutions described in the foregoing embodiments may still be modified, or some technical features may be equivalently replaced; such modifications and substitutions do not substantially depart from the spirit and scope of the embodiments of the present application and are intended to be included within the scope of the present application.

Claims (10)

1. A depth map hole filling method is characterized by comprising the following steps:
acquiring an RGB-D image, wherein the RGB-D image comprises a depth image to be processed and a color image corresponding to the depth image to be processed;
identifying the type of a cavity region of the depth image to be processed;
and based on the color image, carrying out hole filling processing on the depth image to be processed according to a hole filling algorithm corresponding to the type of the hole area to obtain a target depth image.
2. The depth map hole filling method of claim 1, wherein acquiring an RGB-D image comprises:
acquiring a color image;
acquiring a depth image to be processed;
and registering the color image and the depth image to be processed to obtain an RGB-D image.
3. The depth map hole filling method of claim 2, wherein obtaining the depth image to be processed comprises:
acquiring a to-be-processed three-dimensional panoramic image;
and inputting the three-dimensional panoramic image to be processed into a pre-trained depth estimation model, and outputting the depth image to be processed.
4. The depth map hole filling method of claim 1, wherein the hole region types include edge hole sub-regions and non-edge hole sub-regions;
identifying the type of the hole area of the depth image to be processed, including:
inputting the depth image to be processed into a preset classifier, and determining a cavity region in the depth image to be processed;
and determining an edge hole sub-region and a non-edge hole sub-region in the hole region by adopting an edge detection algorithm.
5. The depth map hole filling method of claim 4, wherein the hole filling algorithm comprises a first hole filling algorithm and a second hole filling algorithm;
based on the color image, performing hole filling processing on the depth image to be processed according to a hole filling algorithm corresponding to the type of the hole area to obtain a target depth image, including:
segmenting an edge area and a non-edge area in the color image;
calling a first hole filling algorithm corresponding to a non-edge hole sub-region, and performing first hole filling on the non-edge hole sub-region based on the non-edge region in the color image;
calling a second hole filling algorithm corresponding to the edge hole sub-region, and performing second hole filling on the edge hole sub-region based on a non-edge region in the color image;
and taking the depth image to be processed filled by the first hole and the second hole as a target depth image.
6. A depth map hole filling apparatus, comprising:
the system comprises an acquisition module, a processing module and a display module, wherein the acquisition module is used for acquiring an RGB-D image, and the RGB-D image comprises a depth image to be processed and a color image corresponding to the depth image to be processed;
the identification module is used for identifying the type of the cavity region of the depth image to be processed;
and the filling processing module is used for carrying out cavity filling processing on the depth image to be processed according to a cavity filling algorithm corresponding to the type of the cavity region based on the color image to obtain a target depth image.
7. The depth map hole filling apparatus of claim 6, wherein the obtaining module comprises:
the first acquisition sub-module is used for acquiring a color image;
the second acquisition submodule is used for acquiring a depth image to be processed;
and the registration submodule is used for registering the color image and the depth image to be processed to obtain an RGB-D image.
8. The depth map hole filling apparatus of claim 7, wherein the second acquisition submodule comprises:
the device comprises an acquisition unit, a processing unit and a processing unit, wherein the acquisition unit is used for acquiring a to-be-processed three-dimensional panoramic image;
and the depth estimation unit is used for inputting the three-dimensional panoramic image to be processed into a depth estimation model trained in advance and outputting the depth image to be processed.
9. A server comprising a memory, a processor and a computer program stored in the memory and executable on the processor, characterized in that the processor implements the method according to any of claims 1 to 5 when executing the computer program.
10. A readable storage medium, storing a computer program, characterized in that the computer program, when executed by a processor, implements the method according to any of claims 1 to 5.
CN202111053968.7A 2021-09-09 2021-09-09 Depth map hole filling method and device, server and readable storage medium Pending CN113744256A (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202111053968.7A CN113744256A (en) 2021-09-09 2021-09-09 Depth map hole filling method and device, server and readable storage medium

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202111053968.7A CN113744256A (en) 2021-09-09 2021-09-09 Depth map hole filling method and device, server and readable storage medium

Publications (1)

Publication Number Publication Date
CN113744256A true CN113744256A (en) 2021-12-03

Family

ID=78737432

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202111053968.7A Pending CN113744256A (en) 2021-09-09 2021-09-09 Depth map hole filling method and device, server and readable storage medium

Country Status (1)

Country Link
CN (1) CN113744256A (en)

Cited By (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN115731529A (en) * 2023-01-10 2023-03-03 天津所托瑞安汽车科技有限公司 Image filling method, device, terminal, vehicle and medium for target recognition
WO2023124442A1 (en) * 2021-12-29 2023-07-06 京东方科技集团股份有限公司 Method and device for measuring depth of accumulated water
CN117217626A (en) * 2023-11-09 2023-12-12 济宁矿业集团有限公司霄云煤矿 Intelligent mining method based on green environmental protection
CN117237352A (en) * 2023-11-14 2023-12-15 之江实验室 Gene chip image processing method, data processing device and storage medium

Cited By (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2023124442A1 (en) * 2021-12-29 2023-07-06 京东方科技集团股份有限公司 Method and device for measuring depth of accumulated water
CN115731529A (en) * 2023-01-10 2023-03-03 天津所托瑞安汽车科技有限公司 Image filling method, device, terminal, vehicle and medium for target recognition
CN115731529B (en) * 2023-01-10 2023-04-18 天津所托瑞安汽车科技有限公司 Image filling method, device, terminal, vehicle and medium for target recognition
CN117217626A (en) * 2023-11-09 2023-12-12 济宁矿业集团有限公司霄云煤矿 Intelligent mining method based on green environmental protection
CN117237352A (en) * 2023-11-14 2023-12-15 之江实验室 Gene chip image processing method, data processing device and storage medium
CN117237352B (en) * 2023-11-14 2024-02-09 之江实验室 Gene chip image processing method, data processing device and storage medium

Similar Documents

Publication Publication Date Title
WO2021088473A1 (en) Image super-resolution reconstruction method, image super-resolution reconstruction apparatus, and computer-readable storage medium
CN113744256A (en) Depth map hole filling method and device, server and readable storage medium
CN109461181B (en) Depth image acquisition method and system based on speckle structured light
CN109214996B (en) Image processing method and device
CN111950426A (en) Target detection method and device and delivery vehicle
CN111383254A (en) Depth information acquisition method and system and terminal equipment
CN111563517B (en) Image processing method, device, electronic equipment and storage medium
CN114862929A (en) Three-dimensional target detection method and device, computer readable storage medium and robot
CN113052754B (en) Method and device for blurring picture background
CN110120012B (en) Video stitching method for synchronous key frame extraction based on binocular camera
CN111063029A (en) Map construction method and device, computer readable storage medium and robot
CN110673607A (en) Feature point extraction method and device in dynamic scene and terminal equipment
US6304672B1 (en) Edge detecting method and edge detecting device which detects edges for each individual primary color and employs individual color weighting coefficients
CN113379815A (en) Three-dimensional reconstruction method and device based on RGB camera and laser sensor and server
CN111539975B (en) Method, device, equipment and storage medium for detecting moving object
CN108805838B (en) Image processing method, mobile terminal and computer readable storage medium
CN110765875B (en) Method, equipment and device for detecting boundary of traffic target
CN112102378A (en) Image registration method and device, terminal equipment and computer readable storage medium
CN116977671A (en) Target tracking method, device, equipment and storage medium based on image space positioning
CN110874814A (en) Image processing method, image processing device and terminal equipment
CN114913105A (en) Laser point cloud fusion method and device, server and computer readable storage medium
CN112669346B (en) Pavement emergency determination method and device
CN114898321A (en) Method, device, equipment, medium and system for detecting road travelable area
CN112270693B (en) Method and device for detecting motion artifact of time-of-flight depth camera
CN112907704B (en) Image fusion method, computer equipment and device

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination