CN116188335A

CN116188335A - Multi-window depth image processing method, acceleration method and device and electronic equipment

Info

Publication number: CN116188335A
Application number: CN202111420915.4A
Authority: CN
Inventors: 孙佳睿; 柴晨喜; 李楠; 李健; 金瑞
Original assignee: Zhejiang Sunny Optical Intelligent Technology Co Ltd
Current assignee: Zhejiang Sunny Optical Intelligent Technology Co Ltd
Priority date: 2021-11-26
Filing date: 2021-11-26
Publication date: 2023-05-30

Abstract

The application provides a multi-window depth image processing method, an acceleration method and device and electronic equipment. Acquiring a reference image containing a speckle pattern, the speckle pattern comprising a plurality of feature points; acquiring the position of a central point of the reference image; setting the size of a matching window according to the distance between the point on the reference image and the central point; acquiring a scene image containing the speckle pattern; and matching the reference image with the scene image according to the set matching window to obtain depth information. By matching windows of different sizes for different areas in the image according to the distortion of the projector, the integrity of the depth is ensured.

Description

Multi-window depth image processing method, acceleration method and device and electronic equipment

Technical Field

The present invention relates to the field of image processing, and in particular, to a method, an acceleration method, an apparatus, and an electronic device for multi-window depth image processing.

Background

The depth camera based on binocular stereoscopic vision is sensitive to the ambient illumination intensity and is dependent on the characteristics of the image, so that the effective robust characteristics are difficult to extract under the conditions of insufficient illumination, lack of textures and the like, and the matching error is increased or even the matching is failed.

The depth camera based on the structured light method is proposed to solve the complexity and robustness problems of the binocular matching algorithm, and other processing steps are similar to binocular stereoscopic vision. The structured light method does not depend on the color and texture of the object, adopts the method of actively projecting the known pattern to realize fast and robust matching characteristic points, can achieve higher precision, and greatly expands the application range.

Along with the continuous development of the depth camera and the explosion of the whole industry of machine vision, the requirement standard of the depth camera is gradually improved, high-precision depth information has good promotion effect on the whole industry and related industries, and the current depth camera has the problems of reduced precision along with the increase of distance, blurred boundary and inaccurate data.

Disclosure of Invention

The application aims to provide a multi-window depth image processing method, an acceleration device and electronic equipment, wherein the probability of mismatching can be reduced to a certain extent by adjusting a matching window, and the problems of poor boundary effect, excessive smoothness and reduced accuracy when a filtering window is changed are solved.

The depth camera is guaranteed to have good depth under different distances.

According to an aspect of the present application, a method for multi-window depth image processing is provided, including:

acquiring a reference image containing a speckle pattern;

acquiring the position of a central point of the reference image;

setting the size of a matching window according to the distance between the point on the reference image and the central point;

acquiring a scene image containing the speckle pattern;

and matching the scene images according to the set matching window to obtain depth information.

According to some embodiments, the method further comprises: acquiring the center point position of the reference image comprises the following steps:

setting a matching template;

matching the reference image according to the matching template to obtain an image block with the highest matching degree as a central image block;

and obtaining the center point position of the reference image according to the center image block.

According to some embodiments, the method further comprises: the matching window is set to be larger the farther from the center point.

According to some embodiments, the method further comprises: the matching window contains more than 5 speckles.

According to some embodiments, the method further comprises: after the depth information is obtained, filtering the obtained depth information by using filter windows with different sizes according to different depths.

According to some embodiments, the method further comprises: the filter window decreases with increasing depth distance.

According to some embodiments, the method further comprises:

according to different depths, different post-processing parameters are set, and the post-processing parameters are used for post-processing the depth information.

According to some embodiments, the method further comprises:

acquiring the center of a scene image;

pre-calculating according to the obtained center point position of the reference image and the center of the scene image to obtain parallax information;

and matching according to the pre-calculated parallax information.

According to another aspect of the present application, a method for accelerating multi-window depth image processing is provided, including:

acquiring the center of a scene image;

and matching according to the pre-calculated parallax information.

According to another aspect of the present application, there is provided an apparatus for multi-window depth image processing, including:

the image acquisition module is used for acquiring a reference image containing speckle patterns and a scene image;

the center point calculating module is used for calculating the center point position of the reference image;

the matching module is used for setting the size of a matching window, and matching the scene image according to the set matching window to obtain depth information;

and the post-processing module is used for carrying out post-processing on the depth information.

According to another aspect of the present application, there is provided an electronic device including:

a memory, a processor and a computer program stored in the memory and executable on the processor, the processor implementing the method of any one of the above methods when the computer program is executed.

According to another aspect of the present application, there is provided a computer program product comprising a computer program/instruction which, when executed by a processor, implements the aforementioned method.

According to the embodiment of the application, the integrity of depth is ensured by matching different areas in the image by using windows with different sizes according to the distortion of the projector.

According to the embodiment of the application, according to the actual conditions of the scene, the window sizes of different mean values and median filters and the parameters of post-processing are used according to different depth distances, so that the accuracy of the depth is ensured, excessive post-processing is avoided, and the long-distance depth is prevented from being excessively smoothed.

It is to be understood that both the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the application.

Drawings

In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings that are required to be used in the description of the embodiments will be briefly described below.

FIG. 1 illustrates a multi-window based depth calculation schematic according to an example embodiment of the present application.

Fig. 2 shows a method flowchart of multi-window depth image processing according to an example embodiment of the present application.

Fig. 3 shows a schematic diagram of a structured light based collection module according to an example embodiment of the present application.

Fig. 4 shows a reference image effect map with speckle according to an example embodiment of the present application.

Fig. 5 shows a center image block schematic of a reference image according to an example embodiment of the present application.

Fig. 6 shows a matching window schematic diagram according to an example embodiment of the present application.

Fig. 7 shows a scene image and a processing effect schematic diagram according to an example embodiment of the present application.

Fig. 8 shows a flow chart of post-processing of a multi-window depth image according to an example embodiment of the present application.

Fig. 9 shows a flowchart of an acceleration method for a multi-window depth image according to an example embodiment of the present application.

Fig. 10 shows a block diagram of an apparatus for multi-window depth image processing according to an example embodiment of the present application.

Fig. 11 shows a block diagram of an electronic device according to an example embodiment.

Detailed Description

Example embodiments will now be described more fully with reference to the accompanying drawings. However, the exemplary embodiments can be embodied in many forms and should not be construed as limited to the embodiments set forth herein; rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the concept of the example embodiments to those skilled in the art. The same reference numerals in the drawings denote the same or similar parts, and thus a repetitive description thereof will be omitted.

Furthermore, the described features, structures, or characteristics may be combined in any suitable manner in one or more embodiments. In the following description, numerous specific details are provided to give a thorough understanding of embodiments of the present application. One skilled in the relevant art will recognize, however, that the aspects of the application can be practiced without one or more of the specific details, or with other methods, components, devices, steps, etc. In other instances, well-known methods, devices, implementations, or operations are not shown or described in detail to avoid obscuring aspects of the application.

The block diagrams depicted in the figures are merely functional entities and do not necessarily correspond to physically separate entities. That is, the functional entities may be implemented in software, or in one or more hardware modules or integrated circuits, or in different networks and/or processor devices and/or microcontroller devices.

The flow diagrams depicted in the figures are exemplary only, and do not necessarily include all of the elements and operations/steps, nor must they be performed in the order described. For example, some operations/steps may be decomposed, and some operations/steps may be combined or partially combined, so that the order of actual execution may be changed according to actual situations.

It will be understood that, although the terms first, second, third, etc. may be used herein to describe various components, these components should not be limited by these terms. These terms are used to distinguish one element from another element. Thus, a first component discussed below could be termed a second component without departing from the teachings of the present application concept. As used herein, the term "and/or" includes any one of the associated listed items and all combinations of one or more.

Those skilled in the art will appreciate that the drawings are schematic representations of example embodiments, and that the modules or flows in the drawings are not necessarily required to practice the present application, and therefore, should not be taken to limit the scope of the present application.

In three-dimensional depth acquisition, the most common method is similar to a binocular matching depth acquisition method, the binocular matching is completely based on an image processing technology, and matching points are obtained by searching the same characteristic points in two images, so that a depth value is obtained; the method based on the image matching completely has great difficulty, and the matching precision and accuracy are difficult to ensure; structured light technology has emerged to solve the matching problem.

In depth ranging based on binocular matching, the light source is an uncoded light source such as ambient light or white light, and image recognition is completely dependent on the characteristic points of the photographed object itself, so matching is always a difficulty of binocular.

Whereas structured light based depth ranging differs in that the projection light source is coded or otherwise characterized. Thus, what is captured is an image of the coded light source projected onto the object that is depth modulated by the object surface.

Because the structured light source is provided with a plurality of characteristic points or codes, a plurality of matching angular points or direct code words are provided, and the characteristic points can be conveniently matched. In other words, the structured light actively provides many feature points for matching or directly provides codewords, without using the feature points possessed by the subject itself, and thus can provide a better matching result.

The difference in matching between the two: the binocular uses the feature points of the object itself, while the structured light uses feature points or direct codewords that the light source actively provides. Another difference is that, because of the diversity of photographed objects, each binocular matching faces a different image, and feature points need to be re-extracted; the same pattern is projected by the structured light, the characteristic points are fixed, and the characteristic points do not need to be changed according to the scene change, so that the matching difficulty is reduced.

The structured light method is to project structured light onto the surface of the object, then use the camera to receive the pattern of the structured light reflected by the surface of the object, and because the received pattern is necessarily deformed due to the three-dimensional shape of the object, it is possible to try to calculate the spatial information of the surface of the object by the position and deformation degree of the pattern on the camera. The common structured light method is still a depth calculation which partly adopts the principle of triangular ranging.

Based on the advantage of structure light method depth camera:

1) Because the structured light actively projects the coded light, the structured light is very suitable for being used in a scene with insufficient illumination (even no light) and lacking texture. 2) The structured light projection pattern is generally carefully designed, so that a high measurement accuracy can be achieved within a certain range. 3) The technology is mature, and the depth image can achieve relatively higher resolution.

Structured light is light with a specific pattern, which has a pattern of patterns, e.g. dots, lines, facets, etc. The depth image acquisition principle based on structured light is: structured light is projected onto the scene and a corresponding pattern with structured light is captured by the image sensor. Because the pattern of the structured light deforms according to the shape of the object, the depth information of each point in the scene can be obtained by calculating the position and deformation degree of the pattern image in the captured image by using the triangle principle. Structured light measurement techniques provide high-precision and fast three-dimensional information, which has been widely used in the fields of automobiles, games, medical treatment, etc.

A depth image (depth image), also called range image, refers to an image with the distance (depth) from an image collector to points in a scene as pixel values, which directly reflects the geometry of the scene's visible surface. The depth image can be calculated as point cloud data through coordinate conversion, and the point cloud data with regular and necessary information can also be reversely calculated as depth image data.

With the continuous development of depth cameras and the explosion of the whole industry of machine vision, the requirement standard of the depth cameras is gradually improved. The high-precision depth information has good promotion effect on the whole industry and related industries, and the current depth camera has the problems of reduced precision along with the increase of distance, blurred boundary and inaccurate data.

In order to ensure good depth effect, the application provides a multi-window depth image processing method, which ensures that a depth camera has better depth at different distances.

According to the calculation principle of the structured light depth camera and the speckle characteristic points, windows with different sizes are used for matching according to the distortion of the projector for different areas in the image, so that the integrity of the depth is ensured.

According to the method and the device, according to actual conditions of scenes, different average values, different median filter window sizes and post-processing parameters are used for different depth distances, so that the accuracy of depth is guaranteed, excessive post-processing is avoided, and long-distance depth is prevented from being excessively smoothed.

The technical scheme and beneficial effects of the present application will be described in detail below with reference to the accompanying drawings.

FIG. 1 illustrates a depth calculation schematic based on structured light matching according to an example embodiment of the present application.

The basic structured light measuring system consists of a camera and a projector, wherein the projector projects the coded speckle pattern on the surface of a scene, the camera shoots the scene image with the speckle pattern, the offset between the shot speckle and the shot speckle is calculated, and the depth information of the scene can be obtained.

Referring to fig. 1, a camera C and a projector P, b is a distance between the camera and the projector, and f is a focal length of the camera; the projector P projects a speckle pattern on a reference plane and the surface of a scene object, any point O is taken from the speckle projected on the scene surface by the projector, the corresponding projection point of the point O on the reference plane is R, and the corresponding point of the point O on the scene image shot by the camera is K ₁ The point R corresponds to the point R on the reference image shot by the camera ₁ ，Z ₀ Is the shooting distance of the reference plane, d is the corresponding point R found in the matching ₁ And K ₁ Parallax between the two, Z is the depth corresponding to the O point.

According to triangle similarity principle, delta RKC and DeltaR ₁ K ₁ C is similar, giving:

Δ OKR is similar to Δocp, yielding:

R ₁ K ₁ i.e. parallax d, the depth Z is calculated as follows:

the matching algorithm includes, but is not limited to, the following:

a) Feature-based matching, such as feature point extraction matching methods (SIFT, FAST, etc.);

b) Based on the matching of the gray information and the local features, such as (NCC, ZNCC);

c) Matching based on global information, such as (SGBM, SGM);

taking an NCC matching algorithm based on matching of gray information and local characteristics as an example, the depth calculation process is as follows: and calculating the similarity between the pixel points in the scene graph and the pixel points in the reference image in a certain window, searching according to the row to obtain a corresponding point with the maximum similarity, thereby obtaining parallax, namely the offset between the current pixel and the same point of the reference image, and performing depth calculation by using the offset and parameters of camera equipment to obtain a specific depth value.

The reference image with speckle pattern is a plane (which may be composed of projection cloth, flat plate, etc.) perpendicular to the central axis (Z axis) of the projector and with known perpendicular distance, and the laser beam (including infrared, visible, ultraviolet, invisible, etc.) of fixed pattern projected by the projector forms a non-regular random distribution speckle pattern.

Referring to fig. 2, at S201, a reference image containing a projected speckle pattern is acquired.

According to some embodiments, acquiring an image of the projected pattern requires a projector and a camera, such as an infrared projector and an infrared camera, which may be referred to as a module. Fig. 3 shows a schematic diagram of a structured light based collection module according to an example embodiment of the present application.

FIG. 4 illustrates a reference image effect map with speckle according to an example embodiment of the present application. Wherein the speckle pattern comprises a plurality of speckle features, the distribution of which has a certain randomness, is unique at least in a certain range, and the distribution of the speckle in a region forms a pattern structure, which can be called a speckle pattern, and the whole speckle image is generated by projecting the speckle pattern replicated based on the region.

The central image block diagram of the reference image is shown in fig. 5, wherein the red frame area is the central image block, the size can be 120 x 100 (unit: pixel), the reference image size is 800 x 1280 (unit: pixel), and the size of the central image block can be changed corresponding to different reference image sizes. The distribution pattern of the central image block, i.e. the speckle pattern, may be different for different modules.

At S203, the center point position of the reference image is acquired.

According to some embodiments, first, the set template is used to match, and a central image block as shown in fig. 5 may be used as the template, where the template includes a projected speckle pattern and is sized according to the projector, and the whole reference image is matched by using the template, only the undistorted image block can be matched with the designed template, and the matching degree is highest, so as to obtain the position information of the central image block, and the center point of the reference image is obtained by using the center point of the image block as the center.

In S205, the size of the matching window is set according to the distance of the point on the reference image from the center point.

According to some embodiments, the image block center point is taken as the center, the whole matching window is set according to the distortion degree of different projectors, so that enough speckles in different areas are contained in the matching window, and generally, more than 5 speckles (for example, 8-10 speckles) are required to be at least in the matching window, so that good matching effect can be achieved.

According to the rule of distortion occurrence, the farther the distortion occurrence is from the central area, the larger the distance between the speckles is, so that the size of the window is determined by judging the distance between each pixel point and the center, and the farther the distance is from the window, the larger the distance is.

According to some embodiments, after the size is preliminarily determined, statistics is performed on speckle characteristic points in a window of the current pixel point, 8-10 speckle characteristic points are more suitable in the window, if the number of speckle in the window is less than 8, the window is enlarged, otherwise, if the number of speckle in the window exceeds 10, the window needs to be reduced, and the adjustment of the size of the window can be realized through an algorithm.

For example, the resolution of the image is 800×1280, the window size may be set to 13×13, 17×17, 21×21, 25×25, and the units are pixels. A matching window schematic diagram according to an example embodiment of the present application is shown in fig. 6. Which contains a plurality of windows of different sizes.

At S207, a scene image containing a projected speckle pattern is acquired.

According to some embodiments, the scene image containing the projected speckle pattern is obtained by the same camera and projector that acquired the reference image. As shown in fig. 7, a first image from left to right is a real scene image and a second image is a scene image containing a projected speckle pattern according to an example embodiment of the present application.

In S209, the scene image is matched according to the set matching window, and depth information is obtained.

According to some embodiments, as the projector angle increases, the distortion is more severe, based on the projected pattern center block, and according to different distortion, we use different depth algorithms to match windows on the image, processing from small to large.

Taking an NCC (normalized correlation) matching algorithm based on matching of gray information and local features as an example, according to the similarity between a pixel point in a scene image and a pixel point in a reference image, performing matching search in a search mode in which the center of the image is diffused to the periphery or in a mode of parallel search to obtain a corresponding point with the maximum similarity, thereby obtaining parallax, that is, the offset of the current pixel point in the scene image and the same point as the reference image, and performing depth calculation by using the offset and parameters (such as focal length) of camera equipment to obtain a specific depth value of the current point. The rightmost figure in fig. 7 is a schematic view of the processing effect according to an exemplary embodiment of the present application.

According to some embodiments, feature-based matching, such as feature point extraction matching methods (SIFT, FAST, etc.), gray information and local feature-based matching, such as (NCC, ZNCC), and global information-based matching, such as (SGBM, SGM), may be employed in the matching. All three matching methods are window matching based methods.

According to some embodiments, after obtaining depth information, the obtained depth information is filtered using different sized filter windows according to different depths. As the distance increases, the filter window decreases as the distance increases, larger median and mean filter windows are used at close distances, and smaller filter windows are used at far distances.

According to some embodiments, different post-processing parameters may be set according to different depths, the post-processing parameters being used for post-processing the depth information. For example, in flying spot filtering and cavity filling algorithms, parameters are adjusted to ensure that a dense and accurate point cloud is obtained at all distances, including long distances and short distances.

According to some embodiments, post-processing may include flying spot filtering, hole filling, median filtering, mean filtering, gaussian filtering, and the like.

Taking the flying spot filtering algorithm as an example, referring to fig. 8, in S801, a depth map is input, and depth information subjected to depth calculation is used as input information.

In S803, the region is divided. And traversing the depth image, setting a deviation threshold value, and if the value between two adjacent pixel points is larger than the deviation threshold value, enabling the two points not to belong to the same region. For example, the calculated depth of two adjacent pixels is 800, 850, and the set offset value may be 20, and if the difference between the depths of two pixels is 50, it is determined that the difference is greater than 20, then it is determined that the difference is not the same region point.

At S805, the region size statistics. After the full-graph search is performed in step S803 and the regions with similar depth are divided, the number of effective values in the regions is counted, where the number of effective values refers to the number of points with similar depth and not zero, and the portion where the number of effective values is smaller than the set threshold (for example, the threshold may be set to 200-500) is removed, so that the region where the number of effective values is greater than the threshold is reserved.

The hole filling is similar, for example, the region with the number of effective values smaller than the threshold value is filled, and the region with the number larger than the threshold value is not processed.

Taking a face depth camera as an example (the farthest distance is 1200 mm) according to the difference of the test distances:

when the depth is 300mm-500mm, window setting can be 9*9 or 11 x 11, and the size of flying spot filtering and cavity filling can be 500 pixels;

when the depth is 500mm-800mm, the window can be properly reduced to 7*7 or 5*5, and the size of flying spot filtering and cavity filling can be set to 300 pixels;

when the depth is 800mm-1200mm, the window can be appropriately reduced to 5*5 or 3*3, and the size of flying spot filtering and void filling can be set to 200 pixels.

According to some embodiments, the depth calculation generally has a larger calculation amount, which may result in a lower frame rate, that is, a lower speed of calculating the depth map, so it is necessary to design an acceleration manner to reduce the calculation amount.

When the matching algorithm is used, the center of the projector pattern on the reference image can be found in a matching mode, the center of the projector pattern on the reference image and the center of the scene graph are pre-calculated, parallax information of the area can be obtained, the parallax information is used as reference data, and matching and searching are performed in a small-range parallax, so that the purpose of reducing the calculated amount is achieved.

In S901, the centers of the reference image and the scene image are acquired.

The step of obtaining the center of the reference image is the same as step S203, and is not described herein. The center of the scene image may be obtained by taking the geometric center of the scene image as the center, e.g., dividing the long and wide pixels by 2 to obtain the center point.

In S903, the parallax is calculated.

According to some embodiments, by calculating the center points of the reference image and the scene image, the depth parallax corresponding to the center point can be obtained, and the depth parallax is used as the reference parallax, so that the parallax calculation range of the search is reduced.

In S905, a search range is set according to the pre-calculated parallax.

According to some embodiments, after obtaining the depth parallax corresponding to the center point, the search range of the surrounding adjacent area is set by taking the parallax corresponding to the center point as a basis in a search mode in which the center of the image spreads to the surrounding. For example, if the parallax at the center point is 10, the search range of the adjacent area that spreads from the center to the surrounding area may be 10 as a reference, and searching may be performed within a certain range, after calculating the parallax data within this range, the newly calculated parallax within this range (which may be averaged with the parallax within this range) may be used as a reference for the search range of the surrounding adjacent area, and so on.

According to some embodiments, if the full parallax search range is 100, the parallax search range may be narrowed down to 10 after the reference distance is acquired, reducing the overall calculation amount by 10 times.

It should be clearly understood that this application describes how to make and use particular examples, but is not limited to any details of these examples. Rather, these principles can be applied to many other embodiments based on the teachings of the present disclosure.

Those skilled in the art will appreciate that all or part of the steps implementing the above described embodiments are implemented as a computer program executed by a CPU. When the computer program is executed by the CPU, the program for performing the above-described functions defined by the above-described method provided in the present application may be stored in a computer readable storage medium, which may be a read-only memory, a magnetic disk, or an optical disk.

Furthermore, it should be noted that the above-described figures are merely illustrative of the processes involved in the method according to the exemplary embodiments of the present application, and are not intended to be limiting. It will be readily appreciated that the processes shown in the above figures do not indicate or limit the temporal order of these processes. In addition, it is also readily understood that these processes may be performed synchronously or asynchronously, for example, among a plurality of modules.

Those skilled in the art will readily appreciate from the description of example embodiments that the multi-window depth image processing method according to embodiments of the present application has at least one or more of the following advantages.

According to the example embodiment, through the calculation principle and speckle characteristics of the structured light depth camera, according to the distortion of the projector, windows with different sizes are used for different areas in the image to match, so that the integrity of the depth is ensured, even if post-processing is not performed, the original data has better depth information, and almost no holes exist.

According to the embodiment, the post-processing is performed for the depth at different distances, so that the accuracy of the depth is guaranteed, excessive post-processing is avoided, and the depth at a long distance is prevented from being excessively smoothed.

According to example embodiments, with an acceleration strategy, the frame rate may be increased to a higher level, greatly reducing the amount of computation.

Apparatus embodiments of the present application are described below, which may be used to perform method embodiments of the present application. For details not disclosed in the device embodiments of the present application, reference may be made to the method embodiments of the present application.

Fig. 10 illustrates a block diagram of an apparatus for multi-window depth image processing according to an exemplary embodiment.

As shown in fig. 10, the apparatus of multi-window depth image processing may include: an image acquisition module 1010, a center point calculation module 1020, a matching module 1030, and a post-processing module 1040.

Referring to fig. 10 and with reference to the previous description, an image acquisition module 1010 for acquiring a reference image and a scene image containing a projected speckle pattern;

a center point calculation module 1020 for calculating a center point position of the reference image;

the matching module 1030 is configured to set a size of a matching window, and match the scene image according to the set matching window to obtain depth information;

and the post-processing module 1040 is used for carrying out filtering processing, flying spot filtering processing and cavity filling processing on the depth information.

The apparatus performs functions similar to those provided above, and other functions may be found in the foregoing description and will not be repeated here.

An electronic device 200 according to this embodiment of the present application is described below with reference to fig. 11. The electronic device 200 shown in fig. 11 is only an example and should not be construed as limiting the functionality and scope of use of the embodiments herein.

As shown in fig. 11, the electronic device 200 is in the form of a general purpose computing device. The components of the electronic device 200 may include, but are not limited to: at least one processing unit 210, at least one memory unit 220, a bus 230 connecting the different system components (including the memory unit 220 and the processing unit 210), a display unit 240, and the like.

Wherein the storage unit stores program code that can be executed by the processing unit 210, such that the processing unit 210 performs the methods described herein according to various exemplary embodiments of the present application.

The storage unit 220 may include readable media in the form of volatile storage units, such as Random Access Memory (RAM) 2201 and/or cache memory 2202, and may further include Read Only Memory (ROM) 2203.

The storage unit 220 may also include a program/utility 2204 having a set (at least one) of program modules 2205, such program modules 2205 including, but not limited to: an operating system, one or more application programs, other program modules, and program data, each or some combination of which may include an implementation of a network environment.

Bus 230 may be a bus representing one or more of several types of bus structures including a memory unit bus or memory unit controller, a peripheral bus, an accelerated graphics port, a processing unit, or a local bus using any of a variety of bus architectures.

The electronic device 200 may also communicate with one or more external devices 300 (e.g., keyboard, pointing device, bluetooth device, etc.), one or more devices that enable a user to interact with the electronic device 200, and/or any device (e.g., router, modem, etc.) that enables the electronic device 200 to communicate with one or more other computing devices. Such communication may occur through an input/output (I/O) interface 250. Also, the electronic device 200 may communicate with one or more networks such as a Local Area Network (LAN), a Wide Area Network (WAN), and/or a public network, such as the Internet, through a network adapter 260. Network adapter 260 may communicate with other modules of electronic device 200 via bus 230. It should be appreciated that although not shown, other hardware and/or software modules may be used in connection with electronic device 200, including, but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, data backup storage systems, and the like.

From the above description of embodiments, those skilled in the art will readily appreciate that the example embodiments described herein may be implemented in software, or may be implemented in software in combination with the necessary hardware. The technical solution according to the embodiments of the present application may be embodied in the form of a software product, which may be stored in a non-volatile storage medium (may be a CD-ROM, a usb disk, a mobile hard disk, etc.) or on a network, and includes several instructions to cause a computing device (may be a personal computer, a server, or a network device, etc.) to perform the above-described method according to the embodiments of the present application.

The software product may employ any combination of one or more readable media. The readable medium may be a readable signal medium or a readable storage medium. The readable storage medium can be, for example, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or a combination of any of the foregoing. More specific examples (a non-exhaustive list) of the readable storage medium would include the following: an electrical connection having one or more wires, a portable disk, a hard disk, random Access Memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

The computer readable storage medium may include a data signal propagated in baseband or as part of a carrier wave, with readable program code embodied therein. Such a propagated data signal may take any of a variety of forms, including, but not limited to, electro-magnetic, optical, or any suitable combination of the foregoing. A readable storage medium may also be any readable medium that can communicate, propagate, or transport a program for use by or in connection with an instruction execution system, apparatus, or device. Program code embodied on a readable storage medium may be transmitted using any appropriate medium, including but not limited to wireless, wireline, optical fiber cable, RF, etc., or any suitable combination of the foregoing.

Program code for carrying out operations of the present application may be written in any combination of one or more programming languages, including an object oriented programming language such as Java, C++ or the like and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The program code may execute entirely on the user's computing device, partly on the user's device, as a stand-alone software package, partly on the user's computing device, partly on a remote computing device, or entirely on the remote computing device or server. In the case of remote computing devices, the remote computing device may be connected to the user computing device through any kind of network, including a Local Area Network (LAN) or a Wide Area Network (WAN), or may be connected to an external computing device (e.g., connected via the Internet using an Internet service provider).

Those skilled in the art will appreciate that the modules may be distributed throughout several devices as described in the embodiments, and that corresponding variations may be implemented in one or more devices that are unique to the embodiments. The modules of the above embodiments may be combined into one module, or may be further split into a plurality of sub-modules.

Exemplary embodiments of the present application are specifically illustrated and described above. It is to be understood that this application is not limited to the details of construction, arrangement or method of implementation described herein; on the contrary, the application is intended to cover various modifications and equivalent arrangements included within the spirit and scope of the appended claims.

Claims

1. A method of multi-window depth image processing, comprising:

acquiring a reference image containing a speckle pattern, the speckle pattern comprising a plurality of feature points;

acquiring the position of a central point of the reference image;

acquiring a scene image containing the speckle pattern;

and matching the reference image with the scene image according to the set matching window to obtain depth information.

2. The method of claim 1, wherein obtaining the location of the center point of the reference image comprises:

setting a matching template;

3. The method of claim 1, wherein the matching window is set to be larger the farther from the center point.

4. The method of claim 1, wherein the matching window contains more than 5 speckles.

5. The method as recited in claim 1, further comprising:

after the depth information is obtained, filtering the obtained depth information by using filter windows with different sizes according to different depths.

6. The method of claim 5, wherein the filter window decreases as the depth increases.

7. The method as recited in claim 5, further comprising:

and setting different post-processing parameters according to different depths to post-process the depth information.

8. The method as recited in claim 2, further comprising:

acquiring the center of a scene image;

and matching according to the pre-calculated parallax information.

9. An acceleration method for multi-window depth image processing, comprising:

acquiring the center of a scene image;

and matching according to the pre-calculated parallax information.

10. An apparatus for multi-window depth image processing, comprising:

11. An electronic device, comprising:

memory, a processor and a computer program stored in the memory and executable on the processor, characterized in that the processor implements the method according to any of the preceding claims 1-9 when executing the computer program.