CN111986203B - Depth image segmentation method and device - Google Patents

Abstract

The invention relates to a depth image segmentation method and device, belonging to the technical field of image processing. In the invention, the background in the depth image is regarded as the seabed, the target itself is regarded as a towering island, and the lower limit of the depth value of the depth image is used as the initial level to gradually control the rise of the level. Inundation, a series of puddle maps are generated with the horizontal plane as the boundary, and each puddle map is identified to obtain the isolated island area; based on the process of water level rise, the area of the isolated island area formed by the target hardly changes in a certain water level interval, while the background According to the principle that the area of the island area formed by the raised part will gradually become smaller and quickly submerged, the target can be identified from the island image, so as to realize the segmentation of the target in the depth image. The invention fully considers the situation that the depth difference of the target background surface is large, and can accurately realize the target segmentation of the depth image.

Description

A method and device for segmenting a depth image

technical field

The invention relates to a depth image segmentation method and device, belonging to the technical field of image processing.

Background technique

With the development of artificial intelligence and machine vision technology, the demand for image data in all walks of life is increasing year by year, and the requirements for image segmentation technology are getting higher and higher, thus promoting the development of automation and intelligent technology. In the face of complex image data, how to accurately find and segment the desired target has become an important bottleneck restricting the development of image processing technology. It can be said that accurate image segmentation has become the most basic and important problem in the field of image processing and needs to be solved urgently. . In recent years, with the development of binocular vision technology, depth images are increasingly used in object detection, 3D reconstruction and other fields. A depth image, also known as a range image, is an image that takes the distance (depth) from the image collector to each point in the scene as a pixel value, which directly reflects the geometry of the visible surface of an object. In the field of target detection, the traditional depth image segmentation mainly adopts the threshold segmentation method, which is roughly divided into fixed threshold segmentation and dynamic threshold segmentation. For example, the name "In-situ Measurement Technology of Brown Mushroom Based on SR300 Depth Camera" (Wang Ling, Xu Wei, Du Kaiwei, etc.) proposed a segmentation method for the depth image of brown mushroom, which interferes with the background, and in the depth image Using the mode of the depth value of the substrate surface, combined with the height of the mushroom stipe, adaptively selects the dynamic threshold, and realizes the background segmentation to extract the binary image of the mushroom cap contour, so as to realize the segmentation of the depth image of the brown mushroom. Although this method can realize the segmentation of the depth image of the brown mushroom, in fact, the depth of the substrate surface of the brown mushroom is quite different. If only the mode of the substrate surface depth value is used as the dynamic threshold of the segmentation, the segmentation will be inaccurate. In the face of depth image segmentation under complex background, neither the fixed threshold method nor the dynamic threshold method can achieve a good segmentation effect. Therefore, a depth image segmentation method with wider application range and better segmentation effect is urgently needed.

The invention fully utilizes and mines the rich three-dimensional structure information provided by the depth image, and develops a depth image target segmentation method and device according to the characteristics of the target's depth structure.

SUMMARY OF THE INVENTION

The purpose of the present invention is to provide a depth image segmentation method and device to solve the current problem of inaccurate depth image segmentation.

In order to solve the above-mentioned technical problems, the present invention provides a method for segmenting a depth image, which comprises the following steps:

1) Obtain a depth image containing the target;

2) Perform depth value conversion on the acquired depth image, and convert the depth value of the depth image into the distance between the target object and the reference plane;

3) The lower limit of the depth value of the converted depth image is used as the horizontal plane, and the horizontal plane is gradually raised according to the set step size until reaching the set height; in each horizontal plane rising process, the depth image is binarized with the horizontal plane as the limit. The corresponding puddle map is generated, and the isolated island area separated from the boundary is proposed from each puddle map;

4) Analyze the isolated island area extracted from each puddle map, and select the isolated island area whose area change rate is less than the set threshold during the rise of the water level as the target.

The present invention also provides a depth image segmentation device, the segmentation device includes a processor and a memory, and the processor executes a computer program stored in the memory to implement the depth image segmentation method of the present invention.

In the invention, the background in the depth image is regarded as the seabed, the target itself is regarded as a towering island, and the lower limit of the depth value of the depth image is used as the initial level to gradually control the rise of the level. Inundation, a series of puddle maps are generated with the horizontal plane as the boundary, and each puddle map is identified to obtain an isolated island area; based on the process of water level rise, the area of the isolated island area formed by the target hardly changes in a certain water level interval, while the background According to the principle that the area of the island area formed by the raised part will gradually become smaller and quickly submerged, the target can be identified from the island image, so as to realize the segmentation of the target in the depth image. The invention fully considers the situation that the depth difference of the substrate surface where the target is located is large, and can accurately realize the segmentation of the depth image.

Further, in order to obtain the isolated island region more accurately, the formation process of the isolated island region in the step 3) is as follows: fill the generated puddle map to form a corresponding fill map; subtract the corresponding fill map from the puddle map to obtain the isolated island map, Get the island area from the island map.

Further, in order to screen out the target more accurately, the basis that the target is selected in the described step 4) is:

IS _j is an isolated island area when the water level is j, AIS _j is the area of the isolated island area IS _j , k is the depth of the water level backtracking, R is the area change rate index, T is the island area threshold, IS _jk is the water level backtracking to jk The predecessor of IS _j .

Further, in order to avoid repeated judgments, the method further includes deleting the target from the next island map after determining the target.

Further, in order to improve the accuracy of segmentation, the step 1) also includes the step of removing abnormal points from the acquired depth image.

Description of drawings

Fig. 1 is the flow chart of the segmentation method of the depth image of the present invention;

Fig. 2 is the flow chart of the submerged method in the segmentation method of the depth image of the present invention;

Fig. 3 is the principle schematic diagram of the flooding method of the present invention;

Fig. 4 is the schematic diagram of the evolution process of the puddle map in the first embodiment of the method of the present invention;

5 is a schematic diagram of the evolution process of a sag map, a filled map and an isolated island map in Embodiment 2 of the method of the present invention;

Fig. 6-a is the depth image of Agaricus bisporus obtained in the method embodiment 1 of the present invention;

Figure 6-b is a schematic diagram of the segmentation result of the depth image of Agaricus bisporus in Embodiment 1 of the method of the present invention;

7 is a schematic structural diagram of a depth image segmentation device of the present invention;

Where 1 is the background matrix, 2 is the shadow, 3 is the target, 4 is the bump, and 5 is the horizontal plane.

Detailed ways

The specific embodiments of the present invention will be further described below with reference to the accompanying drawings.

Method Embodiment 1

Aiming at the problems existing in the current depth image segmentation, the present invention provides a depth image segmentation method, which regards the background in the depth image as the seabed, regards the target itself as a towering island, and continuously injects water into the dry sea area. As the water level rises, the background and low targets are gradually submerged, and a series of water puddle maps are generated, and each water puddle map is identified to obtain the island area; based on the process of the water level rising, the island area formed by the target is at a certain water level. There is almost no change in the interval, and the area of the island formed by the convex part of the background where the target is located will gradually become smaller and quickly submerged. The target is identified from the island area, thereby realizing the segmentation of the target in the depth image. The present invention is suitable for accurate segmentation of the image background in the case where the background in the depth image is flat and the target and the background have a certain depth, such as the detection of targets such as Agaricus bisporus and brown mushroom growing on the mushroom bed. The following takes the depth image of Agaricus bisporus as an example. The image segmentation method of the invention is described in detail. The implementation process of the method is shown in Figure 1, and the specific implementation process is as follows:

1. Obtain the depth image of Agaricus bisporus.

The present invention uses the RealSense SR300 RGBD camera to capture the image of Agaricus bisporus. Since this embodiment is aimed at the mushroom house, the depth camera is installed on the Agaricus bisporus rack picking robot, and can move linearly with the robotic arm in the mushroom bed plane. It is installed on the picking robot of Agaricus bisporus rack and can move with the mechanical arm in the plane of the mushroom bed. Since the space above the mushroom stand is 300mm-400mm, the effective shooting distance of the RealSense SR300 RGBD camera is 100mm-1500mm, which meets the requirements of image acquisition. In order to ensure the moving accuracy of the carriage platform, the servo motor is used to drive the 0.02mm sliding table module, the carriage platform moves to the designated position, and the camera captures the image of the designated area.

2. Preprocess the acquired depth image.

There are two kinds of abnormal points in the depth image, one is the point with a depth value of 0 after the structured light is blocked; the other is the pixel point that deviates from the normal value obviously due to the error of the imaging system or the influence of light, etc. ; The main purpose of preprocessing is to eliminate both types of data points. First, count the lower four-digit score (Q1) and the upper four-digit score (Q3) of the depth value of non-zero pixels, then calculate the difference between Q3 and Q1 as ΔQ, and finally keep only the interval [Q1-ΔQ, Q3+ΔQ] pixel points to achieve the removal of outliers.

3. Perform depth value conversion on the preprocessed depth image.

The origin of the depth value of the depth image is at the center of the camera sensor, and the depth value of each point on the image represents the distance from the point to the sensor. In order to facilitate the use of "drowning method" for image processing, the remaining pixels are uniformly added to h, so as to combine the depth image The depth value of is converted to the distance between the target object and the datum. Here, we use the support surface of the mushroom stand on this layer as the reference surface, combine the space range above the mushroom stand 300mm-400mm, the thickness of the mushroom bed substrate 200mm, and the value of h is 550mm to obtain the new depth value of the pixel point.

4. Submerge the depth image and get the island area.

When collecting the depth image of Agaricus bisporus, as shown in Figure 3, the shadow 2 information under the mushroom cap of Agaricus bisporus cannot be obtained. From the top view, the Agaricus bisporus on the depth image is compressed by a cylinder. Therefore, the present invention regards the background matrix 1 (the mushroom bed matrix in this embodiment) in the depth image of Agaricus bisporus as the seabed, and the target 3 (in this embodiment, refers to the Agaricus bisporus) as a towering island, as shown in Figure 3 In this method, in the process of continuously injecting water into the dry sea area, the small hills on the seabed (the bulge 4 in Figure 3) are submerged, and only the standing isolated islands will be stable during the continuous rise of the water level. retained, so as to achieve the segmentation of the target. The specific process is as follows:

Gradually raise the horizontal plane 5 from the lower limit of the image depth value (that is, the seabed) (the length of each rise is set, which is 1mm in this embodiment); taking the horizontal plane (water level value) as the benchmark, reset the pixels below the horizontal plane to 1 , reset the pixels above the horizontal plane to 0, thereby generating a series of water puddle maps, as shown in Figure 4, each time the horizontal plane rises, a corresponding water puddle map is formed. Identify the area where the pixel point is 0 in each puddle map, and determine the island area in each puddle map. The invention can identify the isolated island region from the puddle map by using the regionprops command in Matlab.

5. Analyze the island area to determine the target image.

In order to distinguish the Agaricus bisporus target from the island target, it is necessary to perform island analysis on each island image. Because the stalk of Agaricus bisporus has a certain height and the cap has a certain thickness, the area of the isolated island formed by the target of Agaricus bisporus hardly changes in a certain water level interval during the rise of the water level. will gradually become smaller and quickly submerge, distinguishing mushroom targets from protrusions in the substrate. When an isolated island meets the following conditions during the rising water level, it will be judged as a mushroom target:

In the formula, IS _j is an isolated island area when the water level is j, AIS _j is the area of the isolated island IS _j , k is the depth of the water level backtracking, R is the area change rate index, T is the island area threshold, IS _jk is the water level backtracking to The predecessor of IS _j when jk.

The accuracy of the isolated island analysis is determined by the water level backtracking depth k, the area change rate index R, and the isolated island area threshold T. After statistical analysis, the length of the mushroom stalk of the mature Agaricus bisporus is 8-13 mm. Influence, the traceback value k is selected to be 5mm; in this process, the change rate of the island area is between [0.8-1.1], so the value of R is 0.2. Considering that the diameter of mature Agaricus bisporus is greater than 15mm, and the number of pixels in the depth image is greater than 1600, take T as 1500 to filter the interference target.

6. Extract the contour of the target image and realize the segmentation of the target image.

Store the coordinates of the Agaricus bisporus contour points found in the island analysis in an array. In order to avoid the same Agaricus bisporus target being found repeatedly, when the Agaricus bisporus target is found for the first time, the known Agaricus bisporus target will be subtracted when the next island map is generated. , cycle in turn, find out and store the coordinates of the contour point of Agaricus bisporus in each isolated island map. Finally, draw the segmentation result of "Agaricus bisporus-mushroom bed matrix" with all the outline coordinate points of Agaricus bisporus.

The original depth image of Agaricus bisporus in this embodiment is shown in Figure 6-a, and the segmentation result obtained by the above method is shown in Figure 6-b.

Method Embodiment 2

The identification method of the present invention is basically the same as that in the first embodiment of the identification method, the difference is that the process of obtaining the island image from the puddle map is different in the "flooding method", and the process is shown in FIG. 2 .

Although the regionprops command in Matlab can be used to identify the island area from the puddle map, it will cause the large black area in Figure 4 to be identified as the island area, because the pixel values of the island and the unsubmerged area are both 0 at this time (that is, the display black).

Therefore, in this embodiment, after the puddle map is obtained, the filling map is used to obtain the filling of the island area, even if the closed area is filled (the pixel value of the closed area is filled from 0 to 1, that is, from black to white) , the target circled in the puddle map is filled out. The target circled in the puddle map is an isolated closed area, which is directly filled during the filling operation, as shown in Figure 5, and then the puddle map Subtract the fill map to get the island map, and the white targets in the island map are the closed areas that have been filled and disappeared (these closed areas are temporarily considered to be the targets of Agaricus bisporus, and will be further screened later).

The effect of using the fill map to obtain the island area is: in the island area, the pixel value of the island target is 1 (that is, displayed as white), which can better extract the location and area of the island target.

Device embodiment

The apparatus proposed in this embodiment, as shown in FIG. 7 , includes a processor and a memory. The memory stores a computer program that can be run on the processor, and the processor implements the methods of the above method embodiments when executing the computer program. That is to say, the methods in the above method embodiments should be understood as the flow of the method for segmenting the mushroom depth image by computer program instructions. The computer program instructions may be provided to a processor such that execution by the processor of the instructions results in the implementation of the functions specified by the above-described method flows.

The processor referred to in this embodiment refers to a processing device such as a microprocessor MCU or a programmable logic device FPGA; the memory referred to in this embodiment includes a physical device for storing information, usually after digitizing the information for use electrical, magnetic, or optical media. For example: all kinds of memories that use electrical energy to store information, RAM, ROM, etc.; all kinds of memories that use magnetic energy to store information, hard disks, floppy disks, magnetic tapes, magnetic core memories, magnetic bubble memories, U disks; use optical methods to store information of all kinds of memory, CD or DVD. Of course, there are other ways of memory, such as quantum memory, graphene memory, and so on.

The device constituted by the above-mentioned memory, processor and computer program is realized by the processor executing corresponding program instructions in the computer, and the processor can be equipped with various operating systems, such as windows operating system, linux system, android, iOS system, etc. As other implementation manners, the apparatus may further include a display, which is used for displaying the diagnosis results for the reference of the staff.

Claims (5)

1. A method for segmenting a depth image, the method comprising the steps of:

1) Acquiring a depth image containing a target;

2) Carrying out depth value conversion on the obtained depth image, and converting the depth value of the depth image into the distance between the target object and the reference surface;

3) Taking the lower limit of the depth value of the converted depth image as a horizontal plane, and gradually increasing the horizontal plane according to a set step length until the horizontal plane reaches a set height; in the process of rising the horizontal plane each time, carrying out binarization processing on the depth image by taking the horizontal plane as a boundary to generate corresponding puddle images, and extracting island regions separated from the boundary from each puddle image;

4) Island regions extracted from each water-level depression graph are analyzed, the island regions with the island region area change rate smaller than a set threshold value in the horizontal plane rising process are selected as targets, and the targets are selected according to the following steps:

IS _j is one island region with water level j, AIS _j IS an island region IS _j Area of (a), k IS the depth of water level backtracking, R IS the index of area change rate, T IS the island area threshold, IS _j-k IS for water level backtracking to j-k _j The precursor of (1).

2. The method for segmenting the depth image according to claim 1, wherein the formation of the island region in step 3) is as follows: filling the generated water depression graph to form a corresponding filling graph; and subtracting the corresponding filling map from the water hollow map to obtain an island map, and obtaining an island region from the island map.

3. The method for segmenting the depth image according to claim 2, further comprising determining an object and deleting the object from a next islanding image.

4. The method for segmenting the depth image according to claim 1, wherein the step 1) further comprises a step of removing abnormal points from the acquired depth image.

5. A depth image segmentation apparatus comprising a processor and a memory, the processor executing a computer program stored by the memory to implement the depth image segmentation method as claimed in any one of claims 1 to 4.

Images