KR101981000B1 - Depth Image Based Feature Detection Method Using Hybrid Filter - Google Patents

Abstract

Embodiments of the present invention suggests three hybrid filters to solve a problem that a depth image has. In addition, a lightweight FAST algorithm is suggested by changing a search scheme of the existing FAST algorithm for feature point detection. Therefore, a time required for extracting a feature point can be minimized. Embodiments of the present invention can be applied for automating a parcel warehouse.

Description

[0001] The present invention relates to a depth image-based feature point detection method using a fusion filter,

The following embodiments relate to a depth image based feature point detection technique using a fusion filter.

As the technology of logistics industry develops, the utilization of image processing is increasing. Image processing is used as a method for automating the production of products in factories or automating the delivery logistics warehouse. Among them, the depth image facilitates the distinction between the object and the surrounding, obtains the height value of the object, and has an advantage that the defect of the product or the position of the object can be easily detected through the depth image. However, such a depth image has a disadvantage that noise is generated.

A typical depth image is measured by an IR sensor, which can cause noise to be recognized by the resolution. In order to minimize such noise, a filtering technique is used in image processing.

The feature point detection technique can be used for depth images where the noise is minimized through the filter. Feature point detection is used in various fields. The feature points of the object are detected in the depth image to detect the movement path of the object, and it is studied and used for comparative analysis of images or for machine learning.

Filtering techniques are classified into blurring and sharpening techniques. Blurring techniques include Gaussian filters, median, minimum, and maximum filters. The reason for using the blurring method in the depth image is that noise is generated in the image due to the imbalance between the sensor measurement image and the camera image. In order to remove this, it is possible to restore the hole portion of the lost image using the brightness value of the neighboring pixels. However, when only simple blurring is used, the original image is totally crushed, so that the contour of the object is blurred and there is a problem in detecting the characteristic point.

In order to improve this, there is a technique of applying blurring to only a specific portion where a hole is formed. However, this method shows high performance in a simple image, but in a pixel value that continuously changes according to a frame, the speed improvement is small due to continuous operation. In order to accurately extract the feature points from the blurred image, a sharpening technique is additionally used.

Sharpening techniques include sharpening, unsharp mask, Sobel, and high boost. Sharpening techniques generally emphasize the distinct colors of neighboring pixels and unify the surrounding pixels into a single color, thereby sharpening the outline of the object. If a feature point is detected after the sharpening operation, the performance may be higher than when the original image has a problem. Using these convergent filters, high performance is shown in other general images than depth images.

A feature point is a method for detecting and matching key feature points as a most common method for matching images and images. There are various algorithms to find these feature points. FAST, SIFT, AGAST, SURF, and BRIEF are typical examples. In order to detect feature points, there is a condition in which there is no noise in the input image. Although it is possible to detect feature points from images with noise, the accuracy of feature points is significantly reduced. Therefore, noise must be minimized through filtering techniques. The Features From Accelerated Segment Test (FAST) uses the brightness of the image to extract feature points. The FAST algorithm is disadvantageous in that the feature detection efficiency is highly dependent on the shape of the corners, the order of distribution and query.

The embodiments described below provide a technique for improving the accuracy of detecting feature points of objects in a depth image by using a convergent image filtering technique.

Embodiments compare various fusion filters in order to improve the persistent image of depth image using depth image obtained by using Kinect sensor and compare the various feature points by using the most suitable convergent filter in depth image environment We propose a detection method.

Embodiments can minimize the noise by using the blurring filter a plurality of times because the noise of the image can not be completely removed as a result of using the blurring filter once in the depth image. Embodiments may use an unsharp filter to reconstruct the outer line portion of the crushed image.

Embodiments propose an optimal fused filter and an improved FAST algorithm for improving the performance of a fused filter and detecting feature points in depth images. Embodiments minimize the number of comparison candidates by detecting feature points to improve the speed of FAST.

A feature point detection method according to one aspect includes filtering an input image using a filter fused with a multi-blurring technique and a sharpening technique; And detecting a feature point from the filtered image by comparing pixels of a predetermined orientation on a predetermined shape generated around the target pixel with the target pixel using a predetermined number of pixels.

Wherein the filtering comprises: first filtering the input image using a blurring filter; Second filtering the primary filtered image using the blurring filter; Generating a difference image corresponding to a difference between the primary filtered image and the secondary filtered image; And sharpening the primary filtered image based on the difference image.

Wherein the step of detecting the feature point comprises: generating a circle with the predetermined number of pixels around the target pixel; Determining whether the target pixel is a minutia point by comparing pixels of the predetermined first orientations of the circle - including the east, west, south, and north - with the target pixel; And comparing, when it is determined that the target pixel is not a minutia, the pixels of the predetermined second orientations in the circle, the second orientations being located between the first orientations, respectively, and the target pixel And determining whether the target pixel is a minutia point.

BRIEF DESCRIPTION OF THE DRAWINGS Fig. 1 is a view for explaining a feature point detection method according to an embodiment; Fig.
2 is a view for explaining a waveform change when a mixing filter according to an embodiment is used;
3 is a view for explaining a waveform change caused by sharpening according to an embodiment;
4 is a diagram illustrating a feature point detection method using a weighted FAST algorithm according to an exemplary embodiment;
5 is a view for explaining a logistics management system according to an embodiment;
6 is a view for explaining a method of estimating the position of an object after a feature point is detected according to an embodiment;
7 is a diagram illustrating a hardware installation environment according to an embodiment;
8 is a view for explaining a feature point detection criterion according to an embodiment;
FIG. 9 is a view for explaining the time required for detecting the minutiae point of two fusion type filters and three fusion type filters according to one embodiment; FIG.
10 is a view for explaining images to which two fusion filters and three fusion filters according to one embodiment are applied.
11 is a view for explaining the minutiae point detection times of two fusion type filters and three fusion type filters according to one embodiment.
12 is a diagram comparing MSEs of two fusion filters and three fusion filters according to one embodiment.
Figure 13 compares PSNRs of two fusion filters and three fusion filters according to one embodiment.

It is to be understood that the specific structural or functional descriptions disclosed herein may be implemented by way of example only, and that the embodiments may be embodied in various other forms and are not limited to the embodiments described herein It does not.

The terms first or second may be used to describe various components, but such terms should be understood only for the purpose of distinguishing one component from another. For example, the first component may be referred to as a second component, and similarly, the second component may also be referred to as a first component.

It is to be understood that when an element is referred to as being "connected" or "connected" to another element, it may be directly connected or connected to the other element, . On the other hand, when an element is referred to as being "directly connected" or "directly connected" to another element, it should be understood that there are no other elements in between. Expressions that describe the relationship between components, such as "between" and "between" or "neighboring to" and "directly adjacent to" should be interpreted as well.

The singular expressions include plural expressions unless the context clearly dictates otherwise. In this specification, the terms " comprises ", or " having ", and the like, are used to specify one or more of the features, numbers, steps, operations, elements, But do not preclude the presence or addition of steps, operations, elements, parts, or combinations thereof.

Unless otherwise defined, all terms used herein, including technical or scientific terms, have the same meaning as commonly understood by one of ordinary skill in the art. Terms such as those defined in commonly used dictionaries are to be interpreted as having a meaning consistent with the meaning of the context in the relevant art and, unless explicitly defined herein, are to be interpreted as ideal or overly formal Do not.

Hereinafter, embodiments will be described in detail with reference to the accompanying drawings. Like reference symbols in the drawings denote like elements.

1 is a view for explaining a feature point detection method according to an embodiment. Referring to FIG. 1, an optimal fusion filter and an improved FAST algorithm for improving the performance of existing fusion filters and detecting feature points in depth images are proposed. The flowchart for the detection of the minutiae points is shown in Fig.

System Flowchart

Step 1: Fusion type filter

The median filter can be applied to the input image multiple times. For example, after applying the median filter to the input image, you can apply the median filter again. Then, the unsharp mask method can be used to sharpen the crushed image.

Step 2: MSE, PSNR

The quality of the image is expressed by MSE and PSNR values, and MSE (Mean Square Error) is calculated by Equation (1). In this way, the degree of loss of the original image and the filtered image can be measured.

Mean Square Error (MSE) refers to a mean square error, which is a measure of the difference in pixel values. The closer the MSE value is to zero, the less is lost in the original. MSE is the sum of the square of the difference of pixel values at all the same positions divided by the square of all pixels.

Peak Signal-to-Noise Ratio (PSNR) refers to the maximum signal-to-noise ratio and can be obtained from Equation (2).

s ² is the maximum value of the corresponding image, and can be obtained by subtracting the minimum value from the maximum value of the corresponding channel. For example, for an 8 bit gray scale image, 255 - 0 = 255. It is an evaluation index used mainly for evaluating image loss information in video. The larger the value, the higher the quality of the image. In case of lossless image, PSNR is not defined because MSE is 0.

Step 3: FAST algorithm

By using the lightweight FAST algorithm, which improves the existing FAST algorithm, the feature point searching speed can be increased.

Step 4: Feature point detection

The feature points are detected using the lightweight FAST algorithm and the results are confirmed in the images using the fusion filter.

Hybrid Filter

)로 2차 필터링을 한다. 그 후 도 2(b)에서 도 2(

) 를 빼면 도 2(c)가 만들어진다.2 is a diagram for explaining a waveform change when a mixing filter according to an embodiment is used. Referring to FIG. 2, the conventional fusion type filter is composed of two filters, blurring and sharpening. When Fig. 2 (a) is the input image, it is filtered through Fig. 2 (b) through blurring. However, since we can not remove all the noise of the image as a result of using the blurring filter once in the Kinect depth image, the blurring filter is used twice to remove the noise as much as possible. After primary blurring from Fig. 2 (a) to Fig. 2 (b), Fig. 2 (b)

) To perform the second-order filtering. 2 (b) to 2 (

) Is subtracted from FIG. 2 (c).

3 is a view for explaining a waveform change caused by sharpening according to an embodiment. According to one embodiment, sharpening occurs by adding Figure 2 (c) to Figure 2 (b), resulting in the graph of Figure 3.

The convergent filter method proposed in the embodiments applies the median filter twice but is divided into the pre-processing and post-processing to remove the noise of the image to the minimum and restore the outer line portion of the image that has been crushed using the unsharp filter. Also, the loss rate of the original image can be reduced compared to the conventional method.

Lightweight FAST Algorithm

In the conventional FAST method, the radius of the pixel p is determined, and a circle is drawn based on p, and the feature point is detected according to the ratio when the number of pixels that are lower or higher than the brightness value of p is consecutively n. However, in the depth image, the FAST method changes the values of pixels per frame, and comparing all the 16 pixels per pixel at the time of detecting the feature point consumes a lot of processing speed. In addition, since feature points corresponding to the edges must be additionally calculated among the detected feature point candidates, detection of only the candidates likely to be edges will greatly improve the performance.

4 is a view for explaining a feature point detection method using a lightened FAST algorithm according to an embodiment. Referring to FIG. 4, in the embodiments, candidate points are reduced by changing the feature point detection method to improve the speed of FAST, and feature points near the edge are detected to reduce the number of candidates. Also, the embodiments can improve the detection speed of the feature points by improving the detection method.

The proposed detection method is to detect a rectangle box with a radius of 7 and make a circle with 40 pixels around p pixels. If there are more than 3 pixels among F1, F2, F3, F4 that are lighter or darker than p, L1, L2, L3, and L4, it is discriminated as a corner if any of the two conditions is met. In this way, if the original FAST method compares 40 pixels with 40 pixels, it can be seen that the proposed method is very fast in terms of processing speed.

Object Position Estimation Method

5 is a view for explaining a logistics management system according to an embodiment.

Embodiments analyze various filter combinations to overcome problems with depth images using depth images and to improve problems due to noise in depth images. Therefore, the embodiments propose a lightweight FAST algorithm that improves the detection speed in the combination of the optimal mixing filter and the conventional FAST algorithm. Embodiments can be utilized to improve the limit of existing RFID based warehouses.

We propose an optimal fusion filter and an improved FAST algorithm to improve the performance of existing fusion filters and detect feature points in depth images. Also, an overall system operation method of modeling in 3D so that an object can be visualized using detected minutiae points is shown in Fig.

6 is a view for explaining a method of estimating the position of an object after a feature point is detected according to an embodiment. Referring to FIG. 6, when a box is placed on a pallet for the first time, feature points are detected in areas corresponding to A, B, C, and D as shown in the right figure. The average value of the minutiae concentrated in the A range is set as the position value of the A vertex. Other B, C, and D are the same. If you know the four vertices, you can calculate the size of the object. The height of the box subtracts the height d2 of the box from the initial depth d1 to determine the height of the box. Then, when the next object comes in, the right image is used as the initial image and detection is performed at the changed part. When the box is stacked on the box, the information about the position does not change much, but because the depth value changes, the difference is detected in comparison with the previous information.

Data mapping and visualization (Data Mapping & Visualization)

The database is used for data mapping, and the table information and palette information attached to the article can be visualized by mapping the data to the recognized object. The pallet's unique number, the unique number of the item, the value for the initial depth value coordinate, the object measurement height, and the tag information. The tag information used in this study can be up to 28 characters in the tag ID value and up to 6 characters in the text area of the tag. In order to replace tag information with existing waybill, the tag ID value is configured to serve as an address, and the text field of the tag is configured as an item option. For example, if you write the address of 221 street in Asan city in Asan city, Chungcheongnam-do, AS.TJ.Sumoon.221 as follows, and when you visualize it in 3D, it is used to convert specific characters into a list and convert them to a list. The text area is converted to display various options such as the item's breakage warning and fire warning, and the character is parsed in the same manner as the ID. In order to render the items loaded on the pallet in 3D, the server accesses the database and fetches the corresponding information in the form of JSON. The parsed information is displayed to the administrator through the conversion.

Experiment environment

7 is a diagram illustrating a hardware installation environment according to an embodiment. The equipment used is shown in Table 2 and the experimental environment is shown in FIG. The reader antenna was installed in UHF passive RFID in three directions, and it was experimented by loading one item on the pallet with tag. The server was developed through node.js, and the database used MySQL. The mask size of the image filter used in the experiment is 3 × 3, and the combination of existing fusion filter and various fusion filters is compared. The evaluation criteria were evaluated by MSE value and PSNR value. Since the criterion of the minutiae point is used to detect the courier box, the primary goal is to detect the vertex of the rectangle, and reducing the minutiae candidate is given top priority. In addition, the processing speed is also considered because it must be processed at high speed considering the real time environment. The standard of the goods was based on the warehouse carrying only the courier box, and the experiment was carried out on the condition that the standard of the cargo was various.

The threshold value of FAST is fixed to 11. The reason for this is to exclude an object less than 10 cm from the detection range when the threshold value is above 11 in the depth image. Depth images use Kinect depth images.

Experiment result

The size of the filter mask used in the experiment is 3 × 3, and the combination of existing fusion filter and various fusion filters is compared. The evaluation criteria were evaluated by MSE value and PSNR value. Since the criterion of the feature point is used to detect the rectangular box, reducing the feature point candidate is given priority, and the feature point of each edge is detected as the reference. The threshold value of FAST is fixed to 11. The reason for this is to exclude an object less than 10 cm from the detection range when the threshold value is above 11 in the depth image. Depth images use Kinect depth images.

8 is a diagram for explaining a feature point detection criterion according to an embodiment. Referring to FIG. 8, after the filter is applied, it is compared and analyzed with fused filters consisting only of feature point candidates that are close to the edge in an environment in which noise is minimized.

Table 3 shows the performance of conventional fusion filters. The measurement method is an average value of five measurements at intervals of 5 seconds every 150 frames, and the MSE and PSNR values of the fusion filter of the median filter and the unsharp mask filter are superior.

FIG. 9 is a view for explaining the minutiae point detection times of two fusion filters and three fusion filters according to an embodiment. Referring to FIG. 9, a graph of the feature point detection time per frame is shown. A is a combination of a Gaussian filter and a sharpening filter. B is a combination of a median filter and an unsharp mask filter. It can be seen that the variation range of the processing time is lower than that of the fusion type filters having different processing times. C is a combination of a maximum value filter and an unsharp mask filter.

Table 4 shows the average processing time, feature point, MSE, and PSNR for a given time when three filters are combined. When applying an unsharp mask filter after applying the median filter twice, the feature point candidates are only approximated to the edge and the number of candidates is the smallest when compared with other fusion type filters. The processing speed is higher than other fusion filters, but the difference is small. The loss of the original image is minimized, and the quality of the image can be maximized. Other fusion filters use the blurring method twice so that the original image is more blurred than when two filters are fused together. Therefore, the MSE value is high and the performance quality is low.

10 is a view for explaining images to which two fusion filters and three fusion filters according to one embodiment are applied. Referring to FIG. 10, there is shown an image comparing the performance of two fusion filters and three fusion filters. A is an image using two fusion filters and B is a result using three fusion filters. As can be seen from A, when two fusion filters are used, noise is still present in the image even if the noise generated from the original image is minimized. Feature points In addition to candidates that are close to the edge, candidates are located at different positions. However, B minimizes the noise generated from the original image, and the feature point detection is composed only of candidates close to the edge.

11 is a view for explaining the time required for detecting the minutiae point of two fusion type filters and three fusion type filters according to one embodiment. Referring to FIG. 11, there is shown a result of comparison analysis of the time taken to detect feature points using the FAST algorithm in two fusion filters. A is three fusion filters and B is two fusion filters. As shown in the graph, the three fusion filters have a disadvantage that the detection speed is about 50% slower than the two fusion filters.

12 is a diagram comparing MSEs of two fusion filters and three fusion filters according to one embodiment. Referring to FIG. 12, it can be seen that A is a numerical value for three fusion filters, and the MSE value representing the loss rate of the original image is lower than that of the two types.

13 is a graph comparing PSNRs of two fusion filters and three fusion filters according to an embodiment. Referring to FIG. 13, it can be seen that the three fusion filters are superior to the two fusion filters in terms of image quality.

The embodiments described above may be implemented in hardware components, software components, and / or a combination of hardware components and software components. For example, the devices, methods, and components described in the embodiments may be implemented within a computer system, such as, for example, a processor, a controller, an arithmetic logic unit (ALU), a digital signal processor, such as an array, a programmable logic unit (PLU), a microprocessor, or any other device capable of executing and responding to instructions. The processing device may execute an operating system (OS) and one or more software applications running on the operating system. The processing device may also access, store, manipulate, process, and generate data in response to execution of the software. For ease of understanding, the processing apparatus may be described as being used singly, but those skilled in the art will recognize that the processing apparatus may have a plurality of processing elements and / As shown in FIG. For example, the processing unit may comprise a plurality of processors or one processor and one controller. Other processing configurations are also possible, such as a parallel processor.

The software may include a computer program, code, instructions, or a combination of one or more of the foregoing, and may be configured to configure the processing device to operate as desired or to process it collectively or collectively Device can be commanded. The software and / or data may be in the form of any type of machine, component, physical device, virtual equipment, computer storage media, or device , Or may be permanently or temporarily embodied in a transmitted signal wave. The software may be distributed over a networked computer system and stored or executed in a distributed manner. The software and data may be stored on one or more computer readable recording media.

The method according to an embodiment may be implemented in the form of a program command that can be executed through various computer means and recorded in a computer-readable medium. The computer-readable medium may include program instructions, data files, data structures, and the like, alone or in combination. The program instructions to be recorded on the medium may be those specially designed and configured for the embodiments or may be available to those skilled in the art of computer software. Examples of computer-readable media include magnetic media such as hard disks, floppy disks, and magnetic tape; optical media such as CD-ROMs and DVDs; magnetic media such as floppy disks; Magneto-optical media, and hardware devices specifically configured to store and execute program instructions such as ROM, RAM, flash memory, and the like. Examples of program instructions include machine language code such as those produced by a compiler, as well as high-level language code that can be executed by a computer using an interpreter or the like. The hardware devices described above may be configured to operate as one or more software modules to perform the operations of the embodiments, and vice versa.

Although the embodiments have been described with reference to the drawings, various technical modifications and variations may be applied to those skilled in the art. For example, it is to be understood that the techniques described may be performed in a different order than the described methods, and / or that components of the described systems, structures, devices, circuits, Lt; / RTI > or equivalents, even if it is replaced or replaced.

Claims (3)

delete Filtering an input image using a filter fused with a multi-blurring technique and a sharpening technique; And
Detecting a feature point from the filtered image by comparing pixels of a predetermined orientation on a predetermined shape generated around a target pixel with the target pixel using a predetermined number of pixels
Lt; / RTI >
The filtering step
First filtering the input image using a blurring filter;
Second filtering the primary filtered image using the blurring filter;
Generating a difference image corresponding to a difference between the primary filtered image and the secondary filtered image; And
Based on the difference image, sharpening the first filtered image
Wherein the feature point detection method comprises the steps of: Filtering an input image using a filter fused with a multi-blurring technique and a sharpening technique; And
Detecting a feature point from the filtered image by comparing pixels of a predetermined orientation on a predetermined shape generated around a target pixel with the target pixel using a predetermined number of pixels
Lt; / RTI >
The step of detecting the feature point
Generating a circle with the predetermined number of pixels about the target pixel;
Determining whether the target pixel is a minutia point by comparing pixels of the predetermined first orientations of the circle - including the east, west, south, and north - with the target pixel; And
If it is determined that the target pixel is not a minutia by comparing the pixels of the predetermined second orientations of the circle and the second orientations between the first orientations and the target pixel , Determining whether the target pixel is a minutiae
Wherein the feature point detection method comprises the steps of:

Images