KR101869507B1 - Method of Real-time Virtual Touch Recognition system for Embedded Environment and system For Real-time Virtual Touch Recognition Using This Method - Google Patents

Abstract

The present invention relates to a real-time virtual touch recognition method of an embedded environment and a real-time virtual touch recognition system using the same. In an embedded environment, a virtual touch area is photographed through a depth measurement sensor (10) to obtain a depth image, and the depth image is analyzed to recognize a virtual touch in real time. Then, virtual touch information is transmitted to a control device. The system comprises: a depth image photographing unit (S10) for photographing a depth image having a distance from the depth measurement sensor (10) within a photographing area as a pixel value; a background storage unit (S20) for storing a background depth image of a photographing area; a binarization unit (S30); a labeling unit (S40); a touch sensing unit (S50); a calibration unit (S70); a touch position correction unit (S60); and a touch transmission unit (S80).

Description

TECHNICAL FIELD The present invention relates to a real-time virtual touch recognition method and a real-time virtual touch recognition method using the same.

The present invention relates to a real-time virtual touch recognition system in an embedded environment for recognizing multi-touch in a virtual touch method and a real-time virtual touch recognition method using the same. In the virtual touch, only a simple depth camera is installed, Real-time virtual touch recognition system of embedded environment which improves accuracy of multi-touch by improving algorithm of binarization, labeling and object tracking for multi-touch recognition by recognizing touch only by difference of depth value of background and background. To a touch recognition method.

Unless otherwise indicated herein, the contents set forth in this section are not prior art to the claims of this application and are not to be construed as prior art to be included in this section.

Recently, due to the success of iPhone, touch interface has been applied to various devices. Touch recognition technology has a disadvantage in that the price increases in proportion to the screen size.

In order to overcome this disadvantage, the virtual touch method of detecting the depth of the image and confirming the touch can be applied. In the virtual touch method, the most different element from the conventional touch method is the mobility.

In other words, although the electrostatic touch recognition method most commonly used is a combination of a screen and a touch recognition device, there is a problem that the mobility is degraded in a medium to large screen environment. However, the virtual touch method has a problem in that, Can be easily installed anywhere.

Therefore, it is desirable to implement a virtual touch recognition system in an embedded environment. One of the most important factors in virtual touch recognition is to process a virtual touch in real time.

Here, real-time definition means an information processing action or system that responds accurately within a predetermined time to an input value. In order to satisfy the real-time property in the virtual touch system, in order to capture a depth image, Should react to.

In particular, real-time performance becomes more important because there is a limit to the performance of the processor in the embedded environment.

On the other hand, the method of real-time system implementation can be roughly classified into a method of modifying the scheduling policy that distributes the tasks of the system to a real-time system and a method of improving the response speed of the system.

The scheduling of real-time systems is to prioritize tasks according to a system-determined manner in order to get responses within a certain time to input system resources such as processor occupation for various tasks in the system, It is mainly used in embedded systems which are elements.

The method of applying the real-time scheduling method can satisfy the real-time efficiency efficiently in a system in which a plurality of tasks are performed in parallel or a system in which each reaction is performed on various inputs in a waiting state in order to receive input, If the algorithm in the task is complicated, there is a limitation that the real - time property can not be satisfied even if the real - time scheduling method is applied.

On the other hand, a method for improving the response speed of a system in order to realize a real-time system can be divided into a parallel processing method using a plurality of processors or process cores and a GPU, and a method of improving a reaction speed by improving an algorithm in the system .

Parallel processing refers to processing in a multiprocessor environment or in a multi-core processor by allocating to each processor or core each partitionable task within a system or module.

A system to which the above-described efficient parallel processing is applied has a high processing speed because it efficiently utilizes system resources. However, in order to apply parallel processing, care must be taken to satisfy the mutual exclusion of the critical zone. If attention is not paid to the critical zone, unexpected results arise. The method of improving the algorithm is to reduce the execution time of the algorithm by reducing the complexity of the algorithm by eliminating or optimizing the unnecessary elements in the algorithm by using the characteristics of the system for the part where the generalized algorithm is applied. The execution time can be efficiently reduced.

On the other hand, research on real - time system in image processing field is divided into tasks, and real - time systems are implemented through scheduling.

In addition, due to the development of embedded technology, the GPU may be mounted on an embedded device. By using the GPU mounted on the embedded device, it is possible to perform real-time image processing through parallel processing or parallel processing of H.264 format image , Parallel processing of HEVC coding of UHD images through multiple CPUs and GPUs, and research on estimating the relative position of human beings with images taken by robots through GPU-based parallel processing. We are actively studying the parallel image processing method.

As described above, most of the researches on processing images in real time through parallel processing, and real-time systems are implemented by applying real-time scheduling and parallel processing.

However, there is a lack of research on real - time depth image processing in embedded environment.

1. Korean Patent Registration No. 10-1386655 (Apr. 17, 2014)

2. Korean Patent Registration No. 10-1536673 (November 21, 2013)

3. Kwon, K. and Lee, DS 1986. "Implementation of Real-time Virtual Touch Recognition System in Embedded Environment", Multimedia Journals Vol. 19, No. 10

SUMMARY OF THE INVENTION Accordingly, it is an object of the present invention to provide a real-time virtual touch recognition system in an embedded environment and a real-time virtual touch recognition method using the same.

In addition, one of the most important factors in virtual touch recognition is to efficiently process virtual touches in real time, solve a price increase proportional to the size of a screen, which is a disadvantage of touch recognition technology, It is intended to implement the device.

More specifically, we propose a system that improves the virtual touch system in the binarization and labeling stage, which is deviated from the disadvantages of the existing virtual touch system, which occupies most of the processing time in the binarization and labeling stages. We will present how to implement important real-time systems.

Also, the most differentiating element from the existing touch method in the virtual touch method is mobility. Conventional electrostatic touch recognition methods have a drawback that the mobility is degraded in a medium to large screen environment because the screen and the touch recognition device are combined. We propose a method of implementing a virtual touch recognition system in an embedded environment that overcomes the problem of device mobility degradation regardless of screen size.

In addition, we propose a method for realizing a virtual touch recognition system in an embedded environment that can be applied to any size regardless of the size of the screen, while providing mobility that is a differentiating factor in the conventional touch method.

According to a preferred embodiment of the present invention,

In the real-time virtual touch recognition system of the embedded environment,

A depth image capturing unit (S10) for capturing a background depth image of a touch sensor area of the embedded device; A background storage unit (S20) for storing a background depth image of the touch sensor area; A binarization unit (S30) for generating a binary image for each object by comparing a background depth image stored in the background storage unit (S20) with a depth image taken by the depth image capturing unit (S10); A labeling unit (S40) for labeling and grouping depth images for each object of the binary image generated in the binarization unit (S30) and performing labeling; A touch sensing unit S50 for detecting whether or not each object generated from the labeling unit S40 is touched; A touch position correcting unit (S60) for correcting the touch position according to a positional relationship between the depth image and the touch region, the depth image for each object recognized by the touch sensing unit (S50); A calibration unit (S70) for setting a positional relationship between the depth image detected from the touch position correcting unit and the touch region; And a touch transmission unit (S80) for transmitting the virtual touch information detected from the touch position correcting unit (S60) to the control device. The virtual touch recognition system of the present invention is a real-time virtual touch recognition system of an embedded environment. Wherein the binarizing unit S30 is configured to binarize the partial region for the actual touch region allowed by the maximum movement distance from the first touch point to the next touch point for dragging, The labeling unit S40 may label the position information of the first object among the objects generated in the binarization unit S30 to determine a starting point at which the object is touched. If the size of the touch area of the touch screen is larger than the search area, it is judged that the touch area is not noise, and the labeling is stopped so that the step for touch point detection is performed. A real-time virtual touch recognition method for implementing a real-time virtual touch recognition embedded system in an embedded environment, the method comprising: capturing a background depth image of a touch sensor area of an embedded device through a depth camera; A background storing step of storing a background depth image of the photographing area obtained from the photographing step; A binarization step of generating a binary image for each object by comparing a background depth image of the background storing step and a photographing depth image of the photographing step; A labeling step of classifying and grouping depth images for each object of the binary image generated in the binarization step and performing labeling; A touch sensing step of sensing whether or not each object generated from the labeling step is touched; A touch position correction step of correcting a touch position according to a positional relationship between a depth image and a touch area, the depth image for each object recognized in the touch sensing step; And a touch transmission step of transmitting the virtual touch information detected in the touch position correction step to the control device. The touch position correcting step may include a calibration step of setting a positional relationship between the depth image detected from the touch position correcting step and the touch area, in the real time virtual touch recognition method of the embedded environment.

According to the real-time virtual touch recognition system in the embedded environment and the real-time virtual touch recognition method using the embedded real-time environment according to the embodiment of the present invention, as the resolution increases in the existing embedded environment, the virtual touch processing time becomes larger than the inter- The problem that the virtual touch recognition is not possible is improved, and the virtual touch can be performed in real time in the embedded environment.

In addition, the present invention has an effect of binarizing only a point at which actual touch is possible, so that the processing range is narrowed and the processing speed is remarkably increased as compared with the conventional method.

In addition, the present invention has the effect of significantly increasing the processing speed by applying a method of starting labeling with the start point of the object when it first appears.

In addition, since the present invention realizes a virtual touch in an embedded environment in real time, it can be applied to a variety of fields at low cost and easy mobility.

1 is a configuration diagram of a virtual touch recognition process.
2 is a flowchart of a real-time virtual touch recognition system.
3 is a diagram showing a binarization area.
4 is a diagram for explaining the labeling process.
FIG. 5 is a diagram of a touch information for explaining an equation for taking an error of depth information into account.
Figure 6 is a simulation device produced to illustrate the present invention.
FIG. 7 is a table showing that the processing time of the device is accelerated when touching.

BRIEF DESCRIPTION OF THE DRAWINGS The advantages and features of the present invention and the manner of achieving them will become apparent with reference to the embodiments described in detail below with reference to the accompanying drawings. The present invention may, however, be embodied in many different forms and should not be construed as being limited to the embodiments set forth herein. Rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the concept of the invention to those skilled in the art. Is provided to fully convey the scope of the invention to those skilled in the art, and the invention is only defined by the scope of the claims. Like numbers refer to like elements throughout.

In the following description of the present invention, a detailed description of known functions and configurations incorporated herein will be omitted when it may make the subject matter of the present invention rather unclear. The following terms are defined in consideration of the functions in the embodiments of the present invention, which may vary depending on the intention of the user, the intention or the custom of the operator. Therefore, the definition should be based on the contents throughout this specification.

The present invention relates to a method for recognizing multiple touches in a virtual touch method, wherein a virtual touch includes only a simple depth camera as compared with a physical touch method, and accurately recognizes a touch only by a difference between an object depth value and a background depth value The present invention relates to a system for improving accuracy of multi-touch by improving algorithms of binarization, labeling, and object tracking for multi-object recognition.

In order to realize the present invention, the steps that have a great effect on the processing time in the virtual touch recognition system are explained, the algorithm is analyzed, and the algorithm is improved according to the characteristics of the depth image appearing in the virtual touch.

This virtual touch method is defined as a 'virtual touch recognition system'.

The virtual touch recognition system realizes real-time performance in a hard-to-find embedded environment and can be manufactured at low cost when the device is commercialized as a device, thereby increasing mobility.

A real-time virtual touch recognition embedded system according to the present invention includes an embedded device to which a virtual touch recognition algorithm is applied, a depth measurement sensor (10) connected to an embedded device,

A virtual touch recognition process 20 for receiving information from the depth measurement sensor 10; And

A control device (30) connected to the virtual touch recognition process (20) by wire or wireless and receiving virtual touch information through communication; .

If the depth touch is recognized and the position of the touch point is grasped, the virtual touch recognition process 20 corrects the touch position using the positional relationship between the depth image stored in the calibration unit S70 and the touch region The touch information is transmitted to the control device 30 through wired or wireless communication with the virtual touch information. At this time, not only when the touch is made but also when the touch point is not recognized during the touch, Speed real-time real-time virtual touch recognition method. This is a real - time virtual touch recognition method that implements real - time virtual touch in an embedded environment which is impossible by the improvement of virtual touch recognition algorithm as a method for implementing virtual touch recognition that processes in real time.

The flowchart of the real-time virtual touch recognition system in the embedded environment is as follows.

A depth image capturing unit (S10) for capturing a depth image having a pixel value as a distance from the depth measuring sensor (10) in the photographing area; A background storage unit (S20) for storing a background depth image of a shooting region;

A binarization unit (S30) for generating a binary image by comparing the background depth image stored in the background storage unit (S20) with the depth image taken by the depth image capturing unit (S10) to separate the objects;

A labeling unit (S40) for removing noise from the binary image generated in the binarization unit (S30) and distinguishing each object; A touch sensing unit S50 for detecting whether or not each object is touched;

A calibration unit (S70) for setting a positional relationship between the depth image and the touch region; A touch position correcting unit (S60) for correcting the touch position according to the positional relationship between the depth image and the touch region, the depth image recognized by the touch sensing unit (S50); And a touch transmission unit (S80) for transmitting the virtual touch information to the control device. And a control unit.

In the binarization unit S30, in the binarization process, only the image within the range of the maximum distance to touch the next touch point is binarized, and when there is no virtual touch information of the previous frame, And performs a binarization only for the maximum allowable pixel distance area when the virtual touch information of the previous frame exists, thereby realizing a virtual touch recognition embedded system.

In the labeling unit S40, when the object size is larger than the threshold S, which is the process of extracting the object by labeling as the starting point of the pixel of the object obtained during the binarization process, it is determined that the object is not noise and the labeling is stopped And transmits the object information to the touch point detection step. The real-time virtual touch recognition embedded system improves the existing labeling algorithm.

FIG. 1 is a diagram illustrating an operation method of a virtual touch recognition system of the present invention, and FIG. 2 is an operation flowchart of a virtual touch recognition system according to the present invention. In addition, the order of performing the virtual touch recognition can be summarized as a step of obtaining a depth image, a binarization step, a labeling step, and a touch point extraction step.

3, when the maximum allowable pixel distance between the touch points in the image is designated as Tx in the horizontal direction and Ty in the vertical direction in the image, only the maximum allowable pixel distance area around the previous touch point Pt-1 And performs binarization.

FIG. 4 is a diagram showing that efficient labeling is performed by transmitting the position information bst of the first appearing pixel to the labeling step. If the size of the object is larger than S during the inspection of the object in labeling, it is determined that the object is not noise, and the process goes to the step of detecting whether or not to touch the labeling.

식과 같이 객체 내 한 화소 (x,y)에서의 현재 깊이 값 Dxy과 배경 영상의 깊이 값 Bxy를 비교하는 도면이다. Dxy과 Bxy의 차가 M이내일 때 그 위치에서 터치가 되었다고 판단한다. 이 때 M의 값은 깊이 카메라로 촬영한 깊이 정보의 오차를 고려한 값이다. Figure 5 shows that when an object is considered to have touched when approaching within the screen and M,

(X, y) in the object and the depth value Bxy of the background image. When the difference between Dxy and Bxy is within M, it is judged that the touch is made at the position. In this case, the value of M is a value considering the error of the depth information photographed by the depth camera.

FIG. 6 is a simulation apparatus designed for simulating real-time virtual touches.

FIG. 7 is a table showing that the processing time of the virtual touch recognition is drastically accelerated when touching.

Table 1 shows that labeling and binarization take the largest portion in the processing of virtual touch recognition, and that the touch point recognition step has a smaller weight than the previous two steps.

Therefore, it is essential to improve the algorithm to match the characteristics of the depth image that appears in virtual touch recognition in order to realize a system that recognizes virtual touch in real time in embedded device.

In Table 2, it can be seen that the binarization and labeling steps occupy most of the virtual touch processing time in the binarization and labeling step. The processing time of the binarization and labeling step is proportional to the resolution of the depth image, so it is important to operate at the appropriate depth resolution. In order to realize the real-time virtual touch in the embedded environment, the algorithm can be improved to reduce the processing time, so that the real-time virtual touch can be supported even at a higher resolution. This not only provides a smooth touch interface, but also can increase the virtual touch accuracy.

    Here, the table of the binarization algorithm of the present invention, which differs from the existing full-area binarization method, is shown in the table.

In the existing binarization step, the whole region is binarized. However, when the touch is continuously maintained for dragging, the movement distance from the existing touch point to the next touch point is limited. Using this, only the image within the range of the maximum distance to touch the next touch point is binarized. In this case, when the maximum allowable pixel distance between the touch points in the image is Tx in the horizontal direction and Ty in the vertical direction, the image photographed at the current t frame is It, the image after the binarization is Bt, and the previous touch point is Pt-1 , And the proposed algorithm can be expressed as shown in Table 3.

In the conventional labeling step, labeling is performed on the entire area. However, when the virtual touch is performed, since there is a touch target screen at the center of the image in the process of shooting the image for performing the touch, the probability that the object is also photographed near the center of the image is high. Therefore, when searching for a pixel to perform labeling, if the starting point of the object is known, the number of pixels to be searched for labeling can be reduced by half or more. For this, the binarization step, which is the previous step of labeling, transmits the position information bst where the first object appears to the labeling step to perform efficient labeling. To do this, we modify the binarization algorithm as shown in Table 4.

If the size of the object is larger than S during inspection of the object in the labeling, it is determined that the noise is not noise, and after the labeling is stopped, the process proceeds to the touch point detection step.

Such a labeling algorithm can be represented as shown in Table 5 below.

As a result of applying the proposed algorithm, labeling is performed only on the portion required for touch confirmation as shown in FIG.

In this paper, we study the steps that have a great effect on the processing time in the virtual touch recognition system, analyze the algorithm, and improve the algorithm according to the characteristics of the depth image in virtual touch.

A preferred embodiment of a real-time virtual touch recognition system in an embedded environment by improving the virtual touch recognition algorithm according to the present invention will be described in detail as follows.

In the present invention, the execution time is divided into a labeling process, a binarization process, and a virtual touch recognition process in order to check whether a virtual touch algorithm for an embedded system can be processed in real time. In this case, Raspberry pi 3 is used for the embedded system device to be used for the simulation. The Raspberry pi 3 is an embedded device with a 1.2Ghz quad processor and 1Gb of RAM. In addition, ASUS's Xtion Pro Live was used as a depth camera. This depth camera shoots images at 30 frames per second. In this case, the speed is measured by measuring a total of 100 frames. In this case, since the interval between each frame is about 34 ms, the processing time must be 33 ms or less in order to realize real-time virtual touch. The experimental apparatus is constructed as shown in FIG.

Table 6 compares the processing time when using the existing binarization method and labeling method compared with PC. In this case, the virtual touch algorithm is processed very quickly regardless of the resolution of the PC, but in the case of the embedded device, the processing speed is 78 ms in the case of the resolution of 640 × 480, and the virtual touch can not be recognized in real time.

Table 7 shows the processing speed of each stage when applying the conventional binarization and labeling method when performing a virtual touch by acquiring a depth image of 640x480 in a given embedded environment. At this time, it can be seen that most of the time is consumed in the binarization step and the labeling in the virtual touch recognition.

Table 8 compares the method of binarizing the depth image in the virtual touch recognition with the method of binarizing the entire image, which is a conventional method, and applies a method of binarizing only a partial region around the previous touch point, which is a proposed method of the present invention Is compared with the processing speed at the time.

In the non-touch state, since there is no previous touch point, there is no difference in the processing time between the conventional method and the proposed method. However, in the state of being touched, it can be seen that the time of the binarization process which was 26 ms in the conventional method is remarkably reduced to 0.8 ms in the proposed method

Table 9 shows a method of labeling a binarized image by a method of labeling an entire image, which is an existing method, and a method of starting labeling with a starting point of an object, which is a method proposed in the present invention, Compare the processing speed. When the existing labeling method is applied, the labeling process time is shown to be 26ms. However, when the labeling method is applied, the labeling process time is reduced to 6ms. It is analyzed that the labeling improvement effect is high because the probability that the object to be touched will appear in the center of the image is higher than the probability that the object to be touched will appear on the outer edge of the image.

Table 10 compares the total processing time by applying the methods proposed in the virtual touch recognition system. At this time, when the proposed binarization and labeling method of the present invention is applied, the virtual touch can be performed in real time with a total processing time of 31 ms. In this case, as shown in FIG. 7, it can be seen that the speed of recognition of the virtual touch is drastically increased during the touch.

As described above, the virtual touch recognition system is implemented in an embedded environment using an embedded device. This makes it possible to miniaturize the depth recognition device and perform virtual touch recognition in general. As the resolution increases in the embedded environment, we found a problem that real - time virtual touch recognition processing can not be performed. In order to solve this problem, it is confirmed that most of the time is consumed in the binarization and labeling process in the virtual touch recognition process, and the binarization and the labeling process are improved in accordance with the virtual touch. As a result, in the conventional 640x480 depth image, the processing time per frame was 78ms, and the proposed method was measured as 31ms, which shows that the processing speed is reduced by 60.3%. Also, real - time virtual touch recognition system can be realized by applying the proposed method.

Real-time virtual touch recognition devices are implemented in an embedded environment, which can sufficiently replace existing physical touch screens. Accordingly, the touch interface can be introduced at a low cost even in a large screen environment, and it is expected to be widely used in a touch application field.

Accordingly, according to the real-time virtual touch recognition system of the embedded environment and the real-time virtual touch recognition method using the embedded environment according to the present invention, as the resolution increases in the existing embedded environment, the virtual touch processing time becomes larger than the inter- There is an effect that the virtual touch can be performed in real time in the embedded environment and only the point where the actual touch is possible is binarized so that the touch range is narrower than the existing method and the processing speed is remarkably accelerated, The processing speed is remarkably increased by applying the method of starting the labeling with the starting point as the starting point. In addition, realization of virtual touch in real time in an embedded environment can be applied to various fields at low cost and easy mobility.

It is to be understood that the invention is not limited to the disclosed embodiments, but, on the contrary, is intended to cover various modifications and equivalent arrangements included within the spirit and scope of the appended claims. It is not limited to the embodiment.

10: Depth measuring sensor
20: Virtual Touch Recognition Process
30: Control device
S10: Depth image shooting section
S20: background storage unit
S30: binarization unit
S40: Labeling unit
S50:
S60:
S70: Touch position correcting section
S80:

Claims (6)

A depth image capturing unit (S10) for capturing a background depth image of a touch sensor area of the embedded device;
A background storage unit (S20) for storing a background depth image of the touch sensor area;
A binarization unit (S30) for generating a binary image for each object by comparing a background depth image stored in the background storage unit (S20) with a depth image taken by the depth image capturing unit (S10);
A labeling unit (S40) for labeling and grouping depth images for each object of the binary image generated in the binarization unit (S30) and performing labeling;
A touch sensing unit S50 for detecting whether or not each object generated from the labeling unit S40 is touched;
A touch position correcting unit (S60) for correcting the touch position according to a positional relationship between the depth image and the touch region, the depth image for each object recognized by the touch sensing unit (S50);
A calibration unit (S70) for setting a positional relationship between the depth image detected from the touch position correcting unit and the touch region; And
And a touch transmission unit (S80) for transmitting the virtual touch information detected by the touch position correcting unit (S60) to the control device,
The binarization unit S30 binarizes the partial region for the actual touch region allowed by the maximum movement distance from the first touch point to the next touch point for dragging,
In the binarization process, only an image within a range of the maximum distance to touch the next touch point is binarized,
When there is no virtual touch information of the immediately preceding frame, the entire area is binarized,
When the virtual touch information of the immediately preceding frame exists, binarization is performed only by the maximum allowable pixel distance region,
The labeling unit S40 labels the position information of the first object among the objects generated in the binarization unit S30 to determine a starting point where the object is touched, Is judged to be noise and the labeling is stopped, the step for detecting the touch point is executed.
delete delete delete delete delete

Images