KR101137003B1 - Method for operating touch of touch screen having camera - Google Patents

Abstract

PURPOSE: A method for operating a touch of a touch screen with a camera is provided to obtain information about an unclear touch point by a correction algorithm. CONSTITUTION: A touch screen system initially sets a touch screen. The system checks a touch through a white and black image which is captured by camera sensors(L1,C1,C2,R1) in real time. The system scans an image by a camera sensor which is close to a touch point. When a plurality of touch points are obtained, the system extracts a real touch point. The system corrects a coordinate of the extracted touch point.

Description

Touch operation method of touch screen with camera sensor {METHOD FOR OPERATING TOUCH OF TOUCH SCREEN HAVING CAMERA}

The present invention relates to a touch operation method of a touch screen equipped with a camera sensor. Specifically, a touch provided with a camera sensor provided with a plurality of camera sensors to accurately determine a touch point regardless of the size of the touch screen. It relates to a touch operation method of the screen.

A touch panel, commonly called a touch screen or a touch display, constitutes an interface between an information communication device and a user using various displays, and is generally attached to a display device composed of a touch module to touch a touch object touched by a finger or a pen. The display device configured in hardware in the touch screen unit is one of a user interface (UI) for detecting a touch position by changing electrical characteristics, and is an input device capable of interfacing with a device.

The touch screen is an input device that can be easily used by both men and women by interactively and intuitively operating the computer by simply touching the buttons on the display with their fingers. It is applied in many fields such as government offices, various medical equipment, tourism and major institutions, and traffic guidance.

The touch screen may include a capacitive type, an infrared matrix type, a camera type, and a resistive type.

Among them, the camera method finds the location information of the object by checking the object contacting the screen while the camera (infrared LED is mounted and including the image sensor) mounted on the corner of the screen observes the front of the screen. In general, two or more cameras are used, and the location information of the object is determined by combining two location angle information.

A conventional touch screen system to which the camera method is applied will be described with reference to the drawings.

1 is an exemplary view for explaining a touch screen system to which a conventional camera method is applied.

Referring to FIG. 1, the conventional touch screen system has a camera sensor mounted on one side of a screen and a retroreflective sheet attached to the other three sides. The problem with this form is that the retroreflective sheet reflects the light when the touch object touches the screen and the camera sensor accepts and reflects the reflected light.The touch point touched by the touch object is far from the camera sensor. The farther away the light reflected, there was a problem that is difficult to determine the touch.

That is, in theory, one camera can be installed at each corner of the upper surface of the touch screen and can be seen up to the corner of the lower surface of the touch screen in a diagonal direction.However, when there are two camera sensors, one camera can be viewed per view. If there is a system edge, there is a problem that the butterfly section is generated in the corner.

The present invention has been made in view of the above problems, and provides a touch operation method of a touch screen equipped with a camera sensor provided with a plurality of camera sensors to accurately determine the touch point regardless of the size of the touch screen. Has its purpose.

Another object of the present invention is to provide a touch operation method of a touch screen equipped with a camera sensor to cover the butterfly area and widen the touch area due to the integrated camera configuration.

It is still another object of the present invention to provide a method of operating a touch screen of a touch screen equipped with a camera sensor, which can improve the discriminating power of a touch point according to an angle of view of the camera sensor.

Still another object of the present invention is to integrate a plurality of cameras on one side of the touch screen, and to correct the touch area according to the operation of the touch camera sensor and to extract the butterfly area and accurate coordinate information according to the position of the camera sensor The present invention provides a touch operation method of a touch screen equipped with a camera sensor to obtain information of an unclear touch point.

The objects of the present invention are not limited to the above-mentioned objects, and other objects not mentioned can be clearly understood by those skilled in the art from the following description.

According to an aspect of the present invention, there is provided a touch operation method of a touch screen equipped with a camera sensor, the touch operation of a touch screen system including a plurality of camera sensors to recognize a touch point touched on the touch screen. A method, comprising: a first step of performing initial setup to accept touch information; A second step of confirming a touch through image scanning through a camera sensor and performing image scanning through a camera sensor proximate to the identified touch point; And determining the number of touch points acquired by the image scanning, and if the result of the determination is that the touch points are multi- (multiple), comparing the black and white images to remove the virtual image and then correcting the coordinates. And a third step of extracting the coordinate point and extracting the coordinate point of the touch point through a calculation using the linear interpolation if the single touch point is determined.

Preferably, the first step may include determining a reference point for correcting an error caused by overlapping of touch areas in the touch screen due to the arrangement of the camera sensor; Determining outline information of a portion where a touch area overlaps with respect to the reference point; And determining a correspondence relationship between the determined outline information and the touched touch point, and performing affine transformation on each touch point based on the determined correspondence to display a touch point on a portion where the touch area overlaps. Characterized in that it comprises a step of correcting.

Preferably, the third step includes the steps of selecting a plurality of correction reference points; Measuring a difference between an actual coordinate of the touch point and a coordinate calculated by the touch screen system and storing the difference in a lookup table in advance; It is determined whether the calculated coordinates of the touched point match the calculated coordinates, and if the result of the determination does not match, the coordinates are calculated for the point where the touch is generated, and the corrected value is found after finding a certain number of calculated reference points. Calculating; And extracting correct coordinate points by applying a correction value to the calculated coordinates.

Preferably, after the third step of extracting the coordinate point of the touch point, it is determined whether there is a continuous input, and if the determination result is continuously input, after storing the extracted coordinate point, the initialization of the screen screen and know the stored coordinate point The method further includes the steps of correcting the continuous input coordinates through the second and third steps after receiving the touch input continuously after initializing the performed information, and performing coordinate extraction and displaying the coordinates on the touch screen screen. Characterized in that.

Preferably, the plurality of camera sensors, located on the upper surface of the touch screen, each one located in the corner, in the case of three camera sensors in the middle point between the camera sensor located in the corner, four camera sensors It characterized in that the two camera sensors are positioned at the midpoint between the camera sensor located in the corner.

Preferably, in the case of the four camera sensors, the field of view of the two camera sensors to face the bottom surface of the touch screen and at the same time facing each other, or the field of view of the two camera sensors is the bottom surface of the touch screen The touch screen system by mounting the two camera sensors so as to face each other and to face each other or to have a vertical field of view with respect to the bottom surface of the touch screen. It features.

Preferably, the linear interpolation is characterized by having the same characteristics as in Equation 1 below.

(1)

Here f (xi, yi) means the correction angle.

According to the above-mentioned problem solving means, the present invention integrates a plurality of cameras to one side of a touch screen and extracts butterfly area and accurate coordinate information according to the touch implementation and the position of the camera sensor according to the operation of the touch camera sensor. It is effective to obtain the information of the unclear touch point by applying a risk correction algorithm.

In addition, there is an effect that can accurately determine the touch point regardless of the size of the touch screen.

1 is an exemplary view for explaining a touch screen system to which a conventional camera method is applied.
2 is an exemplary view illustrating a touch screen system to which a touch operation method of a touch screen equipped with a camera sensor is applied according to an exemplary embodiment of the present invention.
3 is an exemplary view for explaining a method of sensing a touch point through a camera sensor according to an embodiment of the present invention.
Figure 4 is an exemplary view for explaining the operation of the camera sensor when using the four camera sensors in accordance with an embodiment of the present invention.
5 is an exemplary view showing a line drawing according to a camera sensor arrangement method according to an embodiment of the present invention.
6 is an exemplary diagram for explaining a portion where a touch region section overlaps according to an exemplary embodiment of the present invention.
7 is an exemplary diagram for explaining a shading phenomenon.
8 is an exemplary view showing a position of a camera sensor according to a coordinate correction algorithm according to an embodiment of the present invention.
Figure 9 is an exemplary view showing a form for making a precise measurement by dividing the entire area of the touch screen by 8 * 8 according to an embodiment of the present invention.
10 is an exemplary view for explaining coordinate correction according to an embodiment of the present invention.
11A and 11B are flowcharts illustrating a touch operation method of a touch screen equipped with a camera sensor according to an exemplary embodiment of the present invention.
12 is an exemplary diagram for explaining touch points of overlapping regions of a touch region according to an exemplary embodiment of the present invention.
13 is a flowchart illustrating correction of overlapping portions using an affine technique in accordance with an embodiment of the present invention.

In the following description, specific details of the touch operation method of the touch screen equipped with the camera sensor of the present invention are shown to provide a more general understanding of the present invention, and the present invention may be easily made without these specific details and by their modifications. It will be apparent to one of ordinary skill in the art that the present invention can be practiced in a timely manner.

In the following description, the affine technique refers to unfolding the outer portion where the touch regions overlap to use a portion where the touch regions overlap due to the arrangement of the camera sensors as a single touch region. We will use it immediately, which means to convert a to a single touch area.

Hereinafter, with reference to the accompanying drawings, preferred embodiments of the present invention will be described in detail, focusing on the parts necessary to understand the operation and action according to the present invention.

2 is an exemplary view illustrating a touch screen system to which a touch operating method of a touch screen equipped with a camera sensor is applied according to an exemplary embodiment of the present invention.

Referring to FIG. 2, three and four camera sensors are mounted, and three or four camera sensors are provided on an upper surface of a touch screen, and a retroreflective sheet is provided on two remaining sides and a bottom surface. It is provided.

First, FIGS. 2A to 2C having four camera sensors will be described. At this time, the upper edge of the touch screen is provided with one camera sensor.

2 (a) is mounted in a direction facing each other while the field of view of the two camera sensors located on the top of the center of the touch screen toward the bottom surface.

FIG. 2B shows the two camera sensors positioned at the top of the center of the touch screen facing the floor and are mounted in opposite directions.

FIG. 2 (c) shows that the viewing directions of two camera sensors positioned at the top of the center of the touch screen may be perpendicular to the bottom surface of the touch screen.

Next, referring to FIG. 2D having three camera sensors, one camera sensor is used at each corner of the touch screen, and one camera sensor is disposed between the camera sensors positioned at each corner. Position the camera sensor. The direction of view of one camera sensor positioned at the center portion may be perpendicular to the bottom surface.

In the above-described present invention, a form in which a plurality of camera sensors are mounted may be proposed in various forms as shown in FIG. 2. In the center of the upper surface of the touch screen, three cases with the camera sensor facing downward and four cases are proposed.

In the present invention is to improve the correct touch discriminating ability by naturally following the overlapping part of the touch area due to the enlarged field of view of the camera according to the position of the camera sensor by the camera sensor used to alleviate the area that the touch can not be detected. There are ways to mount four sensors, but you can also mount three to widen your field of view by 180 degrees.

In the case of four camera sensors, two camera sensors are added in the center, and the sensor is arranged in the top center side by side with the field of view facing downward, and the sensor direction is staggered sideways, When the direction is arranged inwardly, four types are suggested.

That is, the configuration of the camera sensor arrangement configured in this way is to detect a touch point accurately by arranging or staggering more than two camera sensors on the top surface of the touch screen. In addition, the role of the secondary camera is to eliminate the butterfly section due to the placement and operation of the camera sensor, to obtain a more visible view of the screen, thereby detecting the correct touch point.

3 is an exemplary diagram for describing a method of sensing a touch point through a camera sensor according to an exemplary embodiment of the present invention.

Referring to FIG. 3, in FIG. 3A, two camera sensors located on the upper surface of the touch screen are bundled, and even when the L1 and C2 are used, the camera sensors L1, even when only one touch is used in the butterfly section without multi-touch. Because C2 determines the touch point, the camera sensor R1 on the opposite side can be difficult to see. Accordingly, the auxiliary camera C2 processes the touch point that the field of view of R1 does not reach and can clearly identify the touch point.

In FIG. 3B, when three camera sensors are used, the touch points may be determined using the camera sensors L1 and C1. As a result, the auxiliary camera C1 processes the touch point that the shaded R1 field of view does not reach and can clearly determine the touch point.

As described above, according to the arrangement of the plurality of camera sensors, when a single touch or multi-touch is performed on the touch screen, a close camera sensor positioned near the touch point determines touch.

Referring to the case of using the four camera sensors proposed in the present invention with reference to Figure 4 attached, L1, C2 operates in a pair (Pair), C1 and R1 operates in pairs to determine the touch point do. In this case, since the touch coordinates cover the entire area of the touch screen by L1 and R1, the touch coordinates are used as C2 in FIG. 4A and C1 in FIG. 4B.

Of course, the field of view of the camera sensor is wide enough to see even a single camera, but the use of a secondary camera to extract more accurate coordinates.

The field of view of such a camera sensor is configured to be 90 degrees or more to widen the viewable area, thereby alleviating the butterfly area, which is the problem described above.

For example, when a straight line is drawn in the middle of the touch screen as shown in FIG. 5, the line can be naturally drawn regardless of the camera arrangement configuration. This means that the cameras on both sides can be seen widely, but when the screen is bigger, the light received from the retroreflective sheet is not well received if it is far from the camera. Flick is maintained.

In view of the effect, a wide field of view of the camera sensor naturally uses a wide touch section, and a butterfly region may be relaxed to accurately determine touch information of a touch point touched from a touch object on the screen surface.

When the camera sensor is operated to extract the coordinates of the touch point, the above-described touch screen system divides the touch area by tying the camera sensors into L1, C2, and C1 and R1 by two camera sensors. . 6 (b) divides the touch area by tying the camera sensors into L1, C1, C1, and R1 by two camera sensors.

Even if several touch screens are not overlapped as one, even if the touch area is divided into one touch screen, the overlapping portions are generated as shown in FIG.

Accordingly, in the present invention, the portion where the touch region overlaps due to the arrangement of the camera sensor is corrected using the affine technique. This will be described in detail below.

Meanwhile, in the present invention, as shown in FIG. 7, first, when two or three camera sensors have the same shading phenomenon.

In other words, in using the touch screen, the shading phenomenon is recognized, but the touch point is not recognized correctly depending on the angle seen from the position of the camera sensor.

Referring to FIG. 7, (a) of FIG. 7 having two camera sensors and (b) of FIG. 7 having three camera sensors, the first touch seen in the case of camera sensor A when at least one touch point is taken The point obscures the second touch point behind it and recognizes it as only one touch point and receives only the angle. On the contrary, in the case of camera sensor B, the angle seen is opposite to camera sensor A. You can get coordinates and angles.

In the present invention, a coordinate correction algorithm is used.

The coordinate correction algorithm uses the same method regardless of three or four camera sensors. The coordinate correction algorithm is as follows.

Precise measurement is performed by dividing the touch screen screen by 8x8. It calculates angles for physical coordinates, LR coordinates, and RC coordinates to form a correction angle in the form of a look-up table.

Here, LC and RC refer to Left-sensor, Right-sensor, and Center-sensor as described in the coordinate extraction algorithm. Each of them is called LC or RC.

In the case of the center-sensor, since two more camera sensors are mounted among the screens proposed by the present invention, each of them is referred to as C1 and C2, as shown in FIG. 8.

10 is an exemplary view for explaining a form for performing a coordinate correction operation according to an embodiment of the present invention.

Referring to FIG. 10, as an image divided by 8 * 8 for coordinate correction, FIG. 9A when four camera sensors and FIG. 9B when three camera sensors are similarly applied. do.

For coordinate correction, look-up table is created. In this case, linear interpolation is performed inside 8 * 8 to create a look-up table. The calculation method for linear interpolation is divided into LR look-up table and RC look-up table and calculated by physical coordinates and LR and RC coordinates.

Referring to the accompanying FIG. 10, first, the calculation method is as follows.

The touch point A is a finger or pen touch, and the physical coordinate measurement at the touch is (220, 213). However, this coordinate information is incorrectly extracted due to a problem of a camera sensor. The position to be corrected to obtain an accurate coordinate measurement is the point labeled 1 (230, 216).

The reason why the measured value is different from the actual measured value is that the camera sensor does not receive light from the retroreflection so that it does not bring accurate coordinate information. Therefore, we want to correct the direction of the touch point where the touch object touches as indicated by the arrow direction.

The difference between the coordinate measurement value of A and the coordinate measurement value of No. 1 is (-10, -3), and this difference value is called a correction angle.

     The calibration method just described uses the linear interpolation formula. The other touch point correction method also corresponds to this.

The linear interpolation formula is shown in Equation 1 below.

Here, f (xi, yi) means the correction angle, which means LC correction angle or RC correction angle in the look-up table.

11A and 11B are flowcharts illustrating a touch operation method of a touch screen equipped with a camera sensor according to an exemplary embodiment of the present invention.

11A and 11B, when power is supplied to the touch screen, a setting for receiving information of the touch is made. In this state, the screen is initialized before the touch object touches the touch screen, and data is initialized, such as exchanging data to receive information on the touched position. Afterwards, multiple camera sensors capture the image before and after the touch object touches in real time.

If the touch object touches the touch screen during capturing, it is confirmed that the touch is made with the black and white image when the touch is made, and the sensor closest to the touch point scans it and the sensor can see all the touch points. To capture the screen image image as a secondary camera.

When the process of capturing is complete, determine the number of touch points and check whether it is single or multi.

If it is single, it performs coordinate extraction operation. This operation includes a process of correcting using the linear interpolation formula, and the process thereof corresponds to the description of coordinate correction, which is the description of FIG. 11 described above.

If it is multi, the process proceeds to removing the virtual image by comparing the image of the black and white image captured in the previous step. By removing the virtual image, it extracts the coordinate point of the actual touch point rather than the virtual image and plays a correction role.

The correction process of the extracted coordinate points will be described with reference to FIG. 11B. First, a plurality of correction reference points are selected.

Thereafter, the difference between the coordinates of the ideal state of the touched point (hereinafter, referred to as 'P0') and the coordinates calculated by the controller of the touch screen (hereinafter, referred to as 'P1') is measured.

Thereafter, the measured difference is stored in a look-up table (LUT).

When a touch is recognized at an arbitrary point, it is determined whether P0 and P1 coincide.

As a result of the determination, if it does not match, the coordinates are calculated by the controller P1 of the touch screen at a point where an arbitrary touch is generated, and the calculated coordinates are decomposed into four vectors.

After that, the corrected coordinates are calculated and applied to the calculated coordinates and transmitted to the host.

If P0 and P1 coincide, there is no need to calculate the correct coordinate by applying the correction value, so the coordinate correction process is terminated.

After the coordinate correction process is finished, it is determined whether there is a continuous input.

If it is not input continuously, it initializes from the information of the coordinate point extracted in the previous step and goes back to the standby state after initializing process to receive the information data of touch again.

If it is accepted continuously, the data extracted from the coordinate point is stored in the memory and initialized to confirm the touch of the previous process, and the process of extracting the coordinate point and transferring the extracted coordinate information to the host and outputting it to the screen. To be continuous.

Up to this point, the operation flow of single or multi-touch.

Meanwhile, a process of correcting an error caused by overlapping of touch areas is performed in a setting process for receiving information of the touch.

This will be described with reference to the accompanying FIG. 12 to help understand the technical description thereof.

In the case of four camera sensors in FIG. 12, a portion where the regions of L1 and C2 overlap with the regions of C1 and R1 should be corrected. Since the areas overlap, the calibration process must be performed to obtain accurate touch points.

In the present invention, the touch error of overlapping regions is corrected using an affine technique. The Affine technique is called a linear geometric transformation that consists of rotations, transitions, changes in magnitude, and a combination of these, and is called an affine transform. The method flow for correcting overlapping regions using the affine technique is as follows.

13 is a flowchart illustrating correction of overlapped portions by an affine technique in accordance with an embodiment of the present invention.

Referring to FIG. 13, step S1 is a predecessor for determining a reference point for transformation (Model registration). This preceding task corresponds to the above described process of correcting coordinates. This is because it is a series of processes that must be performed to recognize a touch point anywhere in the entire touch screen. In the step S1, a series of processes for correcting coordinates are performed as a prior work.

In step S2, the outer information of the overlapping part is determined and the outer part is passed to the main controller (Input image). In step S3, a correspondence relationship is determined in the input image, which determines a correspondence relationship between an overlapping outer portion and a touch point touched on the outside (Find boundary). In step S1, it is determined whether the coordinate corrected information and the information of the touch point touched on the outside are the same and determine the correspondence in the preceding work performed in the series of steps. In step S4, Affine transformation is performed with information corresponding in step S3 (Affine Transform & matching).

When a touch is made on the overlapping area, the touched touch point is obtained as follows and corrected according to the above description.

For example, if the touch point is overlapped with the areas of L1 and C2, the touch points of the areas L1 and C2 are corrected and used, ignoring the areas of C2 and R1.

While the present invention has been described in connection with what is presently considered to be the most practical and preferred embodiment, it is to be understood that the invention is not limited to the disclosed embodiments, but is capable of various modifications within the scope of the invention. Therefore, the scope of the present invention should not be limited by the illustrated embodiments, but should be determined by the scope of the appended claims and equivalents thereof.

Claims (7)

In the touch operation method of the touch screen system having a plurality of camera sensors to recognize the touch point touched on the touch screen,
A first step of performing initial setting of the touch screen to recognize the touch point;
A second step of checking whether a touch is made through a black and white image captured by the camera sensor in real time and performing image scanning by a camera sensor close to the touch point as a result of the checking; And
The number of touch points acquired by the performed image scanning is determined. When a plurality of touch points are obtained as a result of the determination, the black and white images captured in real time are compared with each other to remove virtual images, and then the actual touch points are extracted and extracted. Correct the coordinates of the actual touch point by using linear interpolation (Linear interpolation) and output, and if the result of the determination is obtained one touch point by correcting the coordinates of the obtained one touch point by using the linear interpolation method Including the third step,
In the first step,
Determining a reference point for correcting an error caused by overlapping of detection areas of a touch screen detected by each camera sensor or a group of camera sensors among the plurality of camera sensors;
Determining outline information of a portion where the sensing region of the touch screen overlaps with respect to the reference point;
Determining a correspondence relationship between the determined outer information and the touch point, and then correcting the touch point of the portion where the sensing region overlaps by performing an affine transformation on the touch point according to the determined correspondence. Touch operation method of a touch screen system equipped with a camera sensor comprising a. delete The method of claim 1, wherein the third step,
Selecting a plurality of calibration reference points;
Measuring a difference between the actual coordinates of the correction reference point and the coordinates calculated by the touch screen system and storing the difference in the lookup table in advance;
It is determined whether the actual coordinates and the calculated coordinates coincide. If the determination does not match, the calculated coordinates are decomposed into a plurality of vectors facing the plurality of correction reference points, respectively, and stored in the lookup table for the decomposed vector. Calculating a correction value using the difference;
And calculating a coordinate obtained by applying the correction value to the calculated coordinates. The touch screen of claim 1, wherein the touch screen is initially set after storing the touch points continuously touched in the memory, and the touch points stored in the memory are stored in the second and third steps. And performing a step of outputting the coordinates. The method of claim 1, wherein the plurality of camera sensors,
Located on the upper surface of the touch screen and located one at each corner, in the case of three camera sensors, one camera sensor is positioned at an intermediate point between two camera sensors located at the corner, and in the case of four camera sensors, A method of operating touch of a touch screen system with a camera sensor, characterized in that two camera sensors are positioned at an intermediate point between two camera sensors located. The method of claim 5, wherein the four camera sensors,
The two camera sensors located at the intermediate point are mounted in a direction facing each other at the same time, with the field of view facing the bottom surface of the touch screen, or the two camera sensors are directed toward the bottom surface of the touch screen. At the same time or mounted in a direction facing each other, the two camera sensors are mounted so as to have a vertical field of view based on the bottom surface of the touch screen, the touch operation method of the touch screen system with a camera sensor. The method of claim 1, wherein the linear interpolation method,
Touch operation method of a touch screen system equipped with a camera sensor, characterized in that performed by Equation 1 below.
(Equation 1)

Where f (xi, yi) means the correction angle.

Images