KR20120018457A - Method and device for estimating touch pressure using image, recording medium for the same, and mobile device comprising the same - Google Patents

Abstract

PURPOSE: An image-based touch strength measurement method, an apparatus thereof, recording medium, and a mobile device are provided to efficiently prevent measurement error according to the external vibrations of an acceleration sensor. CONSTITUTION: An image processing unit(20) detects the blurring degree of an image from a camera unit(10). An operation unit(30) calculates touch strength according to the blurring degree of image frames. The image processing unit quantizes the blurring degree. The operation unit calculates touch strength according to an effective blur value.

Description

Image-based touch intensity measurement method, apparatus, recording medium and mobile device comprising the same {Method and device for estimating touch pressure using image, recording medium for the same, and mobile device comprising the same}

The present invention relates to an image-based touch intensity measurement method, an apparatus, a recording medium for the same, and a mobile device including the same. More specifically, the touch strength can be measured without an expensive touch intensity (pressure) sensor panel, and thus economical touch is possible. The present invention relates to an image-based touch intensity measuring method capable of interface, an apparatus, a recording medium for the same, and a mobile device including the same.

Touch screen means that if a person's hand or an object touches a character or a specific location on the screen (screen) without using a keyboard, the input screen can be input directly on the screen so that the location can be detected and processed by the stored software. It refers to a screen that can be received, and is widely used as an input screen of a user terminal such as a mobile phone or a computer. The touch screen is generally divided into a pressure sensitive type and a capacitive type (capacitive type), and the pressure sensitive type has an advantage that there is no limitation on the input device by detecting a pressure change on the touch screen. The capacitive type senses a touch by using a minute current flowing through the body, and has the advantage of multi-touch and quick response. However, since the capacitive touch screens commercialized so far cannot distinguish the pressure (intensity) to touch, only the on-off type touch control is possible. Furthermore, in the conventional touch input strength measurement, the touch intensity is measured according to the contact between the touch pad and the electrode, but in this case, a problem of deterioration of image quality due to the spaced space between the touch pad and the electrode layer occurs.

On the other hand, Korean Patent Laid-Open Publication No. 10-2009-0127544 measures a contact time using an acceleration sensor, obtains a signal for each time based on signal information of the acceleration sensor during the measured time, and then obtains a dispersion value obtained during ontime. All of them are added to provide a technique for measuring the contact strength. However, since the above technique is a method of summing signal information of all measured acceleration sensors, when the actual vibration is caused by an external vibration (for example, a bus) rather than a human touch, the vibration is generated according to the external vibration. There is a problem in that all sensor signal information is added to the intensity of the touch input.

Accordingly, an object of the present invention is to provide a touch strength measuring method, an apparatus and a recording medium therefor, and a mobile device using the same, which can measure the touch intensity of a user without limiting the type of touch screen.

In order to solve the above problems, the present invention provides a method for measuring the touch strength of a mobile device, characterized in that for measuring the touch intensity to the mobile device through the blur of the image frame obtained by the camera provided in the mobile device. .

In one embodiment of the present invention, the blur of the image frame is extracted from the color difference between the frame pixels, and the method includes: obtaining a plurality of frames of an external image through a camera provided in the mobile device; Extracting a degree of blur of the obtained image frame; And calculating a touch intensity of the mobile device based on the frame having the greatest blurring degree among the frames.

The plurality of frames of the external image may be sequentially obtained during a time domain including a touch generation point and an end point, and the plurality of frames of the external image may include a frame of a time before the touch generation point.

In addition, the smaller the color difference, the greater the blur of the image frame, and the greater the blur, the greater the touch intensity can be calculated.

In order to solve the above further problem, the present invention provides a touch intensity measuring device of a mobile device, the device comprising: a camera unit 10 for taking an external image; An image processor 20 for detecting a degree of blur of an image acquired by the camera unit 10; And an operation unit 30 for calculating the touch intensity according to the difference in the degree of blur between the image frames.

In one embodiment of the present invention, the image processing unit 20 may quantify the blurring degree to a blur value according to Equation (1) below.

는 시간 t에서의 전체 픽셀의 평균 밝기 또는 색상값임)Where d (t) is the inverse dispersion coefficient of the color difference between pixels, n is the total number of pixels in the image frame, y _ti is the average brightness or color value of pixel i at time t,

Is the average brightness or color value of all pixels at time t)

The operation unit may calculate the touch intensity according to an effective blur value excluding the blur value of the image frame obtained before the touch from the maximum blur value of the image frame acquired during the touch. You can calculate the touch strength by intensity.

In another embodiment of the present invention, the touch intensity measuring device may further include an output unit 40 for outputting the calculated touch intensity as an electrical signal.

The present invention provides a mobile device including the above-described touch intensity measuring apparatus and a computer-readable recording medium having recorded thereon program instructions for driving the above-described apparatus and method.

The touch strength measurement method according to the present invention and the touch strength measurement without the expensive touch strength (pressure) sensor panel is possible, an economical touch interface is possible. In addition, it is possible to solve both the problem of deterioration of image quality caused by the provision of a separate pressure sensor and the problem of difficult measurement of touch strength in the electrostatic method. Measurement errors due to external vibrations or movements can be effectively prevented.

1 is a step diagram of a touch intensity measurement method according to an embodiment of the present invention.
2A to 2C and 3A to 3C are diagrams illustrating a degree of blurring according to an embodiment of the present invention.
4 is a view for explaining a blurring degree extraction method according to an embodiment of the present invention.
FIG. 5 is a diagram illustrating a case where two frames (t-1, t-2) acquired before t time are used for calculating a touch intensity.
6 is a photograph showing an example in which the blur degree Blur of an image obtained by the above-described method is quantified by a color dispersion value.
FIG. 7 is a diagram illustrating a method of measuring touch intensity according to the degree of blurring of an image.
8 is a diagram illustrating an example of calculating a degree of blur according to an embodiment of the present invention.
9 is a block diagram of a touch strength measuring device according to an embodiment of the present invention.

In order to fully understand the present invention, the operational advantages of the present invention, and the objects achieved by the practice of the present invention, reference should be made to the accompanying drawings which illustrate preferred embodiments of the present invention and the contents described in the accompanying drawings.

Hereinafter, exemplary embodiments of the present invention will be described in detail with reference to the accompanying drawings. As those skilled in the art would realize, the described embodiments may be modified in various different ways, all without departing from the spirit or scope of the present invention. In addition, in order to clearly describe the present invention, parts irrelevant to the description are omitted, and the same reference numerals in the drawings indicate the same members.

Throughout the specification, when an element is referred to as " including " an element, it does not exclude other elements unless specifically stated to the contrary. The terms "part", "unit", "module", "block", and the like described in the specification mean units for processing at least one function or operation, And a combination of software.

The present invention measures the touch intensity of a mobile device based on a phenomenon in which an image obtained through an image sensor such as a camera is blurred by a touch (blur, blurring, blur). That is, unlike the prior art to remove the blur caused by hand shaking, the present invention measures the touch intensity through the degree of blur of the image generated by the touch. Accordingly, when the blurring degree of the image detected by the touch intensity measuring method and apparatus according to the present invention is large, a stronger touch is calculated, and when the blurring degree is less, a weaker touch is calculated and detected. In the present invention, a plurality of frames of the external image are sequentially obtained during a time domain including a touch generation point and an end point. Hereinafter, a method of measuring touch strength according to the present invention will be described in detail.

1 is a step diagram of a touch intensity measurement method according to an embodiment of the present invention.

Referring to FIG. 1, the method acquires a plurality of frames of an external image during a time domain including before and after a touch time through a camera (S110). Thereafter, after extracting the degree of blur of the obtained image frame (S120), the touch intensity of the mobile device is calculated based on the frame with the largest degree of blur of the frame (S130).

Accordingly, the method and apparatus for measuring touch strength according to the present invention compares the degree of blurring of a plurality of externally obtained image frames sequentially during a time domain including a touch start point and an end point, and then selects a frame having the largest blur level. The touch strength is calculated as a reference.

In order to measure the quantitative touch intensity, the present invention provides a method of quantifying the blurring degree of an image, which will be described in more detail below.

2A to 2C and 3A to 3C are diagrams illustrating a degree of blurring according to an embodiment of the present invention.

2A to 2C, it can be seen that as the shaking becomes stronger, the acquired image is further blurred. That is, it can be seen that the blurring of the image generated by touching the mobile device with the strongest force (FIG. 2C) is larger than that of FIGS. 2A and 2B. This can be confirmed even in units of pixels a1 to a9 (see FIGS. 3A to 3C).

In order to quantify the degree of blurring that increases with the touch intensity, the present invention uses a color difference (the color difference of black and white eventually corresponds to the brightness difference, and thus, the color difference should be interpreted as a term including all the brightness differences). That is, in one embodiment of the present invention, after extracting color blobs of the pixels of the acquired image, the color difference of the pixels is obtained using the extracted blobs. The degree of blurring can then be obtained by obtaining a dispersion value of the color difference obtained and then averaging it for all pixels. For example, the greater the degree of blur, the smaller the color difference between pixels, whereas the smaller the degree of blur, the greater the color difference at the pixel boundary.

4 is a view for explaining a blurring degree extraction method according to an embodiment of the present invention.

Referring to FIG. 4, in the left image, a distinct color difference is seen at the pixel a1 to a9 boundaries, but in the right image, as the shaking increases (that is, even from a touch of strong force), the color at the pixel boundary You can see that the car is gone. Therefore, the blurring degree (blur value) may be calculated as the inverse dispersion coefficient d (t) of the color difference, as shown in Equation 1 below.

는 시간 t에서의 전체 픽셀의 평균 색상값임.Where n is the total number of pixels in the image frame, y _ti is the average color value of pixel i at time t,

Is the average color value of all pixels at time t.

The degree of blur of the image frame at a specific time may be calculated through Equation 1, and as d (t) is larger, the color difference of the image image may be interpreted as distinct.

The present invention can also calculate the touch intensity according to the following Equation 2 through the difference between the blur value before touch generation and the blur value after touch generation obtained from the above equation.

In the above equation, P is a calculated touch intensity, d (t-m) is a blur value for a frame before the touch generation time t, and d (t) is a blur value for a frame after the touch generation time t. Referring to Equation 2, if the blur value difference between the two frames is large, it can be seen that the image is shaken by a strong user touch. Furthermore, in order to not be affected by the shaking of the mobile device generated by the user's touch, the image frame before the touch generation time should be an image frame spaced apart from the touch generation time t in time. . Therefore, in order to distinguish between the shake caused by the user's touch and the shake caused by the external environment, the frame before touch is preferably separated from the touch occurrence time and time in a predetermined level or more.

5 illustrates a case where two frames (t-1, t-2) acquired before t time are used for the touch intensity calculation.

According to another embodiment of the present invention, after determining a local feature of the acquired image, the touch intensity is measured through the color difference of the feature. In the above embodiment, after determining the point where the brightness difference is obvious among the acquired images, the touch intensity is calculated by changing the color difference at the point. To this end, an embodiment of the present invention uses a Harris method for local point specification, which will be described in detail below.

In the case of using the Harris method, first, a point where the change of the color value (or the brightness value) is sharp among the obtained image frames is defined as a corner, which is represented by the Hessian matrix H (p) of Equation 3 below. Can be calculated.

Where I is the image brightness value at coordinates x, y.

From Equation 3, the following autocorrelation matrix can be obtained.

Equation 4 is an autocorrelation matrix equation of a second differential image (see Equation 3) with respect to a window of a point in the image, where w _ij is a weight, K is an x, y point The window size at, x, y represents one point coordinate in the image for corner detection. In the above equation, the weight may be determined in the form of a uniform value or a Gaussian function, and the Harris method determines that the two eigenvalues of the autocorrelation matrix of Equation 4 have a large value and that the corner is a corner response function. It is expressed by subtracting the sum of all diagonal components from the determinant of Equation 4, which is represented by Equation 5 below.

Det (M) is a determinant value of Equation 4, and Trace (M) is a sum of all diagonal components of Equation 4 matrix. In this case, k is appropriately between 0.0.4 and 0.06. According to the above-described Harris method, if the R value is large enough, it is determined that the defined x and y coordinates are corner areas, that is, areas having a large difference in brightness values.

The present invention extracts and judges a feature region (a large brightness difference) in the image using the Harris method. In addition to the Harris method, the present invention may also use a Shi, Tomasi method, and the like. Any way in which it can be extracted can be used in the present invention, which is within the scope of the present invention.

According to the present invention, if a feature area (corner) having a large difference in color or brightness value is extracted according to the above-described method, the touch intensity measuring method according to the present invention compares between frames based on the brightness difference or the color difference in the area. Extract the touch intensity through. The touch intensity extraction method based on the brightness difference or the color difference is the same as described in Equations 1 and 2, and will be omitted below.

6 is a photograph showing an example in which the blur degree Blur of an image obtained by the above-described method is quantified by a color dispersion value.

Referring to FIG. 6, as the blurring becomes stronger, the color difference decreases, so that the total dispersion value decreases.

7 is a view for explaining a method of measuring the touch strength in accordance with the above-described degree of blurring.

Referring to FIG. 7, a plurality of image frames are obtained between a touch generation time t1 and a touch end time t2 of a user. The plurality of image frames have the same exposure time, and the stronger the touch, the greater the shaking of the mobile device, and as a result, the blurring of the frame image becomes greater.

In FIG. 7, it can be seen that the blur in the first frame after the touch occurrence time t1 is greatest. However, in the present invention, when the touch intensity is extracted only by the blurring degree of the image obtained after the touch is generated, the blur due to the existing shake in the environment (for example, the blurring due to the inherent shaking of the hand or the vibration of the bus) Attention has been paid to this problem as if it could be calculated as touch intensity.

To this end, the present invention extracts the degree of blurring before the touch occurs, the blur value that corresponds to the blur before the touch occurs (this corresponds to the inverse dispersion of the color difference in Equation 1), the touch occurs The touch intensity is calculated through the difference in the blur value, which is the maximum blurring degree.

8 is a diagram illustrating an example of calculating a degree of blur according to an embodiment of the present invention.

Referring to FIG. 8, the blur value blur 2 of the image frame acquired immediately before the touch occurs was 0.12, and the blur value blur 4 of the image frame obtained after the touch occurred was 0.42.

In this case, the blur value difference between two frames (Blur 4-Blur 2) corresponds to 0.3, and the blur value (effective blur value) is eventually blurred by the touch of the actual user. Therefore, if the effective blur value is large, the touch intensity is detected as stronger, and if the effective blur value is less, the touch intensity is detected as weaker.

9 is a block diagram of a touch strength measuring device according to an embodiment of the present invention.

9, the touch strength measuring apparatus according to the present invention includes a camera unit 10 for capturing an external image. The camera unit 10 may be provided in the mobile device, and may be any means capable of sequentially photographing an external image over time. In the present invention, as described above, the camera 10 not only captures a plurality of frames in a time range including a touch time (time between touch generation and end point), but also captures at least one image frame even before a touch point. You should be able to shoot and acquire.

Touch strength measuring apparatus according to the present invention includes an image processor 20 for detecting the degree of blur of the obtained image frame. In one embodiment of the present invention, the image processing unit 20 extracts a blur value that is a blurring degree through a color difference in an image frame, which may be obtained through an average of all pixels of the blur value calculated in Equation (1). .

The touch intensity measuring apparatus according to the present invention further includes an operation unit 30 that calculates the touch intensity according to the blur value difference of the image frame acquired by the image processing unit 20. In this case, the touch intensity measuring device according to an embodiment of the present invention calculates the touch intensity as an effective blur value excluding the blur value of the final frame before touch generation from the maximum blur value according to the touch. It is possible to reliably measure the touch strength by excluding the shaking of the mobile device due to the external environment from the shaking due to the touch.

The touch strength measuring apparatus according to the present invention may be provided in a mobile device to measure the touch intensity of the user with respect to the mobile device and output the electrical signal. Therefore, the touch strength measuring apparatus according to another embodiment of the present invention may further include an output unit 40 which receives the touch intensity calculated by the calculating unit and outputs it as an electrical signal therefrom.

The present invention also provides a mobile device comprising the above-described touch intensity measuring device, in order to solve the another problem, the mobile device according to the present invention any electronics having a camera capable of acquiring an external image Devices, in particular mobile electronics. In addition, the touch strength measuring method and apparatus according to the present invention can be embodied as computer readable codes on a computer readable recording medium. The computer-readable recording medium includes all kinds of recording devices in which data that can be read by a computer system is stored. Examples of the recording medium include ROM, RAM, optical disk, magnetic tape, floppy disk, hard disk, nonvolatile memory, and the like, and also include a carrier wave (for example, transmission through the Internet). The computer readable recording medium can also be distributed over network coupled computer systems so that the computer readable code is stored and executed in a distributed fashion.

Although the present invention has been described with reference to the embodiments shown in the drawings, this is merely exemplary, and it will be understood by those skilled in the art that various modifications and equivalent other embodiments are possible.

Therefore, the true technical protection scope of the present invention will be defined by the technical spirit of the appended claims.

Claims (15)

Method of measuring the touch strength of a mobile device, the method
And a method of measuring touch intensity of the mobile device through blurring of an image frame acquired by a camera provided in the mobile device. The method of claim 1,
The blurring of an image frame is extracted from the color difference between the frame pixels. The method of claim 2, wherein the method
Acquiring a plurality of frames of an external image through a camera provided in the mobile device;
Extracting a degree of blur of the obtained image frame;
And calculating a touch intensity with respect to the mobile device based on the frame having the largest blurring degree among the frames. The method of claim 3,
And a plurality of frames of the external image are sequentially obtained during a time domain including a touch generation point and an end point. The method of claim 4, wherein
And a plurality of frames of the external image include frames of a time before a touch occurs. The method of claim 2,
The smaller the color difference, the greater the blurring of the image frame. The method of claim 6,
The greater the blur, the greater the touch intensity is calculated. A computer-readable recording medium having recorded thereon program instructions for carrying out the method according to any one of claims 1 to 7. A device for measuring touch strength of a mobile device, the device comprising
Camera unit 10 for taking an external image;
An image processor 20 for detecting a degree of blur of an image acquired by the camera unit 10; And
Touch intensity measuring device, characterized in that it comprises a calculation unit 30 for calculating the touch intensity according to the difference in the degree of blur between the image frame. The method of claim 9,
The image processing unit 20 is the touch intensity measuring apparatus of the mobile device, characterized in that to quantify the blurring degree to the blur value according to the following equation (1).

Where d (t) is the inverse dispersion coefficient of the color difference between pixels, n is the total number of pixels in the image frame, y _ti is the average brightness or color value of pixel i at time t,

Is the average brightness or color value of all pixels at time t. The method of claim 10,
The calculating unit calculates the touch intensity according to the effective blur value excluding the blur value of the image frame acquired before the touch from the maximum blur value of the image frame acquired during the touch. . 12. The method of claim 11,
And the calculator calculates the touch intensity with a greater intensity as the effective blur value is larger. The method of claim 12,
The touch intensity measuring device further comprises an output unit (40) for outputting the calculated touch intensity as an electrical signal. A mobile device comprising the touch intensity measuring device according to any one of claims 9 to 13. A computer-readable recording medium having recorded thereon program instructions for driving a touch intensity measuring device according to any one of claims 9 to 13.

Images