KR20160004217A - Accelerator for providing visual feedback to touch input, touch input processing device and method for providing visual feedback to touch input - Google Patents

Abstract

A method for providing visual feedback to touch input includes the following steps: receiving a plurality of touch events from a touch sensor coupled to a display; generating an overlay segment based on the touch events; receiving a video frame; coupling the video frame to the overlay segment in accordance with the touch events in order to generate a composite video frame; and supplying the composite video frame to the display.

Description

Field of the Invention [0001] The present invention relates to an accelerator for providing visual feedback to a touch input, a touch input processing device and method for providing visual feedback to a touch input,

Embodiments of the present invention relate to display devices having touch input devices, and more particularly to systems and methods for reducing a display lag between a touch input device and a display device.

Recently, touch screen devices have become commonplace as personal mobile devices such as mobile phones, tablets, laptops, etc. have become popular. In addition to portable devices, touch screens are used in industries where keyboard and mouse systems are not possible for users to interact quickly, intuitively or accurately with the contents of the display and in places such as automobiles and kiosks have.

The touch screen displays recognize user input, for example, by sensing (or tapping) a screen, or by detecting user's fingers or hand gestures that are very close to the touch screen. May be any surface that can be projected and the touch sensed.

Touch events detected by the touch sensor panels are typically processed by high level application software running on the device's application processor (AP). Due to the many processing steps between the touch sensor panel and the AP and the non-deterministic processing time at the AP (including delays due to other computational tasks performed by the AP, such as operating the device's operating system and other applications) High levels of latency (e.g., 70 to 100 milliseconds) are introduced to reduce the responsiveness of the computing device to the user ' s touch inputs.

Some empirical studies have indicated that most people can detect even as much as 30 milliseconds of asynchrony between senses such as touching and seeing (for example, Keetels, M and Vroomen, J of London Taylor & Francis Group Perception of Synchrony Between the Senses and MM Murray and MT Wallace (eds.), Frontiers in the Neural Basis of Multisensory Processes, pp. 147-177). Delays of 50 to 200 milliseconds may be detected for most of the users of these computing devices and this may lead to an increase in user dissatisfaction by not computing the device to immediately provide feedback to the user's input.

However, since the system processor also manages other processes of the device (e.g., by running the operating system of the device and many other applications), the response time required to update the appearance of the virtual keyboard may be slower, There may be a significant rack between the time the key is touched and the time the user is provided with visual feedback on the key press.

It is an object of the present invention to provide an accelerator, a touch input processing device, and a method for providing visual feedback to a touch input, which can reduce the display latency of touch events in detail.

Aspects of embodiments of the present invention relate to reducing the latency between a touch event and a display response to the touch event.

According to one or more embodiments of the present invention, there is provided a method of providing visual feedback to a touch input, the method comprising: receiving a plurality of touch events from a touch sensor coupled to a display; Creating an overlay segment based on the plurality of touch events; Receiving a video frame; Combining a video frame with an overlay segment according to touch events to generate a composite video frame; And providing a composite video frame to the display.

In one embodiment, the video frame includes the displayed line, and the properties of the overlay segment match those of the displayed line.

In one embodiment, generating an overlay segment comprises: calculating an estimated touch path based on locations of touch events; Determining a valid region of video frames corresponding to the estimated touch path; Comparing portions of video frames corresponding to valid regions to determine differences in successive frames of video frames; Storing the differences in a memory buffer; And generating an overlay segment based on the cars to be stored, wherein the overlay segment matches the characteristics of the cars to be stored.

In one embodiment, generating an overlay segment based on stored cars comprises maintaining relative properties of pixels of the cars stored in the overlay segment.

In one embodiment, the relative attributes include the distance to local motion axes along the estimated touch path, spatial separations between pixels, and color gradients between pixels.

In one embodiment, generating an overlay segment based on stored cars comprises shifting the pixels of the cars stored corresponding to the delay introduced by the application processor to locations along the estimated touch path .

In one embodiment, the estimated touch path is calculated based on a subset of touch events corresponding to a fixed number of frames, and a fixed number of frames corresponds to a delay introduced by the application processor .

In one embodiment, combining the video frame with the overlay segment in accordance with the touch events comprises: generating a mask based on the touch events, the mask comprising a matrix of numerical values, Each of the numerical values identifies an operation for producing a composite video frame, and the positions of the numerical values in the matrix correspond to the positions of the pixels in the composite video frame.

In one embodiment, the method further comprises determining, for each pixel in the composite video frame, whether to output the corresponding pixel of the overlay segment or video frame according to a value at a corresponding location in the mask.

In one embodiment, each of the numerical values of the mask corresponds to one pixel in the composite video frame or one pixel in the composite video frame.

In one embodiment, touch events are generated as a result of the interaction between the pointing implement and the touch sensor.

According to one or more embodiments of the present invention, there is provided an accelerator for providing visual feedback to a touch input, the accelerator comprising: a processor; And a memory having instructions stored therein, the instructions, when executed by the processor, cause the processor to: receive a plurality of touch events from a touch sensor coupled to the display; Generate an overlay segment based on the plurality of touch events; Receiving a video frame; Combine a video frame with an overlay segment according to touch events to generate a composite video frame; And to provide a composite video frame to the display.

In one embodiment, creating an overlay segment comprises: calculating an estimated touch path based on locations of touch events; Determining a valid region of video frames corresponding to the estimated touch path; Compare portions of video frames corresponding to valid regions to determine differences in successive frames of video frames; Storing the differences in a memory buffer; And generating an overlay segment based on the cars to be stored, wherein the overlay segment matches properties of the cars to be stored.

In one embodiment, generating the overlay segment based on the stored cars includes maintaining relative properties of the pixels of the cars stored in the overlay segment, and the relative properties include locally shifting axes along the estimated touch path , Spatial separations between pixels, and color gradients between pixels.

In one embodiment, generating an overlay segment based on the stored cars includes shifting the pixels of the cars stored corresponding to the delay introduced by the application processor to locations along the estimated touch path.

In one embodiment, the estimated touch path is computed based on a subset of touch events corresponding to a fixed number of frames, and a fixed number of frames corresponds to a delay introduced by the application processor.

In one embodiment, combining a video frame with an overlay segment in accordance with touch events comprises: generating a mask based on touch events, wherein the mask comprises a matrix of numerical values, Identifies an operation for producing a composite video frame, and the positions of the numerical values in the matrix correspond to the positions of the pixels in the composite video frame.

In one embodiment, the accelerator further comprises, for each pixel in the composite video frame, determining whether to output the corresponding pixel of the overlay segment or video frame according to the value at the corresponding position in the mask.

According to one or more embodiments of the present invention there is provided a touch input processing device for providing visual feedback to a touch input, the touch sensitive display device comprising: a display configured to display video frames; A touch sensor, comprising: a touch sensor coupled to a display and configured to generate a plurality of touch events as a result of interaction between a pointing tool and a touch sensor; A processor; And a memory having instructions stored therein, wherein the instructions, when executed by the processor, cause the processor to: receive a plurality of touch events from the touch sensor; Generate an overlay segment based on the plurality of touch events; Receiving a video frame of video frames; Combine a video frame with an overlay segment according to touch events to generate a composite video frame; And to provide a composite video frame to the display.

In one embodiment, the instructions to be executed further cause the processor to perform receiving a video frame of video frames from an application processor external to the touch sensitive display device.

According to embodiments of the present invention, there is provided a method and apparatus for reducing the perceived response time of a display of a display device (e.g., a touch screen device), and is capable of requiring a change to an existing processor and / There is no.

BRIEF DESCRIPTION OF THE DRAWINGS The accompanying drawings, together with the specification, illustrate exemplary embodiments of the invention and, together with the description, serve to explain the principles of the invention.
FIG. 1A illustrates the response of a device including a conventional touch input processing device, and FIG. 1B illustrates a response of a device including a touch input processing device according to some exemplary embodiments of the present invention.
FIG. 2 is a block diagram of a conventional touch screen display, including a conventional feedback path, and a touch input showing a reduced latency feedback path according to exemplary embodiments of the present invention, from sensing a user touch event to updating a display screen to provide visual feedback to a user. A schematic block diagram of a processing device.
3 is a schematic block diagram of a touch input processing device using an accelerator to reduce the display feedback latency perceived by a user, in accordance with an exemplary embodiment of the present invention.
Figures 4A and 4B illustrate a process for interpolating a touch path and defining an effective area in a video frame by an accelerator in accordance with exemplary embodiments of the present invention.
Figures 5A-5C illustrate three consecutive rendered video frames and a valid region tracking the touch path of a user's touch in accordance with exemplary embodiments of the present invention. Figure 5d illustrates image segments that represent differences between successive video frames as recorded and stored by an accelerator, in accordance with exemplary embodiments of the present invention.
6 illustrates a process for duplicating pixels stored by an accelerator to fill a gap between a video frame rendered by an application processor and a user ' s touch point, in accordance with exemplary embodiments of the present invention.
Figure 7 schematically illustrates a combination of an overlay segment and a current video frame to produce a composite video frame in accordance with exemplary embodiments of the present invention.
8 is a flow diagram of a process for providing visual feedback to a touch input in accordance with exemplary embodiments of the present invention.

In the following detailed description, only certain exemplary embodiments of the invention are shown and described by way of illustration. As will be appreciated by those skilled in the art, the present invention may be embodied in many different forms and should not be construed as limited to the embodiments set forth herein. The same reference numerals designate the same elements throughout the specification.

Embodiments of the present invention enable a user to perceive a faster touch response by displaying an image based on touch locations before the image rendered by the application processor.

Software designed for touch interfaces often provides visual feedback utilizing a metaphor of direct physical manipulation of pseudo "real world" objects and / or a drawing of paths on the screen (e.g., For tracking fingers on a swiping or gesture-based on-screen keyboard, for tracking paths in a drawing or sketchbook application, and for paths drawn in a game).

A common complaint for mobile devices is the lag of the user interface (UI). Current mobile devices may take between 50 and 200 milliseconds to update the display in response to a touching behavior. For example, a typical display response time for a touch event on a mobile device may be greater than 100 milliseconds, or based on a refresh rate of 60 frames per second (FPS) Frames, which can be noticeable to most users.

1A shows the response of a device including a comparable touch input processing device, wherein line 100 displayed in FIG. 1A includes a gap 102 between the position of the finger and the last drawn portion of the displayed line 100, , A significant display rack is generated between the user's touch and the displayed lines. Whether active or passive, you can also face a similar display rack when using the stylus.

Embodiments of the present invention enable a user to perceive a faster touch response by overlaying the image in the touch path prior to the image rendered by the AP. By narrowing the gap between the user's touch points (whether using a finger, stylus, or other tool) and the drawing of lines on the screen, the perceived display rack can be reduced. As used herein, the term "overlay ", when used as a verb, is intended to refer to a video frame or images (e. G., An " overlay " An image or frame rendered by the AP) and additional image data. When the term "overlay" is used as a noun, it may also refer to the appearance of the additional image data within the combined display image.

In addition, by using the overlay method, the application software can also control the rendering operation of the displayed response to an area (e.g., a location on the display), color, and touch events.

1B illustrates a displayed response of a device including a touch input processing device in accordance with an exemplary embodiment of the present invention, and the line displayed in FIG. 1B is displayed on a low latency accelerator 300 in accordance with exemplary embodiments of the present invention. Thereby reducing the display rack perceived by the user. ≪ RTI ID = 0.0 >

2 is a schematic block diagram of a touch input processing device 200 in accordance with exemplary embodiments of the present invention. Figure 2 illustrates a conventional feedback path (e. G., A touch-to-display loop (e. G., A touch-to-display loop TDL) and a reduced latency feedback path (e.g., a reduced latency touch-to-display loop (RTDL)) in accordance with exemplary embodiments of the present invention.

According to the present embodiment, a touch input processing device (e.g., a touch sensitive display device) 200 includes a touch sensor (e.g., touch sensor panel) 202, a touch controller 204, A display driver interface controller (DDIC) 220, and a display unit 230. In addition, The DDIC 220 may include a link receiver 222, a frame buffer 224, and a display driver (also referred to as a timing controller) According to some embodiments, the DDIC 220 further includes an accelerator (also referred to as a visual feedback accelerator or a low latency accelerator or an overlay system) In some embodiments of the invention, the DDIC 220 may be integrated with the touch controller 204. The AP 210 may include a central processing unit (CPU) 212, application software (SW) 214 and a graphical processing unit (GPU).

A touch sensor (e.g., a touch sensor panel or touch screen) 202 detects touch events such as taps or gestures and transmits corresponding touch signals 203 to the touch controller 204. Embodiments of the present invention may utilize a touch sensor 202 that detects a user's touch using any type of pointing tool, such as a body part (e.g., a finger), a stylus, or the like. As used herein, the term "pointing implement" is used herein to refer to a pointing device, including devices (such as active stylus and manual stylus) and body parts (such as a finger or a hand) Refers to objects that can be detected. Embodiments of the present invention can be applied to any one of various types of touch sensors, such as resistive touch panels, surface acoustic wave touch panels, capacitive touch panels, infrared touch panels, optical touch panels, Can be utilized. In some embodiments, the touch signals 203 correspond to the raw data supplied by the touch sensor 202, such as respective site-specific capacitances or measurements of voltage or current within the touch sensor 202 do. The data bus for touch events 205 is connected to both the AP 210 and the low latency accelerator 300. Touch events 205 are a stream of data values corresponding to locations where one or more touches by the user were detected (e.g., a change in voltage or current of a sufficiently high value to effect detection of a touch event) . In some embodiments, the touch events 205 include pressure data indicative of the pressure that was applied to the touch sensor 202.

The AP 210 processes the selection event and the SW 214 operating in the CPU 212 updates the display configuration accordingly (e.g., information to be displayed by the display unit 230). GPU 216 renders appropriate video frames (e.g., images to be displayed) to be transmitted to DDIC 220 to process the display configuration and display on display unit 230. The transmit signal received by the DDIC 220 may be in an analog format and encoded, in which case the link receiver 222 may convert the analog signals transmitted from the AP 210 to image bits Decodes these bits, and generates video frames (e.g., video images). The video frames are then stored in the frame buffer 224 before being sequentially transmitted to the display unit 230 for display to the user. The DDIC 220 may further include a pixel adjustment unit for performing gamma correction on video frames before display. The pixel adjustment unit may adjust the visual quality (e.g., brightness, gamma levels, etc.) of the video frames to be corrected for the characteristics of the display unit (e.g., display panel or light projector) 230 , ≪ / RTI > Display driver 226 includes circuitry for driving the individual pixels of display unit 230 in accordance with the adjusted video frames.

In some embodiments, the touch sensor 202, the touch controller 204, the DDIC 220, and the display unit 230 are both components of a display module that can be separated from the application processor 210. In other embodiments, touch sensor 202, touch controller 204, DDIC 220, and display unit 230, or combinations thereof, may reside within separate modules or may be combined with application processor 210 .

If the accelerator 300 of the present invention is not used, since the path from the user touch input to the display output is long (as represented by the feedback path (TDL)), Can be perceived as a remarkable rack of. This latency may be, for example, about 100 ms or longer, and this latency may be perceived by the user. Part of the delay caused by the feedback path (TDL) may result from the response time of the touch sensor 202 and the processing time of the touch controller 204, which response time and the processing time of the touch controller are deterministic and about 20 ms ≪ / RTI > Moreover, some of the delay may result from the processing time of the AP 210, which may or may not be busy with the CPU 212 running one or more higher priority processes at any given time (E.g., from about 60 ms to about 100 ms).

By using the accelerator 300, embodiments of the present invention are capable of a shorter touch-to-display path (as represented by a reduced latency feedback path (RTDL)) that is interpreted as a reduced response rack or latency. For example, in contrast to a display rack of approximately six frames or more in some comparable devices, embodiments of the present invention may reduce the rack to one or two frames.

In some embodiments of the present invention, accelerator 300 may be enabled and / or disabled via application software.

Accelerator 300 may be implemented as a separate component or integrated into any one of touch controller 204, AP 210 and / or DDIC 220. For example, FIG. 2 illustrates some embodiments in which accelerator 300 is integrated with DDIC 220.

3 is a schematic block diagram of a touch input processing device 200 that uses an accelerator 300 to reduce the display feedback latency perceived by the user, in accordance with exemplary embodiments of the present invention.

Referring to Figure 3, the paths of the image flow between the components are shown with solid arrowheads, while the control paths between the components are shown with arrowheads with dashed lines.

Accelerator 300 includes a touch detector 302, a moving image detector 304, a moving image replicator 306, an output timing controller 308 and an output logic unit 310, according to some embodiments of the present invention. do. Accelerator 300 may be included as a component of DDIC 220 or portions of accelerator 300 may be included in DDIC 220 or AP 210 and other portions may be included in touch controller 204.

In some embodiments, the touch detector 302 is connected to the touch controller 204 and receives touch events 205 from the touch controller 204. The touch detector 302 uses the locations of touch events on the display unit 230 to determine the travel path (or estimated touch path) of the user's finger or tool on or near the display unit 230. Based on the travel path, the touch detector 302 identifies a valid area (e.g., an associated area) surrounding the travel path and provides a valid area to the moving image detector 304 for further analysis.

In some embodiments, the moving image detector 304 is coupled to one or more of a link receiver 222 and a frame buffer 224, which detects the current rendered video frames (< RTI ID = 0.0 > Or video images) and receives video frames to be rendered next from the link receiver 222. [ Moving image detector 304 compares successive video frames of the valid region to detect and store changes (or differences) of video frames appearing in the valid region.

The moving image replicator 306 generates an overlay image segment based on the stored cars to fill the gap between the location of the touch events and the current video frame rendered by the AP 210. [ The overlay image segment is a computed estimate of the difference between the video frame that is currently rendered by the AP 210 and the video frame that the AP 210 would have rendered if it caught up with the current location of the touch event.

Using the information provided by the moving image replicator 306, the output timing controller 308 determines whether the output logic unit 310, one pixel at a time, has received the rendered video frame from the AP 210 (e.g., Video frame) or an overlay image segment.

Based on the control input from the output timing controller 308, the output logic unit 310 generates a composite image using the video frame and overlay image segment rendered by the AP. The output logic unit 310 then sends the composite image to the display driver 226 to generate the corresponding pixel configuration signals to be sent to the display unit 230 to display the composite image to the user. In some embodiments, the DDIC 220 may further process the composite image produced by the output logic unit 310 before supplying it to the display device 226. For example, the DDIC 220 may perform gamma correction on the composite image to be corrected for the characteristics of the display unit 230. For example,

Accelerator 300 therefore accelerates the visual feedback provided to the user by video frames rendered by the AP only with input from AP 210 (or, in the case of a system response to a user touch, Time). In some embodiments, the operations performed by the accelerator 300 may occur within a time span of less than one frame interval.

According to some embodiments, the touch detector 302, the moving image detector 304, the moving image replicator 306, the output timing controller 308, and the output logic unit 310 may each be a different application specific integrated circuit ; ASICs). In other embodiments of the present invention, a single ASIC is used to implement all functions. In another embodiment of the present invention, a field programmable gate array (FPGA) includes a touch detector 302, a moving image detector 304, a moving image replicator 306, an output timing controller 308, And output logic unit 310, respectively. Alternatively, the general purpose processor may be programmed to perform the respective functions of the touch detector 302, the moving image detector 304, the moving image replicator 306, the output timing controller 308 and the output logic unit 310 (E.g., by instructions stored in a memory connected to a general purpose processor). In yet other embodiments, the functionality of one or more of the touch detector 302, the moving image detector 304, the moving image replicator 306, the output timing controller 308, and the output logic unit 310 is controlled by the application processor 210, As shown in FIG.

Furthermore, it is shown that the touch detector 302, the moving image detector 304, the moving image replicator 306, the output timing controller 308 and the output logic unit 310 are components of the DDIC 220 in FIG. 3 However, the embodiments of the present invention are not limited thereto. In some embodiments, a touch detector 302, a moving image detector 304, a moving image replicator 306, an output timing controller 308 and an output logic unit 310 (or a component capable of performing these functions May be located within the touch controller 204, the AP 210, or as separate components, for example. In addition, the components or functions performed by these components may be located in different parts of the device. For example, the touch detector 302 may be implemented as a component or function of the touch controller 204 and may include a moving image detector 304, a moving image replicator 306, an output timing controller 308, (Or components) or functions (or functions) of the AP 210. The APs 310 may be implemented as part

In addition, although the touch controller 204 is shown as a physically separate component, in some embodiments of the present invention, the touch controller 204 is part of a larger integrated circuit. For example, the touch controller may be implemented within the same integrated circuit with the AP 210 and / or the DDIC 220.

The operation of the accelerator 300 and its constituent components will be further described below with reference to Figures 4-7.

4A and 4B illustrate a process of interpolating a touch path and defining an effective area within a video frame by the accelerator 300 (e.g., the touch detector 302) in accordance with an exemplary embodiment of the present invention.

According to some embodiments, the touch detector 302 receives the locations of touch events from the touch controller 204 and maps the touch locations to pixel locations within the video frame 40. 4A and 4B, the touch locations 402 (denoted by 'X' in FIGS. 4A and 4B) provided by the touch controller 204 are used to store the discrete samples of the user's actual touch path And therefore the consecutive touch points 402 can be separated by several pixels. The touch detector 302 interpolates the touch points 402 to fill gaps between consecutive touch points 402 to form interpolation points 404 (labeled 'O' in FIGS. 4A and 4B) Thereby constituting a set.

The touch detector 302 may utilize any one of a variety of suitable curve fitting algorithms to generate the interpolation points 404. For example, interpolation points 404 may be accurate fits to touch points 402 or a result of a smoothing operation performed by touch detector 302. [ The interpolated touch points 404 may follow the estimated touch path, which is a computed estimate of the user's actual touch travel path. The estimated touch path may traverse some or all of the touch locations 402 depending on the curve fitting algorithm used and the uncertainty in place 402. [ The estimated touch path may be computed based on a subset of touch events corresponding to a fixed number of frames that are greater than or equal to the delay (in units of video frames) introduced by the application processor.

According to some embodiments, the touch detector 302 is configured to detect an effective area 406 around each of the touch locations 402 and around interpolation points 404 (collectively referred to as "points " along the estimated touch path) ). A valid area 408 is formed by a collection of valid areas 406 and this valid area 408 is defined by an area around the estimated touch path that can be used by mobile image detector 304 to determine changes in successive frames . In some embodiments, the effective area 406 or the effective area 408 may be set the same as the entire video frame 400. However, processing all pixels in all frames by the accelerator 300 can be expensive in computation and may require significant memory. Therefore, to improve the performance of the accelerator 300, embodiments of the present invention utilize the effective areas 406, which typically comprise only a portion of the video frame. In some embodiments, the effective area 406 is a circle with a radius R (which may be rounded to the nearest pixel). In other embodiments, the effective area 406 is a square or rectangle having a side length that is rounded to the nearest pixel. In still other embodiments, the effective area 406 may be elliptical with lengths that are rounded to the nearest pixel along their major and minor axes. However, the embodiments of the present invention are not limited thereto, and the effective area 406 may have any suitable shape. The dimensions of the effective area 406 may be determined based on, for example, the rate of movement (or speed) of the user's touch and the uncertainty of the touch points 402. The rate of travel or speed may be determined by the distance between successive touch points 402 (e.g., as measured by the number of pixels). The uncertainty in the touch points can be determined by the deviations between the fitted curves and the touch points 402 used to create the interpolation points 404. In some embodiments, the size of the effective area 406 also increases (e.g., the radius R of the circular effective area 406 may increase) as the rate of travel and / or uncertainty increases.

The valid region 408 tracks the touch movement of the user, thereby changing the shape as the user's touch moves along the movement path. In some embodiments, the number of valid areas 406 constituting the valid area 408 (i.e., the length of the valid area 408) may be a fixed pre-set value. In other embodiments, the accelerator 300 may vary (e.g., dynamically) the length of the valid region 408 based on, for example, a system rack (e.g., as represented by the number of video frames) ) Can be set. In some embodiments, the length of the valid area 408 is greater than a value representing the number of frames representing the system rack.

The video frame 400 shown in Figures 4A and 4B shows a small number of pixels 401; However, this is only for ease of explanation, and the number of pixels 401 constituting the frame, as recognized by those skilled in the art, may be arbitrarily high and may be limited only by the technique used.

Figures 5A-5C illustrate three consecutive rendered video frames 502, 504, and 506 in accordance with some embodiments of the present invention and a valid region 408 that traces the movement path of the user's touch. Figure 5d illustrates that the differences between three consecutive video frames 502, 504 and 506 are recorded by the accelerator 300 (e.g., moving image detector 304), in accordance with exemplary embodiments of the present invention ≪ / RTI > showing the image segments represented by stored bars.

In accordance with embodiments of the present invention, moving image detector 304 is configured to detect a valid region of successive video frames rendered by AP 210 to detect any changes in rendered video frames within valid region 408 (408). In some embodiments, the moving image detector 304 compares the next video frame from the link receiver 222 with the current video frame received from the frame buffer 224, within the valid region 408 One). Moving image detector 304 then calculates the differences between the current and next video frames (act 2), for example, by subtracting the current video frame from the next video frame, one pixel by one pixel. The motion picture detector then stores the computed difference in the motion memory buffer (act 3), the outlined process (operations 1 to 3) can be repeated at least N times, where N is the user touch input and AP The expected latency between generated display responses is expressed in units of frames. In some instances, N ranges from 4 to 10, but typically can be 7. In some embodiments, N is a fixed number preset based on empirical performance data of the display device. In other embodiments, the number N can be dynamically adjusted by the accelerator 300 based on changes in latency as measured by the accelerator 300 over time.

5A-5D illustrate one example in which embodiments of the present invention are used in the context of a software application (e.g., a digital sketchbook) for drawing. In such an example, the effective area 408 may represent the active drawing area in which the line is being drawn. The line segments 510,512 and 514 may be moved by the AP 210 into the first video frame 502, the second video frame 504 and the third video frame 506, respectively, Lt; / RTI > 5D, the first image segment S2-1 represents the difference between the second and first video frames 504 and 502, the second image segment S3-2 represents the difference between the third and the third video segments, Lt; RTI ID = 0.0 > 506 < / RTI > Moving image detector 304 may include first and second image segments S2-1 and S3-2 in a moving buffer that may be within accelerator 300 (e.g., within moving image detector 304) (Collectively stored or computed segment 520). At any given time, the mobile buffer may store N-1 image segments representing the differences of valid regions of N consecutive frames, where N is the expected Express latency in units of frames. However, embodiments of the present invention are not limited thereto, and the moving buffer may store N or more image segments.

For ease of description, in Figures 5A-5C, the coverage area 408 is shown as being static, while the AP 210 proceeds through rendering the drawn line; However, in effect, the valid area 408 may be changed from frame to frame to suitably track the touch motion.

FIG. 6 illustrates an example of a video image stored by an accelerator 300 (e.g., a moving image replicator 306) to fill gaps between a user's touch point and an image frame rendered by the AP, in accordance with exemplary embodiments of the present invention. Lt; / RTI > illustrates the process of copying pixels.

In some embodiments, the moving image replicator 306 may use an overlay segment (e.g., an overlay image segment or a duplicated image segment) using the stored segment 520 to cover the gap 602 along the path of travel. (Or extrapolate) it. Gap 602 is used to indicate that touch points (finger, stylus, or other tool) along the path of travel that AP 210 has not yet rendered a corresponding image (e.g., does not follow along drawing a line on the screen) (Such as entity touch locations or interpolated points).

While creating the overlay segment 600, the moving image replicator 306 maintains the relative attributes of the pixels of the stored segment 520. Relative properties may include distances to the moving axes of pixels, spatial separations between pixels, color gradients between pixels, and the like. In some embodiments, the moving image replicator 306 performs a one-to-one mapping of the pixels of the stored segment 520 to the overlay segment 600. 6, the moving image replicator 306 transforms the pixels around the (I) touch point 604 (where I is an integer greater than 0) from the stored segment to (I + M) Th touch point 604 (where M is an integer greater than or equal to N). The corresponding pixels (i.e., shifted pixels) around the (I + M) th touch point 604 have a relative distance from the (I + M) th touch point 604 to the local shift axis 610 (In units of pixels) and this relative distance is equal to the relative distance from the stored difference segment 520 (i.e., stored car pixels) of the pixels to the local moving axis 612. The shifted pixel may have the same color characteristics (e.g., the same color) as the stored difference pixel.

Although some embodiments of the present invention utilize a pixel shift method, embodiments of the present invention are not limited thereto, and any of the many image processing algorithms or combinations thereof known to those skilled in the art may be used to generate the overlay segment 600 Process (e. G., A replication process).

7 is a schematic diagram of combining an overlay segment 600 and a current video frame 702 to produce a composite video frame (e.g., a composite frame image) 704 in accordance with the illustrative embodiments of the present invention .

In some embodiments, the output timing controller 308 may output a video frame 702 generated by the current AP from the frame buffer 224, one pixel at a time, or an overlay segment 600 from the moving image replicator 306, Gt; 706 < / RTI > This determination may be based on video timing, pixel locations from the touch detector 302 on the travel path, and pixel locations of the overlay segment 600. Output logic unit 310 uses mask 706 to generate overlay segment 600 to generate a composite video frame 704 by selecting between overlay segment 600 and current video frame 702. [ 0.0 > 702 < / RTI >

According to some embodiments, the mask 706 is a matrix of numerical values, the position in the matrix corresponds to the position of the pixel (or pixels) in the display unit 230 and the relative positions of the values in the matrix (E.g., mask 706 may be thought of as a two-dimensional matrix corresponding to a two-dimensional map of pixel locations in composite video frame 704). There is therefore a one-to-one relationship between each value in the mask 706 and each pixel in the current and composite video frames 702 and 704.

According to some embodiments, each of the values of the mask 706 is represented by a single bit, and the positions of the values in the mask matrix correspond to positions in the composite video frame 704. [ The locations where the overlay segment 600 may be visible in the composite video frame 704 may be those where the overlay data may not be visible with values set to a first value (e.g., "1 ") (E. G., "0") of the video frame 702 of the video frame 702) may have values set to a second different value In some examples, each of the numerical values of the mask 706 corresponds to exactly one pixel in the composite video frame 704 or to more than one pixel in the composite video frame 704.

In some embodiments, the output logic unit 310 may generate a current video frame 702 or an overlay segment 702 for each pixel based on a value at a location within the mask 706 corresponding to the location of the pixel in the composite video frame 704. [ The multiplexer 600 outputs the multiplexed signal. That is, the output logic unit 310 matches each value in the mask 706 to a corresponding pixel in the current video frame 702 and outputs the pixel or overlay segment 600 of the current video frame 702 to the display unit 600. [ (230). In some embodiments, the output logic unit 310 repeats through each value in the mask 706. If a value of zero is present at a particular position in the mask 706, the output logic unit 310 outputs the corresponding pixel of the current video frame 702. On the other hand, if a value of 1 is present at a particular position in the mask 706, the output logic unit 310 outputs the overlay segment 600. As a result of the iterative process, the output logic unit 310 outputs the composite video frame 704 to the display unit 230. The speed at which the iterative process is performed by the accelerator 300 is controlled by video timing.

Figure 8 is a flow diagram of a process 800 for providing visual feedback to a touch input in accordance with exemplary embodiments of the present invention.

In some embodiments, at operation 802, the accelerator 300 (e.g., the touch detector 302) receives a plurality of touch events from the touch controller 204 coupled to the touch sensor 202. Touch events can be generated by the touch controller 204 as a result of the interaction between the pointing tool (e.g., a user finger, a stylus pen, etc.) and the touch sensor 202.

At operation 804, the accelerator 300 (e.g., the moving image replicator 306) generates the overlay segment 600 based on the received touch events. In some embodiments, the accelerator 300 (e.g., the moving image detector 304) may calculate the estimated touch path based on the locations of touch events and determine the effective area (e.g., 408). Accelerator 300 (e.g., moving image replicator 306) compares the video frames to determine the differences (e.g., S2-1 and S3-2) in successive frames of video frames, (520) in the memory buffer. The overlay segment 600 may match the characteristics of the stored car segment 520. [ For example, accelerator 300 (e.g., moving image replicator 306) may maintain the relative attributes of the pixels of the stored car segment 520 in overlay segment 600. Relative properties may include distances to localized motion axes along the estimated touch path, spatial separations between pixels, and color gradients between pixels. In some embodiments, the accelerator 300 (e.g., the moving image replicator 306) may send the pixels of the stored car segment 520 to a location along the estimated touch path corresponding to the delay introduced by the AP 210 .

In operation 806, accelerator 300 (e.g., output logic unit 310) receives the current video frame 702 from frame buffer 224 to combine with overlay segment 600.

At the operation 808, the accelerator 300 (e.g., the output logic unit 310) generates the current video frame 702 according to the touch events to generate the composite video frame 704 with the overlay segment 600 . In some embodiments, accelerator 300 (e.g., output timing controller 308) generates mask 706 based on touch events. The mask includes a matrix of numerical values (e.g., '0's and' 1's) identifying the operation to produce a composite video frame. The positions of the numerical values in the matrix may correspond to the positions of the pixels in the composite video frame 704. Accelerator 300 (e.g., output logic unit 310) may mask mask 706 the corresponding pixel of current video frame 702 or overlay segment 600 for each pixel in composite video frame 704. [ In accordance with the value in the corresponding position in the output signal.

At act 810, accelerator 300 (e.g., output logic unit 310) provides synthesized video frame 704 to display unit 230 (via display driver 226). The accelerator 300 may perform additional processes for the composite video frame 704 before outputting the resulting frame to the display driver 226. [

Although the terms "first", "second", "third", etc. may be used herein to describe various elements, components, regions, layers and / or sections, It is to be understood that elements, regions, layers and / or sections should not be limited by these terms. These terms are used to distinguish one element, element, region, layer or section from another element, element, region, layer or section. Thus, the first element, component, region, layer or section discussed below may be referred to as a second element, component, region, layer or section without departing from the spirit and scope of the inventive concept.

The terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the inventive concept. As used herein, the singular forms "a" and "an" are intended to also include the plural forms, unless the context clearly indicates otherwise. When the terms "include", "including", "comprises" and / or "comprising" are used in this application, the presence of stated features, integers, steps, operations, elements, and / But do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, elements, and / or groups thereof. As used herein, the term "and / or" includes any and all combinations of one or more of the listed listed items of association. Expressions such as "at least one of" precedes the list of elements, modifying the entire list of elements and not modifying the individual elements of the list. Furthermore, the use of "can" refers to "one or more embodiments of the inventive concept" when describing embodiments of the inventive concept. The term "exemplary" is also intended to refer to an example or instance.

When an element or layer is referred to as being "on," "connected to," "coupled to," or "adjacent to" another element or layer, it is directly on another element or layer, , Or contiguous, or one or more intervening elements or layers may be present. When an element or layer is referred to as being "directly on," "directly connected to," "directly coupled to," or "directly adjacent" to another element or layer, there are no intervening elements or layers.

As used herein, the terms " substantially ", "about" and similar terms are used herein as approximate terms and are not to be taken as severity of terms and should not be construed as limiting the scope of the present invention to variations inherent in measured or calculated values It is intended to speak.

As used herein, the terms "use", "use of" and "used" may be considered synonymous with the terms "utilizing," "utilizing" and "utilizing".

Also, any number of ranges referred to herein are intended to include all subranges of the same number of degrees of accuracy that are included within the stated ranges. For example, a range of "1.0 to 10.0" may be defined as being between a stated minimum value of 1.0 and a stated maximum value of 10.0 (and inclusive), i.e. a minimum value equal to or greater than 1.0 and a maximum Values, for example, 2.4 to 7.6, inclusive. Any maximum number of limitations recited herein is intended to encompass all the lower number limitations encompassed therein, and any minimum number limitations referred to herein may be applied to all higher number limitations . Accordingly, applicants reserve the right to modify this specification including the claims, to explicitly state any subranges included within the scope explicitly recited herein.

The display device and / or any other related devices or components in accordance with embodiments of the invention described herein may be implemented in any suitable hardware, firmware (e.g., on-demand semiconductor), software or software, firmware and hardware May be implemented utilizing appropriate coupling. For example, the various components of the display device may be formed on one integrated circuit (IC) chip or on separate IC chips. Moreover, the various components of the display device may be implemented on a flexible printed circuit film, a tape carrier package (TCP), a printed circuit board (PCB) May be formed on the substrate. Moreover, the various components of the display device may be implemented as a process or thread that operates on one or more processors that execute computer program instructions and interact with other system components to perform the various functions described herein in one or more computing devices lt; / RTI > thread. Computer program instructions are stored in a memory that can be implemented within a computing device using a standard memory device, such as, for example, random access memory (RAM). The computer program instructions may also be, for example, a CD-ROM. Flash drive, and the like. Also, those skilled in the art will appreciate that the functionality of various computing devices may be combined or integrated into a single computing device without departing from the scope of the exemplary embodiments of the present invention, or the functionality of a particular computing device may be distributed across one or more other computing devices .

Although the present disclosure has been described in connection with what is presently considered to be an actual exemplary embodiment of the invention, it is to be understood that the invention is not limited to the disclosed embodiments, but on the contrary, is intended to fall within the spirit and scope of the appended claims and their equivalents It is to be understood that the invention is intended to cover various appropriate modifications and equivalent arrangements.

Claims (20)

CLAIMS 1. A method for providing visual feedback to a touch input,
The method comprising: receiving a plurality of touch events from a touch sensor coupled to a display;
Generating an overlay segment based on the plurality of touch events;
Receiving a video frame;
Combining the video frame with the overlay segment according to the touch events to generate a composite video frame; And
Providing the composite video frame to the display
≪ / RTI > The method according to claim 1,
Wherein the video frame comprises a displayed line, and
Wherein properties of the overlay segment match properties of the displayed line. The method according to claim 1,
Wherein generating the overlay segment comprises:
Calculating an estimated touch path based on locations of the touch events;
Determining a valid region of the video frames corresponding to the estimated touch path;
Comparing portions of the video frames corresponding to the valid region to determine differences in successive frames of the video frames;
Storing the differences in a memory buffer; And
And generating the overlay segment based on the stored cars, wherein the overlay segment matches the properties of the stored cars. The method of claim 3,
Wherein generating the overlay segment based on the stored cars comprises maintaining relative properties of the pixels of the stored cars in the overlay segment. The method of claim 3,
Wherein the relative properties include a distance to local motion axes along the estimated touch path, spatial separations between the pixels, and color gradients between the pixels. The method of claim 3,
Wherein generating an overlay segment based on the stored cars comprises shifting pixels of the stored cars to locations along the estimated touch path in response to a delay introduced by an application processor . The method of claim 3,
The estimated touch path is calculated based on a subset of touch events corresponding to a fixed number of frames, and
Wherein the fixed number of frames corresponds to a delay introduced by the application processor. The method according to claim 1,
Wherein combining the video frame with the overlay segment in accordance with the touch events comprises:
And generating a mask based on the touch events, wherein the mask comprises a matrix of numerical values, each of the numerical values identifying an operation for producing the composite video frame,
Wherein positions of numerical values in the matrix correspond to positions of pixels in the composite video frame. 9. The method of claim 8,
Further comprising determining for each pixel in the composite video frame whether to output the overlay segment or a corresponding pixel of the video frame according to a value at a corresponding location in the mask. 9. The method of claim 8,
Wherein each of the numerical values of the mask corresponds to one pixel in the composite video frame or to more than one pixel in the composite video frame. The method according to claim 1,
Wherein the touch events are generated as a result of an interaction between a pointing implement and a touch sensor. An accelerator for providing visual feedback to a touch input,
A processor; And
A memory having instructions stored therein, the instructions, when executed by the processor, cause the processor to:
Receiving a plurality of touch events from a touch sensor coupled to the display;
Generate an overlay segment based on the plurality of touch events;
Receiving a video frame;
Combine the video frame with the overlay segment according to the touch events to generate a composite video frame; And
And to provide the composite video frame to the display. 13. The method of claim 12,
Creating the overlay segment comprises:
Calculate an estimated touch path based on locations of the touch events;
Determine a valid region of the video frames corresponding to the estimated touch path;
Compare portions of the video frames corresponding to the valid region to determine differences in successive frames of the video frames;
Store the differences in a memory buffer; And
And generating the overlay segment based on the stored cars, wherein the overlay segment matches the properties of the stored cars. 14. The method of claim 13,
Generating the overlay segment based on the stored cars comprises maintaining relative attributes of the pixels of the stored cars in the overlay segment,
Wherein the relative properties include distances to local motion axes along the estimated touch path, spatial separations between the pixels, and color gradients between the pixels. 14. The method of claim 13,
Wherein generating the overlay segment based on the stored cars comprises shifting pixels of the stored cars to locations along the estimated touch path in response to a delay introduced by the application processor. 14. The method of claim 13,
The estimated touch path is calculated based on a subset of the touch events corresponding to a fixed number of frames, and
Wherein the fixed number of frames corresponds to a delay introduced by the application processor. 13. The method of claim 12,
Combining the video frame with the overlay segment according to the touch events comprises:
And generating a mask based on the touch events, wherein the mask comprises a matrix of numerical values, each of the numerical values identifying an operation for producing the composite video frame,
Wherein positions of the numerical values in the matrix correspond to positions of pixels in the composite video frame. 18. The method of claim 17,
Further comprising, for each pixel in the composite video frame, determining whether to output the overlay segment or a corresponding pixel of the video frame according to a value at a corresponding location in the mask. A touch input processing device for providing visual feedback to a touch input,
A display configured to display video frames;
A touch sensor coupled to the display and configured to generate a plurality of touch events as a result of interaction between the pointing tool and the touch sensor;
A processor; And
A memory having instructions stored therein, the instructions, when executed by the processor, cause the processor to:
Receive a plurality of touch events from the touch sensor;
Generate an overlay segment based on the plurality of touch events;
Receiving a video frame of the video frames;
Combine the video frame with the overlay segment according to the touch events to generate a composite video frame; And
And to provide the composite video frame to the display. 20. The method of claim 19,
Wherein the executed instructions additionally cause the processor to perform receiving a video frame of the video frames from an application processor external to the touch sensitive display device.

Images