JP2016170709A - Touch detection device - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a touch detection device that enables a display not having a touch detection function to additionally detect a touch operation.SOLUTION: A touch detection device 100 comprises: an image acquisition part 110 that simultaneously acquires a frame image (20A) photographed by a first camera 20 and a frame image (30A) photographed by a second camera 30; and a touch detection part 140 that, when the coordinates of an indication body in the frame image 20A and the coordinates of an indication body in the frame image 30A substantially match with each other, detects the coordinates as touch coordinates.SELECTED DRAWING: Figure 1

Description

  The present invention relates to a touch detection device that detects a touch of an indicator on a display device.

  A display device equipped with a touch detection function is well known. Various techniques for realizing the touch detection function are also disclosed. For example, a technology for realizing a display device equipped with a touch detection function by integrally forming a touch panel and a display panel, or a “vision-based” touch detection function (display device display as described in Patent Document 1) A technique for realizing touch detection by mounting a screen) and detecting presence / absence of touch by processing a captured image is known.

Patent No. 5122948 (registered on November 2, 2012)

  By the way, in the case of a display device such as an information display installed in a public place, it is desired to perform interactive display for the user, such as changing the display content in response to a touch operation. However, some of the display devices do not have a touch detection function. Therefore, there is an increasing demand for additionally mounting a touch detection function on a display device that does not have a touch detection function. However, when a touch detection function is retrofitted to a display device that does not have a touch detection function, it is necessary to add a complicated and large configuration such as surrounding the display with a reflector.

  The present invention has been made in view of the above problems, and an object thereof is to realize a touch detection device that detects a touch on a display device that does not have a touch detection function.

  In order to solve the above-described problem, a touch detection device according to one aspect of the present invention is a touch detection device that detects a touch of an indicator on a display surface of a display device, and the display surface is captured by a first camera. A video acquisition unit that acquires the first video of the display surface and a second video of the display surface taken from a different angle from the second camera substantially simultaneously with the first camera, and the first video acquired by the video acquisition unit A first position that is a touch position of the indicator on the display surface and a second position that is a touch position of the indicator on the display surface in the second image are substantially the same. A touch detection unit that detects the substantially matched position as a touch position of the indicator.

  According to one embodiment of the present invention, it is possible to detect a touch on a display device that does not have a touch detection function.

It is a block diagram which shows the principal part structure of STB (set top box) which concerns on Embodiment 1 of this invention. It is a figure which shows arrangement | positioning of the camera with respect to the display apparatus connected to the said STB. It is a figure which shows imaging | photography of the frame image by the 1st camera connected to said STB, and distortion correction of the said frame image in the touch detection apparatus mounted in said STB. It is a figure which shows the imaging | photography and corrected image of a frame image in case a indicator is adjoining and non-contacting on a display surface. It is a figure which shows the imaging | photography of the frame image when the indicator is contacting the display surface, and the corrected image. It is a flowchart which shows the flow of the touch detection process in the said touch detection apparatus. It is a block diagram which shows the principal part structure of STB which concerns on Embodiment 2 of this invention. It is a table | surface which shows the relationship between the kind of operation identified with the touch detection apparatus mounted in the said STB, the duration of the touch detection in the said touch detection apparatus, and the change of a touch coordinate. It is a flowchart which shows the flow of the operation specific process in the said touch detection apparatus. It is a figure which shows arrangement | positioning of the camera with respect to the display apparatus connected to STB which concerns on Embodiment 3 of this invention. It is a figure which shows imaging | photography of the frame image when multitouch is performed with respect to the display surface.

Embodiment 1
Hereinafter, a first embodiment of the present invention will be described. In the present embodiment, an example in which the touch detection device according to the present invention is mounted on an STB (set top box) of a display device will be described with reference to FIGS. First, the functions of the STB 40 according to the present embodiment and various devices connected to the STB 40 will be briefly described with reference to FIG. In the following description, an example will be described in which the touch detection device and the display control unit are mounted on one device in the STB. However, the touch detection device and the display control unit may be separate devices.

  FIG. 2 is a diagram illustrating the display device 10 connected to the STB 40 according to the present embodiment and the arrangement of the first camera 20 and the second camera 30 with respect to the display device 10. The STB 40 is connected to the display device 10, the first camera 20, and the second camera 30 as illustrated.

  The display device 10 displays the display data received from the STB 40 on a substantially flat display surface 10A. The display surface 10A is realized by, for example, a liquid crystal display or an organic EL display.

  The first camera 20 and the second camera 30 are cameras that photograph the display surface 10 </ b> A of the display device 10. Each of the first camera 20 and the second camera 30 is fixedly installed so that the entire display surface 10A of the display device 10 can be photographed at a different angle from the other camera substantially at the same time. For example, when the display device 10 is a device having a quadrangular display surface 10A, the first camera 20 and the second camera 30 are provided at any one of the four corners of the display surface 10A as shown in the drawing, and the shooting direction is displayed. It suffices if the entire surface 10A is directed to an angle at which it can be photographed. The first camera 20 and the second camera 30 may be cameras that shoot moving images, or cameras that shoot still images (photos) at predetermined time intervals or at predetermined timings. In the following description, it is assumed that the first camera 20 and the second camera 30 are cameras that shoot the display surface 10A with moving images.

  The STB 40 is a device including a display control device 200 that controls screen display of the display device 10 and a touch detection device 100 that detects a touch on the display device 10. The STB 40 transmits data (display data) displayed by the display device 10 to the display device 10. In addition, the STB 40 receives video captured by the first camera 20 and the second camera 30, and performs touch detection using the video. A method for obtaining display data and a method for touch detection will be described in detail later.

≪Main part composition≫
Next, the main configuration of the STB 40 will be described with reference to FIG. FIG. 1 is a block diagram showing a main configuration of the STB 40. In FIG. 1, the display device 10, the first camera 20, and the second camera 30 connected to the STB 40 are also shown. The STB 40 includes a display control device 200 and a touch detection device 100.

  The display control device 200 acquires data (display data) such as still images and moving images displayed by the display device 10 and transmits the data to the display device 10. The display control apparatus 200 may receive the result of touch detection (whether or not touch detection or touch coordinates) is received from the touch detection apparatus 100 described later, and may change the content of the display data according to the result. In other words, the display control device 200 may change the content displayed on the display device 10 according to the presence or absence of a touch.

  Note that the display data acquisition source of the display control device 200 is not particularly limited. For example, when the STB 40 has a function of receiving a broadcast signal, the display control apparatus 200 may create display data from the received broadcast signal. For example, when the STB 40 has a function of connecting to the Internet, the display control device 200 may acquire display data via the Internet. When the STB 40 has a function of reading display data recorded on a recording medium, the display control device 200 may read display data from the recording medium.

  The touch detection device 100 detects a touch of an indicator on the display surface of the display device 10. More specifically, the touch detection device 100 includes an image acquisition unit (video acquisition unit) 110, a correction unit 120, a pointer specifying unit 130, and a touch detection unit 140. The image acquisition unit 110 acquires a frame image captured by the first camera 20 and the second camera 30 every frame or every predetermined number of frames. Hereinafter, the frame image captured by the first camera 20 is referred to as a frame image 20A (first video), and the frame image captured by the second camera 30 is referred to as a frame image 30A (second video). The image acquisition unit 110 sends the acquired frame images 20A and 30A to the correction unit 120. The correction unit 120 corrects the frame images 20A and 30A. In the following description, “correction” refers to the reference on the frame images 20A and 30A in order to make the shape of the display surface 10A shown in the frame images 20A and 30A the same as the actual shape of the display surface 10A. This means “distortion correction” in which the position of a reference point (a reference point, for example, the four corner points of the display surface 10A) is corrected to a specified position. The correcting unit 120 sends the corrected frame images 20A and 30A to the indicator specifying unit 130. Hereinafter, the corrected frame images 20A and 30A are referred to as corrected images 20B and 30B, respectively.

  Here, photographing and correction of the frame images 20A and 30A will be described in detail with reference to FIG. FIG. 3 is a diagram illustrating photographing of a frame image of the frame image 20A and distortion correction. In the description of FIG. 3, only the shooting of the frame image 20A and distortion correction will be described for the sake of brevity, but the same processing may be performed for the shooting of the frame image 30A and distortion correction.

  FIG. 3A is a diagram illustrating a state where the first camera 20 captures the display surface 10A. The broken line arrows in the figure indicate the shooting direction of the camera, and the x mark in the figure indicates the apparent position at which the indicator appears in the frame image when the camera takes a frame image. The same applies to the subsequent drawings.

  As described above, the first camera 20 captures the frame image 20A of the display surface 10A. Here, it is assumed that the user brings the indicator (user's finger) B closer to the display surface 10A as illustrated. In this case, when the indicator B comes close to the display surface 10A to some extent, the indicator B enters the shooting range of the first camera 20. That is, the indicator B is included in the frame image 20A captured by the first camera 20.

  FIG. 3B is a diagram showing a frame image 20A when the first camera 20 captures the state of FIG. When the entire display surface 10A is photographed from the angle of the first camera 20, the display surface 10A is photographed with the shape distorted as shown in the figure. In addition, the indicator B is reflected at the position indicated by the x mark in FIG.

  FIG. 3C shows a corrected image after the correction unit 120 corrects the frame image 20A shown in FIG. When the correction unit 120 acquires the frame image 20A, first, based on the reference points stored in advance, the correction unit 120 extracts only the region in which the display surface 10A is shown in the frame image 20A. Next, the correcting unit 120 stores the extracted area in advance according to the correction parameters and the correction processing procedure of the display surface 10A, and the apparent shape of the display surface 10A is the same as the actual shape of the display surface 10A. Correct the shape. The method for defining the reference point, the correction parameter, and the correction processing procedure is not particularly limited. For example, the images of the display surface 10A are once taken with the fixed first camera 20 and the second camera 30, and the photographed images (uncorrected (distorted) display surface 10A of each camera are shown. Image) is presented to the user on the display device 10 or another device. Then, the user inputs (registers in the touch detection device 100) reference points such as the four corner points of the display surface 10A or the points on the outer circumference in each of the captured images of the first camera 20 and the second camera 30. Thereby, the touch detection apparatus 100 can set a reference point, and thereafter can specify how the display surface 10A captured by the first camera 20 and the second camera 30 is distorted. Then, the touch detection apparatus 100 can determine a correction parameter and a correction processing procedure for correcting the distortion from the distortion of the reference point registered by the user. Therefore, in the following, when correction is performed by the correction unit 120, distortion correction may be performed using the reference point, the correction parameter, and the correction process.

  The indicator specifying unit 130 specifies a position on the display surface (coordinates on the display surface) designated by the indicator shown in each of the corrected images 20B and 30B acquired from the correcting unit 120. First, the indicator specifying unit 130 specifies the indicator shown in the corrected images 20B and 30B. More specifically, the indicator specifying unit 130 determines whether or not there is a region in the corrected images 20B and 30B where the brightness, saturation, and hue match the indicator color information. Here, the “indicator color information” is information indicating the brightness, saturation, and hue that the indicator shows on the corrected images 20B and 30B. When there is an area indicating brightness, saturation, and hue that matches the indicator color information, the indicator specifying unit 130 specifies that the indicator indicating the indicator color information is included in the area. For example, when the indicator is a user's finger, the indicator specifying unit 130 performs skin color detection in the corrected images 20B and 30B, and if a skin color region is detected, the pointer specifying unit 130 specifies that the finger is reflected in the region. The indicator specifying unit 130 further specifies the position coordinates on the display surface indicated by the indicator from the shape of the indicator shown in the corrected images 20B and 30B. Although the specific method is not particularly limited, for example, the indicator specifying unit 130 specifies a portion showing a convex shape in the shape of the indicator, and the indicator shows the coordinates of one point of the tip of the convex shape. The position coordinates on the surface may be used.

  On the other hand, when there is no area indicating the indicator color information in the corrected image, the indicator specifying unit 130 determines that the indicator is not shown in the corrected image. When the indicator specifying unit 130 specifies the indicator and specifies the coordinates on the display surface indicated by the indicator, the indicator specifying unit 130 sends the coordinates to the touch detection unit 140.

  For example, when the indicator is a user's finger, the brightness, saturation, and hue indicated by the indicator color information may be determined with a range in order to consider individual differences in finger color. The indicator color information and the shape of the indicator may be information that can be registered or modified by the user in advance. For example, the touch detection device 100 acquires an image of a user's finger previously captured by the first camera 20 or the second camera 30, and uses the brightness, saturation, and hue values indicated by the image of the finger as indicator color information. You may remember as.

  As long as the indicator can be specified from the corrected image, the indicator specifying method of the indicator specifying unit 130 is not limited to the above method. For example, when the pointer specifying unit 130 detects an edge of the corrected image and detects a region indicating a predetermined contour (for example, the contour of a hand, the contour of a touch pen), the pointer is reflected in the region. You may judge.

  The touch detection unit 140 includes coordinates on the display surface indicated by the indicator in the corrected image 20B (first coordinate, first position) and coordinates on the display surface indicated by the indicator in the corrected image 30B (second coordinate, From the second position, it is determined whether or not the display device 10 has been touched. Further, when it is determined that the display device 10 has been touched, the touch detection unit 140 detects the position of the touch. When the touch detection unit 140 acquires the first coordinate and the second coordinate from the indicator specifying unit 130, the touch detection unit 140 determines whether or not the first coordinate and the second coordinate match. When the first coordinate and the second coordinate match, the touch detection unit 140 determines that the display surface of the display device 10 has been touched, and detects the matched coordinates (first coordinate and second coordinate) as touch coordinates. . On the other hand, when the first coordinate and the second coordinate do not match, the touch detection unit 140 determines that the display surface of the display device 10 is not touched, and the touch coordinate may not be detected. The touch detection unit 140 transmits the result of touch detection (the presence / absence of a touch and the touch coordinates when there is a touch) to the display control device 200.

≪Indicator contact judgment≫
When the touch detection unit 140 obtains the first coordinate and the second coordinate from the indicator specifying unit 130, that is, when the indicator is reflected in the corrected images 20B and 30B, the touch detection unit 140 is divided into two types. The first is a case where the indicator is close to the display surface 10A to the extent that it enters the imaging range of the first camera 20 and the second camera 30, but is not in contact with the display surface 10A. This is a case where the indicator is touching (touching) the display surface. Hereinafter, the difference between the first coordinate and the second coordinate when the indicator is in proximity (non-contact) to the display surface 10A and when it is in contact will be described with reference to FIGS.

  FIG. 4A shows the shooting direction of the first camera 20 and the second camera 30 when the indicator B is close (and non-contact) with the display surface 10A, and the indicator on the frame images 20A and 30A. B shows the position where the image appears. FIGS. 4B and 4C show the corrected images 20B and 30B of the frame images 20A and 30A taken in FIG. 4A, respectively. Further, FIG. 4D is a superposition of the image of FIG. 4B and the image of FIG. On the other hand, (a) of FIG. 5 shows the shooting direction of the 20 and the second camera 30 when the indicator B is in contact with the display surface 10A, and the position where the indicator B is reflected. 5B and 5C show corrected images 20B and 30B of the frame images 20A and 30A taken in FIG. 5A, respectively. Further, FIG. 5D is an image obtained by superimposing the image of FIG. 5B and the image of FIG.

(When the indicator is not in contact with the display surface)
When the indicator B is not in contact with the display surface 10A, as shown in FIG. 4A, the indicator B is displayed by the indicator B when viewed from the shooting angles of the first camera 20 and the second camera 30. It appears in the position projected on the surface. In other words, the indicator B appears at a position different from the closest position on the display surface 10A (one point on the display surface 10A indicated by the fingertip of the indicator B). As a result, the position indicated by the indicator B in the corrected image 20B (FIG. 4B) does not match the position indicated by the indicator B in the corrected image 30B (FIG. 4C).

(When the indicator is in contact with the display surface)
On the other hand, when the indicator B is in contact with the display surface 10A, as shown in FIG. 5A, the position (touch position) where the indicator B is in contact with the display surface 10A is the first. Even if the image is taken from the camera 20 or the second camera 30, the image is taken unchanged. As a result, the position indicated by the indicator B in the corrected image 20B (FIG. 5B) matches the position indicated by the indicator B in the corrected image 30B (FIG. 5C).

  From the above, when the apparent touch position (first coordinate and second coordinate) of the indicator coincides in the first video and the second video that are photographed substantially simultaneously from different angles, the indicator is displayed on the display surface 10A. Can be regarded as touching. Therefore, the touch detection unit 140 can distinguish and detect the touch of the indicator and the non-contact proximity by determining that the touch is performed when the first coordinate and the second coordinate match.

  Note that the touch detection unit 140 may determine that the display surface 10 </ b> A has been touched even when the first coordinate and the second coordinate are substantially the same. Depending on the correction by the correction unit 120 and the detection accuracy of the indicator region by the indicator specifying unit 130, the first coordinate and the second coordinate may not match even when the indicator is in contact with the display surface 10A. It is. When it is determined that a touch has occurred when the first coordinate and the second coordinate are substantially the same, the touch detection unit 140 detects, for example, the coordinates of an intermediate point between the first coordinate and the second coordinate as the touch coordinates. do it.

  In addition, the indicator specifying unit 130 specifies the indicator, and only when the indicator has a predetermined convex shape (protruding portion), the indicator indicates the coordinates indicated by the protruding portion. It is good also as a position coordinate (1st coordinate or 2nd coordinate mentioned later) on the display surface to show. Here, the “predetermined convex shape” is, for example, the shape of the entire or tip of each fingertip, the shape of the entire or tip portion of a pen, a pointer, or the like.

  For example, when the user touches the display surface 10A with a finger, the indicator specifying unit 130 detects the first coordinate and the second coordinate related to the touch of the finger because the finger has a protruding portion, and the user's fist When the display surface 10A is touched, the fist does not have a protruding portion, and therefore the first coordinate and the second coordinate related to the contact of the fist may not be detected.

  With such a configuration, even if there is a contact of an indicator that does not have a protruding portion, the indicator specifying unit 130 does not detect the first coordinate and the second coordinate, and the first detection unit 140 does not detect the first coordinate and the second coordinate. It is possible not to send the coordinates and the second coordinates. In other words, the touch detection unit 140 can detect no touch even when an indicator that does not have a protruding portion touches. Therefore, the first coordinate and the second coordinate are detected from an unintended contact with the display surface (for example, a fist contact) or a contact with an object (such as a paper memo) attached to the display surface 10A itself. It is possible to prevent a touch from being erroneously detected from the first coordinate and the second coordinate.

≪Touch detection process flow≫
Finally, the flow of touch detection processing (touch detection method) performed by the touch detection device 100 will be described with reference to FIG. FIG. 6 is a flowchart showing a flow of touch detection processing performed by the touch detection device 100.

  The image acquisition unit 110 of the touch detection device 100 acquires the frame images 20A and 30A captured by the first camera 20 and the second camera 30 (S100, video acquisition step). The image acquisition unit 110 sends the frame images 20A and 30A to the correction unit 120, respectively. When the correction unit 120 acquires the frame images 20A and 30A, the correction unit 120 performs correction so that the shape of the display surface 10A in these images is the same as the shape of the actual display surface 10A (S102), and the frame images 20A and 30A. Therefore, the portions where the display surface 10A is shown are cut out and corrected images 20B and 30B. The correction unit 120 sends the corrected images 20B and 30B to the indicator specifying unit 130. The indicator specifying unit 130 specifies the indicator included in the corrected images 20B and 30B (S104), and specifies the coordinates (first coordinate and second coordinate) on the display surface indicated by the indicator. The first coordinate and the second coordinate are transmitted to the touch detection unit 140, respectively. When the touch detection unit 140 acquires the first coordinate and the second coordinate, the touch detection unit 140 determines whether the first coordinate and the second coordinate match (S106). When the first coordinate and the second coordinate match (YES in S106), the touch detection unit 140 determines that the touch has been performed, and detects the matched coordinate (the first coordinate and the second coordinate) as the touch coordinate ( S108, touch detection step). On the other hand, when the first coordinate and the second coordinate do not match (NO in S106), the touch detection unit 140 determines that the touch has not been performed.

  As described above, the indicator appears in the frame images 20A and 30A when the indicator is close to (and is not in contact with) the display surface 10A or when the indicator is in contact. When the indicator is not in contact with the display surface 10A, the indicator appears in a position different from the position on the display surface closest to the indicator in the frame images 20A and 30A. On the other hand, when the indicator is in contact with the display surface 10A, the indicator appears in the closest position on the display surface in the frame images 20A and 30A.

  Therefore, according to the touch detection process of FIG. 6, the touch detection unit 140 can detect the touch of the indicator separately from the non-contact proximity. That is, the touch detection device has an effect of being able to detect a touch (contact) on a display device that does not have a touch detection function.

  In the touch detection process, the correction process (S102) by the correction unit 120 is not essential. For example, when the shooting angle with respect to the display surface 10A of the first camera 20 and the second camera 30 is nearly vertical, the distortion of the display surface 10A in the frame images 20A and 30A is small. In such a case, the touch detection apparatus 100 may not perform correction by the correction unit 120. Specifically, the image acquisition unit 110 sends the acquired frame images 20A and 30A to the indicator specifying unit 130, and the indicator specifying unit 130 specifies the indicator using the frame images 20A and 30A itself, and the first coordinates. The second coordinate may be specified.

  Further, as described above, the first camera 20 and the second camera 30 are fixed, that is, the distortion of the display surface 10A shown in the frame images 20A and 30A does not change. Therefore, for example, when the (apparent) position on the display surface 10A captured (distorted) in each of the frame images 20A and 30A is associated with the position coordinates on the actual display surface 10A, correction is performed. Even if the correction process (S102) by the unit 120 is not performed, the first coordinates and the second coordinates can be specified from the frame images 20A and 30A.

[Embodiment 2]
A second embodiment of the present invention will be described with reference to FIGS. First, the main configuration of the STB 50 according to the present embodiment will be described with reference to FIG. FIG. 7 is a block diagram showing a main configuration of the STB 50. As shown in FIG. For convenience of explanation, members having the same functions as those described in the embodiment are given the same reference numerals, and descriptions thereof are omitted. The STB 50 according to the present embodiment differs from the STB 40 according to the first embodiment in that the touch detection device 300 includes an operation specifying unit 150.

  The operation specifying unit 150 is responsive to at least one of the number of frames (that is, the touch duration) that the touch detection unit 140 continuously detects the touch and a change in touch coordinates within the touch duration. The type of operation indicated by the touch is specified. More specifically, the operation specifying unit 150 includes a duration specifying unit (touch continuation specifying unit) 151 and a coordinate change specifying unit 152.

  The duration specifying unit 151 specifies the touch duration from the determination result of the presence or absence of a touch acquired for each frame from the touch detection unit 140. Specifically, when acquiring the determination result that there is no touch from the touch detection unit 140, the duration specifying unit 151 calculates the time indicated by the number of frames that have been continuously acquired and determined that there is a touch. Identified as the duration of the touch operation. To do. Note that the duration specifying unit 151 may regard the number of frames itself as a value indicating the touch duration. After that, when the determination result indicating that there is a touch is received again from the touch detection unit 140, the duration specifying unit 151 sets the number of frames that have been continuously acquired as the presence of a touch as a new touch operation has started to 0. Measure again.

  In the case where the duration specifying unit 151 specifies the touch duration, the coordinate change specifying unit 152 uses the touch coordinates acquired for each frame from the touch detection unit 140 during the touch duration, in one touch operation. Identify changes in touch coordinates. The operation specifying unit 150 specifies the type of operation corresponding to the touch according to the touch duration specified by the duration specifying unit 151 and the change in touch coordinates specified by the coordinate change specifying unit 152. In addition, although the kind of operation which the operation specific part 150 specifies is not specifically limited, A tap, long press, a swipe, a flick etc. are mentioned as an example. The operation specifying unit 150 transmits data acquired from the touch detection unit 140 such as the specified operation type and touch coordinates to the display control device 200. Furthermore, the display control apparatus 200 may create display data based on the received operation type, touch coordinates, and the like.

  FIG. 8 is a table showing the relationship between the type of operation specified by the operation specifying unit 150, the touch duration, and the magnitude of change in touch coordinates. Here, the tap operation generally means an operation in which the user instantaneously touches the display surface such as a touch panel lightly. Therefore, the operation specifying unit 150 may specify that the touch detected by the touch detection unit 140 is a tap operation when the touch duration is short (for example, less than 0.5 seconds). Note that, if the touch duration time is short, the operation specifying unit 150 may specify the touch operation as a tap operation regardless of the change amount of the touch coordinates. On the other hand, the long press operation generally means an operation in which the user brings the indicator into contact with the display surface for a longer time than the tap. Therefore, the operation specifying unit 150 is configured to detect the touch detection unit when the touch duration is longer than the tap operation (for example, 0.5 seconds or more) and there is no change in touch coordinates (the amount of change is less than a predetermined threshold). The operation related to the touch detected by 140 may be specified as a long press operation.

  The swipe operation means an operation in which the user moves the indicator so as to slide left and right or up and down while keeping the indicator in contact with the display surface. Therefore, the operation specifying unit 150 has a touch duration longer than that in the case of the tap operation (for example, 1 second or more or several tens of frames or more), and there is a change in touch coordinates (the amount of change is larger than a predetermined threshold). In this case, the operation related to the touch detected by the touch detection unit 140 may be specified as the swipe operation. Further, the flick operation means an operation in which the user touches the display body for a short time and moves the screen so as to lightly touch the screen. Therefore, the operation specifying unit 150 flicks an operation related to the touch detected by the touch detection unit 140 when the touch duration is moderately long (for example, 0.5 seconds or more and less than 1 second) and there is a change in touch coordinates. What is necessary is just to specify as operation.

  In the example of FIG. 8, the operation specifying unit 150 considers only the presence or absence of the change in the touch coordinates, but for example, the magnitude of the change in the touch coordinates (the magnitude of the movement) or the change in the touch coordinates. The type of operation may be specified according to the speed. For example, the swipe operation and the flick operation in the above example are not the length of the touch duration, but the magnitude of the change in the touch coordinates (the swipe operation has a larger change in the touch coordinates) or the speed of the change (the direction of the flick operation) May be specified by distinguishing them according to high speed).

<< Operation identification process flow >>
Next, an operation specifying process performed by the touch detection device 100 will be described with reference to FIG. FIG. 9 is a flowchart showing the flow of the operation specifying process performed by the touch detection device 100. The “touch detection process” shown in S200 indicates the touch detection process (S100 to S108) by the touch detection unit 140 shown in FIG.

  When the touch detection process is completed (S200), the touch detection unit 140 of the touch detection device 100 determines whether or not the operation specifying unit 150 is touched (whether or not the touch is performed) and the touch is performed. Sends the specified touch coordinates. The process of S200 is performed for each of the frame images 20A and 30A acquired by the image acquisition unit 110. Then, while the touch detection unit 140 detects a touch (NO in S202), the operation specifying unit 150 acquires a determination result indicating that there is a touch and touch coordinates.

  Here, when the user's touch operation is finished and the touch detection unit 140 no longer detects a touch (NO in S202), the touch detection unit 140 sends a determination result of no touch to the operation specifying unit 150. Upon receiving the determination result, the duration specifying unit 151 of the operation specifying unit 150 specifies the touch duration in a single touch operation from the number of frames that have been continuously acquired as being touched (S204). . Subsequently, the coordinate change specifying unit 152 specifies the magnitude of the change of the touch coordinates during the touch duration (S208). Then, the operation specifying unit 150 specifies the type of operation related to the touch from the touch duration and the magnitude of the change in touch coordinates (S208).

  According to the above processing, the operation specifying unit 150 specifies an operation from the time when the touch lasts and the amount of change in touch coordinates during the time. Thereby, the touch detection apparatus 100 can detect not only the presence / absence of touch and the detection of touch coordinates, but also the type of touch operation.

[Embodiment 3]
Finally, a third embodiment of the present invention will be described with reference to FIGS. The touch detection devices 100 and 300 according to the present invention capture the entire display surface 10A almost simultaneously with the first camera 20 and the second camera 30 from an angle different from both the first camera 20 and the second camera 30. Multi-touch may be detected by acquiring a frame image 60A (third video) from 60 and using the frame image 60A for touch detection processing.

  FIG. 10 is a diagram illustrating an arrangement of the first camera 20, the second camera 30, and the third camera 60 with respect to the display device 10 connected to the STB 40 according to the present embodiment. The STB 40 according to the present embodiment is different from the STB 40 according to the first embodiment and the STB 50 according to the second embodiment in that it is connected to the third camera 60.

  The third camera 60 is a camera that photographs the entire display surface of the display device 10 substantially simultaneously with the first camera 20 and the second camera 30. The third camera 60 is fixedly installed so that the entire display surface 10A can be photographed from an angle different from both the first camera 20 and the second camera 30. For example, as shown in the figure, the third camera 60 is installed at the four corners of the display device 10 at different positions from the first camera 20 and the second camera 30 with the shooting direction facing the display surface 10A.

  As shown in FIG. 10, when the frame images are taken almost simultaneously by three or more cameras, the touch detection device 100 or 300 can detect a plurality of simultaneous touches (multi-touch). The mechanism will be described in detail below with reference to FIG. In the following description, the corrected image of the frame image 60A captured by the third camera 60 is referred to as a corrected image 60B, and the (apparent) position coordinates on the display surface indicated by the indicator in the corrected image 60B are shown. This is called the third coordinate.

  FIG. 11 is a diagram illustrating frame image capturing when multi-touch is performed on the display surface 10A. FIG. 11A shows a case where the first camera 20 and the second camera 30 are used for shooting, and FIG. 11B shows a configuration in which the third camera 60 is added to the configuration of FIG. Show the case. In addition, the solid line arrow in FIG.11 (b) shows the imaging | photography direction of the 3rd camera 60. FIG.

  As shown in FIG. 11 (a), when multi-touch is photographed with two cameras, the indicator C and the indicator D are the first camera 20 or the second camera 30 (the second in FIG. 11 (a)). There is a possibility that the indicator C closer to the first camera 20 or the second camera 30 is large and the other indicator D falls into the shadow of the indicator C in a straight line in the shooting direction of the camera 30). is there. In this case, only one indicator is shown in the frame image of either the first camera 20 or the second camera 30, and therefore multi-touch cannot be detected well, or the detection accuracy may be reduced. .

  On the other hand, as shown in FIG. 11B, when the multi-touch is photographed with three cameras, the indicators C and D are arranged in a straight line as described above, or in the shadow of one indicator Even if the indicator has entered, the image is taken by at least two cameras, the first camera 20, the second camera 30, and the third camera 60, as shown. That is, two indicators appear in at least any two of the frame images 20A, 30A, and 60A. Therefore, the touch detection device 100 or 300 can detect multi-touch from a frame image in which a plurality of indicators are captured.

  More specifically, when the image acquisition unit 110 acquires the frame images 20A, 30A, and 60A, it sends them to the correction unit 120. The correction unit 120 corrects the frame images 20A, 30A, and 60A by the method described in the first embodiment, and sends the corrected images 20B, 30B, and the corrected image 60B to the indicator specifying unit 130.

  The indicator specifying unit 130 specifies the indicator shown in the corrected images 20B, 30B, and 60B by the method described in the first embodiment. Here, the indicator specifying unit 130 may specify a plurality of indicators in each corrected image. Then, the position (first coordinate group to third coordinate group) indicated by each indicator on the display surface 10 </ b> A is specified and sent to the touch detection unit 140.

  When the touch detection unit 140 acquires the first coordinate group to the third coordinate group, any coordinate in the first coordinate group matches any one coordinate in either the second coordinate group or the third coordinate group. It is determined whether or not to do. If there is a matching coordinate, the touch detection unit 140 determines that the touch has been performed, and detects the matching coordinate as the touch coordinate. Similarly, for each coordinate in the second coordinate group, the touch detection unit 140 determines whether or not there is a coordinate that matches the other group. Detect as touch coordinates. As described above, when the apparent touch position matches in at least two of the corrected images 20B, 30B, and 60B, the touch detection unit 140 detects the position as the touch coordinates of the indicator, Multi-touch can be detected.

  Furthermore, the operation specifying unit 150 described in the second embodiment may specify the type of operation for each touch coordinate in accordance with the determination result of the touch detection unit 140. In the case of multi-touch, the operation specifying unit 150 may specify an operation performed on the entire display surface 10A from a combination of operations related to each touch. Specifically, when the operation specifying unit 150 specifies two swipe operations such that the touch coordinates approach each other, the operation specifying unit 150 may specify these two swipe operations as a pinch-in operation. Further, when specifying two swipe operations that cause the touch coordinates to move away from each other, the operation specifying unit 150 may specify these two swipe operations as a pinch-out operation.

[Embodiment 4]
Note that only one of the first camera 20, the second camera 30, and the third camera 60 always shoots the display surface 10 </ b> A, and the other cameras are the pointers that are always shooting. It is good also as driving only when catching.

  For example, in the touch detection device 100 or 300 according to the present embodiment, the image acquisition unit 110 always acquires only the frame image 20A from the first camera 20, and the correction and indicator of the correction unit 120 is obtained for the frame image 20A. The specifying unit 130 specifies the indicator. When the indicator specifying unit 130 determines that the indicator is reflected in the frame image 20A, the touch detection device 100 or 300 according to the present embodiment performs the second camera 30 (and the third camera 60). You may instruct | indicate to start the substantially simultaneous imaging | photography with the 1st camera 20, and to send the frame image 30A (and 60A). Thereby, when the touch is not performed, the display surface 10A is monitored by only one camera, and thus the touch detection device 100 or 300 according to the present embodiment reduces the power consumption related to the touch detection. Can do.

[Embodiment 5]
Further, when it is difficult for the first camera 20 and the second camera 30 (and the third camera 60) to capture the entire display surface 10A, such as when the display device 10 is a large display, the touch detection device 100 or 300 is From the frame images 20A and 30A (and 60A) of the first camera 20 and the second camera 30 (and the third camera 60), an area capable of touch detection on the display surface 10A (capable of photographing with both cameras) The area may be specified and the display control apparatus 200 notified of the area. Then, the display control apparatus 200 may create display data that collectively displays objects such as buttons that prompt the user to perform a touch operation in the notified area, and may display the display data on the display apparatus 10.

  According to said structure, the touch detection apparatus 100 or 300 which concerns on this embodiment cooperates with the display control apparatus 200 which concerns on this embodiment, and a user's touch is detected by the 1st camera 20 and the 2nd camera 30 (and The third camera 60) can be guided within an imageable range. Therefore, the touch detection device 100 or 300 according to the present embodiment can touch the partial area of the display surface 10A even when the entire display surface 10A cannot be within the imaging range of the first camera 20 and the second camera 30. Can be detected. In addition, the display control apparatus 200 according to the present embodiment can perform display according to the touch.

[Embodiment 6]
The control blocks (particularly the image acquisition unit 110, the correction unit 120, the indicator identification unit 130, the touch detection unit 140, and the operation identification unit 150) of the touch detection devices 100 and 300 are formed on an integrated circuit (IC chip) or the like. It may be realized by a logic circuit (hardware), or may be realized by software using a CPU (Central Processing Unit).

  In the latter case, the touch detection devices 100 and 300 include a CPU that executes instructions of a program that is software that implements each function, and a ROM (Read Only) in which the program and various data are recorded so as to be readable by the computer (or CPU). Memory) or a storage device (these are referred to as “recording media”), a RAM (Random Access Memory) for expanding the program, and the like. And the objective of this invention is achieved when a computer (or CPU) reads the said program from the said recording medium and runs it. As the recording medium, a “non-temporary tangible medium” such as a tape, a disk, a card, a semiconductor memory, a programmable logic circuit, or the like can be used. The program may be supplied to the computer via an arbitrary transmission medium (such as a communication network or a broadcast wave) that can transmit the program. The present invention can also be realized in the form of a data signal embedded in a carrier wave in which the program is embodied by electronic transmission.

[Summary]
A touch detection device (touch detection device 100 or 300) according to aspect 1 of the present invention is a touch detection device that detects a touch of an indicator on a display surface (display surface 10A) of a display device (display device 10), The first image (frame image 20A) of the display surface captured by the first camera (first camera 20) and the display captured by the second camera (second camera 30) from a different angle substantially simultaneously with the first camera. A video acquisition unit (image acquisition unit 110) for acquiring a second video (frame image 30A) of the screen, and a touch position of the indicator reflected in the first video acquired by the video acquisition unit with respect to the display surface When the first position (first coordinate) is substantially the same as the second position (second coordinate) that is the touch position of the indicator on the display surface of the indicator in the second video, the substantially same Place The is characterized in that it comprises a touch detection unit which detects a touch position of the pointer (the touch detection unit 140), the.

  When the first video and the second video include an indicator, two ways can be considered. The first is when the indicator is not in contact, and the second is when the indicator is in contact with the display surface.

  When an indicator that is not in contact with the display surface is photographed, the indicator apparently differs from the position projected on the display surface as viewed from the photographing angle, that is, the position on the display surface that is closest to the indicator. It appears in the position. On the other hand, for the indicator that is in contact with the display surface, the position projected on the display surface and the position on the display surface that is closest to the indicator appear in the image.

  Using the above principle, the touch detection unit detects a touch when the first position and the second position substantially coincide with each other in the first video and the second video taken at different angles substantially simultaneously. Thereby, the touch detection part can distinguish and detect the touch of a pointer from non-contact proximity. Therefore, the touch detection device has an effect of being able to detect even a touch on a display device that does not have a touch detection function.

  The touch detection device according to aspect 2 of the present invention is the touch detection device according to aspect 1, in which the touch detection unit is configured so that each of the indicators reflected in the first image and the second image has a protruding portion. The touch position by the protruding portion of the indicator shown in the first video may be the first position, and the touch position by the protruding portion of the indicator shown in the second video may be the second position.

  According to the above configuration, when the positions indicated by the protruding portions of the indicator in the first video and the second video are the first position and the second position, respectively, the touch detection unit substantially matches these positions. In this case, the substantially matched position is set as the touch position. Here, the “protruding portion” means a portion showing a predetermined convex shape in the image of the indicator. More specifically, the “protruding portion” is, for example, the shape of the whole or tip of each fingertip, the shape of the whole or tip portion of a pen, a pointer, or the like. The “position indicated by the protruding portion” is, for example, a position indicated by one point at the tip of the protruding portion.

  More specifically, for example, when the user stretches the finger and touches the display surface, the indicator (user's finger) shown in the first video and the second video has a shape having a protruding portion, so that touch detection is performed. The unit detects the touch of the finger as a touch. On the other hand, when the user bends a finger (touches the display surface) with a fist, the indicator (user's fist) shown in the first video and the second video does not have a protruding portion. Do not detect fist touch as touch. As described above, the touch detection unit detects a touch when the protruding portion of the indicator touches the display surface, and thus can prevent an unintended contact with the display surface from being erroneously detected as a touch.

  In other words, when the user wants to perform a touch operation on the display surface, the user touches the display surface with an indicator such as a fingertip, and when the user does not want to perform the touch operation but wants to touch the display surface (for example, display to other users). For example, when an image of a surface is desired, the touch detection device can detect a touch related to the touch operation only when the touch operation is intended by touching the display surface with an object such as a fist that has no protruding portion. .

  In addition, even when an object (such as a paper memo) that is not an indicator is pasted (contacted) on the display surface itself, the touch detection unit does not detect the touch of the object as a touch. Therefore, the touch detection unit can prevent erroneous detection of a contact of an object other than the indicator as a touch.

  The touch detection device according to aspect 3 of the present invention is the touch continuation specification that specifies the time (touch duration) during which the touch detection unit continuously detects the touch position of the indicator in aspect 1 or 2 above. An operation specifying unit (operation specifying unit) for specifying an operation indicated by a touch corresponding to the continuously detected touch position according to the time specified by the unit (duration specifying unit 151) and the touch continuation specifying unit 150).

  According to said structure, an operation specific | specification part specifies operation according to the time when a touch is continuing. Thereby, for example, a tap operation and a long press operation can be distinguished and specified.

  In the touch detection device according to aspect 4 of the present invention, in the aspect 3, the operation specifying unit includes the time specified by the touch continuation specifying unit, and the indicator detected by the touch detection unit at the time. The operation may be specified according to the change in the touch position.

  According to said structure, an operation specific | specification part specifies operation in consideration of the change of the touch position in the said time in addition to the time when the touch is continuing. By considering the change in the touch position, the operation specifying unit can specify various operations.

  The touch detection device according to aspect 5 of the present invention is the touch detection device according to any one of the aspects 1 to 4, in which the video acquisition unit is different from the first camera and the second camera at the same time in the third camera (third camera 60). A third image (frame image 60A) of the display surface taken from an angle is further acquired, and the touch detection unit is configured to display at least two images of the first image, the second image, and the third image. When a plurality of positions where the touch positions on the display surface of the indicator reflected in the video are substantially coincident are specified, the plurality of positions may be detected as a plurality of touch positions by the plurality of indicators.

  According to the above configuration, each of the first to third cameras captures the display surface from different angles substantially simultaneously. Therefore, even when a plurality of indicators touch the display surface (multi-touch), each indicator It will appear in at least two of the first to third images. Therefore, when the apparent touch position of the indicator substantially matches in at least two images, multi-touch can be detected by detecting each of the substantially matched positions as the touch position of a plurality of indicators. .

  A touch detection device according to an aspect 6 of the present invention is the touch detection device according to any one of the above aspects 1 to 5, wherein the position indicating the outer peripheral line of the display surface set in advance for each of the first video and the second video is a reference In addition, a correction unit (correction unit 120) that performs distortion correction by extracting images of the display surface area from the first video and the second video acquired by the video acquisition unit may be provided.

  When an image is taken from an angle other than the angle perpendicular to the display surface, the display surface is photographed with distortion according to the photographing angle. According to said structure, a correction | amendment part performs distortion correction of a 1st image | video and a 2nd image | video. For example, the shape of the display surface shown in the first image and the second image is corrected to the same shape as the actual shape of the display surface. Then, when the first position in the first video corrected by the correction unit and the second position in the second video corrected by the correction unit substantially coincide with each other, the touch detection unit determines the substantially coincident position of the indicator. Detect as touch position. Thereby, the touch detection unit can detect the touch position from the corrected first video and second video, and can detect the touch position on the display surface more accurately.

  A touch detection method (touch detection process) according to an aspect 7 of the present invention is a touch detection method of a touch detection device that detects a touch of an indicator on a display surface of the display device, and the display surface is captured by a first camera. Acquired in the video acquisition step (S100) for acquiring the first video and the second video of the display surface captured by the second camera at substantially the same angle as the first camera, and the video acquisition step. A first position that is a touch position of the indicator on the display surface of the indicator shown in the first video and a second position that is a touch position of the indicator on the display surface of the second video substantially match. A touch detection step (S108) for detecting the substantially coincident position as the touch position of the indicator. According to said structure, there exists an effect similar to the touch detection apparatus which concerns on aspect 1. FIG.

  The touch detection device according to each aspect of the present invention may be realized by a computer. In this case, the touch detection device is operated on each computer by operating the computer as each unit (software element) included in the touch detection device. The control program for the touch detection device to be realized and the computer-readable recording medium on which the control program is recorded also fall within the scope of the present invention.

  The present invention is not limited to the above-described embodiments, and various modifications are possible within the scope shown in the claims, and embodiments obtained by appropriately combining technical means disclosed in different embodiments. Is also included in the technical scope of the present invention. Furthermore, a new technical feature can be formed by combining the technical means disclosed in each embodiment.

  The present invention can be suitably used to detect a touch of an indicator on a display device. The present invention can be suitably used particularly when a touch detection function is mounted on a display device having no touch detection function as a retrofit.

DESCRIPTION OF SYMBOLS 10 Display apparatus 20 1st camera 30 2nd camera 60 3rd camera 40, 50 STB
100, 300 Touch detection device 200 Display control device 110 Image acquisition unit (video acquisition unit)
DESCRIPTION OF SYMBOLS 120 Correction | amendment part 130 Indicator specification part 140 Touch detection part 150 Operation specification part 151 Duration specification part (touch continuation specification part)
152 Coordinate change specifying part

Claims (5)

A touch detection device that detects a touch of an indicator on a display surface of a display device,
A video acquisition unit that acquires a first video of the display surface captured by the first camera and a second video of the display surface captured by the second camera from a different angle substantially simultaneously with the first camera;
The first position, which is the touch position of the indicator reflected in the first video acquired by the video acquisition unit, with respect to the display surface, and the touch position of the indicator, reflected in the second video, relative to the display surface. A touch detection device comprising: a touch detection unit configured to detect the substantially matched position as the touch position of the indicator when the second position substantially matches.   The touch detection unit is configured to determine a touch position by the protruding portion of the indicator reflected in the first video when the pointer reflected in the first video and the second video each has a protruding portion. The touch detection device according to claim 1, wherein the first position is a touch position by the protruding portion of the indicator in the second video, and the second position is the touch position. A touch continuation identification unit that identifies a time during which the touch detection unit continuously detects the touch position of the indicator;
The operation specifying unit that specifies an operation indicated by the touch corresponding to the continuously detected touch position according to the time specified by the touch continuation specifying unit. 3. The touch detection device according to 2.   The operation specifying unit specifies the operation according to the time specified by the touch continuation specifying unit and a change in the touch position of the indicator detected by the touch detection unit at the time. The touch detection device according to claim 3. The video acquisition unit further acquires a third video of the display surface taken from a different angle by the third camera substantially simultaneously with the first camera and the second camera,
The touch detection unit is configured to determine a position where a touch position of the indicator on the display surface of the first image, the second image, and the third image is substantially coincident with the display surface in the image. 5. The touch detection device according to claim 1, wherein when a plurality of positions are specified, the plurality of positions are respectively detected as a plurality of touch positions by the plurality of indicators.

Images