JP2014032600A - Display input device - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a display input device enabling a natural, non-contact input operation using a finger or the like of an operator, oblivious to a device.SOLUTION: A display input device includes: control means for a flat panel display D; a panel-type imaging optical element 1; sonic wave transmission means (speaker 4) and sonic wave reception means (microphone 5) that send and receive supersonic waves or sonic waves, respectively; and moving body detection means connected thereto. The display D is arranged below the imaging optical element 1 with a display surface Da thereof inclined at a prescribed angle (α). The moving body detection means analyzes a reflected supersonic wave emitted toward a finger H side of an operator from the sonic wave transmission means and reflected at the finger H, and converts the reflected supersonic wave to a detection signal. The control means displays on the display D a projected image (image I) corresponding to a position and movement of the finger H according to the detection signal. A mirror video of the image I is projected above the imaging optical element 1 as a spatial image I'.

Description

  The present invention relates to a display input device capable of displaying a two-dimensional image such as a photograph as a spatial image floating in space and interactively operating the spatial image with an operator's finger or the like.

  As a method for displaying (projecting) an image or video in a space, a binocular method, a multi-view method, a space image method, a volume display method, a hologram method, and the like are known. There has been proposed a display input device capable of intuitively interacting (interacting) a two-dimensional image or a three-dimensional image (aerial image) with a hand or a finger (hereinafter referred to as “finger”).

  As a finger recognition input means (user interface) in such a display input device, a vertical and horizontal light grid is formed in the detection region (plane) by a large number of LEDs, lamps, etc. There has been proposed a system that detects shielding (shielding) of a grating with a light receiving element or the like and detects the position, coordinates, and the like of the input body (see Patent Documents 1 and 2).

JP 2005-141102 A JP 2007-156370 A

  However, as described above, the display input device that detects the shielding of the invisible light grid formed on the detection plane and detects the position and coordinates of the input body, the aerial image is used for the installation of the LED and the light receiving element. Since it is displayed in a frame (hollow frame) that is used, this frame enters the operator's field of view, and there is a problem that it is difficult to obtain a three-dimensional effect or a sense of reality. In addition, the frame may be recognized as an obstacle or a visual obstacle in the input operation. In this case, the input operation (movement of fingers, etc.) by the operator is restricted or smooth. There is a risk of disappearing. Furthermore, it may be considered that external light is incident on the LED or the light receiving element to cause a malfunction.

  The present invention has been made in view of such circumstances, and the provision of a display input device capable of naturally performing non-contact input operation using an operator's fingers and the like without being conscious of the device is provided. Objective.

  In order to achieve the above object, the display input device of the present invention uses an optical element having an imaging function to display an image displayed on a display surface of a flat panel display at a spatial position away from the imaging optical element. A display input device that interactively updates the image of the flat panel display in response to an instruction input using an operator's finger, and that controls the flat panel display And a panel-shaped imaging optical element, a sound wave transmitting means and a sound wave receiving means for transmitting and receiving ultrasonic waves or sound waves, and a moving object detecting means connected thereto, and the flat panel display includes the panel-shaped imaging optical element. The display surface is disposed at a position below the element in a state where the display surface is inclined at a predetermined angle, and the sound wave transmitting means faces the operator side. The ultrasonic wave or sound wave is disposed at a position where the ultrasonic wave or sound wave can be projected, and the sound wave receiving unit detects the reflected ultrasonic wave or the reflected sound wave that is emitted from the sound wave transmitting unit and reflected by the finger and inputs the detected sound to the moving object detection unit. The moving body detection means analyzes the reflected ultrasonic wave or the reflected sound wave to convert the position and movement of the finger into a detection signal, and the control means determines the flat signal based on the detection signal. The panel display is configured to display an image corresponding to the position and movement of the operator's finger.

  That is, the present inventor has conducted research in order to solve the above-described problems, and as a result, using a panel-shaped imaging optical element, two images such as photographs are placed above the imaging optical element. A dimensional image is imaged in a state of rising (standing up), and at an appropriate position around the imaging optical element, using an ultrasonic wave or sound wave (hereinafter referred to as an ultrasonic wave) By providing sound wave transmission / reception means (transceiver) that detects position and movement in a non-contact manner, natural input operations can be performed without being aware of obstacles and systems like conventional optical detection means using a frame. I found it possible.

  The display input device of the present invention comprises a control means for a flat panel display, a panel-shaped imaging optical element, and a case for housing them, and the sound wave transmitting means for transmitting ultrasonic waves or the like is directed toward the operator side. The sound wave receiving means detects the reflected wave such as the ultrasonic wave by the operator's finger or the like and inputs it to the moving object detecting means. In the display input device, the moving body detection unit analyzes the reflected ultrasonic wave and the like to convert the position and movement of a finger into a detection signal. Based on the detection signal, the control unit of the flat panel display On the screen, an appropriate image corresponding to the position and movement of the operator's fingers and the like is displayed / updated. As a result, the display input device of the present invention allows the input operation using the fingers to be performed in a non-contact and natural form without the operator being aware of the means (device or system) for detecting the position or the like of the fingers. Can be done. Accordingly, the display input device of the present invention can interactively update the video on the display surface of the flat panel display in response to the input operation by the operator's finger or the like. Moreover, the display input device can be a user-friendly device that can be operated intuitively by anyone, and has the advantage that it is not easily affected by outside light or the like.

  Further, in the display input device of the present invention, the moving body detection unit extracts a frequency shift component due to the Doppler effect from the reflected ultrasonic wave or the spectrum of the reflected sound wave input from the sound wave receiving unit by calculation, What is designed to analyze the position, direction and speed of the finger of the operator is a small number of sound wave transmitting / receiving means (transmitter / receivers), and the position (coordinates) and movement of the operator's finger can be analyzed. It becomes possible to detect in a dimensional (planar) or three-dimensional (stereoscopic) manner. Therefore, the display input device of the present invention can also cope with gesture operation input of complicated movement by the operator.

  The “ultrasonic wave” used in the present invention is an elastic vibration wave (sound wave) having a high frequency equal to or higher than a human audible frequency (frequency), and generally means a sound wave having a frequency of 16 to 18 kHz or more. . However, the boundary between the sound wave and the ultrasonic wave in the present invention is not particularly required, and both can be used in the same manner.

It is sectional drawing which shows the structure of the display input device of 1st Embodiment of this invention. It is a figure explaining the usage method of the display input device in 1st Embodiment of this invention. It is a partial cross section figure explaining the method of setting a flat panel display to the display input device of 1st Embodiment of this invention. (A) is a schematic diagram which shows the structure of the micromirror array used by embodiment of this invention, (b) is a figure explaining the imaging pattern of the aerial image by a micromirror array. It is sectional drawing which shows the structure of the display input device of 2nd Embodiment of this invention. It is a perspective view which shows the structure of the display input device of 2nd Embodiment of this invention.

  Next, embodiments of the present invention will be described in detail with reference to the drawings. However, the present invention is not limited to this embodiment.

  FIG. 1 is a schematic cross-sectional view showing the configuration of a display input device according to the first embodiment of the present invention, and FIG. 2 is a diagram for explaining how to use the display input device. In this example, a liquid crystal display screen (LCD) of a mobile phone (smart phone or the like) is used for the flat panel display (reference D). Further, reference numeral 10 denotes a case that houses the display input device, reference numeral 2 denotes a transparent plate-like member (transparent plate) that covers the upper opening of the case 10, reference numeral denotes a finger of an operator who performs an input operation (gesture), and the like. H. Furthermore, in these figures, the “image” (thick line arrow I in FIG. 1) displayed on the LCD screen and the projected “aerial image” (thick line arrow I ′ in FIG. It is shown with emphasis.

  As shown in FIG. 1, the display input device according to the present embodiment includes a mounting table 3 for mounting a flat panel display D in a bottomed box-shaped case 10 having an open top surface, and a panel-like imaging. A sound wave transmitting means (ultrasonic oscillator, in this example, a speaker 4) for transmitting ultrasonic waves or the like toward the operator's finger H or the like on the panel 10b that is arranged with the optical element 1 and is attached to the outside, and the speaker Sound wave receiving means (ultrasonic wave receivers, in this example, microphones 5 and 5 ′) for detecting reflected waves such as ultrasonic waves emitted from 4 are arranged. The smartphone (D) placed on the placement table 3 in the case 10 is disposed with its display surface Da inclined at a predetermined angle (α) with respect to the imaging optical element 1. The smartphone (D) controls the display of the flat panel display D and the moving object detection means for analyzing the voice detected by the microphones 5 and 5 ′ (both are programs or applications built in the smartphone) , Not shown). This is a feature of the display input device of the present invention.

  The display input device will be described in detail. As shown in FIG. 1, the case 10 that accommodates each member has the transparent plate 2 disposed in the opening on the upper surface thereof, and the inner surface of the case 10 is made of light. In order to prevent irregular reflection due to, black (chromaticity 0, saturation 0, lightness 0) or a dark color close thereto is used. In addition, as shown in FIGS. 2 and 3, on one side surface of the case 10, a flat panel display D (smart phone in the figure) described later is inserted into the case 10 (on the mounting table 3). A slit-shaped insertion port 10a is provided.

  The transparent plate 2 disposed on the upper surface opening of the case 10 and covering the case 10 transmits light in the case opening direction (up and down direction) (image formation of the aerial image I ′) and the image forming optical element. 1 is provided for mounting (positioning and fixing), and is made of a hard plate-like member having a light transmittance of 80% or more in the visible light region, such as a glass plate or an acrylic resin plate.

  The panel-shaped imaging optical element 1 attached to the transparent plate 2 forms an image, such as an image I or video existing on one side, as a real image (aerial image I ′) on the other side. The display input device of the present invention includes various lenses including a Fresnel lens, a refraction-type imaging element such as an afocal optical micromirror and a corner reflector, and an erecting microlens array. A double imaging element can be used. In particular, in this embodiment, as shown in FIG. 1, a micromirror array (convex corner reflector array, detailed structure is shown in FIG. 1) that forms an image in a plane-symmetric position with respect to the element plane P of the imaging optical element 1. 4) is preferably used. In this example, the micromirror array (imaging optical element 1) is attached to the lower surface (inner surface) side of the transparent plate 2 (lid member) attached to the upper surface opening of the case 10. An opening (hole) having the same shape as that of the imaging optical element 1 may be provided in the transparent plate 2 and may be fixed by being fitted into this opening.

  The micromirror array (corner reflector array) will be described in more detail. As shown in FIG. 4A, the micromirror array is formed on the lower surface of the substrate (substrate) 11 (the lower surface of the imaging optical element 1 in FIG. 1). 1b), a large number of small unit optical elements 12 (corner reflectors) convex downward are arranged in an oblique grid pattern. [FIG. 4 (a) is a view of the array viewed from below. It is. ].

  In each unit optical element 12, the two light reflecting surfaces (first side surface 12a and second side surface 12b) constituting each corner 12c indicate the direction of the operator's viewpoint (right side in FIGS. 1 and 2). It comes to face. When the micromirror array (imaging optical element 1) is viewed from above, as shown in FIG. 4B, the array has an outer edge (outer side) on the front (operator side) of the case 10. The image I on the lower side of the imaging optical element 1 is positioned symmetrically with respect to the imaging optical element 1 (imaging optical element 1). An aerial image I ′ is formed on the upper side of the image.

  Next, as shown in FIG. 1 and FIG. 2, the mounting table 3 for mounting the flat panel display D includes a plate-like member 3a and left and right for supporting the plate-like member 3a in an upright state. It consists of a pair of bases 3b, 3b, and is disposed in the case 10 at a position facing the lower surface 1b of the imaging optical element 1 (below the imaging optical element 1). The plate-like member 3a is placed on the bases 3b and 3b while being inclined at a predetermined angle α with respect to the bottom surface of the case 10 and the element surface P of the imaging optical element 1 (or the lower surface 1b of the imaging optical element 1). The upper surface of the flat panel display D is supported and fixed.

  And the display surface Da of the said flat panel display D becomes imaging optical element by mounting the smart phone etc. which are equipped with the flat panel display D on the mounting surface (above the plate-shaped member 3a) of the said mounting base 3. It is held in a state where the angle α is inclined with respect to the element surface P of one element. The inclination angle α of the mounting table 3 in the case 10 with respect to the element surface P (lower surface 1b) of the imaging optical element 1 is adjusted so that the imaging action by the imaging optical element 1 is optimized. Usually, it is set to 30 ° or more and less than 90 °, preferably 40 ° or more and 80 ° or less.

  Further, as the flat panel display D used for displaying the image I, a liquid crystal display panel (LCD) having a backlight, a plasma display panel, an organic EL display panel, and the like can be as uniform as possible over all visible light wavelengths. It is desirable to use a display panel that can reproduce “white” and “black” when not displayed with good contrast. The flat panel display D may be a display unit such as a mobile phone or a portable information terminal as in this example. Specifically, the flat panel display D may be a smartphone, a tablet PC, a digital photo frame, a portable game machine, Among the portable book readers, PDAs, electronic dictionaries, etc., the size that can be mounted on the mounting table 3 can be used among the types whose display units are always exposed (not covered). In the case of using a display that does not have a backlight, light may be emitted from a separately provided light source toward the display screen and the reflected light may be used.

  Then, as described above, the display input device of the present embodiment extends to the case 10 as shown in FIGS. 1 and 2 in order to detect the position and movement of the operator's fingers H and the like. A sound wave oscillator (speaker 4) for projecting (emitting) ultrasonic waves or the like toward the operator's finger H on the panel 10b, and the speaker's finger H etc. It is characterized in that a sound wave receiver (microphones 5 and 5 ′) for detecting reflected reflected waves is provided.

  The speaker 4 is used for transmitting an ultrasonic wave or a sound wave having a specific frequency in accordance with a signal from the moving body detecting means (a part of an ultrasonic oscillator or a pulsar transducer). Any format can be used as long as it can be converted to. For example, as a typical example, a dynamic speaker having a diaphragm can be used. The speaker 4 may be disposed outside the case 10 or near the operator as long as it can project ultrasonic waves or the like toward the operator, and the directivity (sound wave) (Orientation characteristics) is appropriately selected depending on the use form. The speaker 4 is connected to a power source for driving (not shown) and connected to a smartphone that also serves as the moving object detection means by wireless connection such as Bluetooth (registered trademark) or wireless LAN, or a cable. It is performed by a wired connection using the like.

  The microphones 5 and 5 ′ are configured as a part of an ultrasonic receiver or a pulsar receiver, and detect reflected ultrasonic waves or reflected sound waves emitted from the speaker 4 and reflected by the finger H or the like. It inputs to the smart phone (D) which combines the said moving body detection means. As these microphones 5 and 5 ′, moving coil type, condenser type, ribbon type, etc. microphones used for obtaining voice and the like are used.

  In the above configuration, the panel 10b that protrudes upward from the case 10 is spaced apart from the case 10 so as to easily detect the reflected wave reflected by the operator's finger H or the like. The position of the finger H can be triangulated using the time difference of reflection by the ultrasonic wave or the like. A power supply (not shown) is also connected to the microphones 5 and 5 ′, and a connection with a smartphone that also serves as the moving object detection means is a wireless connection such as Bluetooth (registered trademark) or wireless LAN, or a cable. This is done by a wired connection using.

  Further, in the above embodiment, in order to accurately capture the position and movement of the operator's finger H etc. by triangulation method or the like, the same microphones 5 and 5 ′ are kept at a certain distance as shown in FIG. Although an example in which a plurality (two in this example) are arranged in parallel has been shown, these microphones 5 and 5 'may also be arranged outside the case 10 or close to the operator. The location and number are also selected as appropriate according to the usage form. Furthermore, one or a plurality of ultrasonic distance sensors (for example, 28015 manufactured by PARALLAX) in which the microphone 5 and the speaker 4 are integrated (module) may be used.

  In this embodiment, as described above, a program or application built in the smartphone (D) or the like is used as the moving object detection unit. This moving body detection means (moving body detection engine) analyzes the difference between reflected waves such as ultrasonic waves input from the microphones 5 and 5 ', and moves the position (coordinates) and movement (speed, direction, etc.) of the finger H or the like. Is converted and output as a detection signal, and this detection signal is transmitted to the control means (the controller of the flat panel display D) mounted on the same smartphone. Based on the detection signal, the controller of the flat panel display D causes the flat panel display D to display an image corresponding to the detection signal.

  In the above-described embodiment, the smartphone has both the function as the moving object detection unit, the function as the control unit of the flat panel display D, and the function as the calculation unit and the management unit. Although shown, in addition to the smartphone, a control unit (controller) including a computer or the like may be provided separately, and the control unit may be configured to share functions other than the display function.

  Further, when the display input device includes voice recognition means, the microphone 5 is used for input of external voice or sound (also used for reception of ultrasonic waves, etc.), and the recognition result (voice instruction) is set to the flat signal. It may be used (in combination) for updating the display on the panel display D. Further, in the case where the display input device includes voice synthesis means or voice output means for outputting voice or sound corresponding to the display of the flat panel display D, a character string corresponding to the display using the speaker 4. Can be read aloud, and utterances and pronunciations such as warning sounds can be performed.

  Next, a method of projecting and displaying a real image (spatial image I ′) of the image I on the case 10 using the flat panel display D (smart phone) will be described.

  When displaying (projecting) the image I in the display input device, first, a smartphone including a flat panel display D is prepared, and predetermined processing (image processing such as background color change and contrast improvement) is performed on the flat panel display D. ) Image I (in FIG. 2 and FIG. 3, “dog picture” as an example) is displayed, and the smartphone is placed upside down (top and bottom) of the image I so that the insertion port 10 a It is moved to the vicinity of the side surface of a certain case 10 (see arrow A in FIGS. 2 and 3).

  Next, with the image I upside down, the smartphone is pushed into the case 10 from the insertion port 10a (arrow B in FIGS. 2 and 3), and the smartphone is placed sideways on the placement surface. Slide and set at a predetermined position on the mounting table 3 (position C in FIGS. 2 and 3). Thereby, the planar two-dimensional image I (photograph or the like) displayed on the screen of the flat panel display D is formed into a panel-like image as a spatial image I ′ (stereoscopic two-dimensional image) having a sense of depth. The image is displayed (projected) above the optical element 1 [see FIG. 2].

  In addition, if the flat panel display different from the above is arranged in an inclined manner instead of the plate-like member 3a constituting the mounting surface of the mounting table 3, the flat panel display D is not mounted. The display of the image I and the projection of the aerial image I ′ can be continued. When the flat panel display D is not mounted (inserted) if a self-luminous display device such as an LED display or an LED digital watch is incorporated in the mounting surface (plate member 3a) of the mounting table 3. However, it is preferable because these displays and projections can be continued without interruption. Furthermore, if the mounting table 3 is configured to have a charging (charging stand or cradle) function for the above mobile phone or portable information terminal, it is possible to charge each device while it is inserted into the case 10. The time for charging can be used effectively.

Next, a second embodiment of the present invention will be described.
5 and 6 are diagrams showing the configuration of the display input device according to the second embodiment of the present invention.

  The display input device of the second embodiment also includes a mounting table 3 for mounting the flat panel display D and a panel-shaped imaging optical element 1 in a bottomed box-shaped case 10 having an upper opening. A sound wave transmitting means for transmitting ultrasonic waves or the like toward the operator's finger H or the like on the panel 10b which is disposed and attached to the outside of the case 10, and ultrasonic waves emitted from the sound wave transmitting means, etc. A sound wave receiving means for detecting the reflected wave is provided. In addition, the smartphone (D) placed on the placement table 3 in the case 10 is a moving body that analyzes the sound detected by the control means that controls the display of the flat panel display D and the sound wave receiving means. Detection means (both are programs or applications built in the smartphone) are provided.

  The display input device of the second embodiment is different from the first embodiment in that sound wave transmitting means (speaker) for transmitting ultrasonic waves and the like, and sound wave receiving means (microphone) for detecting reflected waves such as ultrasonic waves, and the like. Is configured as an integrated ultrasonic sensor module M (a pair of a pulsar transducer 6 corresponding to a speaker and a pulsar receiver 7 corresponding to a microphone), and a moving object detection unit to which data is input from the module M is configured as the module M. A frequency shift component (Doppler shift) due to the Doppler effect is extracted from the frequency spectrum of the reflected wave input from the receiver 7 by calculation, and the position of the operator's finger H and the like, and the direction and speed of the movement are represented by 1 The point is that the module M can be analyzed.

  Other configurations are the same as those of the display input device of the first embodiment. That is, the panel-shaped imaging optical element 1 attached to the transparent plate 2 that covers the case 10 includes various lenses including a Fresnel lens and the like, and a refractive imaging element such as an afocal optical micromirror and a corner reflector. An erecting equal-magnification imaging element such as a microlens array can be used, and among them, a micromirror array (convex corner reflector array, see FIG. 4 for the detailed structure) is preferably used.

  Further, as the flat panel display D used for displaying the image I, a liquid crystal display panel (LCD) having a backlight, a plasma display panel, an organic EL display panel, and the like, which is as uniform as possible over the entire visible light wavelength, “white” It is desirable to use a display panel that can reproduce “black” when not displayed and “black” when not displayed.

  Furthermore, if both the sound wave transmitting means (transducer 6) and the sound wave receiving means (receiver 7) of the ultrasonic sensor module M are dynamic types having a diaphragm, these transducer 6 and receiver 7 (ie, speaker) are used. As a microphone), acoustic parts having the same specifications can be used.

  The ultrasonic sensor module M is similarly connected to the smartphone (D) that also serves as the moving object detection means by wireless connection or wired connection. Further, the ultrasonic sensor module M may be disposed outside the case 10 or close to the operator, and the location and number of the ultrasonic sensor module M are appropriately selected depending on the usage form. Further, in addition to the smartphone (D), a control unit (controller) comprising a computer or the like is separately provided, and the functions as the moving object detection means, the control means of the flat panel display D, the calculation means, the management means, etc. You may comprise so that it may share in a part.

  With the above configuration, the display input device according to the second embodiment has a small number of sound wave transmission / reception means (transceivers), and the position (coordinates) and movement of the operator's fingers and the like can be two-dimensional (planar) or tertiary. Original (stereoscopic) detection is possible. In addition, the display input device has an advantage that it can accurately respond to complicated movements (gesture operation input) of the operator's fingers H and the like.

  In the above two embodiments, the program (engine) used in the moving object detection means, the control means, the voice output means, the calculation means, etc. is shown as an example of operating locally on a smartphone or the like. It may be distributed in the separately prepared controller or the like, or may be arranged outside (server or the like) via a network such as a cloud.

  Since the display input device of the present invention can intuitively interact with an aerial image without being aware of the input device, it can be an interface device that is easy for the user to get familiar with.

DESCRIPTION OF SYMBOLS 1 Imaging optical element 4 Speaker 5 Microphone D Flat panel display Da Display surface H Finger I Image I 'Space image

Claims (2)

  The image displayed on the display surface of the flat panel display is imaged using an optical element having an imaging function in a state of being raised in a spatial position away from the imaging optical element, and the operator's fingers are A display input device that interactively updates the image of the flat panel display in response to the used instruction input, the control means for the flat panel display, the panel-shaped imaging optical element, and ultrasonic waves or sound waves. A state in which the flat panel display is inclined at a predetermined angle at a position below the panel-shaped imaging optical element, and includes a sound wave transmitting means and a sound wave receiving means for transmitting and receiving, and a moving body detecting means connected thereto. The sound wave transmitting means is disposed at a position where ultrasonic waves or sound waves can be projected toward the operator side, and The wave receiving means is arranged to detect a reflected ultrasonic wave or reflected sound wave emitted from the sound wave transmitting means and reflected by the finger and to input to the moving object detecting means. The reflected ultrasonic wave or reflected sound wave is analyzed to convert the position and movement of the finger into a detection signal, and the control means determines the position and movement of the operator's finger on the flat panel display based on the detection signal. A display input device characterized in that a corresponding video is displayed.   The moving object detecting means extracts a frequency shift component due to the Doppler effect from the reflected ultrasonic wave or the spectrum of the reflected sound wave inputted from the sound wave receiving means, and calculates the position, direction and speed of the finger of the operator. The display input device according to claim 1, wherein the display input device is analyzed.

Images