JP2013073412A - User interface device for generating projection image signal for pointer projection and image projection method and program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a user interface device for enabling a user to appropriately execute an operation with a finger to a touch panel while visually checking a projected projection image.SOLUTION: The user interface device includes: contact determination means for determining whether or not a touch panel has been touched with a finger; pressing force determination means for determining whether or not the pressing force with the finger is equal to or more than a predetermined threshold; operation/image control means for, when a determination of truth is performed by the contact determination means, and a determination of truth is performed by the pressing force determination means, starting an operation corresponding to a push-in operation with the finger, and for controlling an image displayed on a screen in accordance with the operation; pointer information generation means for generating pointer information for projecting a pointer pointing a position in the projection image, that is, a position which is equivalent to the contact position of the finger; and projection image signal generation means for generating a projection image signal for projecting the pointer information and the image controlled by the operation/image control means to a projection object.

Description

  The present invention relates to a user interface device that includes a display for displaying an image and a touch panel as a user interface, and generates a projection image signal for projecting the image onto a projection target.

  2. Description of the Related Art Conventionally, user interface devices equipped with touch panels that perform an input operation by touching a screen with a finger or a contact instrument such as a stylus have been widely used. In particular, in user interface devices such as smartphones, tablet computers, electronic books, and PDAs (personal digital assistants), so-called portable information devices, touch panels are actively employed as user interfaces that accept operations with fingers and the like.

  In recent years, a portable information device having a projector function for projecting an image including a moving image onto a projection target such as a screen has been developed. By using such a device, for example, the content in the device can be disclosed as a projected image to a large number of people on the spot where the device is carried.

  As a portable information device having such a projector function, for example, Patent Document 1 discloses a projector device that suppresses color variation of output light in an LED light source in a projection optical system that projects an image on a screen. ing. Patent Document 2 discloses a portable projector device that suppresses camera shake using a counterweight and an attenuator.

  Further, Patent Document 3 discloses a portable device with a built-in projector that controls the light emission amount of a projection light source based on a signal from a distance measuring unit that measures a distance to a projection target. Here, it is intended to project an image corresponding to the use condition to the viewer by this light emission amount control. Patent Document 4 discloses a portable communication terminal that controls the light emission amount of a projection light source according to a battery voltage value in order to reduce power consumption during projector operation.

JP 2010-191402 A JP 2010-169966 A JP2011-7948A JP 2007-300321 A

  In a portable information device having this projector function, when projecting an image while operating the touch panel with a finger, inconvenience is likely to occur for the user.

  In general, in a situation where a projector function is used, a user often wants to perform an operation with a finger while viewing a projected image that is projected. For example, there is a case where the projected image is described while confirming the actual state in which the projected image is projected. In some cases, a finger may be operated simultaneously while selecting or moving the projection destination of the projected image. Further, in the presentation using the projector function, the explanation is usually made while facing the projected image and the participants.

  On the other hand, for example, a case where an icon is included in an image displayed on a display screen and a user's finger touches the icon is considered. A projected image including this icon (image) is projected on a projection target such as a screen. However, even if the user's finger touches the vicinity of the icon display position on the screen, no image relating to the finger contact appears in the projected image.

  Therefore, the user must check the contact position of the finger by looking at the screen of the display when operating with the finger. Otherwise, it is highly likely that an unnecessary application is activated by selecting an icon that is not intended by the finger, or that it takes time to activate a desired function by moving the finger blindly.

  Thus, when using the projector function, it is difficult to operate the touch panel with a finger while viewing the projected image. However, in the field of conventional portable projectors as disclosed in Patent Documents 1 to 4, it cannot be denied that such problems relating to user interface and operability have not been sufficiently addressed.

  Therefore, an object of the present invention is to provide a user interface device, an image projection method, and a program capable of appropriately executing a finger operation on the touch panel while viewing and confirming the projected image projected. .

According to the present invention, a display that displays an image on a screen, a touch panel that is arranged on the screen of the display and sequentially outputs a contact position of a finger as time elapses, and a touch given by a finger that touches the touch panel. A user interface device that generates a signal for projecting a projection image corresponding to an image onto a projection target, and a pressure detection unit that detects pressure.
Contact determination means for determining whether or not a finger touches the touch panel;
A pressing force determining means for determining whether the pressing force is equal to or greater than a predetermined threshold;
When the contact determination means makes a true determination, an action corresponding to the operation with the touched finger is activated, and the image displayed on the screen is controlled according to this action, and the pressing force determination means makes a true determination. An action / image control means for activating an action corresponding to the pushing operation with a finger when controlling the image displayed on the screen according to the action;
When the contact determination means makes a true determination, it generates pointer information for displaying a pointer indicating a position corresponding to the contact position of the finger in the projected image projected on the projection target. Pointer information generating means;
A user interface device is provided that includes pointer information and a projection image signal generation unit that generates a projection image signal for projecting an image controlled by the operation / image control unit onto a projection target.

According to the user interface device of the present invention, the contact determination unit determines whether or not the contact position of the finger is superimposed on the display position range of the object that is the operation target included in the image displayed on the display,
The action / image control means responds that an operation for activating the function of the object is performed when the contact determination means makes a true determination regarding the display position range of the object and the pressing force determination means makes a true determination. It is also preferable to activate an operation and control an image displayed on the screen in accordance with the operation.

  In this embodiment, when the contact determination unit makes a true determination regarding the display position range of the object, the operation / image control unit activates a corresponding operation on the assumption that an operation for selecting the object is performed. It is also preferable to control the image displayed on the screen according to the operation.

  The user interface device according to the present invention preferably further includes a projection image signal output terminal for outputting the projection image signal to an external projector.

Further, according to the user interface device of the present invention, the projector optical system that projects the pointer information and the image controlled by the operation / image control means onto the projection target;
It is preferable to further include a projector control means for inputting a projection image signal and controlling the projector optical system in accordance with the projection image signal.

  Furthermore, according to the user interface device of the present invention, the projection with the finger can be performed while seeing and confirming the position indicated by the pointer projected on the projection target as seen from the user's operation. The image signal generation means preferably generates a projection image signal including pointer information corresponding to the pointer.

According to the present invention, the display is further provided by a display that displays an image on the screen, a touch panel that is arranged on the screen of the display and that sequentially outputs the contact position of the finger as time elapses, and a finger that touches the touch panel. A program mounted on a user interface device that generates a signal for projecting a projection image corresponding to an image onto a projection target,
Contact determination means for determining whether or not a finger touches the touch panel;
A pressing force determining means for determining whether the pressing force is equal to or greater than a predetermined threshold;
When the contact determination means makes a true determination, an action corresponding to the operation with the touched finger is activated, and the image displayed on the screen is controlled according to this action, and the pressing force determination means makes a true determination. An action / image control means for activating an action corresponding to the pushing operation with a finger when controlling the image displayed on the screen according to the action;
When the contact determination means makes a true determination, it generates pointer information for displaying a pointer indicating a position corresponding to the contact position of the finger in the projected image projected on the projection target. Pointer information generating means;
There is provided a program for a user interface device that causes a computer to function as projection image signal generation means for generating a projection image signal for projecting pointer information and an image controlled by an operation / image control means onto a projection target.

Further, according to the present invention, a display that displays an image on a screen, a touch panel that is arranged on the screen of the display and sequentially outputs a contact position of a finger as time elapses, and a finger that touches the touch panel An image projecting method in a user interface device that generates a signal for projecting a projection image corresponding to an image onto a projection target, the image sensing method comprising:
A first step of determining whether a finger has touched the touch panel;
When a true determination is made in the first step, a second step of activating an action corresponding to an operation with a touched finger and controlling an image displayed on the screen in accordance with the action;
A third step of determining whether the pressing force is equal to or greater than a predetermined threshold;
When the true determination is made in the first step and the true determination is made in the third step, an action corresponding to the pushing operation by the finger is activated, and the image displayed on the screen is controlled according to this action. A fourth step;
When a true determination is made in the first step, pointer information is generated for projecting a pointer indicating a position in the projected image projected on the projection target and corresponding to the finger contact position. A fifth step to:
An image projection method is provided that includes pointer information and a sixth step of generating a projection image signal for projecting the image controlled in the second step and the fourth step onto a projection target.

  According to the user interface device, the image projecting method, and the program of the present invention, it is possible to appropriately execute an operation with a finger on the touch panel while checking and confirming the projected image.

Schematic diagram showing a portable information device having a projector function according to the present invention and a projection target, a front view of the portable information device showing an image displayed on a display screen, and a projection projected on the projection target It is a front view of the screen which shows an image. It is the schematic which shows operation with a front view of a portable information device, a front view of a screen, and a finger for explaining a situation of a "focus" operation and a projection image corresponding to it. FIG. 4 is a front view of a portable information device and a front view of a screen for explaining a “click” operation and a state of a projection image corresponding to the “click” operation. It is the schematic which shows other embodiment in the portable information device by this invention. The perspective view and functional block diagram which show schematically the structure of one Embodiment (FIG. 1 (A)) of the portable information device by this invention, and other embodiment (FIG. 4 (B) of the portable information device by this invention It is a functional block diagram which shows a part of structure of)) schematically. 5 is a flowchart illustrating an embodiment of an image projection method in the user interface device according to the present invention.

  Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings.

According to the present invention, a user interface device including a display and a touch panel is provided.
(1) An apparatus capable of generating a projection image signal for projecting an image displayed on a display screen. Also,
(2) It is also preferable to have a projector function for projecting an image displayed on the display screen onto a projection target such as a screen. In this user interface device according to the present invention, a projection for projecting a pointer on a projection target, which is a position in the projected image projected on the projection target and corresponding to a position touched by a finger in the display screen. It is characterized in that an image signal is generated.

  Thereby, the user can see the pointer displayed on the projected image displayed and can confirm the position pointed to. As a result, it is possible to appropriately execute an operation with a finger on the touch panel while checking the projected image.

  Here, the “image” is an image including a still image and a moving image displayed on the display screen of the user interface device. The “projection image signal” includes “image” information, and is a signal output to the projector unit or the projector device in order to project the “image” onto a projection target such as a screen. Further, the “projection image” is an image formed on a projection target such as a screen by projection light emitted from the projector unit or the projector apparatus to which the “projection image signal” is input.

  In the user interface device according to the present invention, information is input when a user's finger contacts the touch panel. For this reason, many of the devices are portable information devices such as smartphones and tablet computers that can be carried with one hand or both hands and operated with fingers. Therefore, hereinafter, a portable information device will be described as an embodiment of the present invention.

  FIG. 1 is a schematic diagram showing a portable information device having a projector function according to the present invention and a projection target, a front view of the portable information device showing an image displayed on a display screen, and a projection target. It is a front view of the screen which shows the projected image taken out.

  FIG. 1A shows a state in which projection light is emitted from the portable information device 1 having a projector function, and a projection image 201 is projected on a screen 200 that is a projection target. According to this figure, the portable information device 1 includes a display 101 that displays an image on a screen, and a touch panel 100 arranged on the screen of the display 101. The user performs an input operation with a finger on the touch panel 100. An operation is activated in response to the input operation, and an image corresponding to the activated operation is displayed on the screen of the display 101.

  In the following, a mode in which an input operation is performed by directly contacting a finger will be described, but it is also possible to input using a contact instrument such as a stylus instead of the finger. In this case, in the embodiment described below, the contact position of the finger is replaced with the contact position of the tip of the contact instrument.

  The portable information device 1 further includes a projector optical system 13 having a projector function. Information on the image displayed on the screen of the display 101 is simultaneously converted into a projection image signal. The projection image signal is output to a projector control unit 130 (FIG. 5A) that controls the projector optical system 13. The projector optical system 13 is controlled based on the projection image signal, and projection light including image information displayed on the screen of the display 101 is emitted from the projection lens 133 serving as an output unit of the projector optical system 13. As a result, a projected image 201 corresponding to this image is displayed on the screen 200.

  FIG. 1B shows the correspondence between the image displayed on the screen of the display 101 and the projected image 201 displayed on the screen. According to this figure, a plurality of icons 104 are displayed on the image displayed on the screen of the display 101. The icon 104 is an operation target (object) for the user for starting the application. The user can select the icon 104 by directly touching the icon 104 with a finger.

In FIG. 1B, the user's finger is in contact with the touch panel 100 at the contact position 105. Here, the first quadrant of the xy coordinate system is introduced into the screen of the display 101, and the coordinates of the contact position 105 of the finger are (x ₁ , y ₁ ). Further, the lower left corner of the screen in FIG. 1B is set as the origin (0, 0). Based on the contact position 105 of the finger, pointer information 106 is generated by a pointer information generation unit 123 (FIG. 5A) described later. The pointer information 106 is image information (image data) for projecting the pointer 205 in the projection image 201.

In the pointer information 106, the coordinates of the tip 106a are set to coincide with the coordinates (x ₁ , y ₁ ) of the finger contact position 105. That is, if the pointer information 106 is displayed on the screen of the display 101, the image of the pointer information 106 indicates the contact position 105 of the finger. That is, the tip 106a coincides with the contact position of the finger. Note that the pointer information 106 is not displayed on the screen of the display 101 in this embodiment. However, the pointer information 106 may be displayed on the screen of the display 101 in a form corresponding to the pointer 205 displayed on the screen.

  Next, the formed pointer information 106 and the information on the image displayed on the screen of the display 101 are combined to generate a projection image signal for projection onto the screen 200. Projection light corresponding to the projection image signal is emitted, and as a result, a projection image 201 corresponding to the pointer information 106 and image information displayed on the screen of the display 101 is displayed on the screen 200.

  Similarly in FIG. 1B, a projected image 201 projected on the screen 200 includes a pointer 205 corresponding to the pointer information 106 and a plurality of icon images 204 corresponding to the plurality of icons 104. A pointer 205 points to a position corresponding to the finger contact position 105 in the screen of the display 101 in the projection image 201. That is, the position of the tip 205a of the pointer 205 matches the position corresponding to the finger contact position 105 in the projection image 201.

Here, it is assumed that the projection surface of the screen 200 is flat and perpendicular to the optical axis of the projection light emitted from the projection lens 133. Accordingly, it is assumed that the projected image 201 has a shape similar to that of the screen of the display 101 (the entire image displayed on the screen) or a rectangle having an aspect ratio different from that of the screen. Further, the first quadrant of the XY coordinate system is introduced into the projection surface of the screen 200, and the coordinates of the position of the tip 205a of the pointer 205 are (X ₁ , Y ₁ ). Further, the lower left corner of the projected image 201 in FIG. 1B is defined as the origin (0, 0).

In this case, the following relationship exists between the coordinates (x ₁ , y ₁ ) of the finger contact position 105 and the coordinates (X ₁ , Y ₁ ) of the position of the tip 205a.
(A) X ₁ = α · x ₁ , Y ₁ = β · y ₁
here,
(B) α = (width of the projected image 201 in the X-axis direction) / (width of the screen in the x-axis direction)
(C) β = (height of the projected image 201 in the Y-axis direction) / (height of the screen in the y-axis direction)
It is. For example, when the projected image 201 and the screen of the display 101 (the entire image displayed on the screen) are similar, α = β (= enlargement ratio).

  As described above, the pointer 205 indicates the position corresponding to the finger contact position 105 in the projection image 201 by the tip 205a. Thus, the user can confirm the finger contact position 105 in the screen by looking at the position of the tip 205a in the projection image 201 without looking at the screen of the display 101. Accordingly, it is possible to appropriately perform an operation with a finger on the touch panel 100 while checking and confirming the tip 205a of the pointer 205 in the projected image 201 projected.

  In other words, the projection image signal output to the projector optical system 13 can be operated by a finger while confirming the position indicated by the pointer 205 displayed on the screen 200 as viewed from the user's operation. Thus, the signal includes pointer information 106 corresponding to the pointer 205.

  The pointer 205 is not necessarily limited to an arrow-shaped image. For example, if the image is recognized as pointing to one point, such as an isosceles triangle with a clear vertex, an image of one hand with only the index finger extended, or a minute area image such as a dot or a star, various forms are possible. Can be adopted as the pointer 205. The pointer 205 is also preferably translucent. As a result, it is possible to avoid a situation in which other images (icon image 204 or the like) displayed in the projected image 201 are blocked by the pointer 205 and are difficult to confirm.

  Even when the projection surface of the screen 200 is perpendicular to the optical axis of the projection light emitted from the projection lens 133, the projection image 201 is not necessarily limited to a rectangle. For example, when a circular image is displayed on the screen of the display 101 and a plurality of icons are displayed in the image, the projection image 201 includes a circular image including a plurality of icon images according to the image. You can also Alternatively, it may be a rectangular image corresponding to the screen of the display 101 including such a circular image.

  Further, the user can hold the portable information device 1 with one hand (the right hand in FIG. 1B). In this case, the user directs the projection lens 133 of the portable information device 1 held with one hand toward the screen 200 and performs an appropriate operation with the finger (thumb) of the held one hand while viewing the projected image 201 projected. It is also possible to do this.

Hereinafter, a mode of operation with a finger that can be performed on the touch panel 100 will be described, and a state of the projection image 201 corresponding to the operation will be described. As an example of this finger operation,
(1) “Focus” operation: Select an object such as an icon,
(2) “Click” operation: Activate the function of the object.
Is described below.

  Needless to say, conventional operations other than those described above, for example, a drag operation, a flick operation, a pinch-in operation, a pinch-out operation, and the like can also be set so as to be executed via the touch panel 100. When there are two or more finger contact points at the same time, such as a pinch-in operation and a pinch-out operation, a plurality of pointers respectively pointing to a plurality of positions corresponding to the respective finger contact positions are projected images. It is also preferable that the image is displayed in 201.

  FIG. 2 is a schematic diagram illustrating a front view of the portable information device 1, a front view of the screen 200, and an operation with a finger for explaining the “focus” operation and the state of the projection image 201 corresponding thereto.

  2A and 2B show a “focus” operation. According to FIG. 2A, the user's finger is touching the touch panel 100 and moving (sliding) so as to trace on the touch panel 100 while maintaining the contact. Here, the entire image including the icon 104 does not move despite the movement of the finger and is in a fixed state.

  The user moves while keeping the finger in contact, and sequentially touches the icons 104 whose positions are fixed. In response to this, the pointer 205 in the projected image 201 projected on the screen 200 is also moved, and sequentially points to the icon images 204 whose positions are also fixed. Further, a contact determination unit 121 (FIG. 5A), which will be described later, determines whether or not the finger contact position 105 and the display position range of the icon 104 are overlapped even while the finger is moving. That is, the finger contact position 105 is monitored in a state where the entire image displayed on the screen of the display 101 is fixed.

  The user advances the finger to the display position range of the desired icon 104 while checking whether or not the desired icon 104 exists in the image while tracing the image (touch panel 100) displayed on the screen with the finger. Here, the user does not need to look at the display 101 in order to move (slide) the finger, and only needs to look at the pointer 205 in the projection image 201 in order to confirm the contact position. Therefore, such an operation with a finger is very suitable when the projector function is activated in the portable information device 1.

  Thereafter, the user searches for the desired icon 104 and advances his / her finger within the display position range of the corresponding icon 104. The operation of superimposing the finger contact position on the display position range of the icon 104 is hereinafter referred to as “focus” operation. The “focus” operation is an operation in which the user selects the icon 104, and plays a role as a preparatory operation for surely performing an operation that activates the function of the icon 104 thereafter.

  In response to the “focus” operation by the finger, the pointer 205 in the projection image 201 points to the icon image 204 corresponding to the desired icon 104 (the tip 205a of the pointer 205 is superimposed on the display position range of the icon image 204). To do). Therefore, the user can superimpose the finger contact position 105 on the screen on the display position range of the icon 104 by checking the position of the pointer 205 (the tip 205a) with respect to the icon image 204 for confirmation. That is, the “focus” operation can be appropriately executed without looking at the screen of the display 101.

  Here, it is also preferable that the icon 104 selected by the “focus” operation is highlighted so that the display position range shines more strongly or appears to float above the surroundings. In this case, the corresponding icon image 204 displayed on the projection image 201 is also highlighted. Accordingly, the user can confirm that the “focus” operation has been appropriately executed without looking at the screen of the display 101.

  Further, when the finger contact position 105 is superimposed on the display position range of the icon 104, the tactile response mechanism unit 102 (FIG. 5A) described later makes the finger feel the presence of a physical push button. Can give vibration. Accordingly, the user can confirm that the “focus” operation with the finger has been accepted even while viewing the projection image 201, and can obtain a stable and reliable operational feeling.

  FIG. 3 is a front view of the portable information device 1 and a front view of the screen 200 for explaining the “click” operation and the state of the projection image 201 corresponding thereto.

  According to FIG. 3, the finger advances to the position of the desired icon 104, and as a result, the contact position 105 of the finger is superimposed on the display position range of the icon 104. That is, a “focus” operation is performed. Correspondingly, the position of the tip 205 a of the pointer 205 in the projection image 201 is also superimposed on the display position range of the icon image 204. When the user presses the finger into the touch panel 100 from this state, the function of the corresponding icon 104 is activated. The operation of pushing the icon 104 (touch panel 100) with a finger to activate the function of the icon 104 in this way is hereinafter referred to as a “click” operation.

  When the function of the icon 104, for example, an e-mail application is activated, the image displayed on the screen of the display 101 becomes, for example, the image of the outgoing mail creation screen. Correspondingly, the projected image 201 displayed on the screen 200 also changes from the state in which the icon image 104 is displayed to the image on the outgoing mail creation screen. In other words, the projected image 201 reflects the image displayed on the screen of the display 101 at all times including before and after the function of the icon 104 is activated.

Here, the condition determined as the “click” operation will be described. First, the finger contact position 105 overlaps the display position range of the desired icon 104. That is, a “focus” operation is performed. Next, from this state, when the user strongly presses the finger and the pressing force p _C by the finger is equal to or higher than a predetermined pressing force threshold value p _TH (p _C ≧ p _TH ), the operation / image control unit 124 (FIG. 5) described later In (A)), it is determined that the “click” operation has been performed, and the function of the icon 104 is activated. As described above, the condition that the “push” operation is determined is that the pressing force p _C is _{equal to} or greater than the predetermined threshold value p _TH . Note that the pressing force threshold value p _TH can be set to a value within a range of 0.5 N (Newton) to 2.0 N, for example.

  The “click” operation described above is an operation performed by changing the strength of pressing with a finger. Therefore, the user can appropriately execute the “click” operation without looking at the display 101 for confirmation. Thus, it is understood that the “click” operation is an operation that is very easy to execute when the projector function is activated in the portable information device 1.

  Further, the tactile response mechanism 102 (FIG. 5A), which will be described later, gives a vibration that gives a sensation as if the physical press button was actually pressed to the finger that performed the “click” operation. You can also. Accordingly, the user can confirm that the “click” operation with the finger has been accepted even while viewing the projection image 201, and can obtain a stable and reliable operational feeling.

  FIG. 4 is a schematic view showing another embodiment of the portable information device according to the present invention.

  In FIG. 4A, the portable information device 3 is a tablet computer. The user can operate the portable information device 3 while holding the portable information device 3 with one hand (the left hand in the figure) while bringing the finger of the other hand (the index finger of the right hand in the figure) into contact with the touch panel 30. Also in such a portable information device 3, pointer information is generated based on the finger contact position 35. In addition, an operation corresponding to the operation with the finger is activated, and an image displayed on the screen of the display 31 is controlled according to the operation. Next, a projection image signal for projecting the pointer information and the controlled image onto the screen 40 is generated.

  The generated projection image signal is output to the projector optical system 32 built in the portable information device 3 via the control unit, and the projection light corresponding to the projection image signal is output from the projection lens 320 of the projector optical system 32. Is emitted. This projection light is projected onto the screen 40, and a projection image 41 corresponding to the controlled image displayed on the screen of the display 31 is formed. The projection image 41 further includes a pointer 45 corresponding to the pointer information and indicating a position on the projection image 41 corresponding to the finger contact position 35.

  The user can confirm the contact position 35 of the finger on the screen by looking at the pointer 45 (the tip thereof) in the projection image 41 without looking at the screen of the display 31. Accordingly, it is possible to appropriately perform the operation with the finger on the touch panel 30 while checking and confirming the pointer 45 (the front end) in the projection image 41.

  In FIG. 4B, the portable information device 5 is a device that can generate a projection image signal for projecting an image displayed on the screen of the display 51. However, a projector optical system that receives this projection image signal and projects this image and its control unit are not provided. The portable information device 5 includes a projection image signal output terminal 52 for outputting the generated projection image signal to an external projector 6.

  The projector 6 is controlled by the projection image signal input terminal 60 connected to the projection image signal output terminal 52, the projector control unit 61 that inputs the projection image signal via the projection image signal input terminal 60, and the projector control unit 61. A projector optical system 62 that projects an image related to the projection image signal.

  The user operates the portable information device 5 by bringing the finger of one hand held (the thumb of the right hand in the figure) into contact with the touch panel 50. Also in such a portable information device 5, pointer information is generated based on the contact position 55 of the finger. In addition, an operation corresponding to the operation by the finger is activated, and an image displayed on the screen of the display 51 is controlled according to the operation. Next, a projection image signal for projecting the pointer information and the controlled image onto the screen 70 is generated.

  The generated projection image signal is output from the portable information device 5 via the projection image signal output terminal 52 and input to the projector 6 via the projection image signal input terminal 60. The projector control unit 61 controls the projector optical system 62 to emit projection light corresponding to the input projection image signal from the projection lens 620. This projection light is projected onto the screen 70, and a projection image 71 corresponding to the controlled image displayed on the screen of the display 51 is formed. The projection image 71 further includes a pointer 75 corresponding to the pointer information and indicating a position on the projection image 71 corresponding to the finger contact position 55.

  The user can confirm the contact position 55 of the finger on the screen by looking at the pointer 75 (the tip thereof) in the projection image 71 without looking at the screen of the display 51. Accordingly, it is possible to appropriately perform the operation with the finger on the touch panel 50 while checking and confirming the pointer 75 (tip) in the projected image 71 projected.

  As described above, by separating the projection image signal forming function (portable information device 5) from the image projection function (projector 6), the portable information device 5 is reduced in size and weight. In addition, for example, when a presentation is made by a plurality of users, each user uses his / her projector 6 installed at a fixed position while holding his / her presentation data in his / her portable information device 5, This can be done on the same screen 70.

  FIG. 5 is a perspective view and a functional configuration diagram schematically showing the configuration of the portable information device 1 according to the present invention, and a part of the configuration of the portable information device 5 (FIG. 4B) according to the present invention. It is a functional block diagram shown in FIG.

  FIG. 5A shows the configuration of the portable information device 1. As shown in the figure, the portable information device 1 includes a touch panel 100, a display 101, a tactile response mechanism unit 102, a pressing force detection unit 103, and a processor memory. The portable information device 1 further includes a projector optical system 13. Here, the processor memory realizes its function by executing a program.

  The display 101 displays an image on the screen. Moreover, the touch panel 100 is arrange | positioned on the screen of the display 100, and outputs the contact position of a user's finger | toe sequentially according to progress of time. As the touch panel 100, a projected capacitive touch panel, a surface capacitive touch panel, a resistive touch panel, an ultrasonic surface acoustic wave touch panel, an infrared scanning touch panel, or the like can be employed.

  The tactile response mechanism unit 102 gives a tactile response to the finger touching the touch panel 100 by vibrating the touch panel 100. For example, at the time of the above-mentioned “focus” operation and / or “click” operation, a vibration of a type and intensity corresponding to the operation content is given to the finger. The tactile response mechanism 102 can be a piezoelectric actuator formed using a piezoelectric material such as PZT (lead zirconate titanate).

The pressing force detection unit 103 detects a pressing force p _{C applied} to the touch panel 100 by a finger. The pressing force detection unit 103 is installed, for example, under the four corners of the touch panel 100, and detects the total pressure exerted on the touch panel 100 that is bent by pressing a finger as the pressing force p _C. The pressing force signal output by the pressing force detection unit 103 is input to a pressing force determination unit 122 described later. The pressing force detection unit 103 can be, for example, a piezoelectric sensor formed using a piezoelectric material such as PZT. Further, instead of providing the tactile response mechanism unit 102 formed of a piezoelectric actuator, or at the same time, the pressing force detection unit 103 can be used as a tactile response mechanism unit.

  The processor memory inputs a finger contact position signal output from the touch panel 100 and a pressing force signal output from the pressing force detection unit 103, and recognizes an operation by a user's finger based on these signals. An action corresponding to this operation is activated. Further, the image displayed on the screen of the disc play 101 is controlled according to this operation.

  Further, the processor memory generates pointer information 106 for displaying the pointer 205 on the screen 200. Further, the processor memory generates an operation corresponding projection image signal for projecting the pointer information and the controlled image, and controls the projector optical system 13 based on this signal. Hereinafter, the functional configuration of the processor memory will be described with reference to FIG.

  According to FIG. 5A, the processor memory includes a contact determination unit 121, a pressing force determination unit 122, a pointer information generation unit 123, an operation / image control unit 124, a projection image signal generation unit 125, The tactile response control unit 126, the projector control unit 130, the screen display control unit 111, and the application processing unit 112 are included.

  The contact determination unit 121 receives a finger contact position signal output from the touch panel 100, determines whether or not the finger has touched the touch panel 100, and generates pointer information based on the determination result and the finger contact position information. Output to the unit 123 and the operation / image control unit 124. It is also preferable that the contact determination unit 121 determines whether or not the finger contact position 105 overlaps the display position range of the icon 104 and outputs the determination result to the operation / image control unit 124.

The pressing force determination unit 122 inputs the pressing force signal from the pressing force detection unit 103, and determines whether or not the pressing force p _C by the finger is _{equal to} or greater than a predetermined pressing force threshold value pTH. Further, the determination result is output to the operation / image control unit 124.

  The pointer information generation unit 123 inputs the determination result about the presence / absence of contact and the finger contact position information output from the contact determination unit 122. Next, pointer information (pointer image data) 106 for projecting a pointer 205 indicating a position corresponding to the contact position 105 of the finger in the projected image 201 projected on the screen 200 is generated. Here, the pointer information 106 is generated so that the position of the tip 205 a of the pointer 205 in the projection image 201 corresponds to the contact position 105 of the finger in the screen of the display 101. The pointer information generation unit 123 outputs the generated pointer information 106 to the projection image signal generation unit 125.

  The operation / image control unit 124 inputs a determination result about the presence / absence of contact and finger contact position information from the contact determination unit 122, and further inputs a determination result from the pressing force determination unit 122. The operation / image control unit 124 recognizes an operation (for example, a “focus” operation or a “click” operation) with the touched finger based on the input information, and activates an operation corresponding to the recognized operation. Further, operation instruction information is output to the screen display control unit 111 so as to control the image displayed on the screen of the display 101 in accordance with the activated operation.

Specifically, when the finger contact position 105 overlaps the display position range of the icon 104, the operation / image control unit 124 preferably activates a corresponding operation as a “focus” operation is performed. Furthermore, when the contact position 105 of the finger overlaps the display position range of the icon 104 and the pressing force p _C is equal to or greater than a predetermined threshold value p _TH, it is possible to activate the corresponding action as a “click” operation has been performed. preferable.

  Further, the operation / image control unit 124 outputs image information (displayed on the screen of the display 101) controlled according to the activated operation to the projection image signal generation unit 125.

  The projection image signal generation unit 125 receives the pointer information 106 from the pointer information generation unit 123 and the image information controlled by the operation / image control unit 124. The projection image signal generation unit 125 generates a projection image signal for projecting the pointer information 106 and the controlled image information onto the screen 200. Further, the generated projection image signal is output to the projector control unit 130.

  The tactile response control unit 126 receives the operation instruction information output from the operation / image control unit 124. Next, based on this information, the tactile response mechanism 102 is controlled so as to give a tactile response to the finger touching the touch panel 100.

  The screen display control unit 111 inputs the application processing information from the application processing unit 112 and causes the display 101 to display an image corresponding to the execution of the application. Further, the screen display control unit 111 receives the operation instruction information output from the operation / image control unit 124 and causes the display 101 to display an image corresponding to the instructed operation.

  The projector control unit 130 controls the projector optical system 13 based on the projection image signal input from the projection image signal generation unit 125.

  The projector optical system 13 includes a light source 131, a MEMS (micro electro mechanical system) mirror optical system 132, and a projection lens 133. The light source 131 is a light source for projection light projected on the screen 200. The light source 131 includes three types of LEDs (light emitting diodes) that emit R (red), G (green), and B (blue) light, respectively, or three types of light that emit R, G, and B light, respectively. An LED in which a diode chip is integrated can be used.

  The MEMS mirror optical system 132 includes, for example, a DMD (Digital Micro mirror Device (registered trademark)) element, and is an optical system that forms projection light from a projection image signal by a DLP (Digital Light Processing (registered trademark) system. The DMD element has a large number of micromirrors that are two-dimensionally arranged and whose mirror surface inclination angles are individually controlled.

  In the DLP method, the projector control unit 130 performs resolution conversion or the like according to the DMD element on the input projection image signal, thereby forming a number of signals for driving the micromirrors. Next, the DMD element is controlled and driven by this driving signal. In addition, the projector control unit 130 controls the light source 131 in synchronization with (in synchronization with) the timing of driving each micromirror, and appropriately emits R, G, or B light from the light source 131.

  Light emitted from the light source 131 is irradiated onto the DMD element via a predetermined optical system, reflected by the element, and projected light corresponding to an image composed of a large number of pixels corresponding to the tilt angles of a large number of micromirrors. Form. This projection light is radiated from the projection lens 133 toward the screen 200 via a predetermined optical system. Thereby, the projection image 201 corresponding to the projection image signal including the pointer information 106 (generated by the projection image signal generation unit 125) is displayed on the screen 200.

  FIG. 5B illustrates a configuration of the portable information device 5 illustrated in FIG. Similarly to the portable information device 1 (FIG. 5A), the portable information device 5 also includes a touch panel, a display, a tactile response mechanism unit, a pressing force detection unit (not shown above), a processor memory, It has. However, the projector optical system and its control unit are not provided. Further, the portable information device 5 includes a projection image signal output terminal 52 for outputting the generated projection image signal to an external projector 6 (FIG. 4B).

  Similarly to the portable information device 1 (FIG. 5A), the processor / memory of the portable information device 5 also includes a contact determination unit, a pressing force determination unit, a pointer information generation unit, and an operation / image control unit. A tactile response control unit, a screen display control unit, an application processing unit (not shown above), and a projection image signal generation unit 525. However, no projector control unit is provided.

  Also in such a portable information device 5, by using the external projector 6, the projection image 71 corresponding to the operation-corresponding projection image signal including the pointer information (generated by the projection image signal generation unit 525) is displayed on the screen. 70.

  FIG. 6 is a flowchart showing an embodiment of an image projecting method in the user interface device according to the present invention.

  According to FIG. 6, first, an image including the icon 104 is displayed on the screen of the display 101 (step S600). Next, presence / absence of finger contact with the touch panel 100 is measured (step S601). Next, the contact determination unit 121 determines whether or not the finger has touched the touch panel 100 (step S602).

  Here, when the true determination, that is, the determination that the finger is in contact is made, the finger touch / push operation loop (steps S610 to S622) is entered, and the operation / image control unit 124 scans the entire image. In a fixed state, the contact position of the finger is monitored (monitored) (step S611). On the other hand, when it is determined that the finger is not in contact, the process returns to step S601 again, and the presence or absence of the finger contact is monitored.

  After step S611, the contact determination unit 121 determines whether or not the finger contact position 105 is superimposed on the display position range of the icon 104 (step S612). When the contact determination unit 121 makes a true determination, that is, determines that the finger contact position 105 is superimposed on the display position range of the icon 104, the operation / image control unit 124 determines that the “focus” operation has been performed. Then, the selection operation of the icon 104 is activated (step S613). Further, the image displayed on the screen of the display 101 is controlled (updated) in accordance with the activated operation (step S614). On the other hand, when the contact determination unit 121 makes a false determination, the process immediately proceeds to step S619, and pointer information is generated according to the contact position 105 of the finger.

After step S614, the pressing force detection unit 103 measures the pressing force p _{C applied} to the touch panel 100 by the finger (step S615). Here, the pressing force determination unit 122 determines whether or not the pressing force p _C by the finger is _{equal to} or greater than a predetermined threshold value p _TH (step S616). When the pressing force determination unit 122 makes a true determination, that is, determines that the pressing force p _C is _{equal to} or greater than the predetermined threshold value p _TH , the operation / image control unit 124 determines that the “click” operation has been performed. Then, an operation corresponding to the function of the icon 104 is activated (step S617). Further, the image displayed on the screen of the display 101 is controlled (updated) in accordance with the activated operation (step S618). On the other hand, when the pressing force determination unit 122 makes a false determination, the process immediately proceeds to step S619, and pointer information is generated according to the finger contact position 105.

  After step S618, the pointer information generation unit 123 generates pointer information according to the finger contact position 105 (step S619). Next, the projection image signal generation unit 125 generates a projection image signal for projecting the generated pointer information and the controlled (updated) image onto the screen 200 (step S620). Next, the projector optical system 13 and the projector control unit 130 project the pointer information and the controlled (updated) image onto the screen 200, and the projected image 201 is displayed (step S621).

  Next, the contact determination unit 121 determines whether or not the finger is in contact with the touch panel 100 (step S622). Here, when it is determined that the finger is in contact, the finger touch / push operation loop composed of steps S610 to S622 described above is repeated. On the other hand, if it is determined that the finger is not touching, the image projection method is terminated as if the operation by the user's finger has been completed.

  As described above in detail, according to the user interface device, the image projection method, and the program of the present invention, the pointer 205 indicating the position corresponding to the finger contact position 105 is displayed on the projected image 201. The Thereby, it is possible to appropriately perform an operation with a finger on the touch panel 100 while checking the projected image 201 projected.

  Various changes, modifications, and omissions of the above-described various embodiments of the present invention can be easily made by those skilled in the art. The above description is merely an example, and is not intended to be restrictive. The invention is limited only as defined in the following claims and the equivalents thereto.

1, 3, 5 Portable information device (user interface device)
DESCRIPTION OF SYMBOLS 100, 30, 50 Touch panel 101, 31, 51 Display 102 Tactile response mechanism part 103 Pressing force detection part 104 Icon 105, 35, 55 Contact position 106 Pointer information 107 Search result 111 Screen display control part 112 Application processing part 121 Contact determination part 122 pressing force determination unit 123 pointer information generation unit 124 operation / image control unit 125, 525 projection image signal generation unit 126 tactile response control unit 13, 32, 62 projector optical system 130, 61 projector control unit 131 light source 132 MEMS mirror optical system 133, 320, 620 Projection lens 200, 40, 70 Screen 201, 41, 71 Projected image 204 Icon image 205, 45, 75 Pointer 207 Search result image 52 Projected image signal output terminal 6 Pro Jector 60 Projected image signal input terminal

Claims (8)

A display that displays an image on the screen, a touch panel that is arranged on the screen of the display and that sequentially outputs the contact position of the finger as time elapses, and a pressing force that detects the pressing force applied by the finger that touched the touch panel A user interface device that generates a signal for projecting a projection image corresponding to the image onto a projection target,
Contact determination means for determining whether or not the finger touches the touch panel;
A pressing force determining means for determining whether or not the pressing force is equal to or greater than a predetermined threshold;
When the contact determination means makes a true determination, an action corresponding to the operation by the touched finger is activated, an image displayed on the screen is controlled according to the action, and the pressing force determination means An action / image control means for activating an action corresponding to the pushing operation by the finger when performing a true determination, and controlling an image displayed on the screen according to the action;
Pointer information for projecting a pointer indicating a position corresponding to the contact position of the finger, which is a position in the projected image projected on the projection target when the contact determination unit makes a true determination Pointer information generating means for generating
A user interface device comprising: a projection image signal generation unit configured to generate a projection image signal for projecting the pointer information and an image controlled by the operation / image control unit onto the projection target. The contact determination unit determines whether or not the contact position of the finger is superimposed on a display position range of an object that is an operation target included in an image displayed on the display;
The movement / image control means performs an operation to activate the function of the object when the contact determination means makes a true determination regarding the display position range of the object and the pressing force determination means makes a true determination. 2. The user interface device according to claim 1, wherein a corresponding operation is activated as a failure, and an image displayed on the screen is controlled in accordance with the operation.   When the contact determination means makes a true determination regarding the display position range of the object, the action / image control means activates a corresponding action on the assumption that an operation for selecting the object has been performed. The user interface device according to claim 2, wherein an image displayed on the screen is controlled.   4. The user interface device according to claim 1, further comprising a projection image signal output terminal for outputting the projection image signal to an external projector. 5. A projector optical system that projects the pointer information and the image controlled by the operation / image control means onto the projection target;
5. The user interface device according to claim 1, further comprising a projector control unit that inputs the projection image signal and controls the projector optical system according to the projection image signal. 6. .   The projection image signal generating means is provided with the pointer so that the operation with the finger can be executed while checking the position indicated by the pointer projected on the projection target as seen from the user's operation. The user interface device according to claim 1, wherein a projection image signal including corresponding pointer information is generated. A display that displays an image on the screen, a touch panel that is arranged on the screen of the display and that sequentially outputs the contact position of the finger as time elapses, and a pressing force that detects the pressing force applied by the finger that touched the touch panel A program installed in a user interface device that generates a signal for projecting a projection image corresponding to the image onto a projection target,
Contact determination means for determining whether or not the finger touches the touch panel;
A pressing force determining means for determining whether or not the pressing force is equal to or greater than a predetermined threshold;
When the contact determination means makes a true determination, an action corresponding to the operation by the touched finger is activated, an image displayed on the screen is controlled according to the action, and the pressing force determination means An action / image control means for activating an action corresponding to the pushing operation by the finger when performing a true determination, and controlling an image displayed on the screen according to the action;
Pointer information for projecting a pointer indicating a position corresponding to the contact position of the finger, which is a position in the projected image projected on the projection target when the contact determination unit makes a true determination Pointer information generating means for generating
A user interface device that causes a computer to function as a projection image signal generation unit that generates a projection image signal for projecting the pointer information and an image controlled by the operation / image control unit onto the projection target. Program. A display that displays an image on the screen, a touch panel that is arranged on the screen of the display and that sequentially outputs the contact position of the finger as time elapses, and a pressing force that detects the pressing force applied by the finger that touched the touch panel An image projection method in a user interface device that generates a signal for projecting a projection image corresponding to the image onto a projection target,
A first step of determining whether the finger touches the touch panel;
When a true determination is made in the first step, a second step of activating an action corresponding to an operation by the touched finger and controlling an image displayed on the screen according to the action;
A third step of determining whether or not the pressing force is equal to or greater than a predetermined threshold;
When the true determination is made in the first step and the true determination is made in the third step, an action corresponding to the pressing operation by the finger is activated, and an image displayed on the screen according to the action is displayed. A fourth step to control;
Pointer information for projecting a pointer indicating a position in the projected image projected on the projection target and corresponding to the contact position of the finger when the true determination is made in the first step A fifth step of generating
An image projection method comprising: a sixth step of generating a projection image signal for projecting the pointer information and the image controlled in the second step and the fourth step onto the projection target. .

Images