JP2016018448A - Input device and its control method and control program, and input system - Google Patents

Abstract

PROBLEM TO BE SOLVED: To generate the same input information by the same operation without depending upon a direction of a portable terminal and a posture of a user.SOLUTION: A portable terminal (20) includes a face tilt detection part (32) which performs face recognition on a face image of a user in a photographed image by a photography part (24) and detects a tilt of a midline of a face to the terminal; a terminal movement detection part (31) which detects a movement of the terminal from a detection result of a motion sensor (23); a rotation conversion part (34) which performs rotation conversion of the detected movement of the terminal to a movement of the terminal about the midline of the face using the detected tilt; and an operation data generation part (35) which generates operation data based upon the rotation-converted movement.SELECTED DRAWING: Figure 1

Description

  The present invention relates to an input device that receives input from a user and transmits input information based on the input to an information processing device, a control method therefor, a control program, and an input system.

  A conventional remote controller (remote controller) for operating an external device such as a television (TV) receiver or a game machine is provided with various buttons such as alphanumeric buttons and arrow buttons on a base. When the user operates the various buttons, display of a display device such as a display connected to the TV receiver or the game machine is controlled.

  In recent years, remote controllers equipped with motion sensors such as acceleration sensors (hereinafter referred to as “motion type remote controllers”) are commercially available. For example, in the game system described in Patent Document 1, an acceleration sensor is provided in the controller, the inclination of the controller is detected by the acceleration sensor, and an object on the screen of the display device is rotated by the detected inclination. . In this way, by using the motion type remote controller, an intuitive operation is possible, and display control of the display device is facilitated.

JP 2007-075213 (March 29, 2007)

  If the user changes the grip type of the motion type remote controller, the direction of the remote controller may change, and the detection value of the motion sensor may change. In this case, even if the user operates the remote controller in the same manner, the display control (operation result) on the display device will be different if the gripping form of the remote controller is different. Therefore, conventionally, the grip form of the remote controller has been defined in advance.

  Further, the user recognizes the space with both eyes. Accordingly, examples of the reference line of the user's face include a median line of the face passing between the eyes, a straight line indicating the arrangement direction of the eyes (hereinafter referred to as “eye alignment”), and eye alignment. A straight line indicating the arrangement direction of both ears, which is a parallel straight line (hereinafter referred to as “ear alignment”). Here, the midline of the face refers to a vertical line that equally divides the face left and right.

  On the other hand, the display device is usually arranged so that one of two adjacent sides on the screen is along a vertical line and the other is along a horizontal line. Accordingly, examples of the reference line of the display device include a vertical line and a horizontal line. The operation on the reference line of the user is the operation result on the reference line of the screen.

  At this time, if the user changes his / her posture, the user's reference line (for example, the midline of the face) may change and tilt from the reference line (for example, the vertical line) on the screen. In this case, the operation result of the screen expected by the user is inclined from the actual operation result of the screen. Therefore, conventionally, the posture of the user has been defined in advance.

  The present invention has been made in view of the above problems, and an object thereof is to provide an input device that can create the same input information by the same operation without depending on the orientation of the input device and the posture of the user. There is.

  An input apparatus according to an aspect of the present invention is an input apparatus that receives input from a user and transmits input information based on the input to an information processing apparatus, and performs imaging to solve the above-described problem. A face inclination detecting means for performing face recognition on the face image of the user included in the photographed image obtained by the photographing and detecting a face inclination that is an inclination of the reference line of the face with respect to the own device, An operation movement detecting means for detecting an operation movement that is an operation movement of the apparatus, and an operation movement detected by the operation movement detecting means with respect to the reference line of the face by using the face inclination detected by the face inclination detecting means. It is characterized by comprising rotation conversion means for rotationally converting to an operation motion, and input information creating means for creating the input information based on the operation motion rotationally converted by the rotation conversion means.

  A display system according to another aspect of the present invention is an input system including an input device that receives an input from a user, and an information processing device that receives input information based on the input from the input device. In order to solve the problem, the inclination of the input device relative to the vertical line from the detection result of the acceleration sensor and the face inclination detection means for detecting the face inclination which is the inclination of the reference line of the user's face with respect to the vertical line An apparatus inclination detecting means for detecting an apparatus inclination, an operation movement detecting means for detecting an operation movement of the input device by a user, and an operation movement detected by the operation movement detecting means, the face inclination detecting means and the apparatus inclination detecting Rotation conversion means for rotating and converting the operation motion relative to the reference line of the face using the face inclination and the device inclination detected by the means, respectively, Management device, an operation movement which the rotation conversion unit has rotational transformation as the operation movement relative to the reference line of the apparatus is characterized by performing processing based on the manipulation motion.

  An input device control method according to another aspect of the present invention is a control method for an input device that receives input from a user and transmits input information based on the input to an information processing device. In order to do so, a face inclination detection step of performing face recognition on the face image of the user included in the photographed image by the photographing unit and detecting a face inclination which is an inclination of the reference line of the face with respect to the own device; An operation movement detection step for detecting an operation movement that is an operation movement of the device itself, and an operation movement detected in the operation movement detection step, using the face inclination detected in the face inclination detection step, A rotation conversion step for rotationally converting the operation motion relative to the reference line of the face; an input information creation step for creating the input information based on the operation motion rotationally converted in the rotation conversion step; It is characterized in that it comprises.

  According to one aspect of the present invention, there is an effect that the same input information can be created by the same operation without depending on the orientation of the input device and the posture of the user.

It is a block diagram which shows schematic structure of the input system which concerns on one Embodiment of this invention. It is a figure which shows the outline | summary of the said input system. It is a figure which shows the display apparatus, portable terminal, and user's local coordinate system in the said input system. It is a flowchart which shows the flow of the process in the said portable terminal. It is a perspective view which shows the outline | summary of the portable terminal in the input system which concerns on another embodiment of this invention. It is a block diagram which shows schematic structure of the input system which concerns on another embodiment of this invention. It is a block diagram which shows schematic structure of the input system which concerns on other embodiment of this invention.

  Hereinafter, embodiments of the present invention will be described in detail. For convenience of explanation, members having the same functions as those shown in the embodiments are denoted by the same reference numerals, and description thereof is omitted as appropriate.

Embodiment 1
First, an embodiment of the present invention will be described with reference to FIGS.

(Overview of input system)
FIG. 2 is a diagram showing an overview of the input system according to the present embodiment. As shown in the figure, the input system 1 of the present embodiment is configured to include a display device (information processing device) 10 and a portable terminal (input device) 20. In the following, when each mobile terminal is shown, it is described as a mobile terminal 20a, a mobile terminal 20b, etc., and when the mobile terminals are collectively referred to as a mobile terminal 20. Moreover, when showing an individual user, it describes as user Ua, user Ub, etc., and when referring to a user generically, it describes as user U. The same applies to other cases.

  The display device 10 receives content from an external device such as a broadcasting station, the Internet, or a recording device, and displays the received content on the screen 11. An example of the display device 10 is a TV receiver.

  The portable terminal 20 is a communication terminal that can be carried by the user U. Examples of the mobile terminal 20 include a mobile phone, a smartphone, a tablet terminal, a PDA (Personal Digital Assistant), a notebook PC (Personal Computer), and a PHS (Personal Handyphone System) terminal. In the present embodiment, the mobile terminal 20 functions as a remote control for remotely operating the display device 10 based on the operation of the user U.

  FIG. 3 shows the local coordinate system of the display device 10 (screen 11), the mobile terminal 20, and the user U shown in FIG. As shown in FIG. 3A, in the local coordinate system of the display device 10, the horizontal direction is taken as the xd axis, the vertical direction (opposite to the vertical direction) is taken as the yd axis, and the front-rear direction is taken as the zd axis. Yes. In the local coordinate system of the portable terminal 20, as shown in (b) of the figure, the short side direction is taken as the xt axis, the long side direction is taken as the yt axis, and the longitudinal direction is taken as the zt axis. In the local coordinate system of the user U, the eye alignment direction is taken as the xf axis, the direction of the midline ML as the yf axis, and the front-rear direction as the zf axis, as shown in FIG. Yes. In this embodiment, for convenience of explanation, it is assumed that the zd axis, the zt axis, and the zf axis have the same direction.

  In the present embodiment, the mobile terminal 20 includes an acceleration sensor 30 (see FIG. 1) that detects acceleration in the triaxial directions (xt, yt, zt) of the mobile terminal 20. The mobile terminal 20 uses the acceleration sensor 30 to detect the direction of gravity acting on the mobile terminal 20 (vertical direction) and the amount of movement (movement, movement vector) of the mobile terminal 20 in three axes. Then, the portable terminal 20 uses the detected vertical direction to set the inclination (hereinafter referred to as “terminal inclination”) θ1 of the longitudinal direction (yt axis) of the portable terminal 20 with respect to the vertical direction (yd axis) of the display device 10. Detect.

  Further, as shown in FIG. 3, the mobile terminal 20 is provided with a photographing unit 24 (see FIGS. 1 and 3) that performs photographing and creates a photographed image on the screen side of the display unit 26. Yes. In addition, the mobile terminal 20 performs face recognition on the created captured image, and detects the median line ML of the face of the user U included in the captured image. In addition, since a well-known thing can be utilized for the process of the said face recognition, the description is abbreviate | omitted.

  In addition, the mobile terminal 20 uses the detected midline ML to tilt the user U's midline ML direction (yf axis) with respect to the longitudinal direction (yt axis) of the mobile terminal 20 (hereinafter referred to as “face tilt”). .) Detect θ2. The mobile terminal 20 detects the inclination θ2 ′ of the direction (yf axis) of the midline ML of the user U with respect to the vertical direction (yd axis) of the display device 10 using the detected midline ML and the vertical direction. May be.

  Here, a method of calculating the inclination θ of another straight line with respect to a certain straight line will be described. When the unit vector indicating the slope of the certain straight line is vector a = (a1, a2) and the unit vector indicating the slope of the other straight line is vector b = (b1, b2), the slope θ (0 ≦ θ ≦ 180 °) can be obtained by the following equation (1).

次に、携帯端末２０は、検知した自装置の移動ベクトルｍを、検知した顔傾きθ2を用いて、正中線ＭＬに対する自装置の移動ベクトルに回転変換する。これにより、回転変換された移動ベクトルは、携帯端末２０の傾きに依存しなくなる。また、当該動きは、上記顔の正中線ＭＬに対する携帯端末２０の動きであるので、ユーザの姿勢に依存しない。

Next, the mobile terminal 20 rotationally converts the detected movement vector m of the own apparatus into a movement vector of the own apparatus with respect to the midline ML using the detected face inclination θ2. Thereby, the rotation-converted movement vector does not depend on the inclination of the mobile terminal 20. Moreover, since the said movement is a movement of the portable terminal 20 with respect to the said face midline ML, it does not depend on a user's attitude | position.

  Then, the mobile terminal 20 sets the rotation vector of the own device with respect to the median line ML, which has been rotationally converted, as a movement vector with respect to the vertical direction (yd axis) of the display device 10, and the object on the screen 11 of the display device 10 according to the movement vector. Instruct A to move. Thereby, the same instruction can be performed with the same movement for the user U. Note that examples of the object A include a cursor and an avatar.

  When the mobile terminal 20 cannot recognize the image of the face of the user U from the created captured image, or when the detected face inclination θ2 = 0, the mobile device 20 detects the detected movement vector m of its own device, Using the detected device inclination θ1, it is rotationally converted into a movement vector with respect to the vertical direction (yd axis) of the display device 10. Then, the portable terminal 20 instructs to move the object A on the screen 11 of the display device 10 according to the movement vector of the own device with respect to the vertical direction (yd axis) of the display device 10 that has been rotationally converted. Thereby, an instruction independent of the inclination of the mobile terminal 20 can be performed.

  An operation example according to the present embodiment will be described. As shown by the movement vector m in FIG. 2, it is assumed that the user U is moving the mobile terminal 20 in a direction parallel to the median line ML.

  The user Ua on the left side is in an upright state, holding the mobile terminal 20a in the vertical direction (yd axis) of the display device 10. In this case, the longitudinal direction (yt axis) of the mobile terminal 20a and the median line (yf axis) of the user Ua coincide with the vertical direction (yd axis) of the display device 10. That is, the terminal inclination θ1 = 0 ° and the face inclination θ2 = 0 °.

  Therefore, since the rotation conversion described above is unnecessary, the mobile terminal 20a instructs to move the object A on the screen 11 of the display device 10 according to the detected movement vector m of the own device. As a result, the object A on the screen 11 of the display device 10 moves in a direction parallel to the vertical direction (yd axis).

  On the other hand, the user Ub at the center is in a state of being inclined sideways and holding the mobile terminal 20b in a state inclined by 45 ° from the vertical direction (yd axis) of the display device 10. Furthermore, since the mobile terminal 20b is separated from the face of the user Ub, it is assumed that the created captured image does not include the face image of the user Ub. In this case, the terminal inclination θ1 = 45 °, and the face inclination θ2 is not detected.

  Accordingly, the portable terminal 20b rotates the detected movement vector m of the own apparatus by rotating the terminal inclination θ1 = 45 ° into a movement vector with respect to the vertical direction (yd axis) of the display device 10, and according to the converted movement vector. An instruction is given to move the object A on the screen 11 of the display device 10. As a result, the object A on the screen 11 of the display device 10 moves in a direction parallel to the vertical direction (yd axis).

  On the other hand, the user Uc on the right side is in a lying state, the face is inclined by 30 ° from the vertical direction (yd axis) of the display device 10, and the mobile terminal 20c is placed in the vertical direction (yd axis) of the display device 10. ) And tilted by 45 °. Assuming that the face plane of the user Uc and the screen plane of the mobile terminal 20c are parallel, the terminal inclination θ1 = 45 ° and the face inclination θ2 = 45 ° -30 ° = 15 °.

  Accordingly, the mobile terminal 20c rotates the detected movement vector m of the own device by a face inclination θ2 = 15 ° and converts it into a movement vector with respect to the midline (yf axis) of the user Uc. The converted movement vector is in a direction parallel to the midline (yf axis) as in the above assumption. Next, the mobile terminal 20 uses the converted movement vector as a movement vector with respect to the vertical direction (yd axis) of the display device 10 and instructs the object A on the screen 11 of the display device 10 to move according to the movement vector. Since the movement vector is a direction parallel to the vertical direction (yd axis) of the display device 10, the object A on the screen 11 of the display device 10 moves in a direction parallel to the vertical direction (yd axis).

(Details of input system)
FIG. 1 is a block diagram showing a schematic configuration of the input system 1. As illustrated, the display device 10 includes a content receiving unit 12, a content analysis unit 13, a communication unit 14, a display control unit 15, and a display unit 16.

  One or more contents from the external device are received by the content receiving unit 12, analyzed by the content analysis unit 13, and controlled by the display control unit 15 so as to be displayed on the display unit 16 based on the analysis result. Is done. In addition, the operation data from the mobile terminal 20 is received by the communication unit 14, and the display control unit is configured to display the content on the display unit 16 after changing the position of the content based on the received operation data. 15 is controlled. Thus, the display device 10 in the present embodiment is the same as a known device.

  On the other hand, the mobile terminal 20 includes a control unit 21, a storage unit 22, a motion sensor 23, a photographing unit 24, an input unit 25, a display unit 26, and a communication unit 27.

  The control unit 21 comprehensively controls various components in the mobile terminal 20 and includes a processor such as a CPU (Central Processing Unit). The storage unit 22 stores various data and programs, and includes a storage element such as a random access memory (RAM), a read only memory (ROM), and a flash memory. Details of the control unit 21 will be described later.

  The motion sensor 23 detects the movement of the mobile terminal 20. The motion sensor 23 transmits detection data to the control unit 21. In the present embodiment, the motion sensor 23 includes the acceleration sensor 30 described above. The motion sensor 23 may include a gyro sensor that detects an angular velocity in the triaxial direction of the mobile terminal 20 or may include a geomagnetic sensor that detects geomagnetism. The gyro sensor can detect the amount of rotation (rotation vector) in the three-axis direction of the mobile terminal 20, and the geomagnetic sensor can detect the orientation of the mobile terminal 20.

  As described above, the photographing unit 24 shoots a subject and creates a photographed image such as a still image and a moving image. For example, an optical system such as a lens and light from the outside via the optical system are used. And an image sensor for converting the signal into an electric signal. Examples of the image pickup device include a charge coupled device (CCD), a complementary metal-oxide semiconductor (CMOS) image sensor, and the like. The imaging unit 24 transmits the created captured image data to the control unit 21. The photographed image may be a color image, a black and white image, or an infrared image. The photographing unit 24 preferably has an autofocus mechanism so that the user's face is automatically focused.

  The input unit 25 creates input data and transmits it to the control unit 21 when a user operates an input device such as an operation button provided on the surface of the mobile terminal 20. Examples of input devices include a touch panel in addition to button switches.

  The display unit 26 receives image data from the control unit 21 and displays an image on the display screen based on the received image data. Specifically, the display unit 26 drives a display element based on a display element such as an LCD (Liquid Crystal Display), a PDP (Plasma Display Panel), an EL (Electroluminescence) display, and electronic paper, and received image data. And a driver circuit.

  The communication unit 27 converts the data received from the control unit 21 into a format suitable for wireless transmission, transmits the converted wireless signal to the outside via an antenna (not shown), and from the outside via the antenna. The received radio signal is converted into the original format, and the converted data is transmitted to the control unit 21. Specifically, the communication unit 27 performs data modulation / demodulation processing, RF processing, and the like.

  In the present embodiment, the communication unit 27 performs short-range wireless communication with the display device 10 by wireless LAN, Bluetooth (registered trademark), or the like, but is not limited thereto. For example, the communication unit 27 may perform wireless communication with the display device 10 via a mobile phone line and the Internet, or may perform wireless communication with the display device 10 using infrared rays. The communication unit 27 may perform wired communication with the display device 10 using a cable or the like.

  Next, details of the control unit 21 and the storage unit 22 in the mobile terminal 20 will be described. As illustrated in FIG. 1, the control unit 21 includes a terminal tilt detection unit (device tilt detection unit) 31, a face tilt detection unit (face tilt detection unit) 32, a motion detection unit (operation motion detection unit) 33, and a rotation conversion unit. (Rotation conversion means) 34 and an operation data creation unit (input information creation means) 35 are provided.

  The terminal tilt detection unit 31 detects the terminal tilt θ1 described above based on the detection data from the acceleration sensor 30. The terminal tilt detection unit 31 transmits the detected terminal tilt θ1 to the rotation conversion unit 34.

  The face inclination detection unit 32 performs face recognition using data of the captured image from the imaging unit 24 and detects the above-described face inclination θ2. The face inclination detection unit 32 transmits the detected face inclination θ2 to the rotation conversion unit 34. At this time, when the face recognition cannot be performed, the face inclination detection unit 32 notifies the rotation conversion unit 34 to that effect.

  Based on the detection data from the motion sensor 23, the motion detection unit 33 detects the movement of the own terminal as the movement of the operation of the own terminal by the user. The motion detection unit 33 transmits a motion vector indicating the detected motion to the rotation conversion unit 34.

  The rotation conversion unit 34 converts the motion vector from the motion detection unit 33 using the terminal inclination θ1 from the terminal inclination detection unit 31 or the face inclination θ2 from the face inclination detection unit 32 based on the above-described conditions. To do. The rotation conversion unit 34 transmits the rotation-converted motion vector to the operation data creation unit 35.

  The operation data creation unit 35 creates operation data including a motion vector from the rotation conversion unit 34. The operation data creation unit 35 transmits the created operation data to the display device 10 via the communication unit 27. Thereby, in the display apparatus 10, a display is controlled based on the said operation data.

(Handling of mobile device)
Next, processing (control method) in the mobile terminal 20 having the above-described configuration will be described. FIG. 4 is a flowchart showing the flow of the processing. As shown in the figure, first, the face inclination detection unit 32 performs face recognition using the data of the captured image from the imaging unit 24 (S10), and if the face of the user U can be recognized (YES in S11). ), The face inclination θ2 is detected from the photographed image (S12, face inclination detection step).

  At this time, if the face of user U cannot be recognized (NO in S11), or if face tilt θ2 = 0 (YES in S13), terminal tilt detection unit 31 starts from acceleration sensor 30. The terminal inclination θ1 is detected based on the detected data (S14). Next, the motion detection unit 33 detects the motion vector m of the terminal itself based on the detection data from the motion sensor 23 (S15, operation motion detection step).

  Next, the rotation conversion unit 34 converts the rotation of the motion vector m detected by the motion detection unit 33 using the terminal inclination θ1 detected by the terminal inclination detection unit 31 (S16, rotation conversion step). Then, the operation data creation unit 35 creates operation data including the motion vector rotationally converted by the rotation conversion unit 34, and transmits the operation data to the display device 10 (S17, input information creation step). Thereafter, the processing operation is terminated.

  On the other hand, when the face inclination θ2 is not 0 (NO in S13), the motion detection unit 33 detects the motion vector m of the terminal itself based on the detection data from the motion sensor 23 (S18, operation motion). Detection step). Next, the rotation conversion unit 34 performs rotation conversion on the motion vector m detected by the motion detection unit 33 using the face inclination θ2 detected by the face inclination detection unit 32 (S19, rotation conversion step), and rotation conversion The unit 34 uses the rotation-converted motion vector as a vertical motion vector, and the operation data creation unit 35 creates operation data including the motion vector and transmits it to the display device 10 (S20, input information creation step). ). Thereafter, the processing operation is terminated.

(Additional notes)
In the present embodiment, data is transmitted from the mobile terminal 20 to the display device 10, but it is not necessary to transmit data from the display device 10 to the mobile terminal 20. Accordingly, the communication unit 27 of the mobile terminal 20 may be only a transmission unit that transmits data, and the communication unit 14 of the display device 10 may be only a reception unit that receives data.

  Further, the terminal inclination θ1 detected by the terminal inclination detection unit 31 and the face inclination θ2 detected by the face inclination detection unit 32 are stored in the storage unit 22, and the stored terminal inclination θ1 and face inclination θ2 are rotationally converted. The unit 34 may use it.

  Further, the photographing unit 24 may be provided on the opposite side of the screen of the display unit 26. In this case, the user may hold the mobile terminal 20 so that the side on which the photographing unit 24 is provided, that is, the opposite side of the screen of the display unit 26 faces the user. At this time, the operation of the touch panel on the screen of the display unit 26 may be performed with a finger other than the thumb.

  In the present embodiment, the midline ML (yf axis in FIG. 3C) of the user's face is used as the reference line of the face, and the display device 10 is in the vertical direction (yd axis in FIG. 3A). ), That is, a vertical line is used as the reference line of the display device 10, but is not limited thereto. For example, as the face reference line, eye alignment (the xf axis in (c) of the figure) may be used, or ear alignment substantially parallel to the eye alignment may be used. In this case, the reference line of the display device 10 is a straight line indicated by the horizontal direction of the display device 10 (xd axis in FIG. 3A), that is, a horizontal line.

  However, although the direction of the vertical line is the direction of gravity and is uniquely determined, the direction of the horizontal line is a direction perpendicular to gravity and is not uniquely determined. Therefore, the face midline ML is more preferable as the face reference line than the eye alignment and the ear alignment. In this case, the same instruction can be reliably performed with the same movement for the user.

(Modification 1)
In addition, the user U may perform shooting while changing the orientation of the face in various ways, and the feature data of the face image included in the created captured image may be stored in advance in association with the face inclination θ2. Good. By using this feature data, the face inclination θ2 can be quickly derived.

(Modification 2)
Further, when the mobile terminal 20 is brought closer to the face of the user U, the face image in the captured image becomes larger, while when the mobile terminal 20 is moved away from the user U's face, the face image in the captured image becomes smaller. Therefore, an operation may be performed so that the object A on the screen 11 is enlarged and reduced according to the size of the face image. Accordingly, the size of the object A on the screen 11 can be manipulated by moving the mobile terminal 20 closer to or away from the user U.

(Modification 3)
For the user U, even if the mobile terminal 20 is moved in the same manner, if the distance between the mobile terminal 20 and the face of the user U is short, the mobile terminal 20 appears to move greatly, while the distance is long The mobile terminal 20 appears to move small. Accordingly, the magnification of the length of the motion vector transmitted from the motion detection unit 33 to the rotation conversion unit 34 may be changed in proportion to the size of the face image. That is, when the distance between the mobile terminal 20 and the face of the user U is close, the face image is large, so the motion vector is lengthened. On the other hand, when the distance is far, the face image is small, so the motion vector is shortened. To do.

  Accordingly, even if the mobile terminal 20 is moved in the same manner, the object A moves greatly when the distance between the mobile terminal 20 and the face of the user U is short, while the object A becomes small when the distance is long. It will move. Therefore, since the object A can be moved so as to correspond to the movement of the mobile terminal 20 as seen by the user U, the user U can remotely operate the object A intuitively.

  In this modification, the magnification of the length of the motion vector is continuously changed in proportion to the size of the face image, but may be changed discretely. For example, when the size of the face image is less than or equal to a predetermined value, the magnification is decreased. On the other hand, when the size of the face image is larger than a predetermined value, the magnification is increased. Thus, the magnification may be changed in two steps. The magnification may be changed in multiple stages by setting a plurality of the predetermined values.

(Modification 4)
Moreover, in the said embodiment, although the motion detection part 33 makes the motion of the portable terminal 20 itself detected using the motion sensor 23 the movement of the operation of the portable terminal 20 by a user, it is not limited to this. . For example, the movement of the user's operation at the input unit 25, such as the user's flip operation (flick operation) detected using the touch panel at the input unit 25, may be the movement of the operation of the mobile terminal 20 by the user. In this case, the motion detection unit 33 does not need to use the motion sensor 23.

(Modification 5)
By the way, it is considered that the user usually maintains the posture for at least about one minute. If the posture is maintained, the geometric positional relationship between the user's face and an object located near or around the face is maintained, so that the midline of the face and the reference line of the object It is considered that the geometric positional relationship is also maintained. In addition to the face image, it is highly likely that the captured image includes the object image. That is, it is considered that there is a high possibility that the object becomes a subject other than the face.

  Note that examples of objects located near or around the user's face include furniture, wallpaper affixed to a wall, clothes worn by the user, and the like. Examples of the furniture reference line include vertical lines, horizontal lines, etc., and examples of the wallpaper reference line include straight lines indicating the direction of the wallpaper pattern, and the clothes reference line. As an example, there is a straight line indicating the direction of the pattern of the clothes.

  Therefore, when the face of the user U cannot be recognized from the captured image (NO in S11 of FIG. 4), instead of detecting the terminal inclination θ1 based on the detection data from the acceleration sensor 30 (S14), the following As described above, the face inclination θ2 may be estimated.

  That is, if the user U's face can be recognized from the captured image (YES in S11), the face inclination θ2 is detected from the captured image (S12), and subjects other than the face included in the captured image are detected. Image recognition is performed on the image of the subject, and a subject inclination which is an inclination of the reference line of the subject with respect to the apparatus is detected. If the user U's face cannot be recognized from the captured image (NO in S11), the subject tilt is detected, and the detected subject tilt and the user U's face have been detected previously. The current face inclination θ2 is estimated using the face inclination θ2 and the subject inclination. Note that the subject (the object) used for estimating the face inclination θ2 may be one or plural.

  By the way, many of the approximate portable terminals 20 are provided with the photographing unit 24 (in camera) on the screen side of the display unit 26 and the photographing unit 24 (out camera) on the opposite side. In such a portable terminal 20, when the face of the user U is photographed by the in-camera, the surroundings of the user U are photographed by the out-camera. Therefore, the in-camera may be used for recognizing the face of the user U, while the out-camera may be used for recognizing an object other than the face. Thus, when a plurality of photographing units 24 are provided, the photographing unit 24 used for face recognition and the photographing unit 24 used for recognition of an object other than the face may be the same, May be different.

[Embodiment 2]
Another embodiment of the present invention will be described with reference to FIG. FIG. 5 is a perspective view showing an outline of the mobile terminal 20 in the input system 1 according to the present embodiment.

  As shown in FIG. 5, the portable terminal 20 of the present embodiment is different from the portable terminal 20 shown in FIG. 3B in that a plurality of photographing units 24 are provided, and the other configurations are the same. It is. By using the plurality of captured images respectively created by the plurality of photographing units 24, the face inclination detecting unit 32 can create three-dimensional data of the face of the user U. Therefore, the accuracy of detecting the midline ML of the face of the user U can be improved, and the accuracy of detecting the face inclination θ2 can be improved.

[Embodiment 3]
Still another embodiment of the present invention will be described with reference to FIG. FIG. 6 is a block diagram illustrating a schematic configuration of the input system 1 according to the present embodiment. As illustrated in FIG. 6, the input system 1 according to the present embodiment captures a user's face with the display device 10 and detects a face inclination based on the captured image, as compared with the input system 1 illustrated in FIG. 1. However, it is different in that rotation conversion is performed.

  Specifically, the input system 1 according to the present embodiment has an imaging unit 24, a face tilt detection unit 32, a rotation conversion unit 34, and an operation data creation unit 35 of the mobile terminal 20, as compared with the input system 1 shown in FIG. Instead, the imaging unit 40, the face inclination detection unit 41, the rotation conversion unit 42, and the object operation unit 43 are provided in the display device 10, and the other configurations are the same. For this reason, the terminal inclination θ 1 detected by the terminal inclination detection unit 31 and the motion vector detected by the motion detection unit 33 are transmitted to the display device 10 via the communication unit 27. The imaging unit 40 and the rotation conversion unit 42 are the same as the imaging unit 24 and the rotation conversion unit 34 shown in FIG.

  Compared with the face inclination detection unit 32 shown in FIG. 1, the face inclination detection unit 41 of the present embodiment is a face inclination in the direction of the user's U midline ML (yf axis) with respect to the vertical direction (yd axis) of the display device 10. θ2 'is detected. Therefore, the face inclination detection unit 41 uses the terminal inclination θ1 from the terminal inclination detection unit 31 to change the face inclination θ2 ′ to the direction of the midline ML (yf) of the user U with respect to the longitudinal direction (yt axis) of the mobile terminal 20. Axis) face inclination θ2.

  The object operation unit 43 controls the display unit 16 to operate the object A on the screen 11 based on the motion vector from the rotation conversion unit 42.

  As described above, the detection of the face inclination and the rotation conversion can be performed by the display device 10.

[Embodiment 4]
Another embodiment of the present invention will be described with reference to FIG. FIG. 7 is a block diagram illustrating a schematic configuration of the input system 1 according to the present embodiment. As shown in FIG. 7, the input system 1 according to the present embodiment is different from the input system 1 shown in FIG. Instead of 32, a face inclination conversion unit 51 is provided, and other configurations are the same.

  The glasses-type terminal 50 is worn on the face of the user U and has the same function as the portable terminal 20. For this reason, the glasses-type terminal 50 can detect the inclination of the own device with respect to the vertical direction. This inclination corresponds to the inclination of the face of the user U with respect to the vertical direction, and may be converted into a face inclination θ2 ′ in the direction of the midline ML (yf axis) of the user U with respect to the vertical direction (yd axis) of the display device 10. it can. Accordingly, the glasses-type terminal 50 transmits the face inclination θ2 ′ to the mobile terminal 20.

  The face inclination conversion unit 51 uses the terminal inclination θ1 from the terminal inclination detection unit 31 to change the face inclination θ2 ′ from the glasses-type terminal 50 to the midline ML of the user U with respect to the longitudinal direction (yt axis) of the mobile terminal 20. Is converted into a face inclination θ2 in the direction (yf axis). The face inclination conversion unit 51 transmits the converted face inclination θ2 to the rotation conversion unit 34.

  In the case of this embodiment, the imaging unit 24 and face recognition processing are not necessary.

[Example of software implementation]
The control blocks (particularly the display control unit 15 and the control unit 21) of the display device 10 and the portable terminal 20 may be realized by a logic circuit (hardware) formed in an integrated circuit (IC chip) or the like, or a CPU ( It may be realized by software using a Central Processing Unit.

  In the latter case, the display device 10 and the portable terminal 20 include a CPU that executes instructions of a program that is software that implements each function, and a ROM (Read that records the above-described program and various data so that the computer (or CPU) can read them. Only Memory) or a storage device (these are referred to as “recording media”), a RAM (Random Access Memory) for expanding the program, and the like. And the objective of this invention is achieved when a computer (or CPU) reads the said program from the said recording medium and runs it. As the recording medium, a “non-temporary tangible medium” such as a tape, a disk, a card, a semiconductor memory, a programmable logic circuit, or the like can be used. The program may be supplied to the computer via an arbitrary transmission medium (such as a communication network or a broadcast wave) that can transmit the program. The present invention can also be realized in the form of a data signal embedded in a carrier wave in which the program is embodied by electronic transmission.

(Additional notes)
The present invention is not limited to the above-described embodiments, and various modifications are possible within the scope shown in the claims, and embodiments obtained by appropriately combining technical means disclosed in different embodiments. Is also included in the technical scope of the present invention. Furthermore, a new technical feature can be formed by combining the technical means disclosed in each embodiment.

  For example, in the above-described embodiment, the mobile terminal 20 accepts an operation (input) from the user, but can be applied to any input device. Moreover, although the portable terminal 20 has instruct | indicated the display apparatus 10, it is applicable to the arbitrary information processing apparatuses which process this instruction | indication.

[Summary]
An input device (portable terminal 20) according to aspect 1 of the present invention is an input device that receives input from a user and transmits input information based on the input to an information processing device (display device 10), and performs shooting. A face tilt detection unit (24) for performing face recognition on the face image of the user included in the shot image taken by the shooting and detecting a face tilt that is a tilt of the reference line of the face with respect to the own device; A face inclination detection unit 32), an operation movement detection unit (movement detection unit 33) for detecting an operation movement which is a movement of the user's own device by the user, and an operation movement detected by the operation movement detection unit. Using the face inclination detected by the detection means, rotation conversion means (rotation conversion section 34) for rotating and converting the operation movement relative to the reference line of the face, and the input information based on the operation movement rotated by the rotation conversion means. Create That the input information generating means and (operation data generation unit 35), and a.

  According to the above configuration, face recognition is performed on the face image of the user included in the photographed image photographed by the photographing unit, and the face inclination that is the inclination of the reference line of the face with respect to the own apparatus is detected. Next, an operation movement, which is a movement of the user's own device, is detected, and the detected operation movement is rotationally converted into an operation movement with respect to the face reference line using the face inclination. As a result, the rotation-converted operation movement does not depend on the tilt of the input device. Further, since the operation movement is movement with respect to the reference line of the face, it does not depend on the posture of the user.

  Then, the input information based on the operation movement with respect to the reference line of the face that has been rotationally converted is created and transmitted to the information processing apparatus. As a result, the same input information can be created with the same movement for the user.

  Examples of the face reference line include the face midline, eye alignment in the face, and ear alignment in the face. Further, it is preferable that the input information is created with the operation movement with respect to the reference line of the face as the movement with respect to the reference line (vertical line, horizontal line, etc.) of the apparatus. In this case, for example, the movement of the user's face with respect to the midline can be reflected as the movement of the object displayed on the screen of the display device with respect to the vertical line, and the movement of the user's eyes with respect to the alignment of the object can be reflected. It can be reflected as movement relative to the horizon. As a result, the user can perform an intuitive remote operation.

  Further, the operation movement includes movement and rotation. The operation movement may be the movement of the own apparatus detected by the motion sensor (23) provided in the own apparatus, or may be a movement on a touch panel provided in the own apparatus.

  By the way, when the user stands upright, the direction of the median line of the face becomes the direction of the vertical line, while the direction of the alignment of the eyes and the alignment of the ears becomes the direction of the horizontal line. From this, it can be said that the midline of the face is a reference line corresponding to a vertical line, while the eye alignment and the ear alignment are reference lines corresponding to a horizontal line. The direction of the vertical line is the direction of gravity and is uniquely determined, but the direction of the horizontal line is a direction perpendicular to gravity and is not uniquely determined.

  Therefore, in the input device according to aspect 2 of the present invention, in the aspect 1, it is preferable that the reference line of the face is a median line of the face. According to said structure, the same input information can be produced reliably by the same operation.

  The input device according to aspect 3 of the present invention may include a plurality of the imaging units at different positions in the above aspect 1 or 2. According to said structure, the said face inclination detection means can produce the three-dimensional data of the said user's face by utilizing the some picked-up image which each some said imaging | photography part produced. Therefore, the accuracy of detecting the reference line of the user's face can be improved, and the accuracy of detecting the face inclination can be improved.

  The input device according to aspect 4 of the present invention further includes a motion sensor that detects the movement of the own apparatus in the above-described aspects 1 to 3, and the operation movement detecting means is a movement of the own apparatus detected by the motion sensor. May be detected as the operation movement.

  In the input device according to aspect 5 of the present invention, in the above aspect 4, the motion sensor includes an acceleration sensor (30) for detecting the acceleration of the own device. The apparatus further includes device inclination detection means (terminal inclination detection unit 31) for detecting an apparatus inclination that is an inclination of the apparatus, and when the face inclination detection means cannot perform the face recognition, the rotation conversion means The operation movement detected by the operation movement detection unit may be rotationally converted into an operation movement with respect to the vertical line using the apparatus inclination detected by the apparatus inclination detection unit. According to said structure, the input information which does not depend on the inclination of an input device can be produced.

  In an input device according to aspect 6 of the present invention, in the above aspects 1 to 4, the face inclination detection unit performs image recognition on an image of a subject other than the face included in the captured image, and Detected when the subject inclination, which is the inclination of the reference line of the subject, is detected and the face recognition cannot be performed, and when the subject inclination detected this time and the face recognition can be performed previously. The current face inclination may be estimated from the face inclination and the subject inclination.

  Here, examples of subjects other than the user's face include furniture, wallpaper affixed to a wall, clothes worn by the user, and the like. Examples of the reference line for the furniture include a vertical line and a horizontal line. Examples of the wallpaper reference line include a straight line indicating the direction of the wallpaper pattern. Examples of the clothes reference line include a straight line indicating the direction of the clothes pattern.

  Usually, it is considered that the user maintains the posture for at least about one minute. If the posture is maintained, the geometric positional relationship between the user's face and the subject is maintained, so the geometric positional relationship between the reference line of the user's face and the reference line of the subject is also maintained. It is thought that it is done.

  Therefore, according to the above configuration, if the subject tilt is detected when the face tilt is detected, the subject tilt is detected even if the face tilt cannot be detected because the face recognition cannot be performed. The current face inclination can be estimated by using the detected subject inclination and the previously detected face inclination and subject inclination. Note that the subject used for estimating the face inclination may be one or plural.

  The input device according to aspect 7 of the present invention further includes a storage unit (22) that stores correspondence information in which information of a plurality of face images is associated with the face inclination in the above aspects 1 to 6, The face inclination detection means may acquire the face inclination corresponding to a face image similar to the user's face image with reference to the correspondence information.

  According to said structure, since the process which performs face recognition with respect to the said face image and specifies the reference line of the said face can be abbreviate | omitted, the said face inclination can be acquired rapidly. The face images stored in the storage unit may be those of the same user or may be those of an average person. Moreover, in the case of the said aspect 3, the said corresponding | compatible information can be rapidly produced by utilizing several said imaging | photography part.

  An input system (display system 1) according to an aspect 8 of the present invention is an input system including an input device that receives an input from a user and an information processing device that receives input information based on the input from the input device. A face inclination detecting means (face inclination detecting unit 41, glasses-type terminal 50) for detecting a face inclination which is an inclination of a reference line of the user's face with respect to a vertical line, and an apparatus which is an inclination of the input device with respect to the vertical line An apparatus inclination detecting means for detecting an inclination, an operation movement detecting means for detecting an operation movement of the input device by a user, and an operation movement detected by the operation movement detecting means, the face inclination detecting means and the apparatus inclination detecting means. Rotation conversion means (rotation conversion unit 42) for performing rotation conversion to an operation movement with respect to the reference line of the face using the detected face inclination and apparatus inclination, respectively. Wherein the information processing apparatus, the operation moves to the rotation conversion unit has rotational transformation as the operation movement relative to the reference line of the own device, and performs processing based on the manipulation motion.

  According to said structure, the inclination of the reference line of a user's face with respect to a perpendicular line is detected. In order to perform this detection, a photographed image from a photographing unit provided in a device having a vertical line as a reference line, such as an information processing device or a display device, may be used. Alternatively, the detection can be performed by a portable device attached to the user's face and provided with an acceleration sensor.

  In addition, a device tilt that is the tilt of the input device with respect to the vertical line is detected. In order to perform this detection, the detection result of the acceleration sensor provided in the input device may be used. Or what is necessary is just to utilize the picked-up image containing the said input device image | photographed by the imaging | photography part provided in the said information processing apparatus.

  Next, the detected operation motion is rotationally converted into an operation motion with respect to the reference line of the face using the face tilt and the device tilt. As a result, the rotation-converted operation movement does not depend on the tilt of the input device. Further, since the operation movement is movement with respect to the reference line of the face, it does not depend on the posture of the user.

  Then, the information processing apparatus performs processing based on the movement of the face with respect to the reference line, which has been rotationally converted, as movement with respect to the reference line. Thus, the same instruction can be given to the information processing apparatus with the same movement for the user.

  In addition, each said means may be provided in the said input device, and the said information processing apparatus may be provided. When the input device is provided, the information processing device does not need to add a new function, and thus can use an existing one. Furthermore, each means may be included in a device other than the input device and the information processing device included in the display system.

  An input device control method according to an aspect 9 of the present invention is a control method for an input device that receives input from a user and transmits input information based on the input to an information processing device, and is included in a photographed image by a photographing unit. A face inclination detection step (S12) of performing face recognition on the user's face image to detect a face inclination which is an inclination of the reference line of the face with respect to the own apparatus, and a user's operation of the own apparatus. An operation movement detection step (S15 / S18) for detecting an operation movement, and the operation movement detected in the operation movement detection step is used as a reference for the face using the face inclination detected in the face inclination detection step. Rotation conversion step (S16 / S19) for rotationally converting the operation motion to the line, and input information creation for creating the input information based on the operation motion rotationally converted in the rotation conversion step And step (S17 · S20), contains. In this case, the same effects as those of the first aspect are obtained.

  The input device according to each aspect of the present invention may be realized by a computer. In this case, the input device is realized by the computer by causing the computer to operate as each unit included in the input device. A control program and a computer-readable recording medium on which the control program is recorded also fall within the scope of the present invention.

  In the present invention, a face inclination is detected from a captured image, and using the face inclination, an operation movement, which is a movement of the user's own apparatus, is rotationally converted into an operation movement with respect to the reference line of the face, and the rotation is converted. Since the input information based on the operation motion is created, the same input information can be created by the same operation without depending on the direction of the input device and the posture of the user, so that it can be applied to any input device.

1 Input System 10 Display Device (Information Processing Device)
11 Screen 12 Content receiving unit 13 Content analyzing unit 14 Communication unit 15 Display control unit 16 Display unit 20 Mobile terminal (input device)
21 Control unit 22 Storage unit 23 Motion sensor 24 Imaging unit 25 Input unit 26 Display unit 27 Communication unit 30 Acceleration sensor 31 Terminal tilt detection unit (device tilt detection means)
32 Face tilt detection unit (face tilt detection means)
33 Motion detection unit (operation motion detection means)
34 Rotation conversion unit (rotation conversion means)
35 Operation data creation unit (input information creation means)
40 photographing unit 41 face tilt detecting unit 42 rotation converting unit 43 object operating unit 50 glasses-type terminal 51 face tilt converting unit

Claims (10)

An input device that receives input from a user and transmits input information based on the input to an information processing device,
A shooting unit for shooting;
A face inclination detecting means for performing face recognition on the face image of the user included in the photographed image by the photographing and detecting a face inclination which is an inclination of the reference line of the face with respect to the own device;
An operation movement detecting means for detecting an operation movement that is a movement of the user's own device;
Rotation conversion means for rotating and converting the operation movement detected by the operation movement detection means into an operation movement with respect to a reference line of the face using the face inclination detected by the face inclination detection means;
An input device comprising: input information creating means for creating the input information based on the operation movement rotationally converted by the rotation converting means.   The input device according to claim 1, wherein the reference line of the face is a median line of the face.   The input device according to claim 1, wherein a plurality of the photographing units are provided at different positions. It further includes a motion sensor that detects the movement of its own device,
The input device according to any one of claims 1 to 3, wherein the operation motion detection unit detects the motion of the device detected by the motion sensor as the operation motion. The motion sensor includes an acceleration sensor that detects the acceleration of the device itself,
From the detection result of the acceleration sensor, further comprising a device inclination detecting means for detecting a device inclination that is the inclination of the device relative to the vertical line,
When the face inclination detection unit cannot perform the face recognition, the rotation conversion unit uses the device inclination detected by the device inclination detection unit to detect the operation movement detected by the operation movement detection unit. The input device according to claim 4, wherein the input device is rotationally converted into an operation motion with respect to the vertical line. The face inclination detecting means is
Image recognition is performed on an image of a subject other than the face included in the photographed image, and a subject tilt that is a tilt of a reference line of the subject with respect to the own device is detected;
When the face recognition cannot be performed, the current inclination of the subject is determined from the subject inclination detected this time and the face inclination and the subject inclination detected when the face recognition has been performed previously. The input device according to claim 1, wherein the input device is estimated. A storage unit for storing correspondence information in which information on a plurality of face images is associated with the face inclination;
The face inclination detection unit acquires the face inclination corresponding to a face image similar to the face image of the user with reference to the correspondence information. The input device described in 1. An input system comprising: an input device that receives input from a user; and an information processing device that receives input information based on the input from the input device,
Face inclination detecting means for detecting a face inclination which is an inclination of a reference line of the user's face with respect to a vertical line;
An apparatus inclination detecting means for detecting an apparatus inclination which is an inclination of the input apparatus with respect to the vertical line;
An operation movement detecting means for detecting an operation movement of the input device by a user;
Rotation conversion that rotationally converts the operation movement detected by the operation movement detection means into an operation movement with respect to the reference line of the face, using the face inclination and the apparatus inclination detected by the face inclination detection means and the apparatus inclination detection means, respectively. Means, and
The information processing apparatus performs processing based on the operation movement as the operation movement with respect to the reference line of the own apparatus as the operation movement rotated and converted by the rotation conversion unit. An input device control method for receiving input from a user and transmitting input information based on the input to an information processing device,
A face inclination detection step of performing face recognition on the face image of the user included in the photographed image by the photographing unit and detecting a face inclination which is an inclination of the reference line of the face with respect to the own device;
An operation motion detection step for detecting an operation motion that is a motion of the operation of the device by the user;
A rotation conversion step of rotating and converting the operation movement detected in the operation movement detection step into an operation movement with respect to the reference line of the face using the face inclination detected in the face inclination detection step;
And an input information creating step for creating the input information based on the operation motion rotationally converted in the rotation converting step.   A control program for causing a computer to function as the input device according to any one of claims 1 to 7, wherein the control program causes the computer to function as each of the means.

Images