WO2018083737A1

WO2018083737A1 - Display device and remote operation controller

Info

Publication number: WO2018083737A1
Application number: PCT/JP2016/082467
Authority: WO
Inventors: 吉澤　和彦; 橋本　康宣; 清水　宏; 具徳野村; 岡本　周幸
Original assignee: マクセル株式会社
Priority date: 2016-11-01
Filing date: 2016-11-01
Publication date: 2018-05-11

Abstract

Provided is a feature with which the necessary amount of learning is small and it is possible to improve the ease of use by a user pertaining to a feature for controlling the user's remote operation on the screen of a display device. A display device provided with cameras 21, 22 for imaging an area that includes a user viewing a screen 10, the device detecting the positions of a first point (point P0) that is a user-side reference point against a camera-side reference point (point Q0) and a second point (point F0) of a user's finger by analysis of the captured images of the two cameras, setting a virtual planar space (second space 102) in a space 100 linking the screen 10 and the first point at a position of a prescribed length in the viewing direction from the first point so as to overlap the screen 10 as seen from the user, determining a prescribed remote operation that includes operation of touching the virtual planar space with a finger on the basis of a degree of penetration of a finger into the virtual planar space including the distance between the virtual planar space and the second point, generating operation input information that includes the position of the second point or a position in the screen 10 and operation information representing the prescribed remote operation, and controlling the operation of the display device.

Description

Display device and remote control device

The present invention relates to a technology for a display device, and relates to a technology for controlling a user operation on the display device.

Electronic devices such as display devices including television receivers and projectors have an operation input panel, a remote control device, a graphical user interface (GUI) for a screen, and the like as man-machine interfaces. Various operations and the like can be instructed to the electronic apparatus by user input operations through these operation input means. With the advancement of video display technology, camera imaging technology, semiconductor technology, etc. and the expansion of the environment, a technology has been developed as an operation input means for enabling an intuitive and smooth operation as compared with the operation using a conventional remote control device. For example, the technique etc. which operate an electronic device with a voice recognition function are mentioned. There is a possibility that such advanced interface technology will be introduced to general households in the future. Even in such a situation, the operation using the remote control device is still common. The reason for this is that it can be used easily, can achieve stable operation, and can be manufactured with inexpensive parts.

On the other hand, a technique has been proposed in which a widely used camera is applied and used as a remote operation means for a GUI of a display device screen. For example, there is a system that detects a gesture caused by a predetermined movement of a user using a camera provided in the vicinity of the display device, and controls the operation of the display device using the gesture as an operation instruction by remote operation. In the system, a user at a position away from the screen performs a predetermined gesture such as shaking his / her hand from side to side. The system detects the gesture based on the analysis of the camera-captured image, and gives an operation instruction corresponding to the gesture.

Japanese Patent Laid-Open No. 8-351541 (Patent Document 1) is cited as a prior art example that realizes remote control of a display device or the like using a camera. Patent Document 1 describes the following as a gesture recognition system. The system uses a camera to detect the presence of a predetermined hand gesture and a position in the space to generate a gesture signal, and based on the gesture signal, the on-state corresponding to the position in the space where the hand gesture is detected. Display the hand icon at the position on the screen. The system generates a machine control signal when the hand icon is moved over the machine control icon according to the movement of the hand gesture.

JP-A-8-315154

In a system for controlling a user's remote operation with respect to a screen of a display device in a prior art example, a plurality of gestures defined by a predetermined movement of a user's finger are provided, and an operation instruction or the like is associated with each gesture. In such a system, in order to use a remote operation, the user needs to memorize a plurality of gestures including the correspondence between gestures and operation instructions, and requires a large amount of learning. A certain level of proficiency is also required to prevent misoperation. The necessity for learning or the like in such a system is considerably disadvantageous as compared with the convenience and convenience of the remote control device. The prior art has problems in terms of user convenience and has room for improvement.

Also, in such a system, it is always necessary to perform a process of distinguishing and detecting a plurality of gestures from the user's movement based on the analysis of the captured image of the camera. For this reason, the computer requires a large amount of processing and a high load.

An object of the present invention is to provide a technology that can reduce the amount of learning required and improve the usability of the user with respect to the technology for controlling the user's remote operation on the screen of the display device.

A representative embodiment of the present invention is a display device and a remote control device, and has the following configuration.

A display device according to an embodiment is a display device having a function of controlling a user's remote operation on a screen of the display device, and includes at least two cameras that capture a range including a user who views the screen. The position of a first point that is a user-side reference point associated with the head, face, or eye of the user with respect to the position of the camera-side reference point of the two cameras by analyzing the captured images of the two cameras; The position of the second point of the user's finger is detected, and within the space connecting the screen and the first point, from the user to a position of a predetermined length in the viewing direction from the first point to the screen. The virtual surface space is set so as to overlap with the screen when viewed, and the degree of the finger entering the virtual surface space including the distance between the position of the virtual surface space and the position of the second point is calculated. The approach Based on the degree, a predetermined remote operation including a touch operation of the finger with respect to the virtual surface space is determined, and the position of the second point or the position in the screen associated with the position of the second point Operation input information including coordinates and operation information representing the predetermined remote operation is generated, and the operation of the display device is controlled by the operation input information.

According to a typical embodiment of the present invention, the amount of learning necessary for a technique for controlling the user's remote operation on the screen of the display device is small, and the user's usability can be improved.

1 is a perspective view showing a configuration of a display system including a display device according to a first embodiment of the present invention. 1 is a diagram illustrating a functional block configuration of a display system including a display device according to Embodiment 1. FIG. In Embodiment 1, it is a figure which shows the state which looked at the space from the side. In Embodiment 1, it is explanatory drawing which shows the details, such as a virtual surface space. In Embodiment 1, it is a figure which shows the state which looked at the space from the top. In Embodiment 1, it is a figure which shows the case where a viewing-and-listening position has an angle with respect to the screen center. In Embodiment 1, it is a figure which shows the overlap of the screen and virtual surface in a user's visual field. In Embodiment 1, it is a figure shown about the distance measurement based on binocular parallax. In Embodiment 1, it is a figure shown about the setting of the user side reference point based on the analysis of a camera picked-up image. In Embodiment 1, it is a figure which shows the example of a screen display control according to a finger position. FIG. 10 is a diagram illustrating an example of cursor display control in the first embodiment. FIG. 10 is a diagram illustrating an example of cursor display control in the first embodiment. In Embodiment 1, it is a figure shown about the shape of space. In Embodiment 1, it is explanatory drawing regarding the example of operation. In Embodiment 1, it is a figure shown about the virtual surface adjustment for every user individual. FIG. 11 is a diagram illustrating an example of camera arrangement and posture in a first modification of the first embodiment. It is a figure shown about virtual surface correction in the 1st modification. It is a figure which shows the single virtual surface of the space in the 2nd modification of Embodiment 1, and the example of display control. FIG. 10 is a diagram illustrating an example of virtual surface adjustment in the third modification example of the first embodiment. It is a figure which shows the functional block structure of a display system including the remote control device of Embodiment 2 of this invention. It is a figure which shows the structure of the display system containing the display apparatus of Embodiment 3 of this invention. 6 is a diagram illustrating a configuration of a display device according to Embodiment 3. FIG. FIG. 10 is a diagram showing a first processing flow of the display device according to the third embodiment. FIG. 10 is a diagram showing a second processing flow of the display device according to the third embodiment. It is a figure which shows the structure of the display system containing the display apparatus of Embodiment 4 of this invention. In Embodiment 4, it is a figure which shows the state which looked at the space from the camera. It is a figure which shows the structure of the display system containing the display apparatus of Embodiment 5 of this invention. It is a figure which shows the structure of the display system containing the display apparatus of Embodiment 6 of this invention. It is a figure which shows the structure of the display system containing the display apparatus of Embodiment 7 of this invention.

Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings. Note that components having the same function are denoted by the same reference symbols throughout the drawings for describing the embodiment, and the repetitive description thereof will be omitted.

(Embodiment 1)
The display device according to the first embodiment of the present invention will be described with reference to FIGS. The display device of the first embodiment is a display device having a remote operation control function that enables remote operation input to the screen.

[Display system (1)]
FIG. 1 shows a configuration of a display system including the display device of the first embodiment. In FIG. 1, a state in which the user is looking at the screen 10 of the display device 1 is schematically shown in perspective. In addition, it has (X, Y, Z) as an explanatory direction and a coordinate system, the X direction is the first direction, the Y direction is the second direction, and the Z direction is the third direction. The X direction and the Y direction are two orthogonal directions constituting the screen 10, where the X direction is the horizontal direction within the screen and the Y direction is the vertical direction within the screen. The Z direction is a direction perpendicular to the X direction and the Y direction of the screen 10.

In the display system of FIG. 1, the user is a viewer who views the video on the screen 10 of the display device 1 and is an operator who performs a remote operation on the screen 10. A user-side reference point corresponding to the user's viewpoint is indicated by a point P0. The point P0 is one point of the face or head, for example, the middle point between both eyes. A case where the user is facing the screen 10 as the viewing posture and the viewing direction is along the Z direction is shown. The position of the finger at the time of remote operation by the user is indicated by a point F0. Point F0 is the tip point of the finger that is closest to the screen 10 in the Z direction, for example.

The display device 1 is, for example, a television receiver, and has a function of receiving broadcast waves and reproducing and displaying broadcast programs, a function of reproducing and displaying video based on digital input data, and the like. The screen 10 has a rectangular plane, and a central point is a point Q0 and four corner points are points Q1 to Q4. The middle point on the right side is point Q5, the middle point on the left side is point Q6, the middle point on the upper side is point Q7, and the middle point on the lower side is point Q8.

The display device 1 includes a camera 21 and a camera 22 which are two cameras. The two cameras are arranged on the left and right positions with respect to the axis connecting the points Q7 and Q8. The display device 1 has a function of performing distance measurement based on binocular parallax related to a target object using captured images of two cameras. The target object is the user's eyes and fingers. The camera 21 is a right camera and is disposed at a point Q5 in the middle of the right side of the screen 10. The camera 22 is a left camera and is disposed at a point Q6 in the middle of the left side of the screen 10. The two cameras are arranged in an orientation for photographing a predetermined range including the user in front of the screen 10, and the orientations can be adjusted. The user is included in the moving image or still image of the video captured by the camera. The camera side reference point in the two cameras is a point Q0 that is an intermediate point between the points Q5 and Q6.

The display device 1 detects the distance and position from the target object by distance measurement based on binocular parallax in the analysis processing of the captured images of the two cameras. A known principle can be applied to distance measurement based on binocular parallax. This distance is the distance between the point Q0 that is the camera side reference point and the point P0 that is the user side reference point, or the distance between the point Q0 and the finger point F0. From these distances, the positions of the points P0 and F0 are obtained as the position coordinates of the coordinate system (X, Y, Z) in the three-dimensional space. The display device 1 detects the positions of the point P0 and the point F0 for each time point.

The point P0 which is the user side reference point is set as one point of the head or face, for example, the middle point between both eyes. In FIG. 1, the middle point of both eyes is represented as a point P0 which is a user side reference point. When both eyes can be detected from the images captured by the two cameras, the point P0 can be set.

When the user views the screen 10, the space 100 has a quadrangular pyramid shape with the four points Q1 to Q4 on the screen 10 as the bottom surface and the point P0 as the user side reference point as the apex. The display device 1 grasps the space 100 formed between the point P0 and the screen 10 based on the detection of the position of the point P0. A straight line connecting the point P0 and the point Q0 is indicated as a reference axis J0. The direction of the reference axis J0 corresponds to the viewing direction when the user views the point Q0 on the screen 10. The display device 1 sets a specific virtual plane in the space 100 based on the grasp of the point P0 and the space 100. This virtual surface is a virtual surface that accepts a user's remote operation on the screen 10. In FIG. 1, a space between the first virtual surface 201 and the second virtual surface 202 in the space 100 is set as a second space 102 that is a virtual surface space. In the space 100, a virtual surface reference point is provided at a position of a predetermined length from the point P0 on the reference axis J0 in the direction of the point Q0, and the first virtual surface 201 and the second virtual surface 202 are set as two virtual surfaces. The The virtual plane space is a substantially flat space having a predetermined thickness.

In the space 100, a quadrangular pyramid-shaped space from the point P0 to the first virtual surface 201 is a first space 101, a virtual surface space is a second space 102, and a space from the second virtual surface 202 to the screen 10 is a third space. A space 103 is assumed. In a field of view where the user views the screen 10, the virtual plane is directly overlapped on the screen 10. The processor of the display device 1 grasps such a space 100 and the virtual plane space with the position coordinates of the three-dimensional space by calculation.

The display device 1 determines and detects a predetermined operation by the finger based on the positional relationship between the finger point F0 and the virtual surface space and the degree of the finger entering the virtual surface space. In the viewing direction, the degree of entry of the finger point F0 with respect to the virtual plane is detected as, for example, the distance between the position of the point F0 and the position of the second virtual plane 202. The predetermined operation is, for example, an operation such as touching, tapping, and swiping on the second virtual surface 202. The display device 1 detects the position of a finger and a predetermined operation with respect to the virtual surface space, and generates operation input information representing the detected position and the predetermined operation. The display device 1 controls the operation of the display device 1 and the GUI of the screen 10 using the operation input information. For example, when a touch operation is performed on a GUI object on the screen 10, for example, a selection button, the display device 1 performs a corresponding process according to the touch operation on the object, for example, a selection selection process for the option.

In FIG. 1, the user is viewing the video on the screen 10 in a standard posture. Thus, in normal viewing, there are no obstacles in the space 100 that hinder viewing. In the display system of the first embodiment, the intention of the user's operation on the GUI or the like of the screen 10 is remotely controlled from the movement of the user's fingers in this space 100, particularly the movement in the second space 102 which is a virtual plane space. Detect as.

When the fingertip reaches the first virtual surface 201, the display device 1 automatically shifts to a state of accepting a remote operation, and displays a unique cursor on the screen 10. The cursor is a pointer image representing the presence of the user's finger in the virtual plane space, and is information for feeding back the remote operation state to the user. The position on the XY plane in the virtual plane space and the position on the XY plane such as the GUI of the screen 10 are managed in association with each other. The cursor is displayed at a position such as a GUI on the screen 10 associated with the position of the finger in the virtual plane space.

[Display system (2)]
FIG. 2 shows a functional block configuration of the display device 1 of the display system of FIG. The display device 1 includes a control unit 11, a storage unit 12, a content display unit 13, a GUI display unit 14, a display drive circuit 15, a screen 10, and a remote operation control unit 20. The remote operation control unit 20 includes a camera 21, a camera 22, a binocular parallax calculation unit 23, a virtual surface operation determination unit 24, an operation input information output unit 25, an individual recognition unit 26, a virtual surface setting unit 27, and a virtual surface adjustment unit 28. Have

The control unit 11 controls the entire display device 1. When a remote operation is performed, the control unit 11 controls the operation of the display device 1 according to the operation input information 301. The storage unit 12 stores control information and content data. The content display unit 13 displays content video on the screen 10 based on the content data. The content can be various types such as a broadcast program, a DVD video, and a material file.

The GUI display unit 14 is a part that performs processing for controlling the GUI image display on the screen 10, processing corresponding to regulations according to operations on GUI objects, and the like in the OS or application of the display device 1. The GUI display unit 14 displays a GUI image on the screen 10. The GUI may be a predetermined menu screen or the like. The GUI may be displayed superimposed on the content. Based on the operation of the object indicated by the operation input information 301, the GUI display unit 14 performs a prescribed corresponding process (for example, a selection determination process) in accordance with an operation (for example, a touch operation) for the object (for example, a selection button). The display state (for example, an effect display indicating that the option button is pressed and selected) is controlled. Further, the GUI display unit 14 controls the display state of the cursor on the GUI using the cursor display control information of the operation input information 301.

The display driving circuit 15 generates a video signal based on the input video data from the content display unit 13 or the GUI display unit 14 and displays the video on the screen 10.

The user operates the virtual plane space in FIG. The operation is a remote operation on the screen 10. The operation is detected and determined by the remote operation control unit 20 and converted as an operation input to the GUI of the screen 10. Each of the two cameras captures a range including the user's face and fingers and outputs a captured image. The captured video data is input to the binocular parallax calculation unit 23 and the personal recognition unit 24.

The binocular parallax calculation unit 23 analyzes the captured video, extracts features such as the face and fingers, and based on the principle of distance measurement based on the binocular parallax, The distance to the point F0 is measured, and the position coordinates of the point P0 and the point F0 are obtained. The binocular parallax calculation unit 23 outputs the viewpoint position information including the obtained position coordinates of the point P0 to the virtual surface setting unit 27, and the finger position information including the obtained position coordinates of the point F0 is operated as a virtual surface operation. Output to the determination unit 24.

The virtual plane setting unit 27 sets the virtual plane space, that is, the first virtual plane 201 and the second virtual plane 202 in the space 100 using the viewpoint position information of the user at that time. Virtual surface information representing the virtual surface space set by the virtual surface setting unit 27 is input to the virtual surface operation determination unit 24.

The virtual surface operation determination unit 24 determines the positional relationship of the fingers with respect to the virtual surface space, the state of the approach, and a predetermined operation based on the virtual surface information and the finger position information. The virtual surface operation determination unit 24 determines the positional relationship and the entry, for example, whether or not the finger point F0 has entered the depth after the first virtual surface 201, or whether the point F0 has entered the depth after the second virtual surface 202. Determine the degree. Further, the virtual surface operation determination unit 24 determines whether or not a predetermined operation such as a touch operation has been performed based on the positional relationship and the degree of entry. The predetermined operation is an operation on the virtual surface. The predetermined operation is interpreted as an operation on the GUI object on the GUI display unit 14. The virtual surface operation determination unit 24 provides the determination result information to the operation input information output unit 25.

The operation input information output unit 25 generates operation input information 301 based on the determination result information, and outputs it to the GUI display unit 14 and the like. The operation input information output unit 25 operates based on the position coordinates of the finger point F0 in the virtual plane space, the position coordinates in the screen 10 associated with the position coordinates, and the determination result of the predetermined operation. Is generated.

The operation input information 301 is information including, for example, position coordinate information of the finger point F0 at each time point, operation information indicating a predetermined operation, cursor display control information, and the like. The position coordinate information of the finger ((Xf, Yf, Zf) in FIG. 3) may be the position coordinate information in the screen 10 ((xf, yf) in FIG. 7) by conversion of association. In that case, the operation input information 301 includes position coordinate information in the screen 10. The correspondence conversion may be performed not by the operation input information output unit 25 but by the GUI display unit 14.

The operation input information 301 includes operation information when a predetermined operation is performed. The operation information is information indicating the type, presence or absence of a predetermined operation such as touch, tap, swipe. The operation input information 301 includes cursor display control information according to the finger position. The cursor display control information is information for controlling the cursor display state on the screen 10 and includes designation information such as whether or not the cursor is displayed and the size. In this example, the operation input information 301 is operation input information for a GUI, but may be an operation instruction command or control information for an OS, an application, content, or the like.

The personal recognition unit 26 performs processing for recognizing the individual user by analyzing the video captured by the camera. For example, the individual recognizing unit 26 compares and compares the feature extracted from the face image in the captured video with the feature of the already registered face image to determine and identify the individual user. The recognition result information of the personal recognition unit 26 is input to the virtual surface setting unit 27. The personal recognition unit 26 can be applied to a method other than a method for recognizing an individual from a face image, and a method using a user ID input by a user, various biometric authentication methods, or the like may be applied.

The method for registering the user's personal face image is as follows. The user selects user setting and face image registration from the menu screen of the display device 1. In response to the selection, the display device 1 displays a user setting screen and enters a face image registration mode. In this mode, the display device 1 displays a message indicating that the face image is registered to the user, and asks the camera to face the camera. In this state, the display device 1 shoots a face with a camera, and registers and holds the face image, feature information extracted from the face image, and the like as face information of the individual user. Note that such registration may be omitted, and a face image obtained from a captured image during normal use may be used.

When the individual user is specified by the recognition result information, the virtual surface setting unit 27 sets a virtual surface according to the individual user. In that case, the virtual surface setting unit 27 refers to the virtual surface information of the individual user already set by the virtual surface adjustment unit 28. The virtual plane setting unit 27 sets a virtual plane adjusted according to the user's individual for the finger point F0 of the user's individual, and gives the virtual plane information to the virtual plane operation determination unit 24.

The virtual surface adjustment unit 28 performs processing for adjusting a virtual surface suitable for each individual user, in other words, calibration based on the user setting operation. As will be described later, the virtual plane for each user is obtained by adjusting the position of the virtual plane on the reference axis J0 from the standard position to a suitable position before and after. The virtual surface adjustment unit 28 holds virtual surface information after adjustment set for each individual user.

The specific adjustment method is as follows. The user selects user setting and virtual plane adjustment from the menu screen of the display device 1. In response to the selection, a user setting screen is displayed and the virtual plane adjustment mode is entered. In that mode, a message to the effect of adjustment is displayed to the user, and his / her finger is placed at a position that the user feels suitable. The display device 1 captures the state with a camera and detects the shape and position of a finger from the captured image. The display device 1 adjusts the position of the virtual surface so as to match the position of the finger, and registers and holds the virtual surface information of the individual user.

As a modification, the virtual surface operation determination unit 24 may be integrated with the GUI display unit 14. In that case, the remote operation control unit 20 gives the GUI display unit 14 at least position coordinate information for each point of the finger point F0 as the operation input information 301. The GUI display unit 14 determines a predetermined operation from the operation input information 301.

[Space (1)-Horizontal]
FIG. 3 shows a state in which the space 100 of the display system of FIG. In this state, the space 100 is indicated by a triangle having points Q7, Q8, and P0 as vertices. A second space 102 that is a virtual surface space is set in the space 100. The position coordinates of the point P0 that is the user side reference point are indicated by (Xp, Yp, Zp). The position coordinates of the finger point F0 are indicated by (Xf, Yf, Zf). A half length of the width of the screen 10 in the Y direction is indicated by V1. The distance from the point Q0 that is the camera side reference point to the point P0 that is the user side reference point is indicated by a distance D1. The distance from the point Q0 to the finger point F0 is indicated by a distance D2.

In FIG. 3, the virtual plane is set as a plane that is perpendicular to the viewing direction of the user and parallel to the screen 10. The virtual plane may be set as a plane that is not parallel to the screen 10 according to the viewing direction of the user. For example, a virtual plane shown in FIG. 6 described later is a plane that is perpendicular to the viewing direction of the user and is not parallel to the screen 10. The point P0 that is the user-side reference point is not limited to the middle point between both eyes, and can be set, for example, the center point of the head or face.

The virtual plane space is set as a space for enabling a predetermined operation as a remote operation and detecting the predetermined operation. The first virtual surface 201 is particularly a virtual surface related to cursor display control. In particular, the second virtual surface 202 is a reference virtual surface related to determination of a touch operation or the like. These two virtual planes are set on the reference axis J0 at positions close to the user-side reference point where the user's fingers can reach. Naturally, the space 100 and the virtual plane do not have an entity as an object, and the display device 1 manages the information as calculation information.

The user freely moves his / her finger up / down / left / right / backward with respect to the virtual plane space, for example, moves the finger back and forth in the direction of the reference axis J0 with respect to the position of the GUI object or the like on the screen 10 to perform touch operation or tap operation. I do.

The binocular parallax calculation unit 23 detects the position and movement of the finger in the virtual plane space. The binocular parallax calculation unit 23 measures the distance D1 between the point Q0 and the face point P0 by the distance measurement process based on the binocular parallax in the analysis of the camera-captured image, and calculates the point Q0 and the finger point F0. The distance D2 is measured. The binocular parallax calculation unit 23 detects the position of the point P0 as position coordinates (Xp, Yp, Zp) for each time point, and detects the position of the point F0 as position coordinates (Xf, Yf, Zf) for each time point. Also, the binocular parallax calculation unit 23 grasps the position coordinates (xf, yf) in the screen 10 associated with the position of the finger point F0 in the virtual plane space.

[Space (2)-Distance, Length]
FIG. 4 also shows a virtual surface space, examples of finger positions, various distances and lengths, and the like, while the space 100 is viewed from the side. FIG. 4 shows a state in which the second virtual surface 202 is touched at the point F0 at the tip of the finger. In FIG. 4, the position in the Z direction is 0 with respect to the position Z0 of the screen 10, the position of the point P0 is Zp, the position of the first virtual surface 201 is Z1, the position of the second virtual surface 202 is Z2, and the point F0 Is indicated by Zf.

The finger point F0 enters the space 100 from the outside of the space 100 according to the user's movement. For example, the point F0 first enters the first space 101, and then enters the second space 102 from the first space 101 through the first virtual plane 201. Next, the point F0 enters the third space 103 from the second space 102 through the second virtual plane 202. In addition, the approach of a finger includes not only the Z direction but also free movement up and down, left and right, and back and forth, including the X direction and the Y direction. In the example of FIG. 4, points F1 to F5 are shown as time-series trajectories of finger positions. Initially, the point F0 is outside the space 100. Next, the point moves from a point outside the space 100 to a point F1 in the first space 101. Next, the point F1 moves to a point F2 in the second space 102. Next, the point F2 moves to a point F3 in the third space 103. Next, the point F3 moves to a point F4 in the second space 102. Next, the point F4 moves to a point F5 outside the space 100.

In the example of FIG. 4, the point F0 of the current finger position is at the point F2. The distance from the point Q0 to the point F2 is shown as the distance D2 at that time.

In the Z direction and the reference axis J0, a length L1 and a length L2 are shown as predetermined lengths forward from the point P0. The position of the length L1 from the point P0 in the Z direction is indicated by a position Z1, and the corresponding point is indicated by a point C1. The position of the length L2 from the point P0 in the Z direction is indicated by the position Z2, and the corresponding point is indicated by the point C2. The center of the first virtual surface 201 is arranged at the point C1 at the position Z1. The center of the second virtual plane 202 is arranged at the point C2 at the position Z2.

The distance DST indicates the distance from the point F0 of the fingertip to the second virtual plane 202 in the direction of the reference axis J0, corresponds to the difference between the position Zf and the position Z2, and has a positive / negative sign. If the point F0 is at a position before the second virtual plane 202, the distance DST is positive. The binocular parallax calculation unit 23 calculates the distance DST. The distance DST is a value representing the positional relationship of the fingers with respect to the virtual plane space and the degree of entry.

The thickness M indicates the thickness that is the distance between the first virtual surface 201 and the second virtual surface 202 in the virtual surface space in the direction of the reference axis J0, and is about 5 cm as an example. The thickness M can also be set according to the predetermined lengths L1 and L2.

[Space (3)-Vertical]
FIG. 5 shows a state in which the space 100 and the like are viewed from the top and the vertical direction, that is, the XZ plane viewed from the Y direction. The point P0, which is the user side reference point, is generally set at one end near the screen 10 on the circumference of the user's head. The half length of the width of the screen 10 in the X direction is indicated by H1.

In the example of FIG. 5, points F11, F12, F13, and F14 are shown as examples of the position of the finger point F0. For example, the locus moves from a point F11 in the first space 101 to a point F12 in the second space 102. The user can freely move his / her finger up and down, left and right, that is, in the X direction and the Y direction in the second space 102 in accordance with the position of the GUI object or the like on the screen 10. For example, it can move from point F12 to point F14. When the user performs an operation of selecting a desired object on the GUI of the screen 10, the user moves his / her finger to a position on the object. For example, in the GUI image on the screen 10, it is assumed that an object (for example, an option button) is arranged in an area 311 illustrated in the X direction. An area 312 on the second virtual plane 202 associated with the area 311 of the screen 10 is also shown.

Suppose the user performs a touch operation to select the object. The user moves a finger to a position on the region 312, for example, the point F <b> 12 in the virtual plane space. Then, the user moves his / her finger so as to press the region 312. As a result, the position of the finger moves from the point F12 to the point F13 in the third space 103 through the second virtual plane 202. This movement is determined and detected as a touch operation, and is output as operation input information 301. The GUI display unit 14 performs an object selection determination process in the region 311 based on the operation input information 301 of the touch operation. When the user immediately returns his / her finger from the point F13 to a position in the second space 102, the movement is determined as a tap operation. Further, when the user performs the same second tap operation immediately after that, the movement is determined as a double tap operation. The GUI can be similarly controlled using the operation input information 301 of the tap operation.

[Viewing position]
FIG. 6 similarly shows a case where the viewing position of the user has an angle with respect to the center of the screen 10 with the space 100 viewed from above. The position of the user's eyes or the like is not necessarily limited to just facing the screen 10, and may have an angle with respect to the screen 10 in this way. In such a case, a virtual plane is basically set in the same manner, and remote operation can be realized. In the example of FIG. 6, the reference axis J0 with respect to the point P0 which is the user side reference point is shifted by an angle β with respect to the straight line 320 perpendicular to the center point Q0 of the screen 10. If this deviation is within a certain range of angle β, distance measurement based on binocular parallax can be performed, and remote operation can be similarly realized.

In the state shown in FIG. 6, a point P0 that is a user-side reference point is detected based on the camera-captured image, and a distance D1 between the point Q0 and the point P0 is measured. A straight line connecting the point P0 and the point Q0 is taken as a reference axis J0. On the reference axis J0, the first imaginary plane 201 is set at a point C1 that is located at a length L1 from the point P0. The second virtual surface 202 is set at a point C2 at a position of length L2 from the point P0. Here, the first virtual surface 201 and the second virtual surface 202 are set as planes that are perpendicular to the reference axis J0 and non-parallel to the screen 10.

FIG. 6 shows a case where the viewing direction has a deviation in the left and right directions in the X direction, but a virtual plane can be similarly set when the viewing direction has a deviation in the upper and lower directions in the Y direction. As a modification, when there is a shift in the viewing direction as shown in FIG. 6, a virtual plane parallel to the screen 10 may be set at the positions of the points C1 and C2.

[Overlap of screen and virtual surface]
FIG. 7 shows how the screen 10 and the virtual plane overlap in the field of view when the screen 10 is viewed from the user's viewpoint. As described above, in the display system according to the first embodiment, the first virtual surface 201 and the second virtual surface 202 which are virtual surfaces are exactly the same one-to-one with respect to the rectangle of the screen 10 in the X direction and the Y direction. Set to overlap in size. The position of the content or GUI object displayed on the screen 10 and the position of the finger on the virtual plane are the same one-to-one correspondence. The finger point F0 viewed from the user corresponds to the position when the object or the like on the screen 10 is inserted as it is, and the position of the cursor displayed on the GUI is also the same position. The position coordinates in the screen 10 associated with the position of the point F0 are indicated by (xf, yf).

[Distance measurement based on binocular parallax]
FIG. 8 is an explanatory diagram for supplementing distance measurement based on binocular parallax by two cameras, and shows an XZ plane when the space 100 and the like are viewed from above. In FIG. 8, the user sees the point Q0 at the center of the screen 10 from the front, and the reference axis J0 is in the Z direction and along the normal line of the screen 10. FIG. 8 schematically shows an example of a photographed image of the right camera 21 and a photographed image of the left camera 22. The subject of the two cameras is a user A. The captured image includes user A's head, face, upper body, and the like.

The display system detects the distance and position of the object by applying the principle of distance measurement based on the known binocular parallax. In the principle of distance measurement based on binocular parallax, when two cameras are regarded as both eyes, the distance of the depth from the subject can be measured based on the binocular parallax of the captured image. The camera side reference point Q0, the point Q5 of the camera 21, the position coordinates of the point Q6 of the camera 22, the length H1, etc. are fixed and known values. A point P0 that is a user-side reference point is set as an intermediate point between both eyes. The convergence angle α1 at the point P0 is an angle according to the distance D1, and the convergence angle α2 at the point F0 is an angle according to the distance D2.

The display device 1 measures the distance D1 from the point Q0 to the point P0 and the distance D2 from the point Q0 to the point F0 by calculating the distance measurement based on the binocular parallax in the analysis of the captured images of the two cameras. . The distance D1 is obtained by calculation using the convergence angle α1 (= θ1 + θ2) and the length H1. The angles θ1 and θ2 are obtained by calculation from the deviation of the left and right captured images (FIG. 9).

In FIG. 8, the shooting direction of the two cameras is the direction facing the point P0 and the angle θ1 = θ2 is shown, but the actual direction is not limited to this. The directions of the two cameras may be the same in parallel. It is only necessary that the user is in the shooting range.

The difference between the image 801 taken from the right camera 21 and the image 802 taken from the left camera 22 according to the angles θ1 and θ2 occurs in the image content. The binocular parallax calculation unit 23 counts the position of an object such as an eye in the image content in units of pixels constituting the images 801 and 802 (referred to as pixel positions) as shown in FIG. The binocular parallax calculation unit 23 calculates the difference width of the pixel position of the object in the image content as the difference δ. The distances D1 and D2 can be calculated using the difference δ and the focal lengths (FD1, FD2) of the camera.

For example, when focusing on the right eye point E1 as an object, the difference δ1 between the pixel position of the right eye point E1 of the right image 801 and the pixel position of the right eye point E1 of the left image 802 is calculated. Assume that the difference δ1 is about 2 cm, for example. The head diameter 811 is about 19 cm, and the head circumference 812 is about 60 cm. Then, the angle θ1 is θ1 = 360 ° × (δ1 / 60) / 2 = 6 °. In the triangle formed by the user-side point P0 and the two camera points Q5 and Q6, the distance D1 is obtained by D1 = H1 / tan θ1. For example, when H1 = 1 m, if θ1 = 6 ° is substituted, D1 = 9.5 m. The distance D2 can be similarly measured.

When the screen 10 is large, it is assumed that the length H1 is 10 m. For example, the number of pixels of the camera is set to 12 million pixels. As a shooting range of the camera, for example, the size of a room or a venue is 30 m × 40 m. Even in such a case, the camera can be covered with a resolution of 1 cm, which satisfies the specifications necessary for practical use.

In the display system, the above-mentioned calculation may be performed in the same way at the time of remote operation control, but the calculation may be simplified using a lookup table or the like. In advance, experimental data is taken, an output value is obtained by calculation from the input value, and the correspondence between the input value and the output value is stored in a lookup table.

[User-side reference point setting]
FIG. 9 shows the setting of the point P0, which is the user side reference point, based on the analysis of the camera photographed image. In FIG. 8 and FIG. 9, the right eye point E <b> 1 and the left eye point E <b> 2 are included in the face portions of the user shown in the

images

801 and 802 of the two cameras. A distance and a position may be calculated for an intermediate point between both eyes in the captured image, and the point may be set as the point P0. Alternatively, after calculating the distance and the position for each point of the right eye and the left eye in the captured image, an intermediate point between these points may be set as the point P0.

For example, the binocular parallax calculation unit 23 of the display device 1 includes, for example, the pixel position (x1, y1) of the right eye point E1 in the image 801 of the right camera 21 and the right eye in the image 802 of the left camera 22. The pixel position (x2, y2) of the point E1 is calculated. Then, a difference δ1 (in other words, a pixel distance) between the pixel positions of the right eye is calculated. Similarly, the display device 1 calculates the difference δ2 with respect to the left eye point. The display device 1 calculates the angles θ1 and θ2 from the magnitudes of the pixel position differences δ1 and δ2.

[Screen display control example (1)]
FIG. 10 shows a GUI display control example of the screen 10 according to the position of the finger. The upper side of FIG. 10 shows an example of the position of the finger in a state where the space 100 and the like are viewed from the side, as in FIG. The lower side of FIG. 10 shows a screen G1 and the like as a screen example according to the position of the finger. Screen G1 etc. show the example of a display of the screen 10 seen from the user. As an example of the GUI of the screen 10, a menu screen is shown, and two selection buttons are provided as GUI objects. The user selects an option by touching the button. The GUI display unit 14 performs a selection determination process as a corresponding process according to the button touch operation. This touch operation is an operation of touching the second virtual surface 202. Specifically, this touch operation corresponds to a movement in which the finger point F0 comes into contact with the second virtual surface 202 from the second space 102 and slightly enters the third space 103.

First, when the position of the finger is outside the space 100, the screen 10 is assumed to be in a state in which content or the like is displayed with the display function on. The user moves the fingertip into the space 100, for example, the first space 101.

(1) The display device 1 displays a screen G1, which is a first screen example, when the position of the finger point F0 is the position of the point F1 in the first space 101 from the outside of the space 100, for example. A menu screen is displayed on the screen G1. On the menu screen, GUI objects for giving operation instructions to the OS and applications of the display device 1 are arranged. The GUI objects include various known objects such as option buttons, list boxes, and text input forms. In this example, the screen G1 has two

option buttons

401 and 402 for selecting from “option A” or “option B” as objects. The display device 1 does not display the cursor yet in this state. The user moves the fingertip from the first space 101 to the second space 102 in order to give an operation instruction. When the finger returns from the first space 101 to the outside of the space 100, the menu screen is not displayed.

(2) When the position of the finger point F0 enters from the first space 101 into the back after the first virtual plane 201 and becomes the position of the second space 102, for example, the point F2, this is a second screen example. It changes to the state of the screen G2. The display device 1 displays a cursor K1 for remote operation control on the screen G2 in accordance with the position of the finger in the virtual plane space. The cursor K1 is displayed at a position in the screen 10 associated with the position of the finger point F0 in the virtual plane space. The display state of the cursor K1 is controlled according to the movement of fingers in the second space 102. The user freely moves his / her finger up / down / left / right / front / back in the second space 102. For example, when the user wants to select “option B” of the option button 402, the user moves his / her finger so as to move the cursor K1 onto the option button 402 in order to reflect the intention. The display device 1 displays the cursor K1 at a position above the object.

The display device 1 controls the cursor K1 to be displayed in a predetermined color, size, or shape according to the distance DST in the direction of the reference axis J0. For example, on the second screen G2, the display device 1 displays the cursor K1 in an arrow shape and yellow, and displays a smaller size as the distance DST decreases. When the cursor K1 has a small size, it becomes easier to see and operate the GUI object. The user moves his / her finger further back so as to press the option button 402. When the finger is moved back from the second space 102 into the first space 101, the cursor K1 is not displayed. When the finger is moved from the second space 102 back to the outside of the space 100, the menu screen and the cursor K1 are not displayed.

(3) Third screen example when the position of the finger point F0 reaches the second virtual surface 202 and enters the second virtual surface 202 and beyond and reaches the inside of the third space 103, for example, the point F3 Changes to the state of the screen G3. The display device 1 keeps the cursor K1 displayed in the state of the screen G3 in which the position of the finger is in the third space 103. When the position of the finger point F0 is on the second virtual plane 202, the size of the cursor K1 may be minimized. The display device 1 determines and detects the movement of the fingertip reaching and passing through the second virtual surface 202 as a touch operation in the remote operation. For example, the display device 1 determines and detects a movement from the point F2 to the point F3 as a touch operation on the option button 402. The display device 1 gives operation input information 301 representing the detected touch operation to the GUI display unit 14. In response to the touch operation, the GUI display unit 14 performs a selection decision process of “option B” as a process corresponding to the option button 402, and updates the display state of the object and the cursor K1 on the screen G3. For example, the selection decision effect display of the option button 402 is performed, and the display is made such that the option button 402 is pushed inward. In addition, the color of the cursor K1 changes from yellow to red.

Other screen display control examples may be as follows. Display control in the first space 101 and the second space 102 is the same as described above. The display device 1 hides the cursor K1 when the position of the finger enters the third space 103. For example, in the second space 102, when the position of a finger is on a background area where there is no object such as a button on the screen 10, the user enters the third space 103 from the position via the second virtual plane 202. There is. In that case, the display device 1 switches the cursor K1 to non-display. The display device 1 displays the cursor K1 again when the position of the finger returns to the second space 102. In the case of this control, when the cursor K1 disappears, the user can recognize that the finger is not in the virtual plane space suitable for remote operation as the state feedback, and returns the finger to the front. In addition, the cursor is not hidden in the third space 103, and a specific cursor representing an inappropriate state may be displayed.

As described above, visual feedback regarding the remote operation state is given to the user by the display control of the cursor K1 according to the distance DST. Accordingly, the user can more easily recognize the degree of finger approach to the virtual surface, the positional relationship, and the operation state such as a touch operation, thereby improving usability.

Other screen display control examples may be as follows. Initially, when the position of the finger is outside the space 100, nothing is displayed on the screen 10 due to an off state of the display function of the display device 1 (main power is off). The user moves his / her finger into the space 100, for example, the first space 101. The display device 1 detects the movement of the finger entering the first space 101 based on the analysis of the video captured by the camera. The display device 1 displays the menu screen of the screen G1 for the first time triggered by the detection. The same shall apply thereafter.

[Screen display control example (2)]
FIG. 11 shows a display control example such as a GUI cursor on the screen 10. (A1) shows a screen example in a state in which the position of the finger is relatively far from the second virtual surface 202 in the virtual surface space. In this screen 10, icons 411 and the like are displayed as GUI objects. When the finger is in the virtual plane space, the display device 1 displays the arrow-shaped cursor 410 with the size being changed according to the distance DST. On the screen 10 in this state, since the distance DST is relatively large, the cursor 410 is displayed in a relatively large size. In this state, since the cursor 410 is still large, it is difficult to select an object such as the icon 411. (A2) shows a screen example in a state where the finger approaches the second virtual surface 202 from the state of (A1) and the position of the finger is relatively close to the second virtual surface 202. On the screen 10 in this state, the cursor 410 is displayed in a relatively small size. In this state, since the cursor 410 is small, the icons 411 and the like are easy to select. The user can sense the degree of approach to the second virtual surface 202 from the size of the cursor 410 and the like.

(B1) similarly shows another screen example in a state in which the position of the finger is relatively far from the second virtual surface 202. On this screen 10, a double circular cursor 420 is displayed. The circle inside the double circle is constant and corresponds to the position of the finger point F0. The radius of the outer circle of the double circle is changed depending on the distance DST. In the state (B1), since the radius of the outer circle is relatively large, the cursor 420 is large. (B2) shows an example of a screen in a state in which a finger approaches the second virtual surface 202 from the state of (B1). In the state (B2), the radius of the outer circle is relatively small. When the finger point F0 reaches the position Z2 of the second virtual surface 202, the radius of the outer circle may be the same as the radius of the inner circle.

(C1) similarly shows another screen example in a state in which the position of the finger is relatively far from the second virtual surface 202. On this screen 10, a hand-shaped cursor 430 is displayed. In the state of (C1), an image having a shape in which the palm is widened is used as the first type hand-shaped cursor 430. (C2) shows an example of a screen in a state in which a finger approaches the second virtual surface 202 from the state of (C1). In the state of (C2), the type is changed together with the size according to the distance DST, and in particular, the second type is an image of the shape of one finger. In this way, the cursor type or the like may be changed according to the distance DST. The cursor may be a transparent display where the background can be seen.

[Screen display control example (3)]
FIG. 12 shows display control examples such as a GUI cursor on the screen 10 and various operation examples. An example of displaying in more detail in what state the finger is in contact with the second virtual surface 202 when feedback of the degree of entry of the finger with respect to the virtual surface by cursor display control is shown.

(A) shows a state in which the tip of one finger is in contact with the second virtual surface 202 and a screen example in that state. In this screen 10, an arrow-shaped cursor 410 is displayed at a position on the GUI corresponding to the point F0 of the position of the tip of one finger on the virtual plane, as in (A2) of FIG. In particular, when the fingertip touches the second virtual surface 202, the cursor 410 may be displayed in a unique state. For example, an effect indicating contact may be displayed. An example of GUI operation on this screen 10 is shown below. Examples of operations include touch operations and tap operations. The user performs, for example, a touch operation on an object such as a GUI button in a state where the cursor 410 overlaps. That is, the user moves one finger to the back on the reference axis J0 to bring the fingertip into contact with the second virtual surface 202. This movement is determined as a touch operation. Corresponding processing is performed by a touch operation on the second virtual surface 202. Similarly, a tap operation or the like is possible.

(B) shows a state in which the palm is in contact with the second virtual surface 202 and a screen example in that state. On this screen 10, a cursor 440 having a finger cross-sectional shape is displayed at a position on the GUI in correspondence with the position and cross section of the palm on the virtual plane. An example of GUI operation on this screen 10 is shown below. Examples of operations include a swipe operation. The user performs, for example, a swipe operation on the page or object on the screen 10. That is, the user moves the palm back on the reference axis J0 to be in contact with the second virtual surface 202, and moves the palm so as to slide in a desired direction in the X direction or the Y direction. This movement is determined as a swipe operation. Similarly, a flick operation or the like is possible. The GUI display unit 14 performs page scrolling and object movement processing as corresponding processing in accordance with these operations.

The finger cross-sectional shape of the cursor 440 may be a schematic display. Further, as an application, the size of the finger cross section may be used for operation control. For example, it is possible to determine that the touch operation is performed when the amount of change in the area of the finger cross section is equal to or greater than a threshold. For example, when one finger first comes into contact with the second virtual surface 202, it is a small cross section of one finger, and a large cross section corresponding to the entire hand by entering further into the back.

(C) shows a state in which a plurality of fingers are in contact with the second virtual surface 202 and a screen example in that state. In this screen 10, a finger cross-sectional cursor 450 is displayed at a position on the GUI in correspondence with the positions and cross sections of a plurality of fingers on the virtual plane. For example, the three fingers are in contact with the second virtual surface 202. An example of GUI operation on this screen 10 is shown below. Examples of operations include a pinch operation. The user performs, for example, a pinch-in or pinch-out operation on the image on the screen 10 or the like. For example, the user makes a state in which two fingers are in contact with the second virtual surface 202, and moves the two fingers so as to open and close in a desired direction in the X direction or the Y direction. This movement is determined as a pinch operation. The GUI display unit 14 performs image enlargement / reduction processing in accordance with these operations.

Also, as an application, the number of fingers in the finger section may be used for operation control. For example, when it can be determined that there are two fingers from the finger cross-section, it can be determined as a pinch operation.

[Space shape]
FIG. 13 shows a detailed example of the shape of the space 100 as a supplement to the first embodiment. The space 100 used for the remote operation control may be a substantially quadrangular pyramid-shaped space 100 having points B1 and E2 of both eyes as two vertices as shown in FIG. Similarly, the virtual plane space is set so as to be within the space 100. Both the space 100 in FIG. 1 and the space 100 in FIG. 13 can be used with almost no problem in terms of accuracy. The space 100 is not limited to the above shape, and may be, for example, a cone having a rectangle near the face or head and having the rectangle as four vertices.

[Predetermined operation]
FIG. 14 is an explanatory diagram for explaining details of determination of a predetermined operation, and shows an XZ plane in a state where the space 100 is viewed from above. As examples of the finger position point F0, points f0 to f5 are shown. In addition, an operation control surface 221 is provided at a position Za at a predetermined distance 321 from the position Z2 of the second virtual surface 202 in the Z direction. A surface 222 for operation control is provided at a position Zb at a predetermined distance 322 behind the position Z2 of the second virtual surface 202. The virtual surface operation determination unit 24 may use only the second virtual surface 202 or the second virtual surface 202 and the surface 221 when determining the touch operation. In addition, the virtual surface operation determination unit 24 may use the second virtual surface 202 and the surface 222 when determining a touch operation or a tap operation.

Note that, as a concept of the virtual plane space, the plane 221 at the position Za may be considered as the third virtual plane, and the plane 222 at the position Zb may be considered as the fourth virtual plane. The virtual surface space may be considered to be composed of a plurality of virtual surface layers. The virtual surface space is a space that enables an operation of moving a finger back and forth with respect to an invisible virtual surface in the space 100 as a difference from an existing touch panel. The display device 1 feeds back the remote operation state to the user by controlling the display of the cursor or the like on the screen 10 in real time according to the movement.

Referring to FIG. 14, an example of a predetermined operation that can be performed as a remote operation on the virtual surface of the GUI of the screen 10 in the display system will be described below. Various predetermined operations can be controlled peculiar to the positional relationship with the virtual plane and the degree of entry. In the display system, various operations are determined according to the distance DST between the point F0 of the finger position and the position of each virtual surface in the virtual surface space. In FIG. 14, the distance in a state where the position of the finger point F0 is at the point f2 in the second space 102 is indicated by a distance DST.

(1) Conventional touch panel touch operation (long press operation) and mouse click (hold) operation can be realized as the following specific touch operations in this display system. This touch operation is generally an operation of bringing a finger into contact with the second virtual surface 202. Specifically, this touch operation is performed by maintaining the finger point F0 at a position within the range from the position Z2 to the surface 221 of the position Za, or at a position within the range from the position Z2 to the surface 222 of the position Zb. It is an operation to maintain.

When the position of the finger point F0 is at the point f3 in contact with the second virtual surface 202 from the point f2 in the second space 102, or the surface 221 of the distance 321 in the third space 103 is displayed. If it is within the range up to, it is determined as follows. The movement from the point f2 to the point f3 can be associated as a touch operation on the object when the positions in the X direction and the Y direction are positions on the GUI object. In terms of the condition of the distance DST, the negative value of the distance DST is 0 or more and 321 or less. Whether to use this condition can be set according to the type of object.

In addition, the display device 1 does not associate with the touch operation or the like when the position of the finger in the X direction and the Y direction is a region such as a background region in the GUI, in particular, a region without an object. At that time, the cursor display state is maintained, for example, the same state as the cursor display state when the second virtual surface 202 is touched.

Furthermore, when the position of the finger point F0 enters the back from the second virtual plane 202, for example, within the range from the plane 221 to the plane 222, for example, when the position of the point f4 is reached, Judge as follows. The movement from the point f2 to the point f4 can be associated as a touch operation on the object when the positions in the X direction and the Y direction are positions on the GUI object. In terms of the condition of the distance DST, the negative value of the distance DST is larger than the distance 321 and smaller than or equal to the distance 322. Similarly, in this state, it can be associated as a touch operation of the object according to the type of the object. For example, in the case of the option button in FIG. 10, the touch operation is determined using any of the above conditions.

Also, for example, when the object is a volume adjustment button, the user moves the finger back and forth within the range between the position Za and the position Zb to determine that the operation is a touch operation and adjust the volume according to the distance DST. Can be associated.

The display device 1 may count time when determining the touch operation. The display device 1 may determine that the operation is a long press operation when a state where the position of the finger point F0 is within the range of the position Z2 and the position Za continues for a predetermined time, for example.

Cursor control examples are as follows. The surface 222 at the innermost position Zb is treated as the fourth virtual surface. When the distance DST is within the range up to the surface 222, the display device 1 displays the cursor on the screen 10, and when it exceeds the surface 222, the display device 1 hides the cursor and invalidates the remote operation. . When the finger returns to the front of the surface 222, the cursor is returned to the display state.

(2) With regard to conventional touch panel tap operations and mouse click operations, this display system can be implemented as the following specific tap operations. This tap operation is generally an operation of pushing a finger so as to strike the second virtual surface 202 and immediately returning it. As the transition of the position of the finger point F0, for example, from the point f2 in the second space 102, the point f3 passing through the second virtual plane 202 and within the range of the distance 321 (or the point f4 within the range of the distance 322). And returns to a position in the second space 102 through the second virtual plane 202 within a short time from the point f3. In that case, the movement can be associated as a tap operation.

Same as above, but the conventional double tap and double click operations can be realized as specific double tap operations in this display system. The double tap operation is generally an operation in which the tap operation on the second virtual surface 202 is repeated twice within a short time.

(3) The conventional touch panel swipe operation and mouse drag operation can be realized as the following specific swipe operations in this display system. In general, the swipe operation is performed in a desired direction in the X direction and the Y direction while touching the point F0 of the finger within the distance 321 (or within the distance 322) of the second virtual plane 202. This is a sliding operation. For example, when moving the object, the user pushes the finger at the position on the object on the screen 10 in the second virtual plane 202 and beyond, moves it to a desired position in the X direction and the Y direction, and then moves the finger to the first position in front. 2 Return to a position in the space 102. The display device 1 determines and detects the swipe operation, and the GUI display unit 14 moves the object in accordance with the swipe operation. The same applies to page scrolling and the like.

(4) The conventional touch panel flick operation can be realized as follows as a specific flick operation in this display system. In general, this flick operation is an operation in which a finger is pushed to a position behind the second virtual plane 202 and then returned to a position in the second space 102 while quickly moving in a desired direction in the X direction and the Y direction. It is. For example, in the case of page scrolling, the GUI display unit 14 determines the amount of movement of the page or the like according to the speed of finger movement and the amount of change during the flick operation, and transitions the page display state.

(5) The conventional pinch operation of the touch panel can be realized as follows as a specific pinch operation in this display system. In general, this pinch operation is performed in a desired direction in the X direction and the Y direction while keeping two points of fingers at a position within the range of the distance 321 (or distance 322) behind the second virtual plane 202. This is an operation to open and close the two points. The user, for example, opens two fingers to pinch out or closes two fingers and pinches in with two points entering the range of the distance 321 of the second virtual plane 202 at the same time. Is returned to the second space 102. The GUI display unit 14 can associate the pinch-out and pinch-in operations with the enlargement / reduction processing of the image on the screen 10.

[Virtual surface adjustment]
FIG. 15 shows the virtual surface adjustment for each user in the first embodiment. The display device 1 has a function of setting a virtual plane space suitable for the individual user as one of the functions. In this function, it is possible to adjust the position of each virtual surface to shift back and forth on the reference axis J0 based on the standard virtual surface space, and the thickness M of the virtual surface space can also be adjusted. With the shift of the position of the virtual surface, the size of the virtual surface is also adjusted to a size that fits in the quadrangular pyramid shape of the space 100. As described above, the individual user can be identified using the individual recognition unit 26.

(A) of FIG. 15 shows the standard position etc. in a standard virtual surface first. This virtual surface information is preset as a default value. As standard positions, lengths L1 and L2 having a predetermined length and a thickness M are shown. The virtual plane has a standard size on the XY plane and has a size that matches the ratio of the screen 10. If the user is a new user and the virtual surface information is not registered, a standard virtual surface is set using default values.

(B) of FIG. 15 shows an example of the state of the virtual surface after adjustment with respect to (A). For example, when the user is a child, the position is adjusted so as to be closer to the position in the Z direction of the standard virtual surface for adults. In addition, along with the adjustment of the position in the Z direction, the size of the virtual plane is also reduced to match the space 100. As described above, the virtual plane can be adjusted for each individual user using the virtual plane adjustment unit 28 as the user setting. The user can register a face image or finger image for personal recognition, a user ID, or the like in advance, and can set the position and size of the virtual plane suitable for him.

The position Z2 of the second virtual surface 202 in (A) is shifted to the position Z2s so as to approach the point P0 by a distance of 331 on the reference axis J0 in (B). The center point C2 is the center point C2s. The predetermined length L2 is changed to the length L2s. Similarly, the position Z1 of the first virtual surface 201 is shifted to the position Z1s on the reference axis J0, and the center point C1 of the first virtual surface 201 is the center point C1s. The predetermined length L1 is changed to the length L1s. As the position is shifted, the thickness M of the virtual surface space is changed to the thickness Ms. A second virtual surface 202s is set at a size that fits in the space 100 perpendicular to the center point C2s. A first virtual surface 201s is set at a size that fits in the space 100 perpendicularly to the center point C1s.

The distance 331 is an offset distance for adjustment and can be set by the user. In the above adjustment, the adjusted set value may be obtained by applying a calculation for multiplying the default value by the adjustment coefficient. Adjustment for shifting the virtual plane to the position on the back on the reference axis J0 is also possible.

As a modification, the following virtual surface adjustment may be performed. The display device 1 determines a rough classification such as male / female or adult / child regarding the target user based on the analysis of the camera-captured image. The display device 1 automatically applies a virtual surface having a position and size set for each classification as setting information in advance according to the classification.

For the virtual plane space, it is preferable that the position on the reference axis J0 can be selected and adjusted according to the length of the arm of the individual user. For example, in general, adults have longer arms than children, and men have longer arms than women. It is preferable that the virtual plane space is set at a suitable position for each individual user of the family. This display system can set a suitable virtual plane for each individual user using the adjustment function. The distance between the point P0 and the virtual plane is set to an appropriate length. Thereby, the usability for each user is improved. In this display system, for example, even when an adult and a child have the same standard virtual plane space, the usability is slightly different due to the difference in arm length and the like, but no major problem occurs. The standard virtual surface may be set using a statistical value such as an average value of the user group.

[Effects]
As described above, according to the first embodiment, the user's remote operation on the screen 10 of the display device 1 can be controlled, and the required learning amount is small, and the user's usability can be improved. According to Embodiment 1, there is no need for the user to distinguish and memorize a plurality of gestures for the use of remote operation, the learning amount is minimal, and remote operation is possible by simple finger movement. Yes, it is easy to use. The user can easily perform an operation input to the screen 10 of the television receiver or the like as a remote operation in the state of being empty without using a remote control device. The user only needs to perform an operation according to the conventional mouse or touch panel operation on the virtual surface space, which is convenient. According to the first embodiment, a virtual plane is configured at a suitable position in front of the user's line of sight, and feedback by cursor display is performed according to the position of the finger. Thus, the user can intuitively and smoothly perform remote operation on a space where there is no object.

In the prior art example, the ease of remote operation in the depth direction between the user's viewpoint and the screen is not sufficiently considered. On the other hand, in the first embodiment, a unique touch operation or the like in the depth direction is realized, and the ease of remote operation or the like is sufficiently considered.

Further, according to the first embodiment, since it is not necessary to distinguish and detect a plurality of gestures based on the analysis of the camera-captured image, the processing load on the computer can be relatively reduced, which is advantageous in implementation. In the display device 1 according to the first embodiment, a virtual surface space is set, and a process of detecting a predetermined operation is performed by monitoring the degree of finger entry in the virtual surface space. This processing is less computationally intensive than the processing for distinguishing and detecting a plurality of gestures in the prior art example.

The following are examples of modifications of the first embodiment. First, two cameras may be integrated as one camera having an advanced function of measuring the distance of an object. The processor of the display device 1 performs processing using data from the camera.

[Modification (1)]
FIG. 16 shows the arrangement of two cameras, the space 100, and the like in the display system of the first modification of the first embodiment. In addition, FIG. 16 illustrates a case where the user is viewing the screen 10 in a posture lying on the floor and operates the virtual surface with the right hand as an example of the posture when viewing the user. Thus, the user's posture can change, but remote control is possible in each posture.

In the first modification, the

cameras

21 and 22 are arranged at the position of the upper right point Q1 and the upper left point Q2 with respect to the screen 10. In this arrangement, the camera-side reference point is basically the middle point between the two cameras, that is, the middle point Q7 on the upper side of the screen 10. Even in this arrangement, the virtual plane can be set and controlled as in the first embodiment. In the first modification, the virtual surface of the space 100 is basically as shown in FIG. 17A, and is set as shown in FIG. 17B by correction.

FIG. 17 shows the correction of the virtual surface in the first modification. FIG. 17A shows a state before correction. By analyzing the camera-captured image, the distance D3 between the point Q7 that is the camera side reference point and the point P0 that is the user side reference point is grasped. The reference axis J0 used for setting the virtual plane is a straight line connecting the point Q7 and the point P0. On the reference axis J0, the first virtual plane 201a is set as a vertical plane with the point C1a at the position of the predetermined length L1 from the point P0 toward the point Q7. Similarly, the second virtual surface 202a is set at the point C2a at the position of the length L2. The size of the second virtual surface 202a is a size with a predetermined ratio corresponding to the size of the screen 10.

Note that the first virtual surface 201a and the second virtual surface 202a before correction may be used as they are for control. When the viewing position of the user is sufficiently away from the screen 10, there is almost no problem in terms of accuracy even if such a virtual surface is used.

(B) in FIG. 17 shows a virtual surface after correction from (A). In the first modification, the first virtual surface 201a and the second virtual surface 202a are corrected to obtain the corrected first virtual surface 201b and second virtual surface 202b. Several correction methods are possible, but the following is required when angular rotation is used. Using the point P0 that is the user side reference point as the rotation center point, the reference axis J0 and the virtual plane are rotated at an angle γ so as to be aligned with the center point Q0 of the screen 10. The angle γ is obtained by calculation using the position of the point P0, the length V1, and the like. The reference axis J0 after rotation is a straight line connecting the point Q0 and the point P0. The rotated virtual surfaces are the first virtual surface 201b and the second virtual surface 202b, and the center points thereof are points C1b and C2b.

The other correction methods may be as follows. The position and distance D1 of the point P0 from the center point Q0 of the screen 10 are obtained by calculation using the position and distance D3 of the point P0 from the point Q7 which is the camera side reference point and the length V1. Then, using the lengths L1 and L2 on the reference axis J0 from the point Q0 to the point P0, similarly, the first virtual surface 201b and the second virtual surface 202b, which are corrected virtual surfaces, are set. Also good.

As another correction method, only the second virtual plane is set first between the point Q7 and the point P0, and the second virtual plane is corrected to a position on the reference axis J0 between the point Q0 and the point P0. To do. Then, the first virtual surface is set at a position of a predetermined thickness before the corrected second virtual surface.

The arrangement of the two cameras is not limited to the above form, and various arrangements are possible. What is necessary is just to arrange | position two cameras in arbitrary positions on either side with respect to the center point Q0 of the screen 10. FIG. For example, the positions of the lower right point Q3 and the lower left point Q4 may be used. Further, three or more cameras may be arranged to improve the accuracy of distance measurement based on binocular parallax.

[Modification (2)]
FIG. 18 shows a second modification. In the second modification, a single virtual surface 200 is set in the space 100. On the reference axis J0, the virtual plane 200 is set to a size that fits in the space 100 around the point C0 at the position Zc of a predetermined length L0 from the point P0. The space 100 is divided into a first space 111 on the side closer to the user and a second space 112 on the back side by a single virtual plane 200. Operations such as finger touching and pushing on the virtual surface 200 are possible.

18 shows a display control example of the screen 10 according to the finger position. When the point F0 of the finger position is in the first space 111, for example, at the point Fa, for example, the screen GA is displayed. The screen GA has a GUI object on the menu screen or the like and displays a cursor K1. As in the first embodiment, the display device 1 changes the size of the cursor K1 and the like according to the distance DST that represents the positional relationship of the fingers with respect to the virtual surface 200.

When the point F0 of the finger position is in contact with the virtual plane 200, for example, at the point Fb, or passes through the virtual plane 200 and is in the second space 112, for example, at the point Fc, the position changes as shown in the screen GB, for example. . A movement from the point Fa to the point Fb or the point Fc is determined and detected as a touch operation. Operation input information 301 representing the touch operation is given to the GUI display unit 14. The GUI display unit 14 performs object correspondence processing in accordance with the touch operation, and updates the display state of the object and the cursor K1. Similarly, a tap operation or the like on the virtual surface 200 can be performed.

Further, even when a GUI object or the like is not displayed on the screen 10, a touch operation or the like on the entire virtual surface 200 can be performed. For example, in a state where nothing is displayed on the screen 10, the virtual surface 200 is set based on camera shooting, and a touch operation on the virtual surface 200 is accepted. The touch operation is associated as a predetermined operation instruction to the display device 1, for example, power on (switching the display function to an on state). In that case, feedback such as cursor display control may be omitted.

Furthermore, the entire XY plane of the virtual surface 200 may be divided into a plurality of regions. For example, it may be divided into left and right regions. And the touch operation etc. with respect to each area | region are received. For example, a first operation instruction is associated with a right region tap operation, and a second operation instruction is associated with a left region tap operation.

[Modification (3)]
FIG. 19 shows a state where the screen 10 and the virtual surface overlap in the user's field of view in the third modification. In the third modified example, the setting and adjustment of the virtual plane are limited to a part of the entire XY plane of the screen 10. In FIG. 19, the first virtual plane 201 and the second virtual plane 202 which are virtual planes are set to overlap the right half area 191 in the entire XY plane of the screen 10. In this virtual surface area 191, remote operation such as touch operation is accepted as valid, and the cursor K1 is also displayed. In the left half area 192, remote operation is not accepted as invalid and the cursor K1 is not displayed. Further, a frame or the like indicating the area 191 may be displayed on the screen 10.

For example, when it is desired to remotely control only a partial area of the GUI screen, unnecessary operations outside the partial area can be omitted by setting the virtual plane in this way.

(Embodiment 2)
A display system including the remote control device according to the second embodiment of the present invention will be described with reference to FIG.

FIG. 20 shows a functional block configuration of the display system of the second embodiment. The display system according to the second embodiment is a system in which the display device 1 and the remote control device 3 are connected. The remote operation control device 3, which is an independent device different from the display device 1, has the function of the remote operation control unit 20 of the first embodiment. The remote operation control device 3 controls a user's remote operation on the GUI or the like of the screen 10 of the display device 1. The remote operation control device 3 generates operation input information 301 for remote operation and transmits it to the display device 1. The display device 1 controls the GUI and the like of the screen 10 based on the operation input information 301 as in the first embodiment.

The display device 1 includes a communication unit 16 in addition to the control unit 11 and the like similar to the components in FIG. The communication unit 16 receives the operation input information 301 from the communication unit 33 of the remote operation control device 3 and gives it to the GUI display unit 14 and the like.

The remote operation control device 3 includes similar components corresponding to the remote operation control unit 20 of FIG. 2 and includes a control unit 31, a storage unit 32, a communication unit 33, and the like. The control unit 31 controls the entire remote operation control device 3. The storage unit 32 stores control information and data. The communication unit 33 performs communication processing with the communication unit 16 of the display device 1. The communication unit 16 and the communication unit 33 are parts including a communication interface device corresponding to a predetermined communication interface. The communication unit 33 transmits the operation input information 301 output from the operation input information output unit 25 to the communication unit 16 of the display device 1.

In the second embodiment, remote control can be realized as in the first embodiment. In the second embodiment, it is not necessary to mount a remote control function on the display device 1, and an existing display device can be used. Various display devices 1 can be connected to the remote operation control device 3 as necessary to constitute a display system.

(Embodiment 3)
A display device and the like according to the third embodiment of the present invention will be described with reference to FIGS. The basic configuration of the third embodiment is the same as that of the first embodiment. In the following, the components of the third embodiment that are different from the first embodiment will be described. The display system according to the third embodiment has a function of saving power by controlling operations such as power on / off of the display device 1 in connection with remote operation control.

[Display system]
FIG. 21 is a perspective view showing a configuration of a display system including the display device 1 according to the third embodiment. The display device 1 includes a control board 600, a human sensor 601 and a main power supply unit 602 in a housing, which are connected by a communication line or a power line. FIG. 21 shows a case where two cameras are built in the housing of the display device 1, and the lens portion of the camera is exposed to the outside. 2 is mounted on the control board 600 as an electronic circuit.

The main power supply unit 602 supplies power to each unit. The human sensor 601 is basically always turned on. The two cameras and the display function portion of the display device 1 are normally in a power-off state, in other words, in a standby state.

The human sensor 601 detects the presence of a person in a predetermined range such as around the display device 1 using infrared rays or the like. When the presence sensor 601 detects the presence of a person, a detection signal is sent to the control board 600. The display device 1 turns on the power of the two cameras and turns on the display function according to the detection signal. Thereby, the camera starts shooting, and the display device 1 starts processing using the camera shot image. For example, the display device 1 performs user personal recognition processing based on the analysis of the face image of the captured image. At that time, the display device 1 may determine the brightness of the room based on a camera or another sensor. When the display device 1 determines that the room is dark and the brightness for the personal recognition process is insufficient, the display device 1 turns on the display of the screen 10, and the screen 10 displays a high-intensity image, for example, a high-intensity image. Control to display a menu screen with a background. This supports personal recognition processing. The display device 1 sets a virtual plane in the space 100 based on user personal recognition, and enters a mode in which remote operation is possible.

Further, the display device 1 may control the power-on of the display device 1 by a user's remote operation as follows. The user holds his / her finger over the screen 10 at a position near the front of the screen 10. The display device 1 detects the movement of the finger that has entered the space 100 based on the analysis of the camera-captured video. The display device 1 interprets the movement as a power-on operation instruction. As a result, power is supplied from the main power supply unit 602 to each unit constituting the display function, and the display function is turned on. The screen 10 displays a content or a GUI menu screen.

Even when the presence sensor 601 detects the presence of a person, the power-off state is continued if the user has not entered his / her finger into the space 100. The user can continue actions other than viewing. In addition to cleaning and maintenance of the main body of the display device 1, when the display device 1 has not been used for a long period of time, the main power supply unit 602 completely shifts the display system including the human sensor 601 to the power-off state. .

The human sensor 601 detects that a person has entered a predetermined range such as around the display device 1 or a room entrance. When the camera is turned on in response to human detection, the camera captures a predetermined range and outputs a captured video signal. The display device 1 detects whether there is a user within a predetermined range based on the analysis of the captured image. When there is a user within a predetermined range, the display device 1 determines that the user is highly likely to use the display function, and turns on the display function. That is, power supply from the main power supply unit 602 to each unit of the control board 600 is started.

Alternatively, the following procedure may be used. When a user is detected based on the camera, only the remote control function is turned on and the display function is kept off. The screen 10 is not displayed. The display device 1 determines a predetermined operation by the user using the remote control function and the camera-captured video. The predetermined operation is a predetermined operation for turning on the display function. Although this predetermined operation can be arbitrarily defined, it is, for example, an operation of bringing a finger into the virtual surface space of the space 100, a touch operation of the second virtual surface 202, or the like. In addition, the display device 1 may detect a predetermined operation when a finger is in the virtual plane space for a predetermined time or longer.

[Display device]
FIG. 22 shows a functional block configuration including hardware such as an internal electronic circuit of the display device 1 according to the third embodiment. The display device 1 includes a first antenna 500, a second antenna 501, a tuner circuit 502, a demodulation circuit 503, a video / audio data signal separation circuit 504, a data expansion circuit 505, a camera signal input circuit 510, an image memory 511, an MPU (microprocessor). Unit) 520, nonvolatile data memory 521, video input circuit 530, graphics circuit 540, liquid crystal driving circuit 550, switch 560, display panel screen 10,

cameras

21, 22, human sensor 601, main power supply unit 602, power plug 603 etc. An external PC 700 or the like can be connected to the display device 1.

The MPU 520 performs overall control processing of the display device 1. The MPU 520 handles various processes such as personal recognition, face and finger distance measurement, virtual surface setting, operation determination on the virtual surface, GUI display control, and the like as shown in FIG.

The non-volatile data memory 521 stores control data and information. The non-volatile data memory 521 stores registered face image data for individual user recognition, virtual surface information for each individual user, and the like.

The display device 1 also has a signal line and a power supply line that connect each part. The power supply line 651 is a power supply line for supplying power from the main power supply unit 602 to an electronic circuit such as the MPU 520 in the control board 600 and the human sensor 601. The power supply line 652 is a power supply line for supplying power from the main power supply unit 602 to the

cameras

21 and 22. Control signals and the like are transmitted and received through the signal lines 610 to 613 and the like.

The display device 1 has a display function similar to that of a general television receiver as a basic function. The first antenna 500 is a terrestrial digital broadcast television antenna. The second antenna 501 is a satellite TV antenna. The display device 1 detects a television signal received by the first antenna 500 and the second antenna 501 by the tuner circuit 502 and demodulates the signal by the demodulation circuit 503. The demodulated signal is separated into video, audio, and data signals by a video / audio data signal separation circuit 504. In particular, since the signal is sent in a format in which an enormous amount of data is compressed, it is necessary to expand the signal to match the time scale of a certain synchronization signal. Therefore, the data decompression circuit 505 performs the decompression process. A video signal sent from the external PC 700 or the like is converted into an appropriate format by the video input circuit 530 and transmitted to the graphics circuit 540.

The camera signal input circuit 510 inputs an image signal of a captured video obtained from the

cameras

21 and 22. In the camera signal input circuit 510, the image signal of the captured video data is converted into an image signal of a predetermined format that can be easily recognized and stored in the image memory 511 side by side. An image signal of the captured video data is supplied from the image memory 511 to the MPU 520 through the signal line 610.

The MPU 520 performs the various processes described above using the image signal in the image memory 511. The MPU 520 extracts feature data of a human face, for example, a pattern such as an outline, eyes, nose, and mouth from an image signal, for example. Further, the MPU 520 extracts feature data of arms and fingers from the image signal. The MPU 520 performs personal recognition processing, distance measurement processing based on binocular parallax, and the like using the extracted feature data.

In the personal recognition process, the MPU 520 compares the extracted facial feature data with the registered facial feature data stored in the nonvolatile data memory 521 to identify the individual user. The MPU 520 determines that the subject person corresponds to the individual user of the registered face feature data when there is registered face feature data having a certain degree of similarity or more with the face feature data of the subject person.

The MPU 520 reads virtual surface information and the like corresponding to the identified individual from the nonvolatile data memory 521. The virtual surface information is information for setting a virtual surface space for each individual user in the space 100 described above, and includes setting values related to the lengths L1, L2, and the like. The MPU 520 reads standard virtual surface information from the nonvolatile data memory 521 when the individual user cannot be identified by the personal recognition process. This information includes default values for lengths L1, L2, etc.

The human sensor 601 detects the presence or absence of a person in a predetermined range around the display device 1. The human sensor 601 provides a detection signal indicating human detection to the MPU 520 through the signal line 612 when a person enters the predetermined range. When the MPU 520 receives a detection signal representing human detection from the human sensor 601, the MPU 520 gives a control signal to the switch 560 through the signal line 613, and switches the switch 560 from the off state to the on state. As a result, power is supplied from the main power supply unit 602 to the two cameras through the

power supply lines

651 and 652, and the two cameras are turned on.

The MPU 520 selects and reads out the high-intensity image signal from the nonvolatile data memory 521 when the brightness of the room is less than the brightness necessary for personal recognition based on the image signal from the image memory 511. The MPU 520 sends the high luminance image signal to the graphics circuit 540 through the signal line 611. The graphics circuit 540 controls the liquid crystal driving circuit 550 based on the high luminance image signal, and a high luminance image is displayed on the screen 10 by driving from the liquid crystal driving circuit 550. The high luminance image signal may be data of a menu screen having a high luminance background. The high-intensity image signal may include a message such as “Please brighten the room lighting”. The message is superimposed on the screen 10 and displayed.

When the human sensor 601 detects that the user has left the display device 1 and moved out of the predetermined range, the MPU 520 starts measuring time using an internal timer (not shown). The MPU 520 automatically turns off the power of the display device 1 if a person does not enter the predetermined range again after a predetermined time has elapsed. That is, the display device 1 has a so-called auto shut-off function.

[First processing flow]
FIG. 23 shows a first processing flow of the display device 1 according to the third embodiment. The processing in FIG. 23 is mainly performed by the MPU 520. FIG. 23 includes steps S1 to S6. Hereinafter, it demonstrates in order of a step.

(S1) The processing in FIG. 23 is started based on a detection signal from the human sensor 601. Based on the detection signal, the

cameras

21 and 22 are turned on to start imaging.

(S2) The display device 1 determines from the video captured by the camera whether the lighting state of the room is sufficiently bright for the camera to capture the user's face and head. If the brightness is sufficient (Y), the process proceeds to S4.

(S3) If the brightness is insufficient (N), the display device 1 displays a high-luminance video on the screen 10 in S3. Alternatively, a message or the like is displayed for the user to brighten the room.

(S4) The display device 1 performs a remote operation control process using the camera captured image. This process is shown as a second process flow in FIG.

(S5) The display device 1 determines whether to end the remote operation control of the display system. For example, when the user inputs an end operation, when the absence of the user is detected from the captured video of the camera, or when the presence of a surrounding person is detected by the human sensor 601, the display device 1 ends the display system. It judges that it will make it, and it changes to S6. If not completed (N), the process of S4 is repeated.

(S6) The display device 1 turns off the display function from the main power supply unit 602 and the power supply to the two cameras, and puts the display device 1 into a standby state. After the end, the same is repeated from S1.

[Second processing flow]
FIG. 24 shows a second processing flow of the display device 1 according to the third embodiment, which is mainly processing by the MPU 520. FIG. 24 includes steps S11 to S22. Hereinafter, it demonstrates in order of a step.

(S11) The display device 1 detects the user's face and fingers from the camera-captured video, and detects the distance and position to the point P0 and the point F0 by distance measurement processing based on binocular parallax.

(S12) The display device 1 extracts a facial feature in the facial image from the camera-captured video, compares it with the registered facial feature data, and performs personal recognition processing for identifying the individual user.

(S13) If the individual user can be identified (Y), the process proceeds to S14, and if the user cannot be identified, the process proceeds to S15.

(S14) The display device 1 refers to the virtual surface information set for each individual user from the nonvolatile data memory 521.

(S15) The display device 1 refers to the standard virtual surface information, which is the default value, from the nonvolatile data memory 521. Even when the virtual surface information for each user is not registered, the standard virtual surface information is referred to. The virtual plane information includes lengths L1 and L2.

(S16) The display device 1 uses the virtual plane information to move the virtual plane from the point P0, which is the user side reference point, to the position of the predetermined lengths L1 and L2 on the reference axis J0 toward the point Q0. First virtual surface 201 and second virtual surface 202 are set.

(S17) The display device 1 calculates the distance DST between the position of the finger point F0 and the second virtual surface 202, and the like. The display device 1 uses the distance DST to determine the degree and depth of the finger entering the virtual surface space.

(S18) The display device 1 determines whether or not the finger has entered the second space 102 behind the first virtual plane 201. If it has entered the second space 102 (Y), the process proceeds to S19, and if not (N), the process proceeds to S20.

(S19) The display device 1 displays the cursor in a state corresponding to the distance DST in the menu screen of the screen 10 in correspondence with the position of the finger. At that time, the display device 1 gives operation input information 301 including cursor display control information to the GUI display unit 14.

(S20) The display device 1 determines whether or not the finger has entered the third space 103 behind the second virtual plane 202. If it has entered the third space 103 (Y), the process proceeds to S21, and if not (N), the process ends.

(S21) The display device 1 determines a predetermined operation such as a touch operation on the second virtual surface 202 as shown in FIG. When the display device 1 determines that there is a predetermined operation, the display device 1 gives operation input information 301 including operation information representing the predetermined operation to the GUI display unit 14.

(S22) The display device 1 causes the GUI display unit 14 to perform corresponding processing according to a predetermined operation on the GUI object. Further, a prescribed operation of the display device 1 is executed according to the handling process. In the third embodiment, the GUI display unit 14 is realized by the MPU 520, and the MPU 520 generates the operation input information 301 and performs the GUI display control process by itself.

Examples of the operation of the display device 1 according to a predetermined operation include various operations that can be operated from an operation menu of an existing remote control device in addition to a general switching operation of video, audio, digital input, and the like. Is possible. By using the remote control of this display system, it is possible to give an operation instruction without using an existing remote control device.

As described above, according to the third embodiment, remote control can be realized without using a remote control device, and power saving can also be realized.

(Embodiment 4)
A display device and the like according to the fourth embodiment of the present invention will be described with reference to FIGS. In the fourth embodiment, a projector (projection display device) is applied as the display device 1. The projector has a function of projecting and displaying an image on the screen 10 of the screen 250 based on digital input data or the like.

In the display system of FIG. 25, the display device 1 is installed on the ceiling by the mounting tool 251 and performs projection display on the screen 250 in front of it. In addition, two cameras are provided in the housing of the display device 1.

Cameras

21 and 22 are arranged at the left and right points Qa and Qb of the housing. The orientations of the two cameras are set so that the photographing range is the lower part from the ceiling and the front part from the screen 250, and the orientations can be adjusted. Images taken from the

cameras

21 and 22 are images from slightly above the user.

25 shows a case where there are users A, B, and C as a plurality of users who view the projected video on the screen 10 of the screen 250. A state in which user A performs a remote operation on the virtual surface space of the space 100 as in the first embodiment is shown. Users B and C on both sides of the user A are watching the projected video on the same screen 10. User B has long bangs and user C is wearing a hat with a brim.

FIG. 26 shows an XZ plane corresponding to a photographed image obtained by viewing the space 100 below from the two cameras of the ceiling display device 1 corresponding to the scene of FIG. Similar to the first embodiment, it is possible to detect the user-side reference point and finger position of each user by distance measurement based on binocular parallax in the analysis of camera-captured images. In FIG. 26, the virtual surface space 102a is set in the space 100a of the user A. Similarly, a virtual plane space 102b is set in the user B space 100b. A virtual surface space 102c is set in the space 100c of the user C.

In Embodiment 4, the camera near the ceiling is located above the user's head, but the principle of distance measurement based on binocular parallax can be applied in the same way. However, in the fourth embodiment, the user's face, both eyes, and the like are hardly captured in the camera-captured image. In the photographed image as shown in FIG. 26, both eyes of the face of the user B are hidden by the bangs. User C's face is hidden by a hat. When both eyes are shown, a user-side reference point can be set as in the first embodiment. Even when both eyes are not captured, the user-side reference point can be set by a predetermined method. For example, when a skin color portion of the face can be detected, a user-side reference point may be set for the portion. Further, even when it is difficult to detect a face, a part such as a head, hair, or hat may be detected, and a user-side reference point may be set, for example, at the tip position in the Z direction with respect to that part. In the fourth embodiment, when analyzing the camera-captured video, rather than the face contour and both eyes, the characteristics of the head and body viewed from above are extracted. For example, one point of the tip of the head is point P0. Set as.

The display device 1 also knows in advance the positional relationship between the camera-side reference point near the ceiling and the center point Q0 of the screen 10. A reference axis J0 can be set between the point Q0 and the point P0, which is the user-side reference point of each user, and a virtual plane can be set on the reference axis J0. Also in the fourth embodiment, it is possible to set a virtual plane suitable for the length of an individual arm or the like based on registration data for each individual in accordance with the individual recognition of each user.

In the fourth embodiment, it is easy to determine the state in which the user extends and retracts his / her arm back and forth in the Z-direction screen 10 in the camera-captured video. The display device 1 may perform a remote operation control process in accordance with the determination of the extension / contraction of the user's arm.

As a method of using this display system in a meeting or the like, a cursor corresponding to the position of the finger point F0 can be projected and displayed on the point Kx on the screen 10 as a substitute for a conventional laser pointer or the like. The user can point a portion to be pointed on the screen 10 with the cursor in accordance with the remote operation in the virtual plane space. Designation of the slide material currently projected, etc. can also be realized as a remote operation for a predetermined GUI.

In the conventional technology, when multiple people including viewers and people other than the viewer are shown in the camera video, the movement of people other than the viewer acts as a disturbance, making it difficult to detect the viewer's gesture. , There is a possibility of misoperation due to the movement of people other than the viewer. The person other than the viewer includes, for example, a child playing in the same room, a person passing by, a person who is not watching the screen, and a person who is not aware of the operation. According to the fourth embodiment, even in an environment where there are a plurality of people, it is possible to perform a suitable remote operation with few erroneous operations.

(Embodiment 5)
A display device and the like according to the fifth embodiment of the present invention will be described with reference to FIG. In the fifth embodiment, the projector that is the display device 1 is installed on a table in front of the screen 250. Two

cameras

21 and 22 are arranged at points Qa and Qb which are positions on the left and right sides of the housing of the display device 1. The display device 1 projects and displays an image on the screen 10 of the front screen 250. The

cameras

21 and 22 capture a predetermined range including the user ahead from the screen 10. As described above, in the fifth embodiment, the remote operation can be realized as in the first embodiment.

(Embodiment 6)
A display device and the like according to the sixth embodiment of the present invention will be described with reference to FIG. In the sixth embodiment, as in the fifth embodiment, the projector that is the display device 1 is arranged on a table in front of the screen 250. The two cameras are arranged on the screen 250 at, for example, the upper right and upper left positions. A camera 21 is attached to the upper right point Qa, and a camera 22 is attached to the upper left point Qb. The housing of the display device 1 and the two cameras are connected by

wirings

281 and 282. The

wirings

281 and 282 are wired cables including signal lines and power supply lines. A camera image signal is transmitted to the display device 1 through

wirings

281 and 282.

Thus, in the sixth embodiment, remote control can be realized as in the first embodiment. The arrangement positions of the two cameras can be adjusted. In particular, the distance between the two cameras can be increased, and the shooting range can be increased. When the

cameras

21 and 22 having the wireless communication function are used, the

wirings

281 and 282 can be omitted.

(Embodiment 7)
A display device and the like according to the seventh embodiment of the present invention will be described with reference to FIG. The display device 1 according to the seventh embodiment has a basic configuration similar to that of the first embodiment, and additionally has a remote control function when a plurality of users simultaneously use remote operation.

In the example of FIG. 29, a plurality of users (for example, users A, B, C, D, and E) are viewing the screen 10 of the display device 1 in a meeting or the like. On the screen 10, for example, content such as slide material is displayed. This display system can also be used for a conference system and the like.

As in the first embodiment, a virtual plane of the space 100 is set for each of a plurality of users. In FIG. 29, a part of the virtual surfaces, for example, the second virtual surface 202 of the user B and the second virtual surface 202 of the user C are omitted.

The following controls are possible when multiple users are using the remote control simultaneously. The display device 1 gives an operation authority as a representative operator to only one user. Of the plurality of users, for example, the case where the user B first performed an operation on the virtual surface space at the first time point is shown. Other users are not touching the virtual surface space. The finger of the user B enters the first virtual surface 201 and performs a touch operation on the second virtual surface 202 or the like. The display device 1 gives the operation authority as the current representative operator to the user B who first enters the virtual plane space. Corresponding to the operation of the user B, a cursor 291 of the representative operator is displayed on the screen 10.

In addition, an image 292 representing the user B who is the representative operator who has the current operation authority and is performing remote operation is displayed at a part of the screen 10. As this image 292, a photographed image of the camera is used. For example, a trimmed image of a part of the captured image is used. The image 292 may be a registered face image for personal recognition, or may be other information such as an icon or a mark for each user. With the image 292, the situation of who the current remote operator is can be commonly recognized by all users.

For example, when two users B and C try to operate the virtual surface at the same time, there is a possibility of confusion if the remote operations of the two users are reflected at the same time. Therefore, when it is not desired to reflect the remote operations of a plurality of users at the same time, the operation authority is given only to a single user as described above. That is, the operation authority is given by the first-come-first-served basis. In addition, when a plurality of users try to perform remote operation almost simultaneously, a user who has not been given the operation authority may not be reflected in his / her remote operation, and may misunderstand that the system malfunctions. Therefore, an image 292 representing the user who has the current operation authority and is operating is displayed on a part of the screen 10.

Next, at the second time point, for example, it is assumed that the user C performs an operation on the virtual surface space of the user C. The finger of user C enters the virtual surface space through the first virtual surface 201. At this time, since the user B is operating as a representative operator, the display device 1 does not give the operation authority to the user C and invalidates the remote operation. On the screen 10, the cursor corresponding to the position of the finger of the user C is not displayed.

Next, when the user B's finger goes out of the virtual plane space and the space 100 and stops the operation at the third time point, the operation authority as the representative operator of the user B is released. Thereafter, similarly, an operation authority as a representative operator is given to one user who has entered the virtual plane space.

As a modification, the following control may be performed. Allows simultaneous remote operation of a plurality of users up to a predetermined number of users. A plurality of cursors for each user are displayed on the screen 10. In advance, it is acknowledged that simultaneous remote operation is allowed for each of a plurality of users. Each user performs remote operations in order as necessary. In this case, when the fingers of a plurality of users enter the virtual surface space, a plurality of cursors are displayed on the screen 10 and the display state becomes excessive, but this is useful depending on the usage method. There are no particular problems if a plurality of users cooperate with each other based on the usage method. For example, it may be used in an application that enables collaborative work.

When multiple users are viewing the screen at the same time, when enabling remote control of each user, which user remote control is accepted, how to control simultaneous remote control of multiple users, etc. Is also an issue. According to the seventh embodiment, a suitable remote operation is possible even when a plurality of users perform a remote operation.

As a modification, the following control may be performed. The display device 1 may set priorities among a plurality of users. When a plurality of users put their fingers on each virtual surface at the same time, for example, only the user with the highest priority is given the operation authority according to the priority setting. In addition, when a first user with a lower priority is remotely operated and a second user with a higher priority places a finger on the virtual surface, the operation authority is transferred from the first user to the second user. Good.

As described above, according to the seventh embodiment, simultaneous use of remote operation by a plurality of users can be realized smoothly.

[Determination of body continuity]
The display device 1 detects a face and a head portion and a finger and arm portion of the user based on the analysis of the camera photographed image. At that time, the display device 1 determines the continuity and identity of the user's individual body. That is, for example, it is determined whether or not the arms and fingers are continuously connected from the face area in the photographed image.

For example, in the case of simultaneous use by a plurality of persons shown in FIG. 29, the face in the camera-captured image and the finger in the virtual plane space may not belong to the same user during remote operation. For example, the user A may extend his arm from the side of the user B and touch the virtual surface of the user B with his / her finger. The display device 1 also considers such a possibility, and determines the continuity and identity of the user's individual body when analyzing the camera-captured image. If the display device 1 determines that the face and fingers in the camera-captured image are from different users, the display device 1 invalidates the remote operation.

[When you have an object in your hand]
When performing a remote operation in the display system of each embodiment, basically, a user can easily perform a remote operation with nothing in his hand. However, the present invention is not limited to this, and the user can also perform a remote operation while holding an object such as a stick in his hand. In this case, the display system detects not only a finger but also an object such as a stick during analysis of a camera-captured image, determines the continuity between the finger and the object, for example, determines the position of the tip of the object. A point to be represented is detected as a point F0. The display system may detect the color or shape of the object and use it for control.

As mentioned above, although this invention was concretely demonstrated based on embodiment, this invention is not limited to embodiment mentioned above, A various change is possible in the range which does not deviate from the summary. It is possible to add, delete, separate, merge, replace, and combine components of the embodiment. The numerical values in the specific examples of the embodiments are examples. In the drawing, some lines are shown as lines connecting components. A part or all of the functions and the like of the embodiments may be realized by hardware such as an integrated circuit, or may be realized by software program processing. Software constituting functions of each device may be stored in the device in advance at the time of product shipment, or may be acquired from an external device via communication after the product is shipped. The display system and display device of the embodiment are not limited to a television receiver and a projector, and can be applied as operation input means of various electronic devices.

DESCRIPTION OF SYMBOLS 1 ... Display apparatus, 10 ... Screen, 21, 22 ... Camera, 100 ... Space, 101 ... 1st space, 102 ... 2nd space, 103 ... 3rd space, 201 ... 1st virtual surface, 202 ... 2nd virtual surface , J0: reference axis, P0, Q0 to Q8, F0: point.

Claims

A display device having a function of controlling a user's remote operation on the screen of the display device,
Having at least two cameras that capture a range that includes the user viewing the screen,
A position of a first point that is a user-side reference point associated with the head, face, or eye of the user with respect to a position of a camera-side reference point of the two cameras by analysis of captured images of the two cameras; , Detecting the position of the second point of the user's finger,
In a space connecting the screen and the first point, a virtual plane space is set at a position of a predetermined length in the viewing direction from the first point to the screen so as to overlap the screen when viewed from the user. And
Calculating the degree of entry of the finger into the virtual surface space, including the distance between the position of the virtual surface space and the position of the second point;
Based on the degree of entry, determine a predetermined remote operation including a touch operation of the finger on the virtual surface space,
Operation input information including operation information representing the position of the second point or the position coordinates in the screen associated with the position of the second point, and the operation information representing the predetermined remote operation is generated, and the operation input information Controlling the operation of the display device;
Display device.
The display device according to claim 1,
According to the position of the second point and the degree of entry, control to display a pointer image representing the presence of the finger on the screen,
As the operation input information, including display control information of the pointer image,
Display device.
The display device according to claim 1,
When the distance is relatively long, the image is displayed in a relatively large size, a first color, or a first shape type, and when the distance is relatively short, Display in small size or second color or second shape type,
Display device.
The display device according to claim 1,
When analyzing the captured images of the two cameras, the position of the first point and the position of the second point are detected using the principle of distance measurement based on binocular parallax.
Display device.
The display device according to claim 2, wherein
The virtual surface space has a single virtual surface set at a position of a predetermined length from the first point,
When the finger is in front of the single virtual plane, the pointer image is displayed on the screen,
When the finger has entered the back of the single virtual surface, it is determined as a touch operation.
Display device.
The display device according to claim 2, wherein
The virtual plane space has a first virtual plane set at a position having a first length from the first point and a second virtual plane set at a position having a second length from the first point. And
When the finger is in front of the first virtual plane, the pointer image is not displayed on the screen,
When the finger has entered the back of the first virtual surface or later, the pointer image is displayed on the screen,
When the finger has entered the depth after the second virtual surface, it is determined as a touch operation.
Display device.
The display device according to claim 2, wherein
The virtual plane space has a first virtual plane set at a position having a first length from the first point and a second virtual plane set at a position having a second length from the first point. And
When the finger is in front of the first virtual plane, the pointer image is not displayed on the screen,
When the finger has entered the back of the first virtual surface or later, the pointer image is displayed on the screen,
When the finger has entered the back of the second virtual surface or later, the pointer image is not displayed on the screen.
Display device.
The display device according to claim 1,
The two cameras are arranged at left and right positions in the horizontal direction in the vicinity of the screen.
Display device.
The display device according to claim 1,
The two cameras are arranged at left and right positions in the horizontal direction, away from the screen.
Display device.
The display device according to claim 1,
Perform personal recognition of the user using the camera,
The virtual plane space is set using a standard value as the predetermined length when the user cannot be recognized as an individual, and when the user can be recognized as an individual, the user of the predetermined length according to the individual Set to the adjusted position using the set value,
Display device.
The display device according to claim 1,
The virtual plane space is set so as to overlap only a part of the entire area of the screen viewed from the user, the remote operation is enabled in the partial area, and the other area Is invalid,
Display device.
The display device according to claim 1,
In the virtual plane space, when the position of the finger is at a position overlapping the GUI object on the screen, if the distance is less than or equal to a predetermined value, the touch operation is determined, and the touch operation is further performed. When the distance returns to a value greater than a predetermined value from the state of, it is determined as a tap operation, the corresponding operation according to the touch operation or the tap operation is executed,
Display device.
The display device according to claim 1,
The position of the finger in the virtual surface space is determined to be a touch operation when the distance is equal to or smaller than a predetermined value, and the position of the finger in the virtual surface space is determined from the touch operation state in the virtual surface space. If it moves in a direction parallel to the swipe operation, it is determined that it is a swipe operation, and a corresponding process is executed according to the swipe operation.
Display device.
The display device according to claim 1,
The position of the finger in the virtual surface space is determined as a touch operation when the distance is less than or equal to a predetermined value, and detects a plurality of fingers in the virtual surface space in the state of the touch operation, When the position of each finger of the plurality of fingers moves in a direction parallel to the screen, it is determined as a pinch operation, and a corresponding process is executed according to the pinch operation.
Display device.
The display device according to claim 1,
A human sensor for detecting the presence or absence of a person around the display device;
When the presence of a person around the display device is detected using the human sensor, the camera is turned on to start shooting,
When the presence of the user in the vicinity of the screen or a predetermined operation by the user is detected based on a captured image of the camera, the display function of the display device is turned on.
Display device.
The display device according to claim 1,
When a plurality of users simultaneously use as the user, the virtual surface space is set for each user, and a finger is first entered into one of the plurality of virtual surface spaces of the plurality of users. Only one user who gave the remote control authority,
Displaying information representing the user having the authority at the present time on a part of the screen;
Display device.
A remote operation control device connected to a display device and having a function of controlling a user's remote operation on the screen of the display device,
Comprising at least two cameras for capturing a range including a user viewing the screen;
A position of a first point that is a user-side reference point associated with the head, face, or eye of the user with respect to a position of a camera-side reference point of the two cameras by analysis of captured images of the two cameras; , Detecting the position of the second point of the user's finger,
In a space connecting the screen and the first point, a virtual plane space is set at a position of a predetermined length in the viewing direction from the first point to the screen so as to overlap the screen when viewed from the user. And
Calculating the degree of entry of the finger into the virtual surface space, including the distance between the position of the virtual surface space and the position of the second point;
Based on the degree of entry, determine a predetermined remote operation including a touch operation of the finger on the virtual surface space,
Operation input information including operation information representing the position of the second point or the position coordinates in the screen associated with the position of the second point, and the operation information representing the predetermined remote operation is generated, and the operation input information Controlling the operation of the display device;
Remote control device.