JP6319220B2 - Transparent wearable terminal, data processing apparatus, and data processing system - Google Patents

Description

  The present invention relates to a transparent wearable terminal, a data processing device, and a data processing system, and more particularly to a technique for causing a data processing device to perform desired data processing using a transparent wearable terminal.

  Currently, a head-mounted display or glasses-type wearable device virtually displays a setting screen of another device to be operated, and the user moves the hand to operate the setting screen. It is possible to detect the movement of the hand, accept an operation corresponding to the content of the movement of the hand, and operate the other device. Further, in the wearable device, it is possible to identify the place where the user is gazing by monitoring the center position of the eyeball of the user wearing the device. For example, Patent Document 1 below proposes a means for quickly and stably converging an eyeball center position measurement method necessary for calculating a viewpoint position while considering individual differences in eyeball radii. As technology for accurately observing and calculating human gaze and focal length has developed in this way, functions that use user gaze information and focal length information are realized in head-mounted displays and glasses-type wearable devices. It is possible.

  For example, in Patent Document 2 below, the operation panel of the image forming apparatus is used as a virtual panel, which is virtually displayed to the user using a head-mounted display or glasses-type wearable device, and the user's hand is visually observed and observed. The operation corresponding to the movement of the user's hand on the virtual panel can be set.

  Also, in Patent Document 3 below, in a head mounted display, processing such as transmission of data specified according to the movement of the hand by the user is performed on an object specified by the user's line of sight. At this time, when a user views a device such as an image forming apparatus in the field of view, the image forming apparatus or the like is specified as a data transmission destination.

JP 2008-136789 A JP 2010-145860 A JP 2014-93036 A

  However, the invention of Patent Document 2 is limited to detecting the movement of the user's hand with respect to the displayed virtual panel and has low versatility. In the invention of Patent Document 3, a combination of operations is determined based on what is detected by a line of sight, options are displayed, and a user's hand movement with respect to the options is detected. However, the operation content can be specified only based on the movement of the user's hand with respect to the option.

  The present invention has been made to solve the above problems, and is based on a variety of user hand movements using a head-mounted display or glasses-type wearable device, particularly a transmissive wearable terminal. The operation content can be specified and set so that the user can intuitively operate the data processing device and allow other devices to be operated to perform desired data processing. .

A transmissive wearable terminal according to one aspect of the present invention is a transmissive wearable terminal that displays an image superimposed on a user's field of view,
A display unit that displays an image that can be viewed by the user;
A line-of-sight detector that detects the user's line of sight and focal length;
An imaging unit for capturing a user's field of view image;
Analyzing the captured video of the imaging unit, a data processing device reflected in the video, and a video analysis unit for identifying hand movements by a user;
A communication unit communicating with the data processing device;
The device state is acquired from the data processing device identified by the video analysis unit via the communication unit, and an icon indicating the device state is displayed at a position close to the data processing device in the user's field of view. A device status display control unit to be displayed by the unit,
Based on the detection result of the line-of-sight detection unit, a device specifying unit that specifies a data processing device that the user gazes among the data processing devices identified by the video analysis unit;
A data specifying unit for displaying a data selection GUI in the user's field of view, and specifying data selected by the user in the data selection GUI based on a hand movement by the user identified by the video analysis unit;
A processing specifying unit that specifies data processing according to a gesture of the user's hand identified by the video analysis unit;
Data processing that requests the data processing device specified by the device specifying unit to perform the data processing specified by the processing specifying unit on the data specified by the data specifying unit via the communication unit And a request unit.

  According to the present invention, it is possible to specify and set the operation content based on a variety of user hand movements as compared with the prior art, so that the user can intuitively operate the data processing apparatus and can be set as an operation target. It is possible to cause the apparatus to perform desired data processing. Thereby, the operativity of the data processing apparatus used as the said operation object can also be improved.

1 is a schematic diagram of a data processing system according to an embodiment of the present invention. It is a functional block diagram which shows the main internal structures of a transparent wearable terminal. It is a figure showing a user's field of vision typically. It is a figure showing a user's field of vision typically. It is a figure showing a user's field of vision typically. It is a figure showing a user's field of vision typically. It is a figure showing a user's field of vision typically. It is a flowchart which shows the data processing procedure in a data processing system. It is a detailed flowchart of an initial image display. It is a detailed flowchart of data specification. It is a detailed flowchart for device identification. It is a detailed flowchart of process specification.

  Hereinafter, a data processing system according to an embodiment of the present invention will be described with reference to the drawings. FIG. 1 is a schematic diagram of a data processing system 10 according to an embodiment of the present invention.

  The data processing system 10 includes a transparent wearable terminal 1 and a data processing device 2.

  The transmissive wearable terminal 1 is a terminal that can superimpose and display an image in the user's field of view, and is, for example, a head-mounted display or a glasses-type wearable terminal. The user wears the transmissive wearable terminal 1 on the head and virtually manipulates an image displayed on the field of view by hand. At this time, the transparent wearable terminal 1 analyzes the movement of the hand based on the operation by the user, and operates the data processing device 2 with the operation content corresponding to the content of the hand movement to perform desired data processing. Let it be done.

  The data processing device 2 is a device to be operated by the transparent wearable terminal 1. The data processing apparatus 2 is an image forming apparatus that prints document data and image data on a sheet, for example.

  The transparent wearable terminal 1 and the data processing device 2 are connected to a network 4 such as a LAN or the Internet and can communicate with each other.

  The transparent wearable terminal 1 and the data processing device 2 can directly communicate with each other by short-range wireless communication without using the network 4. Therefore, the data processing device 2 can directly receive data held by the transparent wearable terminal 1 from the transparent wearable terminal 1 and process the received data.

  FIG. 2 is a functional block diagram showing the main internal configuration of the transparent wearable terminal 1. The transmissive wearable terminal 1 includes a control unit 100, a display unit 101, a line-of-sight detection unit 102, an imaging unit 103, a video analysis unit 104, a communication unit 105, a memory 106, a device state display control unit 107, a device specification unit 108, and data specification. A unit 109, a process specifying unit 110, a data processing request unit 111, and an operation panel display control unit 112. These components are connected to each other by an internal bus 113.

  The control unit 100 governs overall operation control of the transparent wearable terminal 1.

  The display unit 101 displays an image that can be viewed by the user. In this embodiment, the display unit 101 is assumed to be a device that projects an image on the retina of the user. The display unit 101 projects an image on the retina of the user, thereby displaying the image generated by the transmissive wearable terminal 1 in front of the user as virtual reality.

  The line-of-sight detection unit 102 is a device that detects a user's line of sight and focal length. Specifically, the line-of-sight detection unit 102 detects the position (line-of-sight position) and focal length at which the person is gazing in the field of view in the direction in which the person is facing by detecting the orientation of the user's pupil. To do. The human visual field is about 90-100 degrees on the ear side, about 60 degrees on the nose side, about 60 degrees on the upper side, and about 70 degrees on the lower side in the horizontal direction of one eye with the pupil facing directly in front. It is said. A human can gaze at an object in the field of view by moving only his eyes.

  A known technique can be used to detect the line-of-sight position. For example, the line-of-sight position can be detected by photographing one eye of a person with a visible light camera (not shown) included in the transmissive wearable terminal 1 and detecting the movement of the pupil with the eye head as a reference point. Alternatively, near-infrared light is irradiated on the surface of one eye from a near-infrared LED light source (not shown) provided in the transmissive wearable terminal 1, and the near-infrared camera (not shown) provided in the transmissive wearable terminal 1 The reflected light (Purkinje image) and the pupil are photographed, and the line-of-sight position is detected from the relative positional relationship between the Purkinje image and the pupil center.

  The imaging unit 103 is a device that captures a user's field of view image, and is, for example, a small CCD camera.

  The video analysis unit 104 is a device that analyzes the video captured by the imaging unit 103 by, for example, pattern matching, etc., and identifies the data processing device 2, the user's hand movements, and gestures captured in the video. (Digital Signal Processor).

  The communication unit 105 is a device that wirelessly communicates with other devices such as the data processing apparatus 2. As a wireless communication system, wireless LAN, Bluetooth (registered trademark), ZigBee (registered trademark), or the like can be used.

  The memory 106 is a device including a nonvolatile memory such as a ROM or a flash memory and a volatile memory such as a RAM. The memory 106 stores an operation program of the transparent wearable terminal 1, data to be processed by the data processing device 2, and temporarily stores captured images of the imaging unit 103.

  The device state display control unit 107 is a module that displays an image on the display unit 101 and displays the device state of the data processing device 2 in the user's field of view as a virtual reality. Specifically, the device state display control unit 107 acquires device state information (hereinafter, simply referred to as device state) indicating the device state from the data processing device 2 identified by the video analysis unit 104 via the communication unit 105. To do. The device state display control unit 107 displays an icon indicating the device state in the vicinity of the data processing device 2 in the user's field of view.

  The device specifying unit 108 is a module that specifies the data processing device 2 that the user gazes according to the user's line of sight. Specifically, the device specifying unit 108 specifies the data processing device 2 to be watched by the user among the data processing devices 2 identified by the video analysis unit 104 based on the detection result of the line-of-sight detection unit 102. Thus, even when there are a plurality of data processing devices 2 in the user's field of view, if the user gazes at any one of them, the device specifying unit 108 specifies one data processing device 2. Will be. Thus, the user can select one data processing device from the plurality of data processing devices by line of sight.

  The data specifying unit 109 is a module that specifies data to be processed in the data processing device 2. Specifically, the data specifying unit 109 causes the display unit 101 to display the data selection GUI so as to overlap the user's field of view, and based on the movement of the user's hand identified by the video analysis unit 104, in the data selection GUI Identify the data selected by the user.

  The process specifying unit 110 is a module that specifies data processing to be performed by the data processing apparatus 2. Specifically, the process specifying unit 110 specifies data processing corresponding to the user's hand gesture identified by the video analysis unit 104.

  The data processing request unit 111 is a module that requests the data processing device 2 to perform data processing. Specifically, the data processing requesting unit 111 sends the processing specifying unit 110 for the data specified by the data specifying unit 109 to the data processing device 2 specified by the device specifying unit 108 via the communication unit 105. Request to perform the data processing specified by.

  The operation panel display control unit 112 is a module that projects an image on the display unit 101 and displays the operation panel for the data processing device 2 in the user's field of view as virtual reality. Specifically, the operation panel display control unit 112 detects that the user gazes at the data processing device 2 identified by the video analysis unit 104 based on the detection result of the line-of-sight detection unit 102. Or when it is detected that the user touches the icon displayed by the display unit 101 in the vicinity of the data processing device 2 in the field of view based on the movement of the user's hand identified by the video analysis unit 104 Then, the operation panel GUI of the data processing device 2 is displayed on the display unit 101 so as to overlap the user's field of view.

  The device status display control unit 107, the device specification unit 108, the data specification unit 109, the process specification unit 110, the data processing request unit 111, and the operation panel display control unit 112 are configured by dedicated hardware circuits. It is stored in the memory 106 as a computer program, and an unillustrated CPU operates according to the computer program, so that a device status display control unit 107, a device specification unit 108, a data specification unit 109, a processing specification unit 110, a data processing request It can also be configured by functioning as the unit 111 and the operation panel display control unit 112.

  Next, a data processing procedure in the data processing system 10 will be described. 3 to 7 are diagrams schematically showing a user's field of view when the data processing system 10 is used. FIG. 8 is a flowchart showing a data processing procedure in the data processing system 10.

  First, an initial image is displayed on the display unit 101 (S1). In this initial video display, as shown in FIG. 3, the data selection GUI 11 is displayed in the field of view of the user wearing the transmissive wearable terminal 1, and the device status of the device is displayed in the vicinity of the data processing device 2 in front of the eyes. An icon 21 is displayed (FIG. 3). Here, first, the initial video display will be described. FIG. 9 is a detailed flowchart of initial video display.

  When the main power switch of the transmissive wearable terminal 1 is turned on, the video analysis unit 104 analyzes the captured video of the imaging unit 103 and identifies the data processing device 2 captured in the video (S11). For example, the video analysis unit 104 captures the image of the imaging unit 103 by pattern matching between the image of the data processing device 2 captured in the captured video of the imaging unit 103 and the image of the data processing device 2 registered in the memory 106 in advance. The data processing device 2 shown in the video is identified.

  When the video analysis unit 104 determines that the data processing device 2 is in the user's field of view (YES in S12), the device state display control unit 107 is identified by the video analysis unit 104 via the communication unit 105. Information indicating the device status and the contents stored in the storage device built in the data processing device 2 is acquired from the data processing device 2 (S13), and the icon 21 indicating the device status is displayed in the data processing device within the user's field of view. 2 is displayed by the display unit 101 in the vicinity of 2 (S14). The data specifying unit 109 first displays an image indicating the storage device at a predetermined position in the user's field of view based on information indicating the content stored in the storage device built in the data processing device 2. The data selection GUI 11 is displayed on the display unit 101 (S15). The data selection GUI 11 and the icon 21 are visible when the display unit 101 projects an image on the user's retina.

  Even if the position of the data processing device 2 in the field of view changes, for example, when the user moves his head, the data specifying unit 109 does not move the data selection GUI 11 in the field of view by the display unit 101. The device state display control unit 107 causes the display unit 101 to display the icon 21 in the vicinity of the data processing device 2 following the movement of the data processing device 2 in the field of view. Thereby, the display of the initial video by the display unit 101 is completed.

  Returning to FIG. When the user performs an operation such as superimposing his / her hand on the display position of the data selection GUI 11 in the field of view, the data specifying unit 109 specifies data to be processed by the data processing device 2 (S2). Here, the data specifying process will be described. FIG. 10 is a detailed flowchart of data specification.

  The data specifying unit 109 causes the display unit 101 to display the data selection GUI 11 at a predetermined position in the user's field of view (S21). The data selection GUI 11 is displayed when the initial video is displayed. For example, as the data selection GUI 11, an image of a folder indicating a storage device built in the data processing device 2 is displayed (FIG. 3). At this time, when the user performs an operation such as superimposing his / her hand on the data selection GUI 11, the video analysis unit 104 analyzes the captured video of the imaging unit 103, and the movement of the hand 5 by the user reflected in the video Is identified (YES in S22), the data specifying unit 109 causes the display unit 101 to change the display of the data selection GUI 11 to a display that appears as if the folder has been expanded, and stores the data in the storage device built in the data processing device 2. Based on the information indicating the stored contents, icons indicating various data in the folder are displayed (S23). For example, as illustrated in FIG. 4, the data specifying unit 109 causes the display unit 101 to display an image indicating a state in which the image data 11 a and the text data 11 b are stored in the folder.

  The video analysis unit 104 continues to analyze the captured video of the imaging unit 103 and determines whether the user has selected data. For example, as shown in FIG. 4, when the user performs an operation of putting the hand 5 on the display position of the image data 11 a and putting the image data 11 a outside the folder image (YES in S 24), the data specifying unit 109 displays on the display unit 101 a display for moving the icon of the image data 11a in accordance with the movement of the hand 5 by the user based on the analysis result of the movement of the hand 5 by the video analysis unit 104 (S25). The image data 11a is specified as data to be processed by the data processing device 2 (S26).

  Returning to FIG. After S2, when the user gazes at the data processing device 2 in the field of view, the device specifying unit 108 selects the data processing device 2 that the user gazes on based on the detection result of the gaze detection unit 102. It is specified as a device to be processed (S3). Here, the apparatus specific process will be described. FIG. 11 is a detailed flowchart of device identification.

  The line-of-sight detection unit 102 detects the user's line of sight 6 and the focal length (S31). The device specifying unit 108 determines whether the user is gazing at the data processing device 2 identified by the video analysis unit 104 based on the detection result of the line-of-sight detection unit 102. When it is determined that the user is gazing at the data processing device 2 (YES in S32), the device specifying unit 108 communicates with the data processing device 2 via the communication unit 105, and from the data processing device 2 The gesture information 22 indicating the data processing that can be executed by the data processing device 2 and the corresponding gesture is acquired (S33). Then, the device specifying unit 108 changes the display of the icon 21 indicating the operation state of the data processing device 2 to the display unit 101 (S34), and displays the gesture information 22 in the vicinity of the data processing device 2 in the user's field of view. (S35). For example, as illustrated in FIG. 5, the device specifying unit 108 changes the display content of the icon 21 from “ready” shown in FIG. 4 to “data reception OK” (FIG. 5). Also, as the gesture information 22, information on two data processings, BOX storage and printing (part), is displayed on the display unit 101.

  When the position of the data processing device 2 in the field of view changes, for example, when the user moves his / her head, the device specifying unit 108 causes the gesture information 22 to follow the movement of the data processing device 2 in the field of view to perform data processing. The image is displayed on the display unit 101 in the vicinity of the device 2.

  Returning to FIG. When the user moves the hand 5 to perform a gesture, the video analysis unit 104 analyzes the gesture based on the video of the gesture captured by the imaging unit 103, and the process specifying unit 110 determines the gesture based on the analysis result. The data processing corresponding to is specified. The processing specifying unit 110 specifies the data processing as processing to be performed by the data processing device 2 watched by the user for the data selected by the user in S2 (S4).

  FIG. 12 is a detailed flowchart of the process specification. The video analysis unit 104 analyzes the captured video of the imaging unit 103 (S41), and identifies the gesture of the user's hand 5 captured in the video (S42). The process specifying unit 110 determines whether or not the gesture identified by the video analysis unit 104 matches the gesture information 22 acquired by the device specifying unit 108. If they match (YES in S43), The process corresponding to the gesture is specified as the process to be performed by the data processing device 2 (S44). For example, as illustrated in FIG. 6, if the gesture of the user's hand 5 is the same as the print gesture, the process specifying unit 110 specifies a print as a process to be executed by the data processing device 2.

  Returning to FIG. The data processing requesting unit 111 sends the data processing specified by the processing specifying unit 110 for the data specified by the data specifying unit 109 to the data processing device 2 specified by the device specifying unit 108 via the communication unit 105. Is requested (S5). For example, as shown in FIG. 6, when the user makes a print gesture, the data processing request unit 111 sends a request to print the image data 11a to the data processing apparatus 2 as shown in FIG. Upon receiving the request, the data processing device 2 executes processing for printing the specified image data 11a in accordance with the request.

  Further, for example, when the user gazes at the data processing device 2 or touches the icon 21, the operation panel display control unit 112 displays the operation panel GUI as a virtual reality in the user's field of view. Also good. When the user gazes at the data processing device 2 or touches the icon 21, the device state display control unit 107 starts from the data processing device 2 identified by the video analysis unit 104 via the communication unit 105. As the device state, information indicating an operation that can be executed in the data processing device 2 and the operation instruction content is acquired, and the device state display control unit 107 performs an operation instruction for each operation that can be executed in the data processing device 2. Is displayed on the display unit 101. Similarly to the above, the process specifying unit 110 specifies the data process corresponding to each identified gesture according to the gesture of the user's hand with respect to the operation panel GUI identified by the video analysis unit 104. The data processing request unit 111 requests the data processing apparatus 2 to execute the specified data processing via the communication unit 105. As a result, the user can operate the data processing apparatus 2 through the operation panel GUI.

  According to the present embodiment, using a head-mounted display or glasses-type wearable device, particularly a transmissive wearable terminal, it is possible to specify and set the operation content based on a variety of user hand movements than before, As a result, the user can intuitively operate the data processing apparatus, and the data processing apparatus 2 as another apparatus to be operated can be made to perform desired data processing. As a result, the transparent wearable terminal 1 can be used to intuitively handle data by a user's hand gesture, and the data operability in the data processing device 2 can be improved to improve convenience. . In addition, since the gesture information possessed by the data processing device 2 is acquired from the data processing device 2 specified by the user's line of sight and the gesture pattern is recognized for each specified data processing device 2, a wide variety of data It is possible to cope with the processing apparatus 2 and an effect of improving versatility is expected.

  The present invention is not limited to the configuration of the above embodiment, and various modifications can be made. For example, a plurality of data to be processed by the data processing device 2 may be selected simultaneously. In this case, the data specifying unit 109 moves the first finger (for example, the index finger) of the user's hand according to the movement of the first finger and the second finger of the user identified by the video analysis unit 104. The first data is specified based on the second data, and the second data is specified based on the movement of the second finger (for example, the middle finger), thereby specifying a plurality of data as the processing target data.

  Note that the configuration and processing shown in the above embodiment using FIGS. 1 to 12 are only one embodiment of the present invention, and the present invention is not limited to the configuration and processing.

DESCRIPTION OF SYMBOLS 1 Transparent wearable terminal 2 Data processing apparatus 3 Data holding apparatus 5 User's hand 10 Data processing system 11 Data selection GUI
21 Icon 22 Gesture information 101 Display unit 102 Gaze detection unit 103 Imaging unit 104 Video analysis unit 105 Communication unit 107 Device state display control unit 108 Device identification unit 109 Data identification unit 110 Processing identification unit 111 Data processing request unit 112 Operation panel display control Part

Claims (6)

A transmissive wearable device that overlays and displays video in the user ’s field of view,
A display unit that displays an image that can be viewed by the user;
A line-of-sight detector that detects the user's line of sight and focal length;
An imaging unit for capturing a user's field of view image;
Analyzing the captured video of the imaging unit, a data processing device reflected in the video, and a video analysis unit for identifying hand movements by a user;
A communication unit communicating with the data processing device;
The device state is acquired from the data processing device identified by the video analysis unit via the communication unit, and an icon indicating the device state is displayed at a position close to the data processing device in the user's field of view. A device status display control unit to be displayed by the unit,
Based on the detection result of the line-of-sight detection unit, a device specifying unit that specifies a data processing device that the user gazes among the data processing devices identified by the video analysis unit;
A data specifying unit for displaying a data selection GUI in the user's field of view, and specifying data selected by the user in the data selection GUI based on a hand movement by the user identified by the video analysis unit;
A processing specifying unit that specifies data processing according to a gesture of the user's hand identified by the video analysis unit;
Data processing that requests the data processing device specified by the device specifying unit to perform the data processing specified by the processing specifying unit on the data specified by the data specifying unit via the communication unit A transparent wearable terminal comprising a request unit.   When the data processing device to be watched by the user is specified, the device specifying unit acquires, from the data processing device, data processing that can be executed by the data processing device and gesture information corresponding to the data processing device. The transmission-type wearable terminal according to claim 1, wherein the display unit displays the display unit at a position near the data processing device in the field of view.   The data specifying unit specifies the first data based on the movement of the first finger of the user's hand and specifies the second data based on the movement of the second finger. The transparent wearable terminal according to claim 1 or 2, wherein a plurality of data selected by the user as data is specified.   When the device identifying unit detects that the data processing device identified by the video analysis unit is gaze based on the detection result of the visual line detection unit, or the user identified by the video analysis unit When it is detected that the user touches the icon displayed by the display unit at a position in the field of view near the data processing device, the operation panel GUI of the data processing device is displayed. The transmission type wearable terminal according to any one of claims 1 to 3, further comprising an operation panel display control unit for displaying in the field of view.   A data processing device that receives a data processing request from the transparent wearable terminal according to any one of claims 1 to 4 and performs data processing according to the request. The transmissive wearable terminal according to any one of claims 1 to 4,
A data processing system comprising: the data processing device according to claim 5.