JP7331345B2 - Operation support device, method and program - Google Patents

Description

The present disclosure relates to an operation support device, method, and program.

Conventionally, in electronic devices such as smartphones and tablet terminals, a touch panel is used as an input device on the front side (visual field side) of the display part such as a liquid crystal display, and there is known a technology that enables touch operation with a user's finger or a stylus pen. It is

Further, in recent years, there has been known an interactive technology that detects an operation on a projection surface projected by an image projection device such as a projector by image recognition, and enables instruction operation, drawing operation, and the like (see Patent Document 1). .

JP 2013-97177 A

However, in the conventional technology as described above, when a plurality of users can operate the same operation surface, there is a problem that it is difficult to distinguish and detect which user performed the operation.

SUMMARY OF THE INVENTION Accordingly, an object of the present invention is to determine which of a plurality of users to associate operation information for an operation surface that can be operated by a plurality of users.

In order to achieve the above object, in one aspect of the present invention, a plurality of users each having a user terminal can operate, and each of the plurality of users selects one of at least one function switch. a receiving means for receiving operation information of said two or more operations on an operation surface formed such that two or more operations involving a series of movements up to an identification switch for identifying said user are executed; , based on acquisition means for acquiring motion information representing movement of the user terminal, motion information relating to the plurality of user terminals acquired by the acquisition means, and the operation information received by the reception means and determining means for determining a user, out of the plurality of users, to be associated with the operation information, wherein the determining means selects, out of the operation information relating to the plurality of user terminals acquired by the acquiring means, the Provided is an operation support device that identifies motion information that best matches a series of motions, and determines a user terminal associated with the identified motion information among the plurality of user terminals as the user to be associated with the operation information. be.

In one aspect, according to the present invention, it is possible to determine which of the plurality of users to associate the operation information for the operation surface that can be operated by a plurality of users.

1 is a diagram schematically showing the configuration of a user interface system to which an operation support device according to an embodiment is applied; FIG. 1 is a schematic diagram of an electronic device according to one embodiment; FIG. It is a block diagram which shows the hardware constitutions of an electronic device. 2A and 2B are diagrams showing two examples of images projected by a projector; FIG. 3 is a diagram illustrating an example of a hardware configuration of a processing device of a data transfer terminal; FIG. 3 is a diagram illustrating an example of a functional configuration of a processing device of a data transfer terminal; FIG. FIG. 4 is an explanatory diagram of master motion information in a master motion information storage unit; 4 is a schematic flow chart showing an example of processing executed by a data transfer terminal; 9 is a schematic flow chart showing an example of processing executed by an electronic device in relation to the processing shown in FIG. 8;

Each embodiment will be described in detail below with reference to the accompanying drawings.

FIG. 1 is a diagram schematically showing the configuration of a user interface system 10 to which an operation support device according to one embodiment is applied.
A user interface system 10 includes an electronic device 1, a projector 2, and a data transfer terminal 3 (an example of an operation support device).

The electronic device 1 is capable of two-way communication with the projector 2 and the data transfer terminal 3, respectively. This form of two-way communication is wireless communication. Also, the projector 2 and the data transfer terminal 3 are capable of two-way communication with each other. The form of this two-way communication may be wireless communication or wired communication.

(Electronics)
FIG. 2 is a schematic diagram of an electronic device 1 according to one embodiment. As shown in FIG. 2, the electronic device 1 of this embodiment is a wearable terminal (an example of a user terminal) and is configured as a wristwatch-type device (smartwatch, etc.). Note that, in a modification, the electronic device 1 may have another form that can accompany the user.

The electronic device 1 includes a first display section 18 and a second display section 24 (described later), and the second display section 24 is laminated on the first display section 18 . Furthermore, a touch panel 17 to be described later is provided on the second display section 24 . Therefore, in the electronic device 1, it is possible to superimpose the display of the second display section 24 on the display of the first display section 18, and to perform a touch operation on the displayed content. . Below, the user is assumed to be the user of the electronic device 1 . A user typically uses the electronic device 1 by wearing it on the arm.

FIG. 3 is a block diagram showing the hardware configuration of the electronic device 1. As shown in FIG. As shown in FIG. 3, the electronic device 1 includes a CPU (Central Processing Unit) 11, a ROM (Read Only Memory) 12, a RAM (Random Access Memory) 13, a storage section 14, and an RTC (Real Time Clock). A unit 15, a drive 16, a touch panel 17, a first display unit 18, a first input unit 19, a Bluetooth (registered trademark) antenna 20, a Bluetooth module 21, and a wireless LAN (Local Area Network) antenna 22. , wireless LAN module 23, second display unit 24, pulse sensor 25, geomagnetic sensor 26, acceleration sensor 27, gyro sensor 28, illuminance sensor 29, second input unit 30, GPS (Global Positioning System) Antenna 31 and GPS module 32 are provided.

The CPU 11 is composed of a first CPU 11A and a second CPU 11B. The first CPU 11A performs various kinds of arithmetic processing and executes processing of an OS (Operating System), thereby controlling smartphone-like functions in the electronic device 1 . In this embodiment, the first CPU 11A causes the first display unit 18 to display incoming e-mails, weather information messages, and the like received via the Bluetooth module 21 or the wireless LAN module 23, and displays the messages received via the touch panel 17. It accepts operations that The first CPU 11A also recognizes voice input via the first input unit 19, and performs processing related to various functions implemented as functions similar to smartphones.

Also, in this embodiment, the first CPU 11A acquires a time signal from the RTC unit 15 at a predetermined timing.

By executing specific program processing, the second CPU 11B issues display instructions to the second display unit 24, acquires detection results from various sensors, and performs various other functions implemented as functions of the wristwatch. Such processing is performed. In this embodiment, the second CPU 11B calculates the time based on the time signal input from the first CPU 11A, and causes the second display section 24 to display the time, day of the week, date, or the like. The specific program processing (time calculation, etc.) executed by the second CPU 11B is simpler than the OS processing executed by the first CPU 11A. Also, for this reason, hardware specifications required for the second CPU 11B are lower than those for the first CPU 11A.

The ROM 12 can read data from each of the first CPU 11A and the second CPU 11B, and stores various programs executed by the first CPU 11A and the second CPU 11B and initial setting data. For example, the ROM 12 stores an OS program executed by the first CPU 11A, various programs executed under the control of the OS, or a specific program executed by the second CPU 11B (here, a built-in program that realizes the functions of the wristwatch). Store the program.

The RAM 13 can read and write data from the first CPU 11A and the second CPU 11B, provides working memory space to the first CPU 11A and the second CPU 11B, and stores temporary working data. For example, the RAM 13 provides a system area and work area when the first CPU 11A executes the OS, and provides a storage area when the second CPU 11B executes a specific program.

The storage unit 14 is a non-volatile memory from which data can be read and written by the first CPU 11A and the second CPU 11B, and is, for example, a flash memory or an EEPROM (Electrically Erasable and Programmable Read Only Memory). The storage unit 14 stores various data (such as data of various settings) generated in various functions similar to smartphones and functions of wristwatches.

The RTC unit 15 generates a time signal.

A removable medium 41 such as a magnetic disk, an optical disk, a magneto-optical disk, or a semiconductor memory is mounted in the drive 16 as appropriate. The removable medium 41 can store various data such as data detected by various sensors.

The touch panel 17 is a capacitive touch panel, a resistive touch panel, or the like provided on the display screen of the second display section 24 . The touch panel 17 detects a user's touch operation position and operation content on the operation surface, generates a signal corresponding to the operation, and outputs the signal as an input signal to the first CPU 11A.

The first display unit 18 is composed of an organic EL display (OLED), and displays various information on the display screen under the control of the first CPU 11A.

The first input unit 19 has a microphone that converts voice into an electrical signal, and outputs a signal indicating the input voice (voice command for operation, etc.) to the first CPU 11A.

The Bluetooth antenna 20 is an antenna for transmitting and receiving electromagnetic waves based on the Bluetooth standard, and is composed of, for example, a monopole antenna. The Bluetooth antenna 20 transmits an electric signal for wireless communication input from the Bluetooth module 21 as an electromagnetic wave, converts the received electromagnetic wave into an electric signal, and outputs the electric signal to the Bluetooth module 21 .

The Bluetooth module 21 transmits signals to other devices via the Bluetooth antenna 20 according to instructions from the first CPU 11A. The Bluetooth module 21 also receives signals transmitted from other devices and outputs information indicated by the received signals to the first CPU 11A. The wireless LAN antenna 22 is an antenna capable of receiving radio waves of a frequency corresponding to wireless communication used by the wireless LAN module 23, and is configured by, for example, a loop antenna or a rod antenna. The wireless LAN antenna 22 transmits an electrical signal for wireless communication input from the wireless LAN module 23 as an electromagnetic wave, converts the received electromagnetic wave into an electrical signal, and outputs the electrical signal to the wireless LAN module 23 .

The wireless LAN module 23 transmits signals to other devices via the wireless LAN antenna 22 according to instructions from the first CPU 11A. The wireless LAN module 23 also receives a signal transmitted from another device and outputs information indicated by the received signal to the first CPU 11A.

The second display unit 24 is composed of a PN (Polymer Network) liquid crystal display that can transmit light partially or wholly, and displays various information on the display screen (in this case, segment display) according to the control of the second CPU 11B. .

In this embodiment, the PN liquid crystal display, which is the second display section 24, is laminated on the display screen of the organic EL display, which is the above-described first display section 18, for example. In this PN liquid crystal display, the liquid crystal molecules are arranged irregularly in the areas where no potential is applied, and reflect light. In other words, the PN liquid crystal display is displayed at the portion to which this potential is not applied. On the other hand, in the portion to which the potential is applied, the liquid crystal molecules are aligned vertically with respect to the display screen, so that light can be transmitted. In other words, since light from the above-mentioned organic EL display can be transmitted through the portion to which this potential is applied, display by the organic EL display can be visually recognized through the PN liquid crystal display. That is, in the display area of the electronic device 1, the display by the first display unit 18 and the display by the second display unit 24 can be displayed in a superimposed state.

The pulse sensor 25 is installed on the back side of the electronic device 1 (the side facing the user's arm) and detects the pulse of the user wearing the electronic device 1 . The pulse sensor 25 outputs the detected pulse to the second CPU 11B.

The geomagnetic sensor 26 detects the direction of geomagnetism and outputs information indicating the direction of the detected geomagnetism to the second CPU 11B.

The acceleration sensor 27 detects acceleration in three axial directions of the electronic device 1, and outputs information indicating the detected acceleration to the second CPU 11B.

The gyro sensor 28 detects angular velocities in three axial directions in the electronic device 1, and outputs information indicating the detected angular velocities to the second CPU 11B.

The illuminance sensor 29 is installed at a predetermined position on the back side of the first display unit 18, detects the brightness (illuminance) in the display area of the electronic device 1, and outputs information indicating the detected brightness to the second CPU 11B.

The second input unit 30 is composed of various buttons, and inputs various information in accordance with user's instruction operation.

The GPS antenna 31 receives radio waves transmitted from GPS satellites, converts them into electric signals, and outputs the converted electric signals (hereinafter referred to as “GPS signals”) to the GPS module 32 .

The GPS module 32 detects the position (latitude, longitude, altitude) of the electronic device 1 and the current time indicated by GPS based on the GPS signal input from the GPS antenna 31 . The GPS module 32 also outputs information indicating the detected position and current time to the second CPU 11B.

Note that the configuration shown in FIG. 3 is merely an example, and various modifications are possible. For example, the electronic device 1 includes the first display section 18 and the second display section 24 , but the electronic device 1 may include only the first display section 18 . Moreover, although the first display unit 18 on which the display contents are displayed is composed of an OLED, it may be composed of other display means such as a liquid crystal display. Further, although the CPU 11 of the electronic device 1 is composed of the first CPU 11A and the second CPU 11B, the CPU 11 is not limited to this and may be a single CPU having both the functions of the first CPU 11A and the functions of the second CPU 11B. Also, the wireless LAN module 23, the gyro sensor 28, the illuminance sensor 29, the GPS module 32, etc. may be omitted.

In this embodiment, the electronic device 1 transmits acceleration information from the acceleration sensor 27 to the data transfer terminal 3 via the Bluetooth module 21 in response to a request from the data transfer terminal 3, which will also be described in detail below. At this time, the electronic device 1 sends transmission data including acceleration information and an identifier such as a device ID to the data transfer terminal 3 so that the data transfer terminal 3 can distinguish it from acceleration information sent from other electronic devices 1. Send.

(projector)
The projector 2 has an unillustrated processing unit (formed by, for example, a microcomputer), and the processing unit generates and projects an image. The projector 2 functions the projection surface on which the image is projected as an operation surface.

The projector 2 is arranged so as to form an operation surface on an appropriate target (for example, a wall surface or the surface of a desk) in a space where a plurality of users wearing the electronic device 1 (an example of attachment) are present.

The operation surface that can be formed by the projector 2 includes an operation surface that can be operated by a plurality of users wearing the electronic device 1 (hereinafter also referred to as "an operation surface that can be operated by multiple users"). The operation surface that can be formed by the projector 2 may include an operation surface that can be operated by a single user in addition to an operation surface that can be operated by a plurality of users. For example, when a communication state is established with the Bluetooth modules 21 of a plurality of electronic devices 1, the projector 2 outputs an operation surface that can be operated by a plurality of users, and the Bluetooth module 21 of the single electronic device 1 If a communication state has been established with the , an operation surface that can be operated by a single user may be output. Alternatively, the projector 2 may switch between an operation surface that can be operated by multiple users and an operation surface that can be operated by a single user according to the operation state of a predetermined changeover switch (not shown).

4A and 4B are diagrams showing two examples of images projected by the projector 2. FIG.

The projection image G40 forms an operation surface by being projected, and has an image portion functioning as a plurality of switches (buttons) SW1 to SW6 on the operation surface. Each of the switches SW1 to SW6 is drawn with a notation or the like suggesting the content of the function assigned thereto (the function realized by operating the switch). For example, in the example shown in FIG. 4, the notation "image transfer" is drawn on the switch SW1. In this case, the user performs an operation such as pressing (tapping) or pointing the switch SW1 with a finger when desiring to perform image transfer. Hereinafter, such an operation of the user pointing or pressing a finger on any of the switches SW1 to SW6 (the same applies to the specifying switch SW7, which will be described later) is also simply referred to as a "tap operation". The operation surface formed by the projection image G40 is an example of an operation surface that can be operated by multiple users.

Projection image G41 is output when any one of switches SW1 to SW6 is tapped on the operation surface related to projection image G40. The projected image G41 forms an operation surface for identifying the operator by being projected, and has an image portion that functions as the identification switch SW7 on the operation surface. Note that a specific notation such as a ☆ mark may be drawn on the switch SW7. The projection image G41 may also include a guidance display GD1 that functions as guidance on the operation surface. As shown in FIG. 4, the guidance display GD1 includes a message to the effect that "the operator should press the lower right switch".

In the following description, it is assumed that the user refers to the user wearing the electronic device 1, and the user performs the tap operation described below with the hand wearing the electronic device 1.

Here, for an operation surface that can be operated by a plurality of users, it is difficult to accurately determine which user among the plurality of users has performed the tap operation. For this reason, in this embodiment, by forming the operation surface with the projected image G41, the user (operator) who has performed the tap operation is caused to make a specific hand movement. In other words, the operation surface formed by the projection image G41 functions so as to perform a tap operation on the switch SW7 after a tap operation on any one of the switches SW1 to SW6. Further in other words, the operation surface formed by the projection image G41 is such that the user can perform two or more operations (a tap operation on any of the switches SW1 to SW6 and a subsequent tap operation on the switch SW7) accompanied by a series of movements. configured to run. Below, such two or more operations accompanied by a series of movements by the user are also referred to as "a series of tap operations".

Upon detecting the tap operation, the processing unit of the projector 2 generates operation information and transmits it to the data transfer terminal 3 . The operation information is information indicating which switch has been operated. The operation information may be any information that allows the data transfer terminal 3 to derive which switch has been operated based on the operation information, and may be, for example, touch position information on the operation surface.

Any method can be used to detect the tap operation, for example, a method based on image recognition (i.e., a method of specifying the position pointed by the operating finger and the switch based on the image of the operating finger captured in the image of the operating surface). Alternatively, a method using infrared rays or the like may be used.

Note that part of the functions described above by the processing unit of the projector 2 may be realized by the processing unit of the data transfer terminal 3, which will be described later.

(Data transfer terminal 3)
The data transfer terminal 3 is any electronic device, and may be, for example, a portable terminal such as a smart phone or a tablet, or a fixed terminal such as a personal computer. Also, in another embodiment, the data transfer terminal 3 may be in the form of a server. In this case, the data transfer terminal 3, the electronic device 1, and the projector 2 are connected via a network (not shown). In this case, the network includes a wireless communication network, and may also include the Internet, VPN (Virtual Private Network), WAN (Wide Area Network), wired network, or any combination thereof.

The data transfer terminal 3 has a function of transferring data to the electronic device 1 . For example, the data transfer terminal 3 transfers data to the electronic device 1 through communication with the Bluetooth module 21 of the electronic device 1 . The data to be transferred is arbitrary, but may be determined based on the operation information from the projector 2 described above. For example, when operation information representing a tap operation on the switch SW1 is received, predetermined image data is transferred. In this case, the operation information may further include information designating predetermined image data.

FIG. 5 is a diagram showing an example of the hardware configuration of the processing device 100 of the data transfer terminal 3. As shown in FIG.

In the example shown in FIG. 5, the processing device 100 includes a control unit 101, a main storage unit 102, an auxiliary storage unit 103, a drive device 104, a network I/F unit 106, and an input unit 107.

The control unit 101 is an arithmetic device that executes programs stored in the main memory unit 102 and the auxiliary memory unit 103, receives data from the input unit 107 and the memory device, calculates and processes the data, and outputs the data to the memory device or the like. do.

The main storage unit 102 is a ROM (Read Only Memory), a RAM (Random Access Memory), or the like. The main storage unit 102 is a storage device that stores or temporarily stores programs and data such as an OS (Operating System) that is basic software executed by the control unit 101 and application software.

The auxiliary storage unit 103 is an HDD (Hard Disk Drive), an SSD (Solid State Drive), or the like, and is a storage device that stores data related to application software and the like.

A drive device 104 reads a program from a recording medium 105, such as a flexible disk, and installs it in a storage device.

A recording medium 105 stores a predetermined program. A program stored in this recording medium 105 is installed in the processing device 100 via the drive device 104 . The installed predetermined program can be executed by the processing device 100 .

A network I/F unit 106 is an interface between a peripheral device having a communication function and the processing device 100 connected via a network constructed by a data transmission line such as a wired and/or wireless circuit.

The input unit 107 includes a keyboard, a mouse, a touch pad, and the like having cursor keys, numeric input, various function keys, and the like.

Note that, in the example shown in FIG. 5, various types of processing described below can be realized by causing the processing device 100 to execute a program. It is also possible to record a program on the recording medium 105 and cause the processing apparatus 100 to read the recording medium 105 on which the program is recorded, thereby realizing various processes described below. Various types of recording media can be used as the recording medium 105 . For example, the recording medium 105 is a recording medium for optically, electrically or magnetically recording information such as a CD (Compact Disc)-ROM, a flexible disk, a magneto-optical disk, or a recording medium such as a ROM, a flash memory, or the like. It may be a semiconductor memory or the like that electrically records .

FIG. 6 is a diagram showing an example of the functional configuration of the processing device 100 of the data transfer terminal 3. As shown in FIG.

The processing device 100 of the data transfer terminal 3 includes an operation information receiving section 301 (an example of receiving means), an action information acquiring section 302 (an example of acquiring means), an operator specifying section 303 (an example of determining means), a master A motion information storage unit 304 is included. The operation information reception unit 301, the operation information acquisition unit 302, and the operator identification unit 303 are configured by the control unit 101 in FIG. 5 executing one or more programs in a storage device (for example, the main storage unit 102 in FIG. 5). can be realized by The master operation information storage unit 304 can be realized by the main storage unit 102 or the auxiliary storage unit 103 in FIG.

The operation information receiving section 301 receives operation information from the projector 2 . Operation information is as described above. In addition, a situation in which a plurality of users wearing the electronic devices 1 can perform the above-described tap operation on an operation surface that can be operated by a plurality of users (hereinafter referred to as "situation in which a plurality of users can perform a tap operation"). ), the operation information receiving unit 301 can receive operation information from any one of the plurality of electronic devices 1 .

The motion information acquisition unit 302 acquires motion information representing motion of the electronic device 1 . In this embodiment, as an example, the motion information acquisition unit 302 acquires acceleration information from the acceleration sensor 27 of the electronic device 1 as motion information. The motion information acquisition unit 302 may acquire acceleration information through communication with the Bluetooth module 21 of the electronic device 1 .

Note that in a situation where a plurality of users can perform tap operations, the motion information acquisition unit 302 can acquire motion information from any one of the plurality of electronic devices 1 .

The operator identification unit 303 functions in a situation where multiple users can perform tap operations. In the data transfer terminal 3 , a situation in which a plurality of users can perform tap operations can be detected based on the number of electronic devices 1 with which communication with the Bluetooth module 21 has been established.

The operator specifying unit 303 selects one of the plurality of electronic devices 1 based on the operation information related to the plurality of electronic devices 1 acquired by the operation information acquiring unit 302 and the operation information received by the operation information receiving unit 301. or a user among a plurality of users to be associated with the operation information.

Here, in a situation where a plurality of users can perform tap operations, even if the operation information receiving unit 301 receives the operation information, only from the operation information, which user (which user wearing the electronic device 1) can perform the operation. It is not possible to specify whether information has been generated (that is, whether a tap operation has been performed). This is because, as described above, the operation information is information transmitted from the projector 2 and does not include information indicating which user (user wearing which electronic device 1) the operation information pertains to. .

In this regard, in this embodiment, the operator identification unit 303 uses the motion information related to the plurality of electronic devices 1 acquired by the motion information acquisition unit 302 to determine which user (which user wearing the electronic device 1 ) has generated the operation information (that is, whether a tap operation has been performed).

As described above, the user performs a tap operation with the hand on which the electronic device 1 is worn. Therefore, when performing the tap operation, the electronic device 1 generates an acceleration corresponding to the tap operation. In other words, the motion information from the electronic device 1 of the user who has performed the tapping operation and the motion information from the electronic device 1 of the user who has not performed the tapping operation are time-series waveforms (for example, time-series waveforms of acceleration on each axis). waveform) are clearly different. In the present embodiment, by using this point, it is possible to accurately identify which user (which user wearing which electronic device 1) generated the operation information (that is, performed the tap operation).

Specifically, if the motion information related to the plurality of electronic devices 1 acquired by the motion information acquisition unit 302 includes motion information representing a motion that appears when a tap operation is performed, the operator identification unit 303 The electronic device 1 that has transmitted the action information (which can be identified based on the identifier included in the transmitted data) can be identified as the electronic device 1 that generated the operation information (that is, the user who performed the tap operation).

For example, the operator identifying unit 303 compares master motion information, which is motion information that should be obtained when a tap operation is performed, with motion information related to the plurality of electronic devices 1 acquired by the motion information acquiring unit 302. to identify the motion information that most matches the master motion information. In this case, the operator identifying unit 303 identifies the electronic device 1 that has transmitted the motion information most consistent with the master motion information as the electronic device 1 that generated the operation information (that is, the user who performed the tap operation). may

Here, the master motion information is prepared in advance and stored in the master motion information storage unit 304 . Note that the master operation information may be acquired through learning or the like.

Master action information is preferably prepared for each series of tap operations. FIG. 7 is an explanatory diagram of the master motion information in the master motion information storage unit 304. As shown in FIG. FIG. 7 shows master action information for a series of tap operations on the operation surface shown in FIG. In this case, as a series of tap operations, for example, "tap operation on switch SW1+tap operation on switch SW7" is associated with acceleration information A1 as master motion information, and "tap operation on switch SW2+tap operation on switch SW7" is associated with acceleration information A1. Acceleration information A2 is associated as master operation information with "tap operation on SW7", and so on. The acceleration information A1 is generated based on the acceleration information from the acceleration sensor 27 obtained when the switch SW7 is tapped after the switch SW1 is tapped with the hand wearing the electronic device 1. good. The acceleration information A2 may be generated based on acceleration information from the acceleration sensor 27 obtained when the switch SW7 is tapped after the switch SW2 is tapped with the hand wearing the electronic device 1.

Here, regarding a series of tap operations, for example, "tap operation on switch SW1+tap operation on switch SW7" and "tap operation on switch SW2+tap operation on switch SW7" are as shown in FIG. Since the switches SW1 and SW2 are at different positions, the acceleration information A1 and the acceleration information A2 have different characteristics. Therefore, by preparing master operation information for each series of tap operations, it is possible to accurately identify the electronic device 1 that generated the operation information (that is, the user who performed the tap operation).

In this case, the operator identifying unit 303 extracts from the master motion information storage unit 304 the master motion information corresponding to the series of tap operations represented by the operation information received by the operation information receiving unit 301, and extracts the extracted master motion information. , the operation information of the plurality of electronic devices 1 acquired by the operation information acquisition unit 302 is compared, and the operation information most consistent with the master operation information is specified. Then, the operator identifying unit 303 identifies the electronic device 1 that has transmitted the motion information most consistent with the master motion information as the electronic device 1 that generated the operation information (that is, the user who performed the tap operation). good too. In this way, the operator identifying unit 303, based on the relationship between the operation information received by the operation information receiving unit 301 and the operation information related to the plurality of electronic devices 1 acquired by the operation information acquiring unit 302, It can be specified as the electronic device 1 that generated the operation information (that is, the user who performed the tap operation). Note that specifying the electronic device 1 that generated the operation information (that is, the user who performed the tap operation) means “determining the electronic device 1 to be associated with the operation information among the plurality of electronic devices 1, or determining the electronic device 1 associated with the operation information. Determining a user to be associated with the operation information among the users."

After identifying the electronic device 1 that generated the operation information, the operator identification unit 303 executes a predetermined process on the electronic device 1 . The predetermined process may be determined based on the operation information. For example, in the case of the operation information representing the switch SW1, the predetermined processing is image transfer processing.

(Operation example)
Next, an operation example of the user interface system 10 will be described with reference to schematic flow charts in FIG. 8 and subsequent figures. In the subsequent process flow charts (flowcharts), the processing order of each step may be changed as long as the relationship between the input and output of each step is not impaired.

FIG. 8 is a schematic flowchart showing an example of processing executed by the data transfer terminal 3. As shown in FIG. The processing shown in FIG. 8 is executed in a state where communication (communication by the Bluetooth module 21) is established between the data transfer terminal 3 and the plurality of electronic devices 1. FIG. Here, it is assumed that communication has been established with N1 pieces of electronic equipment 1 .

In step S800, the data transfer terminal 3 outputs an operation screen including a file transfer switch (see image G40 in FIG. 4) via the projector 2. FIG. For example, the data transfer terminal 3 sends a command to the projector 2 to output an operation screen including a file transfer switch (see image G40 in FIG. 4).

In step S<b>802 , the data transfer terminal 3 determines whether operation information indicating a tap operation on the file transfer switch has been received from the projector 2 . If the determination result is "YES", the process advances to step S804; otherwise, an operation waiting state for the file transfer switch is entered. It should be noted that if the operation waiting state for the file transfer switch continues for a predetermined time or longer, the process may be terminated.

In step S804, the data transfer terminal 3 outputs an operation screen (see image G41 in FIG. 4) including a specification switch via the projector 2. FIG.

In step S<b>806 , the data transfer terminal 3 determines whether operation information indicating a tap operation on the specification switch has been received from the projector 2 . If the determination result is "YES", the process advances to step S808. Otherwise, an operation waiting state for the specification switch is entered. It should be noted that if the operation waiting state for the specification switch continues for a predetermined time or longer, the process may end.

In step S808, the data transfer terminal 3 requests operation information from the plurality of electronic devices 1. FIG.

In step S<b>810 , the data transfer terminal 3 determines whether operation information has been received from a plurality of electronic devices 1 . If the determination result is "YES", the process proceeds to step S812;

In step S812, the data transfer terminal 3 specifies the current series of tap operations based on the operation information received in steps S802 and S806, and converts master action information corresponding to the specified series of tap operations into master action information. It extracts from within the storage unit 304 .

In step S<b>814 , the data transfer terminal 3 selects one specific electronic device 1 from among the plurality of electronic devices 1 .

In step S815, based on the operation information of one specific electronic device 1 and the master operation information in the master operation information storage unit 304 obtained in step S812, the operation information of one specific electronic device 1 is changed to master operation. It is determined whether or not it matches the information. The method of selecting one specific electronic device 1 is arbitrary, and may be, for example, the first-come-first-served order of operation information. Whether or not they match is determined, for example, by deriving a correlation coefficient between the waveform related to the master operation information and the waveform related to the operation information of a specific electronic device 1, and determining whether the correlation coefficient is equal to or greater than a predetermined threshold. may be determined based on If the determination result is "YES", the process proceeds to step S816; otherwise, the process proceeds to step S818.

In step S816, the data transfer terminal 3 associates the operation information received in steps S802 and S806 with the specific electronic device 1 (or the user wearing the electronic device 1) selected this time. For example, the data transfer terminal 3 executes file transfer processing related to the operation information received in step S802 for the specific electronic device 1 selected this time.

In step S818, the data transfer terminal 3 determines whether or not there is another electronic device 1 that has not been selected. That is, the data transfer terminal 3 determines whether or not there is an electronic device 1 for which the determination process of step S814 has not been performed among the N1 electronic devices 1 . If the determination result is "YES", the process proceeds to step S820; otherwise, the process ends. Note that if the determination result is “NO”, a message or the like may be sent via the projector 2 to prompt the user to perform the operation again.

In step S<b>820 , the data transfer terminal 3 selects one next specific electronic device 1 from among other unselected electronic devices 1 . After step S820 ends, the determination process of step S815 is executed based on the operation information from the newly selected one specific electronic device 1 .

FIG. 9 is a schematic flowchart showing an example of processing executed by the electronic device 1 in relation to the processing shown in FIG. The processing shown in FIG. 9 is executed in each of the N1 electronic devices 1. In FIG.

In step S900, the electronic device 1 determines whether or not there is a request for operation information (see step S808 in FIG. 8). If the determination result is "YES", the process proceeds to step S902; otherwise, the request is waited for.

In step S<b>902 , the electronic device 1 transmits to the data transfer terminal 3 acceleration information from the acceleration sensor 27 from a predetermined time ago to the current time. The predetermined period corresponds to the time required for a series of tap operations and may be adapted by testing or the like. In a modification, in FIG. 8, the data transfer terminal 3 may transmit an acceleration information recording start instruction to the electronic device 1 when the determination result in step S802 is "YES". In this case, in step S<b>902 , the electronic device 1 transmits to the data transfer terminal 3 the acceleration information from the acceleration sensor 27 from the reception of the recording start instruction to the current time.

Here, among the N1 electronic devices 1, the electronic device 1 that generated the operation information received by the data transfer terminal 3 in steps S802 and S806 in FIG. is transferred. Therefore, among the N1 electronic devices 1, the electronic device 1 that generated the operation information received by the data transfer terminal 3 in steps S802 and S806 of FIG. 8 will receive the file after step S902. Become.

The series of processes described above can be executed by hardware or by software.

In other words, the functional configuration as described above is merely an example and is not particularly limited. That is, it is sufficient if the electronic device 1 has a function capable of executing the series of processes described above as a whole, and what kind of functional block is used to realize this function depends particularly on the functional configuration as described above. is not limited to Also, one functional block may be composed of hardware alone, software alone, or a combination thereof.

The functional configuration in this embodiment is realized by a processor that executes arithmetic processing, and processors that can be used in this embodiment are composed of various single processing units such as single processors, multiprocessors, and multicore processors. In addition, it includes a combination of these various processing devices and a processing circuit such as ASIC (Application Specific Integrated Circuit) or FPGA (Field-Programmable Gate Array).

When a series of processes is to be executed by software, a program constituting the software is installed in a computer or the like from a network or a recording medium. The computer may be a computer built into dedicated hardware. The computer may also be a computer capable of executing various functions by installing various programs, such as a general-purpose personal computer.

A recording medium containing such a program is not only constituted by the removable medium 41 of FIG. It consists of a recording medium, etc. provided to The removable medium 41 is composed of, for example, a magnetic disk (including a floppy disk), an optical disk, or a magneto-optical disk. Optical discs are composed of, for example, CD-ROMs (Compact Disk-Read Only Memory), DVDs (Digital Versatile Disks), Blu-ray (registered trademark) Discs, and the like. The magneto-optical disk is composed of an MD (Mini-Disk) or the like. Further, the recording medium provided to the user in a state of being pre-installed in the device main body is composed of, for example, the ROM 12 in FIG. .

In this specification, the steps of writing a program recorded on a recording medium refer not only to processes performed in chronological order, but also to parallel or individual processes even if they are not necessarily performed chronologically. It also includes the processing executed in

Although each embodiment has been described in detail above, it is not limited to a specific embodiment, and various modifications and changes are possible within the scope described in the claims. It is also possible to combine all or more of the constituent elements of the above-described embodiments.

For example, in the above-described embodiments, some functions of the data transfer terminal 3 may be realized by each electronic device 1 . For example, each electronic device 1 may implement a part of the operation information receiving section 301 and the operator identifying section 303 which are functions of the data transfer terminal 3 . In this case, each electronic device 1 has a storage unit corresponding to the master operation information storage unit 304, and based on the function corresponding to the operator identification unit 303, based on the acceleration information from the acceleration sensor 27, the operation information is generated by the own electronic device 1 or not. Then, each electronic device 1 transmits the determination result to the data transfer terminal 3 . In this case, the data transfer terminal 3 can specify the electronic device 1 to be the file transfer destination among the plurality of electronic devices 1 based on the determination result from each electronic device 1 .

Further, in the above-described embodiment, the operation surface that can be operated by a plurality of users is formed by the projector 2, but it is not limited to this. For example, an operation surface that can be operated by multiple users may be realized by an input device having a touch panel. That is, instead of the projector 2, an input device having a touch panel may be used. In this case, the input device can form an operation surface that can be operated by a plurality of users by outputting an image as shown in FIG. In this case, the input device may be communicably connected to the data transfer terminal 3 and transmit the operation information to the data transfer terminal 3 so that the operation information receiving section 301 of the data transfer terminal 3 can receive it. Alternatively, the input device may be implemented as part of the data transfer terminal 3 . For example, the input device may be implemented by a touch panel of the data transfer terminal 3 . In this case, the input device may implement the operation information receiving unit 301 .

The invention described in the scope of claims originally attached to the application form of this application is additionally described below. The claim numbers in the appendix are as in the claims originally attached to the filing of this application.
<Claim 1>
receiving means for receiving operation information for an operation surface operable by a plurality of users each having a user terminal;
Acquisition means for acquiring motion information representing motion of the user terminal;
determining a user among the plurality of users to be associated with the operation information based on the operation information relating to the plurality of user terminals acquired by the acquisition means and the operation information received by the reception means; and determining means.
<Claim 2>
The determination means associates the operation information of the plurality of users based on the operation information of the plurality of user terminals acquired by the acquisition means and the operation information received by the reception means. 2. The operation support device according to claim 1, wherein, as a user, a user terminal associated with said operation information is determined from among said plurality of user terminals.
<Claim 3>
The determination means determines a user associated with the operation information based on the relationship between each of the operation information relating to the plurality of user terminals acquired by the acquisition means and the operation information received by the reception means. 3. The operation support device according to claim 1 or 2.
<Claim 4>
the operating surface is configured to allow a user to perform two or more operations involving a series of movements;
The determining means identifies motion information that most matches the series of motions among the motion information related to the plurality of user terminals acquired by the acquiring means, 4. The operation support device according to claim 3, wherein the user terminal associated with said action information is determined as said user associated with said operation information.
<Claim 5>
The operation according to any one of claims 1 to 4, further comprising processing means for transmitting, to said user determined by said determining means, information corresponding to said operation information associated with said user. support equipment.
<Claim 6>
The operation support device according to any one of claims 1 to 5, wherein said user terminal is a wearable terminal worn by a user.
<Claim 7>
7. The operation support device according to claim 1, wherein said operation surface is formed by a projection surface on which an operation screen is projected by a projector, or a touch panel surface.
<Claim 8>
Acquiring motion information representing motion of a user terminal attached to each user,
Receive operation information for the displayed operation surface,
An operation executed by a computer, including determining a user among the plurality of users to be associated with the operation information based on the acquired operation information related to the plurality of user terminals and the received operation information. how to help.
<Claim 9>
Acquiring motion information representing motion of a user terminal attached to each user,
Receive operation information for the displayed operation surface,
An operation support program that causes a computer to execute a process of determining a user to be associated with the operation information among the plurality of users based on the acquired operation information related to the plurality of user terminals and the received operation information.

1 electronic device 2 projector 3 data transfer terminal 11A first CPU
11B Second CPU
14 storage unit 15 RTC unit 16 drive 17 touch panel 18 first display unit 19 first input unit 20 Bluetooth antenna 21 Bluetooth module 22 wireless LAN antenna 23 wireless LAN module 24 second display unit 25 pulse sensor 26 geomagnetic sensor 27 acceleration sensor 28 Gyro sensor 29 Illuminance sensor 30 Second input unit 31 GPS antenna 32 GPS module 41 Removable media 301 Operation information reception unit 302 Operation information acquisition unit 303 Operator identification unit 304 Master operation information storage unit

Claims (8)

Operation is possible by a plurality of users each having a user terminal, and each of the plurality of users performs a series of movements from any one of at least one function switch to an identification switch for identifying the user. receiving means for receiving operation information of said two or more operations on an operation surface configured to perform two or more operations involving
Acquisition means for acquiring motion information representing motion of the user terminal;
determining a user among the plurality of users to be associated with the operation information based on the operation information relating to the plurality of user terminals acquired by the acquisition means and the operation information received by the reception means; a determining means;
with
The determining means identifies motion information that most matches the series of motions among the motion information related to the plurality of user terminals acquired by the acquiring means, determining the user terminal associated with the action information as the user associated with the operation information;
Operation support device. The determination means associates the operation information of the plurality of users based on the operation information of the plurality of user terminals acquired by the acquisition means and the operation information received by the reception means. Determining, as a user, a user terminal to be associated with the operation information among the plurality of user terminals;
The operation support device according to claim 1. The determination means determines a user associated with the operation information based on the relationship between each of the operation information relating to the plurality of user terminals acquired by the acquisition means and the operation information received by the reception means. do,
The operation support device according to claim 1 or 2. Further comprising processing means for transmitting, to the user determined by the determination means, information according to the operation information associated with the user;
The operation support device according to any one of claims 1 to 3. The user terminal is a wearable terminal worn by the user,
The operation support device according to any one of claims 1 to 4. The operation surface is formed by a projection surface on which an operation screen is projected by a projector, or a touch panel surface.
The operation support device according to any one of claims 1 to 5. Acquiring motion information representing the motion of a user terminal attached to each of a plurality of users,
Two or more operations involving a series of movements from any one of at least one function switch to an identification switch for identifying the user are performed by each of the plurality of users. receive operation information of the two or more operations on the operation surface,
Determining a user among the plurality of users to be associated with the operation information based on the acquired operation information related to the plurality of user terminals and the received operation information,
Identifying motion information that most matches the series of motions among the acquired motion information related to the plurality of user terminals, determining as the user associated with the operation information;
A computer-implemented operational support method. Acquiring motion information representing the motion of a user terminal attached to each of a plurality of users,
Two or more operations involving a series of movements from any one of at least one function switch to an identification switch for identifying the user are performed by each of the plurality of users. receive operation information of the two or more operations on the operation surface,
determining a user among the plurality of users to be associated with the operation information based on the acquired operation information related to the plurality of user terminals and the received operation information;
Identifying motion information that most matches the series of motions among the acquired motion information related to the plurality of user terminals, determining as the user associated with the operation information;
An operation support program that causes a computer to execute processing.

Images