JP7217841B2 - Operation input device and operation input method - Google Patents

Description

The present disclosure relates to an operation input device and an operation input method.

In recent years, various techniques have been proposed for operation input devices that set an operation target device, which is a device to be operated from among a plurality of devices, using a user's line of sight (for example, Patent Documents 1 to 4). .

JP 2016-184238 A JP 2012-105034 A JP 2012-006552 A JP 2016-184238 A

It is preferable that the intention of the user to continue the operation be reflected in the maintenance and cancellation of the setting of the operation target device. However, in the conventional technology, such an intention is not fully reflected, and there is a problem that the setting of the operation target device is not properly maintained.

Accordingly, the present disclosure has been made in view of the above-described problems, and aims to provide a technology capable of appropriately maintaining settings of an operation target device.

An operation input device according to the present disclosure is an operation input device that inputs a user's operation to an operation target device, and includes a first acquisition unit that acquires the user's operation and a second acquisition unit that acquires the user's line of sight. and a control unit that sets and cancels an operation target device from among a plurality of predetermined devices based on the operation and line of sight respectively acquired by the first acquisition unit and the second acquisition unit, and controls After setting the operation target device, the unit determines that the line of sight is not directed to any of the plurality of devices, and the operation elapsed time, which is the time that has elapsed since the operation was input to the operation target device, is the allowable time. If it is determined that it is within the allowable time, the setting of the operation target device is maintained, and after setting the operation target device, it is determined that the line of sight is not directed to any of the plurality of devices, and the operation elapsed time is the allowable time From the point of time , if it is determined that the line of sight continues to face a specific line-of-sight target that is at least one of the first acquisition unit and the operation manual, the setting of the operation target device is maintained.

According to the present disclosure, after setting the operation target device, when it is determined that the line of sight is not directed to any of the plurality of devices and it is determined that the operation elapsed time is within the allowable time, the operation target device settings. Further, after setting the operation target device, it is determined that the line of sight is not directed to any of the plurality of devices, and from the time when the operation elapsed time reaches the allowable time, the line of sight continues to be directed to the specific line of sight target. If it is determined that the device to be operated is determined to be According to such a configuration, it is possible to appropriately maintain the settings of the operation target device.

Objects, features, aspects and advantages of the present disclosure will become more apparent with the following detailed description and accompanying drawings.

1 is a block diagram showing the configuration of an operation input device according to Embodiment 1; FIG. FIG. 9 is a block diagram showing the configuration of an operation input device according to Embodiment 2; FIG. 9 is a flow chart showing the operation of the operation input device according to Embodiment 2; FIG. 10 is a diagram showing an overview of the operation of the operation input device according to the second embodiment; FIG. FIG. 11 is a block diagram showing a hardware configuration of an operation input device according to another modified example; FIG. FIG. 11 is a block diagram showing a hardware configuration of an operation input device according to another modified example; FIG. FIG. 11 is a block diagram showing the configuration of a server according to another modified example; FIG. 11 is a block diagram showing the configuration of a communication terminal according to another modified example;

<Embodiment 1>
FIG. 1 is a block diagram showing the configuration of an operation input device 1 according to the first embodiment. The operation input device 1 of FIG. 1 is communicably connected to a plurality of candidate devices 51, which are a plurality of predetermined devices, by at least one of wired and wireless communication. An acquisition unit 12 and a control unit 13 are provided. As described below, the operation input device 1 sets an operation target device, which is an operation target device, from among a plurality of candidate devices 51, and inputs a user's operation (operation information) to the operation target device. do.

The first acquisition unit 11 acquires a user's operation (operation information). For example, a remote control, a touch pad, a joystick, buttons, and the like may be used for the first acquisition unit 11 .

The second acquisition unit 12 acquires the line of sight of the user. For the second acquisition unit 12, for example, a camera that captures the user's face and an image recognition device that detects the user's line of sight as a line of sight vector from the image captured by the camera is used.

The control unit 13 sets and cancels the operation target device from among the plurality of candidate devices 51 based on the operation and line of sight respectively acquired by the first acquisition unit 11 and the second acquisition unit 12 . For example, when the control unit 13 determines that the first acquisition unit 11 has acquired an operation and determines that the line of sight acquired by the second acquisition unit 12 is directed toward one candidate device 51, The one candidate device 51 is set as the operation target device.

Further, when the control unit 13 determines that the line of sight is not directed to any of the plurality of candidate devices 51 after setting the operation target device, the control unit 13 determines that the line of sight is not directed to any of the plurality of candidate devices 51, the control unit 13 determines that the operation target device is set in the time elapsed after the operation is input to the operation target device. It is determined whether or not a given operation elapsed time is within the allowable time. When the control unit 13 determines that the line of sight is not directed to any of the plurality of candidate devices 51 and determines that the operation elapsed time is within the allowable time, the control unit 13 maintains the setting of the operation target device. do.

In addition, the control unit 13 determines that the line of sight is not directed to any of the plurality of candidate devices 51, and from the time when the operation elapsed time reaches the allowable time, the line of sight continues to be directed to the specific line-of-sight target. determine whether or not there is The specific line-of-sight target referred to here is at least one of the first acquisition unit 11 and the operation manual. The time when the operation elapsed time becomes the allowable time may be the time when the operation elapsed time becomes completely the same as the allowable time, or the time when it becomes substantially the same as the allowable time.

When the control unit 13 determines that the line of sight continues to face the specific line-of-sight target from the time when the operation elapsed time reaches the allowable time, the control unit 13 maintains the setting of the operation target device.

<Summary of Embodiment 1>
According to the operation input device 1 according to the first embodiment as described above, even if the line of sight is not directed to any of the plurality of candidate devices 51 after setting the operation target device, the operation elapsed time is within the allowable time. , the setting of the operation target device is maintained. According to such a configuration, for example, a user who is accustomed to the operation can operate the operation target device while turning the line of sight away from the operation target device if the operation elapsed time is within the allowable time.

Further, according to the first embodiment, even if the operation elapsed time exceeds the allowable time, when the line of sight is directed to a specific gaze target from the time when the operation elapsed time reaches the allowable time, the operation target device Settings are preserved. According to such a configuration, for example, a user who is inexperienced in operation can operate the operation target device while carefully looking at the operation buttons and the operation manual of the first acquisition unit 11 .

As described above, it is possible to realize the operation input device 1 that appropriately reflects the user's operation intention, that is, the operation input device 1 that can appropriately maintain the settings of the operation target device.

<Embodiment 2>
FIG. 2 is a block diagram showing the configuration of the operation input device 1 according to the second embodiment. In the second embodiment, it is assumed that the operation input device 1 is used in a vehicle, and that the specific line-of-sight target is the input device 11a. Hereinafter, among the constituent elements according to the second embodiment, constituent elements that are the same as or similar to the above-described constituent elements are denoted by the same or similar reference numerals, and different constituent elements will be mainly described.

The operation input device 1 of FIG. 2 includes an input device 11a, a line-of-sight detection device 12a, and an operation target determination device 13a. The input device 11a, the line-of-sight detection device 12a, and the operation target determination device 13a are included in the concepts of the first acquisition unit 11, the second acquisition unit 12, and the control unit 13 described in the first embodiment, respectively.

The input device 11a acquires a user's operation (operation information).

The line-of-sight detection device 12a acquires the user's line of sight. The line-of-sight detection device 12a, for example, captures an image of the user and detects the user's line of sight from the image as a line-of-sight vector.

The operation target determination device 13a sets and cancels the operation target device from among the plurality of candidate devices 51 based on the operation detected by the input device 11a and the line of sight detected by the line of sight detection device 12a. For example, IVI (In-Vehicle Infotainment) is used for the operation target determination device 13a. The plurality of candidate devices 51 are, for example, devices used in vehicles, such as air conditioners, rearview mirrors, side mirrors, car navigation devices, sunroofs, instrument panel display devices, room lights, and various other lights.

The operation target determination device 13a of FIG. 2 is communicably connected to the plurality of candidate devices 51, the input device 11a, and the line-of-sight detection device 12a by at least one of wired and wireless. The operation target determination device 13a includes a spatial information storage unit 13b, a line-of-sight determination unit 13c, an operation target determination unit 13d, and an information transmission unit 13e.

The spatial information storage unit 13b stores spatial information indicating positions of a plurality of candidate devices 51 and seats in the vehicle.

Based on the line of sight detected by the line of sight detection device 12a and the spatial information, the line of sight determination unit 13c determines whether or not the user's line of sight is directed toward any of the plurality of candidate devices 51 and the input device 11a. do. For example, the line-of-sight determination unit 13c determines that the user's line of sight is directed toward the candidate device 51 when the line of sight of the user reaches the candidate device 51 without being blocked by the seat. On the other hand, if the user's line of sight is blocked by the seat and does not reach the candidate device 51, the line of sight determination unit 13c determines that the user's line of sight is not directed toward the candidate device 51.

The operation target determination unit 13d sets and cancels the operation target device from among the plurality of candidate devices 51 based on the operation acquired by the input device 11a and the determination result of the line-of-sight determination unit 13c. The operation of the operation target determination unit 13d will be described later using a flowchart.

The information transmission unit 13e transmits (inputs) an operation acquired by the input device 11a to the operation target device under the control of the operation target determination unit 13d. For convenience of explanation, the information transmitting unit 13e may be described as transmitting the operation to the operation target device, and the operation target determination unit 13d may be described as transmitting the operation to the operation target device.

<Action>
FIG. 3 is a flowchart showing the operation of the operation input device 1 according to the second embodiment, and generally shows the operation of the operation target determination unit 13d.

In step S1, the operation target determination unit 13d determines whether or not an operation has been acquired with the input device 11a. If it is determined that the operation has been acquired, the process proceeds to step S2, and if it is not determined that the operation has been acquired, the process of step S1 is performed again.

In step S2, the operation target determination unit 13d determines whether or not the user's line of sight is directed to any one of the plurality of candidate devices 51 based on the determination result of the line of sight determination unit 13c. If it is determined that the line of sight is directed to any of the plurality of candidate devices 51, the process proceeds to step S3, and if it is determined that the line of sight is not directed to any of the plurality of candidate devices 51, The process proceeds to step S5.

In step S3, the operation target determination unit 13d sets the candidate device 51 determined as the line of sight to the operation target device. That is, when it is determined in step S1 that the input device 11a has acquired an operation, and in step S2 it is determined that the line of sight of the user is directed to one candidate device 51, the one candidate device 51 It is set as the operation target device. The operation target determination unit 13d transmits the operation of step S1 acquired by the input device 11a to the operation target device. As a result, as shown in FIG. 4, when the user continues to look at one candidate device 51, the one candidate device 51 is set as the operation target device, and the user can perform the operation as the operation target device. can be entered into the device.

In step S4 of FIG. 3, the operation target determination unit 13d resets (initializes) the timer, and the timer starts timekeeping. After that, the process returns to step S1.

In step S5, the operation target determination unit 13d determines whether or not the setting of the operation target device is maintained. If it is determined that the setting of the operation target device is maintained, the process proceeds to step S6. If it is not determined that the setting of the operation target device is maintained, the operation in step S1 acquired by the input device 11a is discarded, and the process returns to step S1.

In step S6, the operation target determination unit 13d determines whether or not the timer time is within the allowable time. Note that the time of the timer corresponds to the operation elapsed time, that is, the time that has elapsed since the operation was input to the operation target device. If it is determined that the timer time is within the allowable time, the process proceeds to step S7, and if it is not determined that the timer time is within the allowable time, the process proceeds to step S8.

In step S7, the operation target determination unit 13d maintains the setting of the operation target device, and transmits the operation of step S1 acquired by the input device 11a to the operation target device. As a result, a user accustomed to the operation can operate the operation target device while directing the line of sight away from the operation target device. After that, the process proceeds to step S4, the operation target determination unit 13d resets (initializes) the timer, and the timer starts timekeeping.

In step S8, the operation target determination unit 13d determines whether or not the line of sight continues to face the input device 11a from the time when the time of the timer reaches the allowable time. The position of the input device 11a used for this determination may be detected based on a transmission signal such as an infrared signal that the input device 11a transmits at any time, or may be used for image recognition of an image of the interior of the vehicle captured by a camera. may be detected based on

If it is determined that the line of sight continues to face the input device 11a, the process proceeds to step S9, and if it is not determined that the line of sight continues to face the input device 11a, the process proceeds to step S10. proceed to

In step S9, the operation target determination unit 13d maintains the setting of the operation target device, and transmits the operation of step S1 acquired by the input device 11a to the operation target device. As a result, as shown in FIG. 4, when a user unfamiliar with operation continues to look at the input device 11a, the setting of the operation target device is maintained. You can type slowly into the device. After that, the process proceeds to step S4 in FIG. 3, the operation target determination unit 13d resets (initializes) the timer, and the timer starts timekeeping.

In step S10, the operation target determination unit 13d cancels the setting of the operation target device. That is, after setting the operation target device, the operation target determination unit 13d determines that the line of sight is not directed to any of the plurality of candidate devices 51, and from the time when the operation elapsed time reaches the allowable time, the line of sight is When it is determined that the user does not continue to face a specific line-of-sight target, the setting of the operation target device is canceled. The operation of step S1 acquired by the input device 11a is discarded, and the process returns to step S1.

In the operation described above, an operation corresponding to the operation target device is input to the operation target device. For example, if the device to be operated is an air conditioner, an operation such as increasing the set temperature or decreasing the set temperature is input to the air conditioner. Then, when the user presses the "increase set temperature" button of the input device 11a once while looking at the air conditioner, and then presses the button repeatedly while looking away from the air conditioner (Yes in step S6), the air conditioner The operation to raise the set temperature is repeatedly input. Further, when the user presses once the "increase set temperature" button of the input device 11a while looking at the air conditioner, and presses the button while looking at the input device 11a (Yes in step S8), the air conditioner An operation to raise the set temperature is input.

<Summary of Embodiment 2>
The operation input device 1 according to the second embodiment as described above is used in a vehicle. According to such a configuration, since the positions of the plurality of candidate devices 51 and the seats of the vehicle do not substantially change, it is possible to improve the determination accuracy of the line-of-sight determination unit 13c.

<Modification 1>
A user who is unfamiliar with the operation has a long time interval between operations, so it is considered that there are many cases where the process does not actually proceed from step S6 to step S7 in FIG. Therefore, the allowable time may be changed based on the time interval of the operation. For example, if the operation time interval is long, the allowable time may be lengthened, and if the operation time interval is short, the allowable time may be shortened. According to such a configuration, even if the user is unfamiliar with the operation, the process can proceed from step S6 to step S7 in FIG. can be done.

In addition, it is considered that the user becomes accustomed to the operation as the cumulative number of operations increases. Therefore, the allowable time may be changed based on the number of operations. For example, if the number of operations is small, the allowable time may be lengthened, and if the number of operations is large, the allowable time may be shortened. According to such a configuration, it is possible to suppress a state in which the allowable time is longer than necessary. Therefore, an erroneous operation that occurs when the allowable time is longer than necessary, such as touching the input device 11a without intention of operation. It is possible to suppress erroneous operations due to

Note that a lower limit value may be set for the allowable time. With such a configuration, the process can proceed from step S6 to step S7 in FIG. 3, so that the setting of the operation target device can be maintained as much as possible.

<Modification 2>
In the second embodiment, the specific line-of-sight target is the input device 11a. The operation manual may consist of printed paper, or may be an image displayed by a device other than the candidate device 51 . When the operation manual is made of printed paper, the position of the operation manual may be detected based on an infrared signal from an infrared element provided in the operation manual, or may be detected by a vehicle photographed by a camera. It may be detected based on image recognition for an image of the room.

<Modification 3>
In the operation of FIG. 3, unless it is determined in step S1 that the input device 11a has acquired an operation, the setting of the operation target device is maintained, but the setting is not limited to this. For example, the process may proceed to step S6 when the time during which it is not determined that an operation has been acquired by the input device 11a exceeds a predetermined threshold time.

<Other Modifications>
The above-described first acquisition unit 11, second acquisition unit 12, and control unit 13 in FIG. 1 are hereinafter referred to as "first acquisition unit 11 and the like". The first acquisition unit 11 and the like are realized by the processing circuit 81 shown in FIG. That is, the processing circuit 81 determines that the first acquisition unit 11 that acquires the user's operation, the second acquisition unit 12 that acquires the line of sight of the user, and that the line of sight is directed to none of the plurality of devices. and a function of maintaining the setting of the operation target device when it is determined that the operation elapsed time is within the allowable time; and a control unit 13 having a function of maintaining the setting of the operation target device when it is determined that the line of sight continues to be directed to a specific line-of-sight target from the point in time when becomes the allowable time. Dedicated hardware may be applied to the processing circuit 81, or a processor that executes a program stored in a memory may be applied. Processors include, for example, central processing units, processing units, arithmetic units, microprocessors, microcomputers, and DSPs (Digital Signal Processors).

When the processing circuit 81 is dedicated hardware, the processing circuit 81 may be, for example, a single circuit, a composite circuit, a programmed processor, a parallel programmed processor, an ASIC (Application Specific Integrated Circuit), an FPGA (Field Programmable Gate Array), or a combination of these. Each function of each unit such as the first acquisition unit 11 may be realized by a circuit in which processing circuits are distributed, or the functions of each unit may be collectively realized by one processing circuit.

When the processing circuit 81 is a processor, the functions of the first acquisition unit 11 and the like are realized by combining software and the like. Software and the like correspond to, for example, software, firmware, or software and firmware. Software or the like is written as a program and stored in memory. As shown in FIG. 6, a processor 82 applied to a processing circuit 81 reads out and executes a program stored in a memory 83 to implement the functions of each section. That is, when the operation input device 1 is executed by the processing circuit 81, a step in which the first acquisition unit 11 acquires the user's operation, a step in which the second acquisition unit 12 acquires the line of sight of the user, maintaining the setting of the operation target device when it is determined that the line of sight is not directed to any of the plurality of devices and that the operation elapsed time is within the allowable time; If it is determined that the line of sight is not directed to any of them and the line of sight continues to be directed to a specific line of sight target from the time when the operation elapsed time reaches the allowable time, the setting of the operation target device is maintained. A memory 83 is provided for storing the program that will result in the execution of the steps. In other words, it can be said that this program causes a computer to execute the procedures and methods of the first acquisition unit 11 and the like. Here, the memory 83 is, for example, a non-volatile or Volatile semiconductor memories, HDDs (Hard Disk Drives), magnetic disks, flexible disks, optical disks, compact disks, mini disks, DVDs (Digital Versatile Disks), their drive devices, etc., or any storage medium that will be used in the future. may

The configuration in which each function of the first acquisition unit 11 and the like is realized by either hardware or software has been described above. However, the configuration is not limited to this, and a configuration in which a part of the first acquisition unit 11 and the like is realized by dedicated hardware and another part is realized by software or the like may be employed. For example, the functions of the first acquisition unit 11 and the second acquisition unit 12 are realized by a processing circuit 81 as dedicated hardware, an interface, a receiver, and the like. The function can be realized by reading out and executing the program stored in the .

As described above, the processing circuit 81 can implement each of the functions described above using hardware, software, etc., or a combination thereof.

Further, the operation input device 1 described above includes vehicle devices such as IVI, PND (Portable Navigation Device), navigation device and DMS (Driver Monitoring System), and communication terminals including mobile terminals such as mobile phones, smart phones and tablets. , an application function installed in at least one of the vehicle device and the communication terminal, and an operation input system constructed as a system by appropriately combining a server. In this case, each function or each component of the operation input device 1 described above may be arranged dispersedly in each device constituting the system, or may be centrally arranged in one of the devices. good.

FIG. 7 is a block diagram showing the configuration of the server 91 according to this modification. A server 91 in FIG. 7 includes a communication unit 91a and a control unit 91b, and is capable of wirelessly communicating with a vehicle device 93 such as an IVI of a vehicle 92, a navigation device, and a DMS device.

The communication unit 91 a that is the first acquisition unit and the second acquisition unit receives the user's operation and line of sight acquired by the vehicle device 93 by performing wireless communication with the vehicle device 93 .

The control unit 91b has the same function as the control unit 13 of FIG. That is, the control unit 91b generates a control signal for setting and canceling the operation target device from among the plurality of candidate devices 51 based on the user's operation and line of sight received by the communication unit 91a. Then, the communication unit 91 a transmits the control signal of the control unit 91 b to the vehicle device 93 . According to the server 91 configured in this way, the same effects as those of the operation input device 1 described in the first embodiment can be obtained.

FIG. 8 is a block diagram showing the configuration of a communication terminal 96 according to this modification. The communication terminal 96 of FIG. 8 includes a communication unit 96a similar to the communication unit 91a and a control unit 96b similar to the control unit 91b, and can perform wireless communication with the vehicle device 98 of the vehicle 97. ing. As the communication terminal 96, for example, a portable terminal such as a mobile phone, a smart phone, or a tablet carried by the driver of the vehicle 97 is applied. According to the communication terminal 96 configured in this way, the same effects as those of the operation input device 1 described in the first embodiment can be obtained.

It should be noted that it is possible to freely combine each embodiment and each modification, and to modify or omit each embodiment and each modification as appropriate.

The above description is, in all aspects, illustrative and not restrictive. It is understood that innumerable variations not illustrated can be envisaged.

1 operation input device, 11 first acquisition unit, 12 second acquisition unit, 13 control unit, 51 candidate device.

Claims (6)

An operation input device for inputting a user's operation to an operation target device,
a first acquisition unit that acquires the operation of the user;
a second acquisition unit that acquires the line of sight of the user;
a control unit that sets and cancels the operation target device from among a plurality of predetermined devices based on the operation and the line of sight respectively acquired by the first acquisition unit and the second acquisition unit; ,
The control unit
After setting the operation target device, it is determined that the line of sight is not directed to any of the plurality of devices, and an operation elapsed time is a time that has elapsed since the operation was input to the operation target device. when it is determined that the time is within the allowable time, maintaining the setting of the operation target device;
After setting the operation target device, it is determined that the line of sight is not directed to any of the plurality of devices, and from the time when the operation elapsed time becomes the allowable time, the line of sight is set to the first acquisition unit and the An operation input device that maintains the setting of the operation target device when it is determined that the user continues to face a specific line-of-sight target that is at least one of the operation manuals. The operation input device according to claim 1,
The control unit
After setting the operation target device, the line of sight is determined to be directed to none of the plurality of devices, and the line of sight is directed to the specific line-of-sight target from the time when the operation elapsed time becomes the allowable time. An operation input device that cancels the setting of the operation target device when it is determined that it is not facing continuously. The operation input device according to claim 1,
The operation input device, wherein the allowable time is changed based on a time interval of the operation. The operation input device according to claim 1,
The operation input device, wherein the allowable time is changed based on the number of operations. The operation input device according to claim 1,
The operation input device, wherein a lower limit value is set for the allowable time. An operation input method for inputting a user's operation to an operation target device,
A first acquisition unit acquires the operation of the user,
A second acquisition unit acquires a line of sight of the user, and selects from among a plurality of predetermined devices based on the operation and the line of sight respectively acquired by the first acquisition unit and the second acquisition unit. Set and cancel the operation target device,
After setting the operation target device, it is determined that the line of sight is not directed to any of the plurality of devices, and an operation elapsed time is a time that has elapsed since the operation was input to the operation target device. when it is determined that the time is within the allowable time, maintaining the setting of the operation target device;
After setting the operation target device, it is determined that the line of sight is not directed to any of the plurality of devices, and from the time when the operation elapsed time becomes the allowable time, the line of sight is set to the first acquisition unit and the An operation input method that maintains the setting of the operation target device when it is determined that the user continues to face a specific gaze target that is at least one of the operation manuals.

Images