WO2014103217A1

WO2014103217A1 - Operation device and operation detection method

Info

Publication number: WO2014103217A1
Application number: PCT/JP2013/007304
Authority: WO
Inventors: 洋一成瀬
Original assignee: 株式会社デンソー
Priority date: 2012-12-26
Filing date: 2013-12-12
Publication date: 2014-07-03
Also published as: JP2014126997A

Abstract

According to the present invention, a line-of-sight detection device (30), on the basis of an image of a driver's face photographed from below, detects the driver's fixation point and the position coordinates of the driver's right and left eyes in the photographic image, and provides the fixation point and position coordinates to an onboard device (10). The onboard device displays on a display device (20) an operation screen in which is depicted an icon or other such object, and, in addition, on the basis of the position coordinates acquired from the line-of-sight detection device, calculates the vertical spacing between the positions of the right and left eyes in the photographic image, and on the basis thereof, detects a behavior in which the driver tilts the orientation of his or her face sideways. Then, when the driver is gazing at the same object when he or she engages in this behavior, it is assumed that an operation that corresponds to the object has been performed. This makes it possible for the operation to be carried out easily and freely without using the hands.

Description

Operation device and operation detection method

Cross-reference of related applications

This disclosure is based on Japanese Patent Application No. 2012-282842 filed on December 26, 2012, the contents of which are incorporated herein.

The present disclosure relates to an operation device and an operation detection method for receiving an operation based on a user's line of sight.

Conventionally, in an electronic device such as an in-vehicle device or a portable device, a button or the like has been used as an interface between the device and a user. However, in recent years, many devices using a touch panel arranged on a display are seen.

By using such a touch panel, it is possible to operate by touching an icon or the like displayed on the display, and more intuitive operation is possible. However, the physical burden of performing the operation with the fingertip is not reduced, and it is difficult to perform the operation freely when both hands are closed, for example, during driving.

On the other hand, Patent Document 1 describes a system that detects an operation based on a user's line of sight. The system detects the coordinates of the position on the screen that is on the user's line of sight (line-of-sight coordinates), receives a coordinate confirmation instruction from the user, and performs processing corresponding to the line-of-sight coordinate when the coordinate confirmation instruction is received Execute.

Japanese Laid-Open Patent Publication No. 5-108251

According to the system described in Patent Document 1, since an operation can be selected by line of sight, a troublesome key operation is not necessary, but an operation for confirming the selection of the operation needs to be performed by hand, and both hands are still required. It is difficult to operate freely in a state where is closed.

The present disclosure has been made in view of the above points, and an object thereof is to provide an operation device that can be operated easily and freely without using a hand, and an operation detection method.

According to the first aspect of the present disclosure, the operating device includes a display unit that displays an image on the screen of the display device, and a gaze point on the screen that the user is gazing on based on a captured image of the user's face. A specific part to be identified, a detection part that detects a behavior of moving the entire user's head based on a photographed image, and a user's behavior detected by the detection part, the specific part is identified when the behavior is made An operation unit that considers that an operation corresponding to the gaze point specified by the unit has been performed.

According to such a configuration, when the user moves the entire head in a predetermined manner, such as turning the face in a predetermined direction while gazing at a specific position on the screen of the display device, An operation corresponding to the gaze point is detected. For this reason, after selecting an operation by gazing at the screen, the user can confirm the selection by an extremely simple behavior of moving the entire head, and can perform the operation easily and freely without using a hand. Can do.

In addition, as described above, an electronic device that can perform various operations by arranging a touch panel on a display and touching an object such as an icon displayed on the screen is known. If applied, a more intuitive operation is possible.

According to the second aspect of the present disclosure, in the operation device according to the first aspect, the display unit displays an image on which one or more objects are drawn on the screen of the display device, and the operation unit includes the detection unit. When the behavior of the user is detected by the above, when the gaze point specified by the specifying unit when the behavior is performed is in the area where any one object is drawn, an operation corresponding to the object is performed. It is considered to have been done.

According to such a configuration, when the user moves the entire head in a predetermined manner while viewing a specific object on the screen, an operation corresponding to the object is detected. For this reason, the user can confirm the selection of the object by gaze by an extremely simple behavior of moving the entire head, and can perform a simple and free operation more intuitively without using a hand. .

According to the third aspect of the present disclosure, the operation detection method displays an image on a screen of a display device, and based on a captured image of the user's face, a gaze point on the screen on which the user is gazing. Identifying and detecting a predetermined behavior of moving the entire user's head based on the captured image, and detecting the user's behavior in the detection, when the behavior is made It is provided that an operation corresponding to the gaze point specified in specifying is performed.

According to such a method, when the user moves the entire head in a predetermined manner, such as turning the face in a predetermined direction while gazing at a specific position on the screen of the display device, the user's An operation corresponding to the gaze point is detected. For this reason, after selecting an operation by gazing at the screen, the user can confirm the selection by an extremely simple behavior of moving the entire head, and can perform the operation easily and freely without using a hand. Can do.

The above and other objects, features and advantages of the present disclosure will become more apparent from the following detailed description with reference to the accompanying drawings. In the drawing
It is a block diagram which shows the structure of the vehicle-mounted system in 1st Embodiment. It is explanatory drawing of the arrangement position of the camera with which the gaze detection apparatus of the vehicle-mounted system in 1st Embodiment is provided. It is explanatory drawing about the process which detects direction of the face of the driver in 1st Embodiment. It is explanatory drawing about the process which confirms operation according to the operation confirmation behavior of the driver in 1st Embodiment. It is explanatory drawing about an example of the detection method of the operation confirmation behavior of the driver in 1st Embodiment. It is explanatory drawing about an example of the detection method of the operation confirmation behavior of the driver in 1st Embodiment. It is explanatory drawing about an example of the detection method of the operation confirmation behavior of the driver in 1st Embodiment. It is explanatory drawing about an example of the detection method of the operation confirmation behavior of the driver in 1st Embodiment. It is a flowchart of the operation detection process in 1st Embodiment. It is a flowchart of the operation detection process in 1st Embodiment. It is a block diagram which shows the structure of the portable apparatus in 2nd Embodiment. It is the schematic which shows the external appearance of the portable apparatus in 2nd Embodiment.

Hereinafter, embodiments of the present disclosure will be described with reference to the drawings. Embodiments of the present disclosure are not limited to the following embodiments, and various forms can be adopted as long as they belong to the technical scope of the present disclosure.

[First Embodiment]
[Description of configuration]
FIG. 1 is a block diagram illustrating a configuration of an in-vehicle system 1 according to the first embodiment. The in-vehicle system 1 includes an in-vehicle device 10 that displays various images on the display device 20, and a line-of-sight detection device 30 (eye tracker) that detects a position or the like where the driver is gazing based on a captured image of the driver's face.

The line-of-sight detection device 30 includes a light projecting unit that projects infrared rays toward the driver's face, a camera 31 that captures the driver's face, and the like. Analyzing the eye image. Then, the line-of-sight direction of the driver is detected from the analysis result, and the coordinates (gaze point) of the position on the screen of the display device 20 being watched by the driver are detected based on the line-of-sight direction. The camera 31 is disposed at a position adjacent to the meter of the host vehicle in a state tilted upward by a predetermined angle from the horizontal direction, and images the driver's face from a viewpoint closer to the lower side than the front of the face (see FIG. 2).

The in-vehicle device 10 includes a CPU, a ROM, a RAM, an I / O, and the like. The in-vehicle device 10 outputs video signals to the control unit 12 that performs overall control of the in-vehicle device 10 and the display device 20. And a video signal output unit 11 for displaying an image.

The control unit 12 is configured to generate various image data, and outputs a video signal based on the generated image data to the display device 20 via the video signal output unit 11, so that the display device 20 Display various images on the screen.

[Description of operation]
(1) Outline of Operation Next, an outline of the operation of the in-vehicle device 10 will be described.

The in-vehicle device 10 generates image data of an operation screen for accepting an operation on the own device, and causes the display device 20 to display the operation screen. One or a plurality of objects (icons, etc.) are drawn on this operation screen, and the in-vehicle device 10 behaves in a manner that the driver inclines the face sideways while gazing at any object (in other words, The operation corresponding to the object is detected in response to the behavior of shaking the face in the horizontal direction (hereinafter, this behavior is also referred to as an operation confirmation behavior).

Note that, for example, a behavior in which the driver tilts the face upward or downward or a behavior that moves the face back and forth may be regarded as an operation confirmation behavior. However, there is a possibility that such a behavior is unexpectedly performed by the driver, and the intention of the driver can be more accurately reflected by setting the behavior of tilting the face in the horizontal direction as the operation confirmation behavior. Further, for example, the behavior of the entire other head may be detected as the operation confirmation behavior, such as a behavior of repeatedly shaking the entire head left and right or up and down.

The line-of-sight detection device 30 detects the position coordinates of the left and right eyes of the driver on the photographed image and the gazing point on the operation screen based on the photographed image of the driver's face in response to an instruction from the vehicle-mounted device 10. Provided to the apparatus 10.

The photographed image of the driver's face is viewed from a position closer to the lower side than the front of the face, and is projected with the driver's face slightly looked up. Therefore, when the position coordinate of the left eye of the driver is L (lx, ly) and the position coordinate of the right eye is R (rx, ry), the vertical direction (y-axis direction) is obtained when the driver's face is facing directly in front. The difference Δd = ry−ly, which is the interval between the left and right eye positions, is a value near 0 (see FIG. 3). As the driver's face is tilted laterally, the absolute value of Δd increases.

Therefore, the control unit 12 of the in-vehicle device 10 that acquired the position coordinates L and R calculates Δd based on this, detects the driver's face orientation based on Δd, and further determines the operation confirmation behavior based on the face orientation. Is detected.

Of course, in addition to this, it is also conceivable to detect the orientation of the face using, for example, a photographed image of the driver's face from the front or above, a photographed image of the entire head, and the like. In addition to the position coordinates of the driver's eyes, for example, it is conceivable to detect feature points from the driver's face image and detect the face orientation based on the position of the feature points.

However, by paying attention to the position of the driver's eyes, the orientation of the driver's face can be detected more easily. In addition, by using an image of the driver's face taken from below and paying attention to the distance between the vertical positions of the left and right eyes, the face orientation can be detected easily and accurately. The processing load when detecting the behavior can be reduced.

4 and 5, the control unit 12 displays the operation screen 100 on the display device 20, and then based on the position coordinates L and R, the absolute value | Δd ( t) | Then, when | Δd (t) | becomes equal to or greater than a predetermined threshold value D1, the control unit 12 sets the time at that time as Tsta. Further, the control unit 12 specifies an object on the operation screen 100 that the driver is gazing at time Tsta based on the gazing point acquired from the line-of-sight detection device 30, and highlights the object 101 on the display device 20. You may let them.

Thereafter, the control unit 12 periodically calculates | Δd (t) |, and when | Δd (t) | becomes equal to or greater than the threshold value D2 by the time Tend (before a predetermined time elapses from the time Tsta). It is assumed that the operation confirmation behavior has been made.

Further, during the detection of the operation confirmation behavior, the control unit 12 specifies the object on the operation screen on which the driver is gazing based on the gazing point acquired from the line-of-sight detection device 30. When the driver is gazing at the same object 101 (the highlighted object 101) until the operation confirmation behavior is detected, the control unit 12 performs an operation corresponding to the object 101. The processing corresponding to the object 101 is executed. At this time, the control unit 12 may further change the display mode of the object 101.

In addition to this, the operation confirmation behavior may be detected as follows. That is, after the time Tsta, the control unit 12 periodically calculates | Δd (t) |, calculates an integrated value of | Δd (t) | starting from the time Tsta, and performs the integration until the time Tend. When the value is equal to or greater than the threshold value D3, it may be considered that the operation confirmation behavior has been made (see FIG. 6).

In addition, during the detection of the operation confirmation behavior, the control unit 12 may similarly specify an object on the operation screen that the driver is gazing at. Then, when the driver is gazing at the same object 101 until the operation confirmation behavior is detected, the control unit 12 may consider that an operation corresponding to the object 101 has been performed.

In addition to this, the control unit 12 detects the behavior of tilting the face to the right (right operation confirmation behavior) and the behavior of tilting to the left (left operation confirmation behavior) as different operation confirmation behaviors. May be.

As described above, since Δd (t) = ry−ly, when the driver tilts the face to the right, Δd (t) is a negative value, and when the face tilts to the left , Δd (t) is a positive value. For this reason, the direction in which the driver tilted the face can be determined based on the sign of Δd (t), and the control unit 12 can detect the right and left operation confirmation behavior based on Δd (t).

Therefore, when the right operation confirmation behavior is detected while the driver is gazing at any object, the control unit 12 detects the operation R corresponding to the object, and the left operation confirmation behavior is detected. In this case, the operation L corresponding to the object may be detected.

More specifically, regarding the left-side operation determination behavior, when Δd (t) is equal to or greater than a predetermined threshold DL1, the control unit 12 determines the time (time Tsta) based on the gazing point acquired from the line-of-sight detection device 30. The object on the operation screen that the driver is gazing at is specified (see FIG. 7).

Thereafter, the control unit 12 periodically calculates | Δd (t) |, and if | Δd (t) | is equal to or greater than the threshold DL2 by time Tend, the left-side operation confirmation behavior is assumed to have been performed. I reckon. If the driver is gazing at the same object during the detection of the left-side operation determination behavior, the control unit 12 regards that the operation L corresponding to the object has been performed, and corresponds to the object and the operation L. Execute the process.

On the other hand, regarding the right operation confirmed behavior, as in the left operation confirmed behavior, when Δd (t) is equal to or less than a predetermined threshold DR1, the control unit 12 identifies the object that the driver is watching (see FIG. 7). ). Thereafter, when | Δd (t) | becomes equal to or less than the threshold value DR2 by time Tend, the control unit 12 considers that the right operation confirmation behavior has been performed.

If the driver is gazing at the same object during the detection of the right-side operation determination behavior, the control unit 12 regards that the operation R corresponding to the object has been performed, and corresponds to the object and the operation R. Execute the process.

Regarding the left-side operation confirmation behavior, the control unit 12 periodically calculates Δd (t) after time Tsta when Δd (t) becomes equal to or greater than the threshold value DL1, and Δd (t) starting from time Tsta. The integral value of may be calculated. Then, when the integrated value becomes equal to or greater than the threshold value DL3 by time Tend, it may be considered that the left operation confirmation behavior has been made (see FIG. 8).

Similarly, the control unit 12 may calculate the integral value of Δd (t) starting from the time Tsta after the time Tsta when Δd (t) becomes equal to or less than the threshold value DR1 with respect to the right-side operation confirmation behavior. . Then, when the integrated value becomes equal to or less than the threshold value DR2 by time Tend, it may be considered that the left operation confirmation behavior has been performed (see FIG. 8).

Of course, during the detection of the left or right operation confirmation behavior by the above method, the control unit 12 similarly identifies the object on the operation screen that the driver is watching, and the driver is watching the same object during the detection. In this case, it is needless to say that the operation L or R has been performed.

By detecting the operation of the driver in this way, the driver can easily and freely operate the in-vehicle device 10 even when both hands are closed by driving.

(2) Operation detection process Next, as an example, after time Tsta, an integrated value of | Δd (t) | is calculated, and it is determined whether or not the integrated value is equal to or greater than a threshold value D3. The operation detection process for detecting the above will be described with reference to the flowcharts shown in FIGS. This process is executed when the in-vehicle device 10 starts accepting an operation from a driver or the like.

In S200, the control unit 12 of the in-vehicle device 10 causes the display device 20 to display an operation screen, and the process proceeds to S205.

In S205, the control unit 12 instructs the line-of-sight detection device 30 to detect the position coordinates L and R of the left and right eyes of the driver. In the line-of-sight detection device 30, the position coordinates L and R are detected according to the instruction, and when the detection is successful, the position coordinates L and R are provided to the in-vehicle device 10.

In subsequent S210, the control unit 12 determines whether or not the position coordinates L and R have been successfully detected and acquired. If a positive determination is obtained (S210: Yes), the control unit 12 shifts the process to S205, and if a negative determination is obtained (S210: No), the control unit 12 shifts the process to S215. .

In S215, the control unit 12 specifies the position coordinates of the left and right eyes of the driver, calculates | Δd (t) | described above based on these coordinates (S220), and shifts the processing to S225.

In S225, the control unit 12 determines whether or not | Δd (t) | is greater than or equal to the threshold value D1, and when an affirmative determination is obtained (S225: Yes), the process proceeds to S230 and a negative determination is made. Is obtained (S225: No), the process proceeds to S205.

In S230, the control unit 12 instructs the line-of-sight detection device 30 to detect the driver's gaze point. The line-of-sight detection device 30 detects a gazing point according to the instruction, and when the detection is successful, the gazing point is provided to the in-vehicle device 10.

In subsequent S235, the control unit 12 determines whether or not the gaze point has been successfully detected and the gaze point has been acquired. Then, when an affirmative determination is obtained (S235: Yes), the control unit 12 proceeds to S240, and when a negative determination is obtained (S235: No), the control unit 12 proceeds to S230. .

In S240, the control unit 12 sets the current time as Tsta, determines the provided gaze point as the gaze point at time Tsta, and shifts the processing to S245.

In S <b> 245, the control unit 12 specifies an object (an object being watched by the driver) drawn in an area including the gazing point on the operation screen displayed on the display device 20, and the object is displayed on the display device 20. Is highlighted, and the process proceeds to S250. Note that if there is no corresponding object, the control unit 12 may shift the process to S205 again.

In S250, the control unit 12 determines whether or not the current time t has reached Tend (whether or not a predetermined time has elapsed from time Tsta). If a positive determination is obtained (S250: Yes), the control unit 12 shifts the process to S255, and if a negative determination is obtained (S250: No), the control unit 12 shifts the process to S310. .

In S255, the control unit 12 instructs the line-of-sight detection device 30 to detect the gaze point of the driver, and the process proceeds to S260.

In S260, the control unit 12 determines whether or not the gazing point has been successfully detected and the gazing point has been acquired from the line-of-sight detection device 30. When a positive determination is obtained (S260: Yes), the control unit 12 shifts the process to S265, and when a negative determination is obtained (S260: No), the control unit 12 shifts the process to S255. To do.

In S265, the control unit 12 determines the provided gazing point as the gazing point of the current driver, and the process proceeds to S270.

In S270, the control unit 12 specifies an object (an object that the driver is gazing at) drawn in an area including the above-mentioned gazing point on the operation screen, and the object and the highlighted object (at time Tsta). It is determined whether or not (objects being watched) match. Then, when an affirmative determination is obtained (S270: Yes), the control unit 12 proceeds to S275, and when a negative determination is obtained (S270: No), the control unit 12 proceeds to S310. .

In S275, the control unit 12 instructs the line-of-sight detection device 30 to detect the position coordinates L and R of the left and right eyes of the driver, and the process proceeds to S280.

In S280, the control unit 12 determines whether or not the position coordinates L and R have been successfully detected and can be acquired from the line-of-sight detection device 30. If an affirmative determination is obtained (S280: Yes), the control unit 12 shifts the process to S285, and if a negative determination is obtained (S280: No), the control unit 12 shifts the process to S275. .

In S285, the control unit 12 specifies the position coordinates of the left and right eyes of the driver, calculates | Δd (t) | described above based on these coordinates (S290), and shifts the processing to S295.

In S295, the control unit 12 calculates an integrated value of | Δd (t) | from time Tsta to the present time, and proceeds to S300.

In S300, the control unit 12 determines whether or not the integrated value of | Δd (t) | is equal to or greater than the threshold value D3. Then, when an affirmative determination is obtained (S300: Yes), the control unit 12 proceeds to S305, and when a negative determination is obtained (S300: No), the control unit 12 proceeds to S250. .

In S305, the control unit 12 considers that an operation corresponding to the identified object (in other words, an object that was being watched by the driver while the operation confirmation behavior was performed) was performed, and responds to the operation. The process is executed and the present process is terminated.

On the other hand, in S310, which shifts to the case where the current time t reaches Tend, the control unit 12 causes the display device 20 to release the highlighted display of the object being watched at time Tsta, and ends this processing. To do.

Note that after the time Tsta, if it is a case where an operation confirmation behavior is detected by determining whether or not | Δd (t) | is equal to or greater than the threshold value D2, the control unit 12 determines that | Δd (t ) | Instead of determining the integral value, it may be determined whether | Δd (t) | is equal to or greater than the threshold value D2.

If the right and left operation fixed behaviors are detected as different operation fixed behaviors, the control unit 12 calculates Δd (t) instead of | Δd (t) | in S220 and S290, In S295, instead of the integrated value of | Δd (t) |, the integrated value of Δd (t) may be calculated.

In S225, the control unit 12 may compare Δd (t) with the threshold value DR1 or DL1 instead of comparing | Δd (t) | (specifically, Δd (t) is If it is a positive value, it is determined whether or not Δd (t) is greater than or equal to the threshold value DL1, and if it is a negative value, it may be determined whether or not Δd (t) is less than or equal to the threshold value DR1.

In S300, the control unit 12 may compare Δd (t) with the threshold value DR2 or DL2 instead of comparing the integrated value of | Δd (t) | (specifically, Δd ( If t) is a positive value, it is determined whether or not Δd (t) is greater than or equal to the threshold DL2, and if it is a negative value, it may be determined whether or not Δd (t) is less than or equal to the threshold DR2. .

In S300, the control unit 12 may compare the integrated value of Δd (t) with the threshold value DR3 or DL3 instead of comparing the integrated value of | Δd (t) | If the integrated value of Δd (t) is a positive value, it is determined whether the integrated value of Δd (t) is greater than or equal to the threshold value DL3. If the integrated value of Δd (t) is a negative value, the integrated value of Δd (t) is the threshold value. It may be determined whether or not DR3 or less).

[Second Embodiment]
FIG. 11 is a block diagram illustrating a configuration of a mobile device 50 (for example, a smartphone) according to the second embodiment. The portable device 50 includes a display unit 51 that displays various images, a CPU, a ROM, a RAM, an I / O, and the like. The portable device 50 is based on a control unit 52 that performs overall control of the portable device 50 and a photographed image of the user's face. A line-of-sight detection unit 53 (eye tracker) that detects a position where the user is gazing.

As shown in FIG. 12, the display unit 51 has a substantially rectangular screen 51 a, and the screen 51 a is arranged on the surface of the portable device 50 having a rectangular plate shape, One of the short sides in the screen 51a is the upper side and the other is the lower side. The portable device 50 is configured to be used in a state where the upper short side of the screen 51a is positioned upward.

The line-of-sight detection unit 53 includes a light projecting unit that projects infrared rays toward the user's face, a camera 53a that captures the user's face, and the like, and specifies the position of the user's eyes from the captured image. Analyzing the image of the user's eyes. And a user's gaze direction is detected from an analysis result, and the coordinate (gaze point) of the position on the screen 51a of the display part 51 which the user is gazing based on this gaze direction is detected. The camera 53a is disposed adjacent to the lower side of the screen 51a.

The control unit 12 is configured to generate various image data, and displays various images on the screen 51 a of the display unit 51.

In the portable device 50 of the second embodiment, the same operation detection process as that of the first embodiment is performed, and when the user performs an operation confirmation behavior while gazing at an object on the operation screen, an operation corresponding to the object is performed. Detected.

That is, the control unit 52 of the portable device 50 generates image data of an operation screen for accepting an operation on the own device, and displays the operation screen on the screen 51 a of the display unit 51. Further, since the camera 53a is arranged adjacent to the lower short side of the screen 51a, the user's face that is gazing at the screen 51a is usually photographed from a lower viewpoint from the front of the face. It becomes a state to do.

For this reason, the photographed image of the user's face by the camera 53a is projected with the face slightly looked up, as in the first embodiment, and is set to Δd (t) which is the interval between the positions of the left and right eyes in the vertical direction. Based on this, the orientation of the user's face can be detected.

Therefore, the line-of-sight detection unit 53 detects the position coordinates L and R of the left and right eyes of the driver and the gazing point on the captured image based on the captured image of the user's face in accordance with an instruction from the control unit 52. This is provided to the control unit 52.

On the other hand, the control unit 52 that has acquired the position coordinates L and R calculates Δd based on them in the same manner as in the first embodiment, detects the driver's face orientation based on Δd, and further determines the operation confirmation behavior. To detect. Then, when the user is gazing at the same object during the detection of the operation confirmation behavior, the control unit 52 regards that the operation corresponding to the object has been performed.

[Other Embodiments]
(1) In the operation detection process according to the first and second embodiments, when the same object on the operation screen is being watched by the driver or the like while the operation is confirmed by the driver or the user, the object is detected. An operation corresponding to is detected.

However, the present invention is not limited to the case where such an operation screen is displayed. For example, the driver is determined in advance on the screen of the display device 20 (or the display unit 51 of the portable device 50) while the operation confirmation behavior is performed. When an area is watched, an operation corresponding to the area may be detected. Even in such a case, the same effect can be obtained.

(2) Further, in the operation detection processing in the first and second embodiments, an object that the driver or the like is gazing at when the face orientation of the driver or the like is inclined by a predetermined amount in the lateral direction is specified. The Then, when an operation confirmation behavior is made while gazing at the object, an operation corresponding to the object is detected.

However, the present invention is not limited to this. For example, when the control unit 12 of the in-vehicle device 10 or the control unit 52 of the portable device 50 detects that a driver or the like is gazing at a specific object, The degree of inclination of the face direction of a driver or the like in the horizontal direction may be detected. Then, when the degree of inclination of the driver or the like changes by a predetermined amount from what was initially detected while gazing at the object, these control units consider that the operation confirmation behavior has been made, and A corresponding operation may be detected. Even in such a case, the same effect can be obtained.

[Correspondence with Claims]
The correspondence between the terms used in the description of the above embodiment and the terms used in the description of the claims is shown.

The in-vehicle device 10 in the first embodiment and the portable device 50 in the second embodiment correspond to an operation device, and the display unit 51 of the portable device 50 corresponds to a display device.

In addition, S200 of the operation detection process is displayed on the display unit, S230 and S255 are specified on the specified unit, S205 to S225 and S275 to S300 are detected on the detection unit, and S270 and S305 are displayed on the operating unit. , Equivalent to operating.
The flowchart in the present disclosure or the process of the flowchart is configured by a plurality of sections (or referred to as steps), and each section is expressed as, for example, S100. Furthermore, each section can be divided into multiple subsections, while multiple sections can be combined into a single section. Further, each section configured in this manner can be referred to as a device, module, or means.

Although the present disclosure has been described based on the embodiments, it is understood that the present disclosure is not limited to the embodiments and structures. The present disclosure includes various modifications and modifications within the equivalent range. In addition, various combinations and forms, as well as other combinations and forms including only one element, more or less, are within the scope and spirit of the present disclosure.

Claims

A display unit (S200) for displaying an image on the screen of the display device (20, 51);
A specifying unit (S230, S255) for specifying a gaze point on the screen on which the user is gazing based on a photographed image of the user's face;
A detection unit (S205 to S225, S275 to S300) that detects a behavior of moving the entire head of the user based on the captured image;
When the behavior of the user is detected by the detection unit, an operation unit (S270, S305) that considers that an operation corresponding to the gazing point identified by the identification unit is performed when the behavior is performed;
An operating device (10, 50).
The operating device according to claim 1,
The display unit displays an image on which one or more objects are drawn on the screen of the display device,
When the behavior of the user is detected by the detection unit, the operation unit is an area in which any one of the objects is drawn with the point of interest identified by the identification unit when the behavior is performed In the case of the operation device, it is assumed that an operation corresponding to the object has been performed.
In the operating device according to claim 1 or 2,
The behavior is an operating device that is a movement that tilts the face in a lateral direction.
The operating device according to any one of claims 1 to 3,
The operation unit detects the user's eye position based on the captured image, and detects the behavior based on the eye position.
In the operating device according to claim 4, which refers to claim 3,
The captured image is an image of the user's face taken from below,
The detection unit detects an interval between vertical positions of the left and right eyes of the user in the captured image, and determines a degree by which the user tilts the face in the horizontal direction based on the interval. An operating device that detects the behavior based on the degree.
In the operating device according to any one of claims 1 to 5,
The operating device is an operating device mounted on a vehicle.
Displaying an image on the screen of the display device (20, 51) (S200);
Identifying a point of interest on the screen on which the user is gazing based on a captured image of the user's face (S230, S255);
Detecting a predetermined behavior of moving the entire user's head based on the captured image (S205 to S225, S275 to S300);
When the behavior of the user is detected in the detection, it is considered that an operation corresponding to the gaze point specified in the specification is performed when the behavior is performed (S270, S305); ,
An operation detection method comprising: