WO2021147897A1

WO2021147897A1 - Apparatus and method for controlling a vehicle-mounted device, and vehicle-mounted device

Info

Publication number: WO2021147897A1
Application number: PCT/CN2021/072867
Authority: WO
Inventors: Kazuhiko Sakai; Weicheng Zhou
Original assignee: Faurecia Clarion Electronics (Xiamen) Co., Ltd.
Priority date: 2020-01-21
Filing date: 2021-01-20
Publication date: 2021-07-29
Also published as: CN113220111A

Abstract

An apparatus for controlling a vehicle-mounted device are provided. The apparatus includes: a sightline detection unit used to detect a sightline of a driver; a first display unit used to display a selection cursor over a target control in response to detecting that the sightline of the driver falls into a target region of a first display screen, the target control being a control displayed in the target region; and a gesture detection unit used to detect a hand motion of the driver after the selection cursor is displayed over the target control. The first display unit is further used to move the selection cursor according to the hand motion of the driver.

Description

APPARATUS AND METHOD FOR CONTROLLING A VEHICLE-MOUNTED DEVICE, AND VEHICLE-MOUNTED DEVICE

This application claims priority to Chinese Patent Application No. 202010072018.8, filed on January 21, 2020, which is incorporated herein by reference in its entirety.

TECHNICAL FIELD

The present application relates to the field of vehicle-mounted devices, and in particular, to an apparatus and a method for controlling a vehicle-mounted device, and a vehicle-mounted device.

BACKGROUND

A vehicle is usually equipped with a vehicle-mounted device.

SUMMARY

The present application provides an apparatus and a method for controlling a vehicle-mounted device, and a vehicle-mounted device, which may improve an operational convenience of the vehicle-mounted device.

In order to achieve the above purpose, in embodiments of the present application, the following technical solutions are adopted.

In a first aspect, embodiments of the present application provide an apparatus for controlling a vehicle-mounted device. The apparatus for controlling the vehicle-mounted device includes: a sightline detection unit used to detect a sightline of a driver; a first display unit used to display a selection cursor over a target control in response to detecting that the sightline of the driver falls into a target region of a first display screen, the target control being a control displayed in the target region; a gesture detection unit used to detect a hand motion of the driver after the selection cursor is displayed over the target control. The first display unit is further used to move the selection cursor according to the hand motion of the driver.

In a second aspect, embodiments of the present application provide a method for controlling a vehicle-mounted device, which includes: detecting a sightline of a driver; displaying a selection cursor over a target control in response to detecting that the sightline of the driver falls into a target region of a first display screen, the target control being a control displayed in the target region; detecting a hand motion of the driver after displaying the selection cursor over the target control; and moving the selection cursor according to the hand motion of the driver.

In a third aspect, embodiments of the present application provide an apparatus for controlling a vehicle-mounted device, which includes: a processor and a memory that is used for storing instructions that are capable of being executed by the processor; the processor is configured to execute the instructions to implement the method for controlling the vehicle-mounted device provided in the second aspect.

In a fourth aspect, embodiments of the present application provide a computer-readable storage medium, which stores instructions that, when executed by a computer, cause the computer to perform the method for controlling the vehicle-mounted device provided in the second aspect. The computer-readable storage medium may be a non-transitory computer-readable storage medium.

In a fifth aspect, embodiments of the present application provide a computer program product that, when run on a computer, causes the computer to perform the method for controlling the vehicle-mounted device provided in the second aspect.

In a sixth aspect, embodiments of the present application provide a vehicle-mounted device. The vehicle-mounted device includes the apparatus for controlling the vehicle-mounted device provided in the first aspect and the first display screen.

The present application take into account that in a case where the area of the vehicle-mounted display screen is too large or the position of the vehicle-mounted display screen is far away from the driver, the driver may need to make a larger range of movement to realize the operations on a part, away from the driver, of the vehicle-mounted display screen. For the above problem, the present application proposes to combine the sightline recognition and the gesture recognition to detect the sightline of the driver, and thus the position range of the control that the driver wants to operate on the vehicle-mounted display screen is determined. After the sightline of the driver is detected to fall into a target region of the vehicle-mounted display screen, a selection cursor is displayed over a target control in the target region. Then, by detecting a hand motion of the driver, a position of the selection cursor is moved, so that the selection cursor can be accurately positioned on the control that the driver wants to operate. In this way, after the selection cursor is moved over the control that the driver wants to operate, the vehicle-mounted device may be controlled by triggering the control. The solution provided by the present application can not only avoid a problem of large control errors caused by the incapability of accurately recognize the position corresponding to the sightline when using the sightline recognition, but also avoid a problem of complicated operations caused by only using gesture recognition for control.

BRIEF DESCRIPTION OF THE DRAWINGS

In order to describe technical solutions in embodiments of the present application or in the prior art more clearly, the accompanying drawings to be used in the description of the embodiments or the prior art will be introduced briefly below.

FIG. 1 is a first schematic diagram showing a structure of a vehicle-mounted device, in accordance with embodiments of the present application;

FIG. 2 is a first schematic diagram showing a structure of an apparatus for controlling a vehicle-mounted device, in accordance with embodiments of the present application;

FIG. 3 is a diagram showing a working process of an apparatus for controlling a vehicle-mounted device, in accordance with embodiments of the present application;

FIG. 4 is a first schematic diagram of a display interface, in accordance with embodiments of the present application;

FIG. 5 is a second schematic diagram of a display interface, in accordance with embodiments of the present application;

FIG. 6 is a schematic diagram showing a correspondence between gestures and control actions, in accordance with embodiments of the present application;

FIG. 7 is a third schematic diagram showing a structure of an apparatus for controlling a vehicle-mounted device, in accordance with embodiments of the present application;

FIG. 8 is a fourth schematic diagram showing a structure of an apparatus for controlling a vehicle-mounted device, in accordance with embodiments of the present application;

FIG. 9 is a second schematic diagram showing a structure of a vehicle-mounted device, in accordance with embodiments of the present application;

FIG. 10 is a schematic flow diagram of a detection of a sightline, in accordance with embodiments of the present application;

FIG. 11 is a schematic diagram of a detection of a sightline of a driver, in accordance with embodiments of the present application;

FIG. 12 is a fifth schematic diagram showing a structure of an apparatus for controlling a vehicle-mounted device, in accordance with embodiments of the present application; and

FIG. 13 is a sixth schematic diagram showing a structure of an apparatus for controlling a vehicle-mounted device, in accordance with embodiments of the present application.

DETAILED DESCRIPTION

The embodiments of the present application will be described below with reference to the accompanying drawings.

In the embodiments of the present application, the term "for example" or "such as" is used to give an example, an illustration, or an explanation. Any embodiment or design described as "for example" or "such as" in the embodiments of the present application will not be construed as more preferred or advantageous over other embodiments or designs. Rather, use of the term "for example" or "such as" is intended to present relevant concepts in a specific manner. In addition, in the description of the embodiments of the present application, the term "a plurality of" means two or more unless otherwise specified.

Terms used in the embodiments of the present application are only for a purpose of describing specific embodiments, and are not intended to limit the present application. Singular forms such as "a" , "an" and "the" used in the embodiments of the present application and the appended claims are also intended to include plural forms, unless the context clearly indicates other meanings.

It will be understood that, although the terms such as "first" , "second" and "third" may be used in the embodiments of the present application to describe various markers, thresholds, signals and instructions, these markers, thresholds, signals and instructions will not be limited to these terms. These terms are only used to distinguish the markers, thresholds, signals, and instructions from one another. For example, without departing from the scope of the embodiments of the present application, a first target marker may also be referred to as a second target marker or the like.

Depending on the context, the word "if" used herein may be construed as "in a case where" or "when" or "in response to determining" or "in response to detecting" . Similarly, depending on the context, the phrase "if (a stated condition or event) is determined" or "if (a stated condition or event) is detected" may be construed as" in a case where the stated condition or event is determined" or "in response to determining the stated condition or event" or "in a case where the stated condition or event is detected" or "in response to detecting the stated condition or event" .

At present, a function of a vehicle-mounted device may be realized by performing a series of touch operations on a vehicle-mounted display screen. However, as the vehicle-mounted display screen becomes wider and longer, it may be difficult for a user (e.g., the driver) to touch an edge region of the vehicle-mounted display screen so as to control the vehicle-mounted device to perform some of its functions, which may cost the user more time and more large movement to touch the edge region of the vehicle-mounted display screen. That is to say, as an area of the vehicle-mounted display screen increases, the operational convenience of the vehicle-mounted device may be reduced.

Considering that in a case where a position where the vehicle-mounted display screen is located is far away from a driver, if the control of a vehicle-mounted device is realized through touch operations performed by the driver on the vehicle-mounted display screen, it becomes more inconvenient for the driver to perform direct touch operations on the vehicle-mounted display screen. Therefore, there is a need to control the vehicle-mounted device in an indirect touch manner. However, in existing control methods, whether the control is realized by using gesture recognition or voice recognition, there are problems such as low recognition precision and complicated operations.

Therefore, the present application provides an apparatus for controlling a vehicle-mounted device and a method for controlling a vehicle-mounted device. In the present application, a sightline of the driver is detected to determine a position range of a control on the vehicle-mounted display screen that the driver wants to operate. Then, after it is detected that the sightline of the driver falls into a target region of the vehicle-mounted display screen, a selection cursor is displayed over a target control in the target region. Then, a hand motion of the driver is detected to move a position of the selection cursor, so that the selection cursor can be accurately positioned on the control that the driver wants to operate. In this way, after the selection cursor is moved over the control that the driver wants to operate, the control of the vehicle-mounted device may be realized by triggering the control.

Based on the above inventive concept, embodiments of the present application provide a method for controlling the vehicle-mounted device, which is applied to the apparatus for controlling the vehicle-mounted device. For example, FIG. 1 is a schematic diagram showing a structure of a vehicle-mounted device provided by the present application. The vehicle-mounted device 10 includes a first camera 102, a second camera 103, and a first display screen 104. The first camera 102 is used for capturing facial images of the driver, so as to detect the sightline of the driver according to the facial images of the driver. The second camera 103 is used for capturing hand images of the driver, so as to detect the hand motion of the driver according to the hand images of the driver. The first display screen 104 is used for displaying various types of information. For example, the first display screen 104 may be a display screen of a center console of a vehicle.

In some examples, the first camera 102 and the second camera 103 are two different cameras. Of course, in some other scenarios, the facial image and the hand image of the driver may be captured by a same camera. That is, the first camera 102 and the second camera 103 are the same camera.

In addition, as shown in FIG. 1, the vehicle-mounted device 10 further includes the apparatus 101 for controlling the vehicle-mounted device which is used to apply methods for controlling the vehicle-mounted device provided by the present application.

As shown in FIG. 2, the apparatus 101 for controlling the vehicle-mounted device may include a sightline detection unit 1011, a first display unit 1012, and a gesture detection unit 1013.

The sightline detection unit 1011 is used for detecting the sightline of the driver. For example, after the first camera 102 in FIG. 1 transmits a captured facial image of the driver to the apparatus 101 for controlling the vehicle-mounted device, the sightline detection unit 1011 detects the sightline of the driver by using the obtained facial image of the driver. The first display unit 1012 is used for controlling contents displayed on the vehicle-mounted display screen. For example, the first display unit 1012 is used for controlling contents displayed on the first display screen 104 in FIG. 1. The gesture detection unit 1013 is used for detecting the hand motion of the driver. For example, after the second camera 103 in FIG. 1 transmits a captured hand image of the driver to the apparatus 101 for controlling the vehicle-mounted device, the gesture detection unit 1013 detects the hand motion of the driver by using the obtained hand image of the driver.

In some implementations, the above functional units may be integrated in one processing unit, or each unit may physically exist separately, or two or more of the units may be integrated in one unit. The present application does not limit this.

Hereinafter, a working process of the apparatus 101 for controlling the vehicle-mounted device provided by the embodiments of the present application will be described below. As shown in FIG. 3, the working process of the apparatus 101 for controlling the vehicle-mounted device may include the following steps, i.e., S201 to S204.

In S201, the sightline detection unit 1011 detects the sightline of the driver.

In S202, in response to detecting that the sightline of the driver falls into a target region of the first display screen, the first display unit 1012 displays a selection cursor over a target control.

The target control may include a control displayed in the target region.

For example, FIG. 4 is a schematic diagram of a display interface of the first display screen, in accordance with embodiments of the present application. As can be seen, the first display screen is a long and narrow rectangular display screen. When the driver directly touches the display screen, it is difficult for the driver to perform touch operations on a part of the display screen away from the driver. Therefore, in the present application, a display region where the driver wants to perform operations is determined by detecting the sightline of the driver. For example, when the sightline of the driver is determined, according to the sightline of the driver, to be located in the target region A in FIG. 4, the selection cursor is displayed over a target control, i.e., "button A" , in the target region A. For example, the button A is highlighted. In this case, the contents displayed on the first display screen are as shown in FIG. 5, in which the "button A" is highlighted (the shaded portion in the figure is used for indicating the highlighting) .

It will be noted that, the "control" referred to in the present application is used for indicating a pattern that is displayed on the display interface and that performs a corresponding operation or function after being triggered. In some scenarios, the "control" may also be referred to as a control component, a button, or other names. It will be understood that, these names may each be considered as the "control" referred to in the present application.

In addition, the "selection cursor" referred to in the present application refers to a cursor used for displaying a position of a control that is to be triggered currently on the display interface. For example, the selection cursor may be a pattern mark with a specific shape (e.g., an arrow-shaped pattern, or a finger-shaped image) . For another example, the selection cursor may also be represented in a manner of changing a display form of the control to be triggered (e.g., changing it to a display form different from other controls' display forms, such as highlighting the control to be triggered) .

In S203, after the selection cursor is displayed over the target control, the gesture detection unit 1013 detects the hand motion of the driver.

In S204, the first display unit 1012 moves the selection cursor according to the hand motion of the driver.

For example, FIG. 6 shows a correspondence between hand motions and methods of controlling the selection cursor, in accordance with embodiments of the present application. In (a) of FIG. 6, waving the hand leftwards will move the selection cursor to the left; waving the hand rightwards will move the selection cursor to the right; moving the hand upwards will move the selection cursor upwards; moving the hand downwards will move the selection cursor downwards; and making a fist means a determination, i.e., triggering a function corresponding to the selection cursor. In (b) of FIG. 6, pointing upwards, downwards, leftwards and rightwards with a finger correspond to moving the selection cursor upwards, downwards, leftwards and rightwards, respectively; and making an "OK" motion by hand means a determination, i.e., triggering a function corresponding to the selection cursor.

In this way, after the selection cursor is moved over the control that the driver wants to operate, the vehicle-mounted device may be controlled by triggering the control. The solution provided by the present application can not only avoid a problem of large control error caused by the incapability of accurately recognizing a position corresponding to the sightline when sightline recognition is used, but also avoid a problem of complicated operations caused by only using the gesture recognition for control.

In an implementation, as shown in FIG. 7, the apparatus 101 for controlling the vehicle-mounted device provided by the present application further includes a function trigger unit 1014. The function trigger unit 1014 is used for triggering functions corresponding to various controls displayed on the display screen.

As shown in FIG. 3, the working process of the apparatus 101 for controlling the vehicle-mounted device may further include:

S205, in response to the gesture detection unit 1013 detecting a first target gesture of the driver, triggering, by the function trigger unit 1014, the function of a control currently corresponding to the selection cursor.

For example, the first target gesture may be the motion of making a fist shown in (a) of FIG. 6, or the "OK" motion shown in (b) of FIG. 6. When it is detected that the driver makes the motion of making a fist by hand, the function of the control currently corresponding to the selection cursor is triggered. Or, when it is detected that the driver makes the "OK" motion by hand, the function of the control currently corresponding to the selection cursor is triggered.

In some embodiments, after the function of the control currently corresponding to the selection cursor is triggered, the first display unit 1012 is further used to control the first display screen 104 to display a display interface corresponding to the control. The function triggering unit 1014 is further used to trigger a function of a control displayed on the display screen 104, according to a gesture of the driver detected by the gesture detection unit 1013.

For example, after a control for playing audio files is triggered by the function triggering unit 1014, the vehicle-mounted device displays an audio file (e.g., the audio file of a song) , and the display interface (called music playing interface) corresponding to the control may be displayed on the first display screen 104 under the control of the first display unit 1012. According to the gesture of the driver detected by the gesture detection unit 1013, the function triggering unit 1014 may trigger the function of a control displayed on the music playing interface, such as a function of playing the next song, playing the previous song, increasing the volume, or decreasing the volume.

In one implementation, when it is unsuitable to perform operations by using gestures, the target region determined according to the sightline of the driver may be displayed on another screen closer to the driver to facilitate the driver to perform the touch operations.

In a possible design, in the present application, two display screens may be considered to be provided in the vehicle, and one display screen is farther from the driver, and the other display screen is closer to the driver. The display screen farther from the driver may have a larger display area for displaying more richer contents, and the display screen closer to the driver may be used to display the contents in the target region that are determined according to the sightline of the driver.

For example, as shown in FIG. 9, besides the first display screen 104, the vehicle-mounted device 10 may further include the second display screen 105 used to display contents displayed in the target region of the first display screen 104. The first display screen 104 may be the display screen closer to the driver, and the second display screen 105 may be the display screen farther from the driver. And the second display screen 105 is disposed at a distance from the driver greater than a distance from the first display screen 104 to the driver.

Further, as shown in FIG. 8, the apparatus 101 for controlling the vehicle-mounted device may further include a second display unit 1015. The second display unit 1015 is used to control the second display screen 105 to display contents. As shown in FIG. 3, the working process of the apparatus 101 for controlling the vehicle-mounted device may further include:

S206, in response to the gesture detection unit 1013 detecting a second target gesture of the driver, displaying, by the second display unit, contents currently displayed in the target region on the second display screen.

For example, the second target gesture may be a gesture of stretching five fingers from a fist-making state. The second display screen may be a display screen of an electronic device in wireless or wired connection with the vehicle-mounted device, such as a mobile phone, or a tablet computer. As such, after the driver makes a gesture of stretching five fingers from the fist-making state, the contents displayed in the target region are displayed on the display screen of the electronic device, and then the driver may perform relevant touch operations after placing the electronic device at a place convenient to operate for himself/herself.

In addition, in an implementation, the target region may be set as a rectangular region, and the present application further provides a method for determining whether the sightline of the driver falls into the target region. For example, as shown in FIG. 10, S201 in the present application may include the following steps.

In S2011, a position of an eye of the driver and a direction of the sightline of the driver are detected.

For example, as described above, the position of the eye of the driver and the direction of the sightline of the driver may be determined through the facial image of the driver captured by the camera by using an image recognition technology.

In S2012, coordinate values of four vertices of the target region in a target coordinate system are calculated according to the position of the eye of the driver.

The target coordinate system includes a space rectangular coordinate system which takes the position of the eye of the driver as an origin and takes a vertical direction, a horizontal direction and a direction right ahead of the vehicle as coordinate axes.

For example, FIG. 11 is a schematic diagram of the detection of the sightline of the driver, in accordance with embodiments of the present application. It is assumed that the position of the eye of the driver is at point O, a position of the camera for detecting the sightline of the driver is at point O', and the target region is a region shown by the rectangle abcd.

Since relative position between the camera and the target region is fixed, coordinates of four vertices a, b, c, and d of the target region in a space coordinate system with the point O' as an origin may be known. In addition, according to the position of the eye of the driver, the coordinates of a, b, c, and d in the space coordinate system with the point O' as the origin may be converted to coordinates in the target coordinate system with the point O as the origin. For example, the target coordinate system may be the space rectangular coordinate system which takes the position of the eye of the driver as the origin and takes the vertical direction, the horizontal direction and the direction right ahead of the vehicle as the coordinate axes.

It is assumed that the coordinates of the four vertices a, b, c, and d, relative to the target coordinate system are (x1, y1, z1) , (x2, y2, z2) , (x3, y3, z3) , and (x4, y4, z4) , and a coordinate of the point O is (0, 0, 0) .

For convenience of description, in the following description of the present application, in the target coordinate system, a coordinate axis in the vertical direction is referred to as a z-axis, a coordinate axis pointing to the front of the vehicle is referred to as an x-axis, and a coordinate axis in the horizontal direction is referred to as a y-axis.

In S2013, it is determined whether the sightline of the driver falls into the target region, according to the coordinate values of the four vertices of the target region in the target coordinate system and the direction of the sightline of the driver.

For example, after the coordinate values of the four vertices of the target region in the target coordinate system and the direction of the sightline of the driver are known, it may be determined whether the sightline of the driver falls into the target region. For example, by determining whether an point of intersection, at which the sightline of the driver and a plane where the display screen is located meet, falls into the rectangle formed by the a, b, c, and d, it may be determined whether the sightline of the driver falls into the target region.

In an implementation, S2013 may include: determining whether the sightline of the driver falls into the target region if the following conditions 1 to 4 are met.

Condition 1: an included angle between the sightline E of the driver and an Oxy plane is greater than an included angle between an Oab plane and the Oxy plane.

Condition 2: the included angle between the sightline E of the driver and the Oxy plane is less than an included angle between an Ocd plane and the Oxy plane.

Condition 3: an included angle between the sightline E of the driver and an Oyz plane is greater than an included angle between an Oac plane and the Oyz plane.

Condition 4: the included angle between the sightline E of the driver and the Oyz plane is less than an included angle between an Obd plane and the Oyz plane.

The Oxy plane is a plane formed by the x-axis and the y-axis in the target coordinate system. The Oab plane is a plane formed by the O point, the a point, and the b point. The Oyz plane is a plane formed by the y-axis and the z-axis in the target coordinate system. The Ocd plane is a plane formed by the O point, the c point, and the d point. The Oac plane is a plane formed by the O point, the a point, and the c point. The Obd plane is a plane formed by the O point, the b point, and the d point.

Further, in a possible design, the condition 1 may be that the included angle between the sightline E of the driver and the Oxy plane is greater than θ1;

where x1, y1, and z1 are coordinates of the vertex a of the target region on the x-axis, y-axis, and z-axis in the target coordinate system, respectively; and x3, y3, and z3 are coordinates of the vertex c of the target region on the x-axis, y-axis, and z-axis in the target coordinate system, respectively. In addition, θ1 is greater than or equal to 0 degrees and is less than or equal to 180 degrees (i.e., 0≤θ1≤180°) .

For example, in a case where the coordinates of the a piont and the c point relative to the target coordinate system are (x1, y1, z1) and (x3, y3, z3) , respectively, according to the formula for cross product, it will be known that a normal vector of the plane Oac is Oac = Oa × Oc. That is, Oac= (y1z3-z1y3, z1x3-z3x1, x1y3-y1x3) .

Further, according to the cosine formula, it will be known that:

Further, it will be known that:

Similarly, in a possible design, the condition 2 may be that the included angle between the sightline E of the driver and the Oxy plane is less than θ2.

where x1, y1, and z1 are the coordinates of the vertex a of the target region on the x-axis, y-axis, and z-axis in the target coordinate system, respectively; x2, y2, and z2 are coordinates of the vertex b of the target region on the x-axis, y-axis, and z-axis in the target coordinate system, respectively; and x3, y3, and z3 are the coordinates of the vertex c of the target region on the x-axis, y-axis, and z-axis in the target coordinate system, respectively. In addition, θ2 is greater than or equal to 0 degrees and is less than or equal to 180 degrees (i.e., 0≤θ2≤180°) .

The condition 3 may be that the included angle θ3 between the sightline E of the driver and the Oyz plane meets:

where x1, y1, and z1 are the coordinates of the vertex a of the target region on the x-axis, y-axis, and z-axis in the target coordinate system, respectively; x3, y3, and z3 are the coordinates of the vertex c of the target region on the x-axis, y-axis, and z-axis in the target coordinate system, respectively; and x4, y4, and z4 are coordinates of the vertex d of the target region on the x-axis, y-axis, and z-axis in the target coordinate system, respectively. In addition, θ3 is greater than or equal to 0 degrees and is less than or equal to 180 degrees (i.e., 0≤θ3≤180°) .

The condition 4 may be that the included angle between the sightline E of the driver and the Oyz plane is less than θ4.

where x2, y2, and z2 are the coordinates of the vertex b of the target region on the x-axis, y-axis, and z-axis in the target coordinate system, respectively; x3, y3, and z3 are the coordinates of the vertex c of the target region on the x-axis, y-axis, and z-axis in the target coordinate system, respectively; and x4, y4, and z4 are the coordinates of the vertex d of the target region on the x-axis, y-axis, and z-axis in the target coordinate system, respectively. In addition, θ4 is greater than or equal to 0 degrees and is less than or equal to 180 degrees (i.e., 0≤ θ4≤180°) .

As for the apparatus and the method for controlling the vehicle-mounted device provided by the present application, considering that in the case where the area of the vehicle-mounted display screen is excessively large or the position where the vehicle-mounted display screen is located is far away from the driver, the driver needs to make a large movement to perform operations on the part of the vehicle-mounted display screen away from the driver. For the above problem, in the present application, it is provided that the sightline recognition and the gesture recognition are combined, and by detecting the sightline of the driver, the position range of the control that the driver wants to operate on the vehicle-mounted display screen is determined. After it is detected that the sightline of the driver falls into the target region of the vehicle-mounted display screen, the selection cursor is displayed over the target control in the target region. Then, by detecting the hand motion of the driver, the position of the selection cursor is moved, so that the selection cursor can be accurately positioned on the control that the driver wants to operate. In this way, after the selection cursor is moved over the control that the driver wants to operate, the vehicle-mounted device may be controlled by triggering the control. The solutions provided by the present application can not only avoid the problem of large control errors caused by the incapability of accurately recognizing the position corresponding to the sightline when the sightline recognition is used, but also avoid the problem of complicated operations caused by only using the gesture recognition for control.

In another embodiment, the present application further provides a method for controlling the vehicle-mounted device. The method includes the following steps.

In S301, the sightline of the driver is detected.

In S302, in response to detecting that the sightline of the driver falls into the target region of the first display screen, the selection cursor is displayed over the target control which is a control displayed in the target region.

In S303, after the selection cursor is displayed over the target control, the hand motion of the driver is detected.

In S304, the selection cursor is moved according to the hand motion of the driver.

Optionally, after the selection cursor is displayed over the target control and the hand motion of the driver is detected, the method further includes:

S305, in response to detecting the first target gesture of the driver, triggering the function of the control currently corresponding to the selection cursor.

Optionally, the target region is a rectangular region.

The step of detecting the sightline of the driver includes: detecting the position of the eye of the driver and the direction of the sightline of the driver; calculating the coordinate values of the four vertices of the target region in the target coordinate system according to the position of the eye of the driver, wherein the target coordinate system includes the space rectangular coordinate system which takes the position of the eye of the driver as the origin and takes the vertical direction, the horizontal direction and the direction right ahead of the vehicle as the coordinate axes; and determining whether the sightline of the driver falls into the target region according to the coordinate values of the four vertices of the target region in the target coordinate system and the direction of the sightline of the driver.

S306, in response to detecting the second target gesture of the driver, displaying the contents currently displayed in the target region on the second display screen.

Optionally, the second display screen includes the vehicle-mounted display screen at the distance from the driver greater than the distance from the first display screen to the driver.

Based on a same inventive concept, as for the specific implementation process, the technical problem solved, and the technical effects achieved, of the method for controlling the vehicle-mounted device provided by the present application, reference may be made to the relevant description of the apparatus for controlling the vehicle-mounted device, and details will not be repeated here.

In a case where an integrated unit is applied, FIG. 12 is a schematic diagram showing another possible structure of the apparatus for controlling the vehicle-mounted device involved in the above embodiments. The apparatus 40 for controlling the vehicle-mounted device includes a processing module 401, a communication module 402, and a storage module 403. The processing module 401 is used for controlling and managing actions of the apparatus 40 for controlling the vehicle-mounted device. For example, the processing module 401 is used for supporting the apparatus 40 for controlling the vehicle-mounted device to perform the steps in the above embodiments. The communication module 402 is used for supporting communication between the apparatus 40 for controlling the vehicle-mounted device and other entities. The storage module 403 is used for storing program codes and data of the apparatus 40 for controlling the vehicle-mounted device.

The processing module 401 may be a processor or a controller, such as a central processing unit (CPU) , a general-purpose processor, a digital signal processor (DSP) , an application-specific integrated circuit (ASIC) , a field programmable gate array (FPGA) or other programmable logic devices, transistor logic devices, hardware components or any combination thereof. It may implement or execute various illustrative logical blocks, modules and circuits described in the disclosure of the present application. The processor may also be a combination of implementation of computing functions, such as a combination including N microprocessors, or a combination of a DSP and a microprocessor. The communication module 402 may be a transceiver, a transceiver circuit, a communication interface, or the like. The storage module 403 may be a memory.

In a case where the processing module 401 is the processor 501 shown in FIG. 13, the communication module 402 is the transceiver 502 in FIG. 13, and the storage module 403 is the memory 503 in FIG. 13, the apparatus for controlling the vehicle-mounted device involved in the embodiments of the present application may be the apparatus 50 for controlling a vehicle-mounted device as follows.

Referring to FIG. 13, the apparatus 50 for controlling the vehicle-mounted device includes a processor 501 and a memory 503. The apparatus 50 for controlling the vehicle-mounted device may further include a transceiver 502 and a bus 504.

The processor 501, the transceiver 502, and the memory 503 are connected to each other through the bus 504. The bus 504 may be a peripheral component interconnect (PCI) bus, an extended industry standard architecture (EISA) bus, or the like. The bus may be divided into an address bus, a data bus, a control bus, and the like. For ease of representation, in the figure, only one thick line is used to represent the bus, but this does not mean that there is only one bus or one type of bus.

The processor 501 may be a general-purpose central processing unit (CPU) , a microprocessor, an application-specific integrated circuit (ASIC) , or N integrated circuits for controlling execution of programs of the solutions in the present application.

The memory 503 may be a read-only memory (ROM) or other types of static storage devices that may store static information and instructions, a random access memory (RAM) or other types of dynamic storage devices that may store information and instructions, an electrically erasable programmable read-only memory (EEPROM) , a compact disc read-only memory (CD-ROM) or other compact disk storages (including a compact disc, a laser disc, an optical disc, a digital versatile disc, a blu-ray disc) , a magnetic disk storage medium or other magnetic storage devices, or any other medium that can be used for carrying or storing desired program codes in a form of instructions or data structures and that can be accessed by a computer, which is not limited thereto. The memory may exist separately and is connected to the processor through the bus. The memory may also be integrated with the processor.

The memory 503 is used for storing application program codes for performing the solutions in the present application, and the solutions in the present application are controlled and performed by the processor 501. The transceiver 502 is used for receiving contents input by an external device, and the processor 501 is used for executing the application program codes stored in the memory 503, thereby implementing functions of various virtual units in the apparatus for controlling the vehicle-mounted device in the embodiments of the present application.

It will be understood that, in the embodiments of the present application, a magnitude of serial numbers of the above processes does not mean an order of execution, and the order of execution of the processes will be determined by their functions and inherent logic, and will not constitute any limitation to the implementation process of the embodiments of the present application.

A person of ordinary skill in the art will appreciate that, various illustrative units and algorithm steps described in the embodiments disclosed by the present disclosure can be implemented through electronic hardware or a combination of computer software and electronic hardware. Whether these functions are implemented through hardware or software depends on specific applications and design constraints imposed on the technical solutions. For each specific application, the described functions may be implemented in different ways by a skilled person, but such implementation will not be considered to exceed the scope of the present application.

A person skilled in the art will clearly understand that, for convenience and conciseness of description, specific working processes of the system, apparatus, and units that are described above, reference may be made to the corresponding processes in the foregoing method embodiments, and details will not be repeated herein.

In addition, the functional units in the embodiments of the present application may be integrated in one processing unit, or each unit may physically exist separately, or two or more of the units may be integrated in one unit.

In the above embodiments, the functions may be implemented in whole or in part through software, hardware, firmware, or any combination thereof. When implemented by using a software program, the functions can be implemented in a form of a computer program product in whole or in part. The computer program product includes N computer instructions. When computer program instructions are loaded and executed on a computer, the processes or functions described in the embodiments of the present application are generated in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions may be stored in a computer-readable storage medium, or may be transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions may be transmitted from one website, computer, server or data center to another website, computer, server or data center through wired means (e.g., coaxial cable, optical fiber, digital subscriber line (DSL) ) or wireless means (e.g., infrared, wireless, or microwave) . The computer-readable storage medium may be any available medium that can be accessed by the computer or a data storage device including N servers or data centers that may be integrated by media. The available medium may be a magnetic medium (e.g., a floppy disk, a hard disk, or a magnetic tape) , an optical medium (e.g., DVD) , a semiconductor medium (e.g., a solid state disk (SSD) ) , or the like.

The foregoing descriptions are merely specific implementations of the present application, but the protection scope of the present application is not limited thereto. Any changes or replacements that a person skilled in the art can easily conceive of within the technical scope disclosed by the present application shall be included in the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the protection scope of the claims.

Claims

An apparatus for controlling a vehicle-mounted device comprising a first display screen, the apparatus comprising:

a sightline detection unit used to detect a sightline of a driver;

a first display unit used to display a selection cursor over a target control in response to detecting that the sightline of the driver falls into a target region of the first display screen, the target control being a control displayed in the target region; and

a gesture detection unit used to detect a hand motion of the driver after the selection cursor is displayed over the target control, wherein

the first display unit is further used to move the selection cursor according to the hand motion of the driver.
The apparatus according to claim 1, further comprising:

a function triggering unit used to trigger a function of a control currently corresponding to the selection cursor in response to the gesture detection unit detecting a first target gesture of the driver.
The apparatus according to claim 2, wherein after the function of the control is triggered,

the first display unit is further used to control the first display screen to display a display interface corresponding to the control; and

the function triggering unit is further used to trigger a function of a control displayed on the display interface, according to a gesture of the driver detected by the gesture detection unit.
The apparatus according to claim 1, wherein the target region is a rectangular region; and

the sightline detection unit is used to:

detect a position of an eye of the driver and a direction of the sightline of the driver;

calculate coordinate values of four vertices of the target region in a target coordinate system according to the position of the eye of the driver, the target coordinate system including a space rectangular coordinate system which takes the position of the eye of the driver as an origin and takes a vertical direction, a horizontal direction and a direction right ahead of a vehicle as coordinate axes; and

determine whether the sightline of the driver falls into the target region, according to the coordinate values of the four vertices of the target region in the target coordinate system and the direction of the sightline of the driver.
The apparatus according to claim 1, wherein the vehicle-mounted device further comprises a second display screen; and

the apparatus further comprises:

a second display unit used to control the second display screen to display contents currently displayed in the target region in response to the gesture detection unit detecting a second target gesture of the driver.
The apparatus according to claim 1, wherein the vehicle-mounted device further comprises a first camera used to capture facial images of the driver and a second camera used to capture hand images of the driver;

the sightline detection unit is used to detect the sightline of the driver, according to the facial images of the driver; and

the gesture detection unit is used to detect the hand motion of the driver, according to the hand images of the driver.
A vehicle-mounted device comprising:

the apparatus according to any one of claims 1 to 6;

the first display screen;

a first camera used to capture facial images of the driver; and

a second camera used to capture hand images of the driver.
The vehicle-mounted device according to claim 7, wherein the first camera and the second camera are a same camera or different cameras.
The vehicle-mounted device according to claim 7, further comprising:

a second display screen at a distance from the driver greater than a distance between the first display screen and the driver; and

the second display screen is used to display contents displayed in the target region.
A method for controlling a vehicle-mounted device comprising a first display screen, the method comprising:

detecting a sightline of a driver;

displaying a selection cursor over a target control in response to detecting that the sightline of the driver falls into a target region of the first display screen, the target control being a control displayed in the target region;

detecting a hand motion of the driver after the selection cursor is displayed over the target control; and

moving the selection cursor according to the hand motion of the driver.
The method according to claim 10, wherein after displaying the selection cursor over the target control and detecting the hand motion of the driver, the method further comprises:

triggering a function of a control currently corresponding to the selection cursor in response to detecting a first target gesture of the driver.
The method according to claim 10, wherein the target region is a rectangular region; and

detecting the sightline of the driver includes:

detecting a position of an eye of the driver and a direction of the sightline of the driver;

calculating coordinate values of four vertices of the target region in a target coordinate system according to the position of the eye of the driver, the target coordinate system including a space rectangular coordinate system which takes the position of the eye of the driver as an origin and takes a vertical direction, a horizontal direction and a direction right ahead of a vehicle as coordinate axes; and

determining whether the sightline of the driver falls into the target region, according to the coordinate values of the four vertices of the target region in the target coordinate system and the direction of the sightline of the driver.
The method according to claim 10, wherein after displaying the selection cursor over the target control and detecting the hand motion of the driver, the method further comprises:

controlling a second display screen to display contents currently displayed in the target region in response to detecting a second target gesture of the driver.
The method according to claim 13, wherein

the second display screen is disposed at a distance from the driver greater than a distance between the first display screen and the driver.
An apparatus for controlling the vehicle-mounted device, the apparatus comprising a processor and a memory for storing instructions executable by the processor; wherein

the processor is configured to execute the instructions to implement the method for controlling the vehicle-mounted device according to any one of claims 10 to 14.
A computer-readable storage medium storing instructions that, when executed by a computer, cause the computer to perform the method for controlling the vehicle-mounted device according to any one of claims 10 to 14.