CN112114671A - Human-vehicle interaction method and device based on human eye sight and storage medium - Google Patents

Abstract

The application discloses a human-vehicle interaction method, a human-vehicle interaction device and a storage medium based on human eye sight, wherein the method and the device are applied to a vehicle, in particular to the acquisition of a human face image of a driver of the vehicle; processing the face image by using a neural network algorithm to obtain the current sight direction and eye movement of the driver; and executing operation on the in-vehicle equipment in the vehicle according to the current sight line direction and the eye movement. Therefore, the driver does not need to operate the corresponding in-vehicle equipment by hands, and the adverse effect on the safe driving of the automobile caused by the fact that the driver executes the operation action of releasing the steering wheel is avoided.

Description

A human-vehicle interaction method, device and storage medium based on human eye sight

technical field

The present application relates to the technical field of vehicles, and more particularly, to a human-vehicle interaction method, device and storage medium based on the sight line of human eyes.

Background technique

We know that in the normal driving process of the car, in order to ensure safe driving, the driver's hands are best limited to the steering wheel and its accessories, and it is best not to move other parts. However, when the car is actually driving, the driver is often required to perform actions that require hand-eye coordination, such as opening and setting navigation, switching audio and video playback, and opening and closing windows.

The inventors of the present application have found in practice that these manipulation actions requiring hand-eye coordination affect the driver's effective grasp of the steering wheel and its accessories, thereby adversely affecting the safe driving of the vehicle.

SUMMARY OF THE INVENTION

In view of this, the present application provides a human-vehicle interaction method, device and storage medium based on the sight of human eyes, which are used to avoid the adverse impact on the safe driving of the vehicle caused by the driver performing the manipulation action that requires releasing the steering wheel.

In order to achieve the above purpose, the proposed scheme is as follows:

A human-vehicle interaction method based on human eye sight, applied to a vehicle, the human-vehicle interaction method includes the steps:

obtaining a face image of the driver of the vehicle;

Utilize neural network algorithm to process described face image, obtain described driver's current sight direction and eye movement;

The in-vehicle device in the vehicle is operated according to the current gaze direction and the eye movement.

Optionally, the obtaining the face image of the driver of the vehicle includes the steps of:

capture an image of the driver using at least one camera device in the vehicle;

The image is processed to obtain the face image.

Optionally, the image is a visible light image and/or an infrared image.

Optionally, the processing of the face image using a neural network algorithm includes the steps of:

Use three cascaded Hourglass modules to process the face image to obtain face information;

Utilize a convolutional layer module to process the face information to obtain a feature map including a plurality of eye key points, and the feature map includes the coordinates of each of the eye key points;

Using the Resnet network to process the feature map with a direct regression algorithm to obtain the current line of sight direction;

The feature map is processed by using the Lenet two-class network to obtain the eye action.

Optionally, the performing an operation on the in-vehicle device in the vehicle according to the current line of sight direction and the eye movement includes the steps of:

Selecting a target in-vehicle device from a plurality of in-vehicle devices in the vehicle according to the current line-of-sight direction;

The target in-vehicle device is controlled to perform an operation matching the eye movement.

Optionally, the controlling the target in-vehicle device to perform an operation matching the eye movement includes the steps of:

Detecting that the current sight direction is at the focal position of the target in-vehicle device;

detecting the eye movement;

When the eye movement meets a preset standard, the target in-vehicle device is controlled to perform an operation matching the focus position.

Optionally, the human-vehicle interaction method further includes the steps:

The current line of sight direction is processed by a binary support vector machine to obtain a conclusion of whether the driver is distracted. When the driver is found distracted, a warning message is sent to the driver.

A human-vehicle interaction device based on human eye sight, applied to a vehicle, the human-vehicle interaction device includes:

a face acquisition module for acquiring a face image of the driver of the vehicle;

an image processing module for processing the face image by using a neural network algorithm to obtain the current sight direction and eye movements of the driver;

The operation execution module is configured to perform an operation on the in-vehicle device in the vehicle according to the current line of sight direction and the eye movement.

Optionally, the face acquisition module includes:

a camera device for capturing images of the driver;

The processor is configured to process the image to obtain the face image.

Optionally, the image is a visible light image and/or an infrared image.

Optionally, the image processing module includes:

Three cascaded Hourglass modules are used to process the face image to obtain face information;

a convolutional layer module for processing the face information to obtain a feature map including a plurality of eye key points, the feature map including the coordinates of each of the eye key points;

The Resnet network is used to process the feature map by using a direct regression algorithm to obtain the current line of sight direction;

The Lenet two-classification network is used to process the feature map to obtain the eye action.

Optionally, the operation execution module includes:

a target selection unit, configured to select a target in-vehicle device from a plurality of in-vehicle devices in the vehicle according to the current line of sight direction;

A device control unit, configured to control the target in-vehicle device to perform an operation matching the eye action.

Optionally, the device control unit includes:

a focus detection subunit, configured to detect that the current line of sight direction is at the focus position of the target in-vehicle device;

an action judging subunit for detecting the eye action;

A control execution sub-unit is configured to control the target in-vehicle device to perform an operation matching the focus position when the eye movement meets a preset standard.

Optionally, the human-vehicle interaction device further includes:

A distraction judging module is used to process the current line of sight direction by using a binary support vector machine to obtain a conclusion of whether the driver is distracted, and when it is found that the driver is distracted, a warning message is sent to the driver.

A storage medium, on which program codes are stored, and when the program codes are executed, each step of the above-mentioned human-vehicle interaction method is implemented.

It can be seen from the above technical solutions that the present application discloses a human-vehicle interaction method, device and storage medium based on human eye sight. The neural network algorithm is used to process the face image to obtain the driver's current sight direction and eye movement; according to the current sight direction and eye movement, the in-vehicle equipment in the vehicle is operated. In this way, the driver does not need to operate the corresponding in-vehicle equipment by hand, thereby avoiding the adverse impact on the safe driving of the vehicle caused by the driver's execution of a manipulation action that requires the driver to release the steering wheel.

Description of drawings

In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the following briefly introduces the accompanying drawings required for the description of the embodiments or the prior art. Obviously, the drawings in the following description are only These are some embodiments of the present application. For those of ordinary skill in the art, other drawings can also be obtained based on these drawings without any creative effort.

FIG. 1 is a flowchart of a human-vehicle interaction method based on human eye sight according to an embodiment of the application;

2 is a block diagram of a neural network model according to an embodiment of the application;

3 is a flowchart of another method for human-vehicle interaction based on human eye sight according to an embodiment of the present application;

FIG. 4 is a block diagram of a human-vehicle interaction device based on human eye sight according to an embodiment of the present application;

FIG. 5 is a block diagram of another human-vehicle interaction device based on human eye sight according to an embodiment of the present application.

Detailed ways

The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, but not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative work fall within the protection scope of the present application.

Example 1

FIG. 1 is a flowchart of a method for human-vehicle interaction based on human eye sight according to an embodiment of the present application.

As shown in FIG. 1 , the human-vehicle interaction method of the present application is applied to a manned vehicle, and a driver exists in the vehicle, that is, the method operates the equipment in the vehicle based on the human-eye line of sight of the driver, and the human-eye interaction The release method includes the following steps:

S1. Acquire a face image of the driver.

That is, during the normal driving process of the driver, the camera device in the vehicle is used to obtain the driver's face image. Specifically, the face image is obtained through the following steps:

First, use at least one camera device in the vehicle to take a picture of the driver to obtain an image of the driver. The camera device can be a visible light camera device or an infrared camera device. Therefore, the obtained image is also a visible light image or an infrared image, and an infrared camera is used. The advantage of the device is that it can capture effective images even when the light inside the car is dim.

Then, the visible light image or the infrared image is processed by cropping and perspective processing by a processor to obtain the driver's face image.

S2, using a neural network algorithm to calculate the driver's current sight direction and eye movements.

That is, the pre-trained neural network model is used to process the driver's face image to obtain the driver's current line of sight and eye movements. The eye movements here can be eye movements, eye opening, eye closing or blinking. . The neural network model includes three cascaded Hourglass modules, one convolutional layer module and two branches, one branch is Resnet network and the other branch is Lenet two-classification network, as shown in Figure 2.

Specifically, the current gaze direction and eye movements are obtained by the following methods:

First, use three cascaded Hourglass modules to process the face image to obtain face information;

Then, a convolutional layer module is used to process the face information, and a feature map including multiple eye key points is obtained, and the feature map contains the coordinates of each eye key point;

Then, use the Resnet network to process the feature map with a direct regression algorithm to obtain the current line of sight direction;

While using the Resnet network to process the feature map, the Lenet two-classification network is used to process the feature map to obtain eye movements.

S3. Operate the in-vehicle device according to the current sight direction and eye movements.

That is, after obtaining the driver's current sight direction and eye movements, the target in-vehicle equipment selected in the car can be operated based on the current sight direction and the corresponding eye movements, that is, the manual participation of wireless driving and Realize the operation of in-vehicle equipment. The specific process of this step is as follows:

First, select the target in-vehicle device in the car according to the current line of sight. For example, there are multiple in-vehicle devices in the vehicle, such as the main control screen, window, air conditioner, etc. When the current line of sight falls on a certain position, such as When landing on the main control screen or window, select the main control screen or window as the target in-vehicle device.

Then, on the basis of obtaining the eye action, the target in-vehicle device is controlled to perform an operation matching the eye action.

In this embodiment, the specific process of controlling the target in-vehicle equipment is:

First, it is detected that the current sight direction is at the focus position of the in-vehicle device, that is, the coordinates of the in-vehicle device in the current sight direction are detected. For example, for the main control screen, which button is its focal position; for a car window, whether its focal position is the bottom, middle, or top of the car window.

Then, the eye action is detected to determine whether the eye action is a predetermined action. For example, if we take eye blinking as a starting action, then only eye blinking is an effective action for starting subsequent operations.

Finally, when the eye action is found to be an effective action, the target in-vehicle device is controlled to perform an operation corresponding to the focus position.

For example, for the main control screen, if the main control screen is in the off-screen state, when it is found that the current line of sight is located at any point on the main control screen, the main control screen will be turned on; When the play button on the screen is controlled and a blinking action is performed, the play button is driven to perform a pressing action, and further a playback operation is performed corresponding to the pressing action. When it is found that the current line of sight has left the main control screen for a period of time, the main control screen will be turned off.

For the car window, when it is detected that the current sight direction is located at a certain position of the car window, and the driver is found to blink, the car window is driven to open or close the window, so that the upper edge of the glass reaches the current window. Where the line of sight is located on the window, so as to realize the automatic opening and closing of the window.

It can be seen from the above technical solutions that the present embodiment provides a human-vehicle interaction method based on the sight of the human eye. The method is applied to a vehicle, specifically acquiring a face image of the driver of the vehicle; The image is processed to obtain the driver's current sight direction and eye movements; operations are performed on the in-vehicle equipment in the vehicle according to the current sight direction and eye movements. In this way, the driver does not need to operate the corresponding in-vehicle equipment by hand, thereby avoiding the adverse impact on the safe driving of the vehicle caused by the driver's execution of a manipulation action that requires the driver to release the steering wheel.

In addition, in a specific embodiment of the present application, the following steps are also included, as shown in Figure 3:

S4. Detect whether the driver is distracted.

That is, after obtaining the driver's current line of sight direction, the current line of sight direction is processed by a binary support vector machine, and a conclusion of whether the driver is distracted is obtained. If it is found that the driver is distracted, a warning message will be sent to the driver or other people in the vehicle in time to warn the driver to concentrate on driving, or to warn others to remind the driver, thereby further ensuring the safe driving of the vehicle. .

Embodiment 2

FIG. 4 is a block diagram of a human-vehicle interaction device based on the line of sight of a human eye according to an embodiment of the present application.

As shown in FIG. 4 , the human-vehicle interaction device of the present application is applied to a manned vehicle, and a driver exists in the vehicle, that is, the method operates the equipment in the vehicle based on the human-eye line of sight of the driver, and the human-eye interaction The playback device includes the following face acquisition module 10 , image processing module 20 and operation execution module 30 .

The face acquisition module is used to acquire the driver's face image.

That is, during the normal driving process of the driver, the camera device in the vehicle is used to obtain the driver's face image. Specifically, the module specifically includes a camera device and a processor.

The camera equipment is used to photograph the driver to obtain the image of the driver. The camera equipment can be a visible light camera equipment or an infrared camera equipment. Therefore, the obtained image is also a visible light image or an infrared image. The advantage of using infrared camera equipment is that Effective images can be captured even when the interior light is dim.

The processor performs processing such as cropping and perspective processing on the visible light image or the infrared image to obtain the driver's face image.

The image processing module is used to calculate the driver's current line of sight direction and eye movements using neural network algorithms.

That is, the pre-trained neural network model is used to process the driver's face image to obtain the driver's current line of sight and eye movements. The eye movements here can be eye movements, eye opening, eye closing or blinking. . The neural network model includes three cascaded Hourglass modules, one convolutional layer module and two branches, one branch is Resnet network and the other branch is Lenet two-classification network, as shown in Figure 2.

Specifically, the work of each module in the model is as follows:

Three cascaded Hourglass modules are used to process face images to obtain face information;

The convolutional layer module is used to process the face information to obtain a feature map including multiple eye key points, and the feature map contains the coordinates of each eye key point;

The Resnet network direct regression algorithm processes the feature map to obtain the current line of sight direction;

While using the Resnet network to process the feature map, the Lenet two-classification network is used to process the feature map to obtain eye movements.

The operation execution module is used to operate the in-vehicle equipment according to the current line of sight direction and eye movements.

That is, after obtaining the driver's current sight direction and eye movements, the target in-vehicle equipment selected in the car can be operated based on the current sight direction and the corresponding eye movements, that is, the manual participation of wireless driving and Realize the operation of in-vehicle equipment. The module specifically includes a target selection unit and a device control unit.

The target selection unit is used to select the target in-vehicle device in the car according to the current sight direction. For example, there are multiple in-vehicle devices in the vehicle, such as the main control screen, window, air conditioner, etc. When the location is selected, such as when it falls on the main control screen or the window, select the main control screen or the window as the target in-vehicle device.

The device control unit is configured to control the target in-vehicle device to perform an operation matching the eye action on the basis of obtaining the eye action.

In this embodiment, the device control unit specifically includes a focus detection subunit, an action judgment subunit, and a control execution subunit.

The focus detection subunit is used to detect that the current line of sight direction is at the focus position of the in-vehicle device, that is, to detect the coordinates of the current line of sight direction of the in-vehicle device. For example, for the main control screen, which button is its focal position; for a car window, whether its focal position is the bottom, middle, or top of the car window.

The action judgment subunit is used to detect the eye action to determine whether the eye action is a predetermined action.

The action execution subunit is used to control the target in-vehicle device to perform an operation corresponding to the focus position after the eye action is found to be an effective action.

For example, for the main control screen, if the main control screen is in the off-screen state, when it is found that the current line of sight is located at any point on the main control screen, the main control screen will be turned on; When the play button on the screen is controlled and a blinking action is performed, the play button is driven to perform a pressing action, and further a playback operation is performed corresponding to the pressing action. When it is found that the current line of sight has left the main control screen for a period of time, the main control screen will be turned off.

For the car window, when it is detected that the current sight direction is located at a certain position of the car window, and the driver is found to blink, the car window is driven to open or close the window, so that the upper edge of the glass reaches the current window. Where the line of sight is located on the window, so as to realize the automatic opening and closing of the window.

It can be seen from the above technical solutions that this embodiment provides a human-vehicle interaction device based on human eye sight. The image is processed to obtain the driver's current sight direction and eye movements; operations are performed on the in-vehicle equipment in the vehicle according to the current sight direction and eye movements. In this way, the driver does not need to operate the corresponding in-vehicle equipment by hand, thereby avoiding the adverse impact on the safe driving of the vehicle caused by the driver's execution of a manipulation action that requires the driver to release the steering wheel.

In addition, in a specific embodiment of the present application, a distraction judgment module 40 is also included, as shown in FIG. 5 :

The distraction judgment module is used to detect whether the driver is distracted.

That is, after obtaining the driver's current line of sight direction, the current line of sight direction is processed by a binary support vector machine, and a conclusion of whether the driver is distracted is obtained. If it is found that the driver is distracted, a warning message will be sent to the driver or other people in the vehicle in time to warn the driver to concentrate on driving, or to warn others to remind the driver, thereby further ensuring the safe driving of the vehicle. .

Correspondingly, an embodiment of the present application further provides a storage medium, where a program code suitable for execution by a processor is stored on the storage medium, and the program code is used for:

obtaining a face image of the driver of the vehicle;

Utilize neural network algorithm to process described face image, obtain described driver's current sight direction and eye movement;

The in-vehicle device in the vehicle is operated according to the current gaze direction and the eye movement.

The refined functions and extended functions of the program code may refer to the above description.

The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments, and the same and similar parts between the various embodiments may be referred to each other.

It should be understood by those skilled in the art that the embodiments of the embodiments of the present invention may be provided as a method, an apparatus, or a computer program product. Accordingly, embodiments of the present invention may take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, embodiments of the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) having computer-usable program code embodied therein.

Embodiments of the present invention are described with reference to flowcharts and/or block diagrams of methods, terminal devices (systems), and computer program products according to embodiments of the present invention. It will be understood that each process and/or block in the flowchart illustrations and/or block diagrams, and combinations of processes and/or blocks in the flowchart illustrations and/or block diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to the processor of a general purpose computer, special purpose computer, embedded processor or other programmable data processing terminal equipment to produce a machine that causes the instructions to be executed by the processor of the computer or other programmable data processing terminal equipment Means are created for implementing the functions specified in the flow or flows of the flowcharts and/or the blocks or blocks of the block diagrams.

These computer program instructions may also be stored in a computer readable memory capable of directing a computer or other programmable data processing terminal equipment to operate in a particular manner, such that the instructions stored in the computer readable memory result in an article of manufacture comprising instruction means, the The instruction means implement the functions specified in the flow or flow of the flowcharts and/or the block or blocks of the block diagrams.

These computer program instructions can also be loaded on a computer or other programmable data processing terminal equipment, so that a series of operational steps are performed on the computer or other programmable terminal equipment to produce a computer-implemented process, thereby executing on the computer or other programmable terminal equipment The instructions executed on the above provide steps for implementing the functions specified in the flowchart or blocks and/or the block or blocks of the block diagrams.

Although preferred embodiments of the embodiments of the present invention have been described, additional changes and modifications to these embodiments may be made by those skilled in the art once the basic inventive concepts are known. Therefore, the appended claims are intended to be construed to include the preferred embodiment as well as all changes and modifications that fall within the scope of the embodiments of the present invention.

Finally, it should also be noted that in this document, relational terms such as first and second are used only to distinguish one entity or operation from another, and do not necessarily require or imply these entities or there is any such actual relationship or sequence between operations. Moreover, the terms "comprising", "comprising" or any other variation thereof are intended to encompass non-exclusive inclusion such that a process, method, article or terminal device that includes a list of elements includes not only those elements, but also a non-exclusive list of elements. other elements, or also include elements inherent to such a process, method, article or terminal equipment. Without further limitation, an element defined by the phrase "comprises a..." does not preclude the presence of additional identical elements in the process, method, article, or terminal device that includes the element.

The technical solutions provided by the present invention are described in detail above, and specific examples are used in this paper to illustrate the principles and implementations of the present invention. The descriptions of the above embodiments are only used to help understand the method of the present invention and its core idea; Meanwhile, for those of ordinary skill in the art, according to the idea of the present invention, there will be changes in the specific embodiments and application scope. In summary, the contents of this specification should not be construed as limiting the present invention.

Claims (15)

1. a human-vehicle interaction method based on human eye sight, applied to a vehicle, is characterized in that, described human-vehicle interaction method comprises the steps: obtaining a face image of the driver of the vehicle; Utilize neural network algorithm to process described face image, obtain described driver's current sight direction and eye movement; The in-vehicle device in the vehicle is operated according to the current gaze direction and the eye movement. 2. The human-vehicle interaction method as claimed in claim 1, wherein the obtaining the face image of the driver of the vehicle comprises the steps of: capture an image of the driver using at least one camera device in the vehicle; The image is processed to obtain the face image. 3. The human-vehicle interaction method according to claim 2, wherein the image is a visible light image and/or an infrared image. 4. The human-vehicle interaction method as claimed in claim 1, characterized in that, the described human face image is processed using a neural network algorithm, comprising the steps of: Use three cascaded Hourglass modules to process the face image to obtain face information; Utilize a convolutional layer module to process the face information to obtain a feature map including a plurality of eye key points, and the feature map includes the coordinates of each of the eye key points; Using the Resnet network to process the feature map with a direct regression algorithm to obtain the current line of sight direction; The feature map is processed by using the Lenet two-class network to obtain the eye action. 5. The human-vehicle interaction method according to claim 1, wherein the performing an operation on the in-vehicle equipment in the vehicle according to the current line of sight direction and the eye movement, comprises the steps of: Selecting a target in-vehicle device from a plurality of in-vehicle devices in the vehicle according to the current line-of-sight direction; The target in-vehicle device is controlled to perform an operation matching the eye movement. 6. The human-vehicle interaction method according to claim 5, wherein the controlling the target in-vehicle device to perform an operation matching the eye action comprises the steps of: Detecting that the current sight direction is at the focal position of the target in-vehicle device; detecting the eye movement; When the eye movement meets a preset standard, the target in-vehicle device is controlled to perform an operation matching the focus position. 7. The human-vehicle interaction method according to any one of claims 1 to 6, wherein the human-vehicle interaction method further comprises the steps of: The current line of sight direction is processed by a binary support vector machine to obtain a conclusion of whether the driver is distracted. When the driver is found distracted, a warning message is sent to the driver. 8. A human-vehicle interaction device based on human eye sight, applied to a vehicle, wherein the human-vehicle interaction device comprises: a face acquisition module for acquiring a face image of the driver of the vehicle; an image processing module for processing the face image by using a neural network algorithm to obtain the current sight direction and eye movements of the driver; The operation execution module is configured to perform an operation on the in-vehicle device in the vehicle according to the current line of sight direction and the eye movement. 9. The human-vehicle interaction device according to claim 8, wherein the face acquisition module comprises: a camera device for capturing images of the driver; The processor is configured to process the image to obtain the face image. 10 . The human-vehicle interaction device according to claim 9 , wherein the image is a visible light image and/or an infrared image. 11 . 11. The human-vehicle interaction device according to claim 8, wherein the image processing module comprises: Three cascaded Hourglass modules are used to process the face image to obtain face information; a convolutional layer module for processing the face information to obtain a feature map including a plurality of eye key points, the feature map including the coordinates of each of the eye key points; Resnet network, for using the direct regression algorithm to process the feature map to obtain the current line of sight direction; The Lenet two-classification network is used to process the feature map to obtain the eye action. 12. The human-vehicle interaction device according to claim 8, wherein the operation execution module comprises: a target selection unit, configured to select a target in-vehicle device from a plurality of in-vehicle devices in the vehicle according to the current line of sight direction; A device control unit, configured to control the target in-vehicle device to perform an operation matching the eye action. 13. The human-vehicle interaction device according to claim 12, wherein the device control unit comprises: a focus detection subunit, configured to detect that the current line of sight direction is at the focus position of the target in-vehicle device; an action judging subunit for detecting the eye action; A control execution sub-unit is configured to control the target in-vehicle device to perform an operation matching the focus position when the eye movement meets a preset standard. The human-vehicle interaction device according to any one of claims 8 to 13, wherein the human-vehicle interaction device further comprises: A distraction judging module is used to process the current line of sight direction by using a binary support vector machine to obtain a conclusion of whether the driver is distracted, and when it is found that the driver is distracted, a warning message is sent to the driver. 15 . A storage medium, characterized in that, program codes are stored on the storage medium, and when the program codes are executed, each step of the human-vehicle interaction method according to any one of claims 1-7 is implemented.

Images