CN117315879A

CN117315879A - Driving environment monitoring method and device, computer storage medium and vehicle

Info

Publication number: CN117315879A
Application number: CN202311149879.1A
Authority: CN
Inventors: 罗智; 王珏华; 王庚
Original assignee: Great Wall Motor Co Ltd
Current assignee: Great Wall Motor Co Ltd
Priority date: 2023-09-07
Filing date: 2023-09-07
Publication date: 2023-12-29

Abstract

The application discloses a driving environment monitoring method, a device, a computer storage medium and a vehicle, wherein the method is applied to the technical field of safe driving monitoring and comprises the following steps: monitoring a driving environment image of a driver inside the vehicle, and monitoring whether specific disturbance behaviors aiming at the driver exist or not based on the driving environment image; when specific interference behaviors exist, voice information in the vehicle is collected, and the driving environment state of the driver is judged based on the voice information; and if the driving environment state is a dangerous state, performing preset early warning processing. By adopting the method, whether the driving environment of the driver is abnormal or not can be monitored, the safety of the driver and passengers of the driver is ensured, and the accuracy of judgment is improved by combining the image data with the voice recognition.

Description

Driving environment monitoring method and device, computer storage medium and vehicle

Technical Field

The present disclosure relates to the field of safe driving monitoring technologies, and in particular, to a driving environment monitoring method and apparatus, a computer storage medium, and a vehicle.

Background

In recent years, news about casualties in which drivers are disturbed to drive are increasing. Along with the rapid increase of the number of private cars and commercial cars, real-time detection and early warning for violence threat are required to be carried out on the safety of a vehicle driver in the running process of the vehicle so as to ensure the safety of the driver and passengers. Therefore, a monitoring scheme of driving environment state is needed to ensure the driving safety of the vehicle and the personnel in the vehicle.

Disclosure of Invention

The embodiment of the application provides a driving environment monitoring method, a driving environment monitoring device, a computer storage medium and a vehicle, which can monitor a vehicle driver and the driving environment thereof and ensure the safety of the driver and passengers.

In a first aspect, an embodiment of the present application provides a driving environment monitoring method, including:

monitoring a driving environment image of a driver inside the vehicle, and monitoring whether specific disturbance behaviors aiming at the driver exist or not based on the driving environment image;

when specific interference behaviors exist, voice information in the vehicle is collected, and the driving environment state of the driver is judged based on the voice information;

and if the driving environment state is a dangerous state, performing preset early warning processing.

In a second aspect, an embodiment of the present application provides a driving environment monitoring device, including:

the image acquisition module is used for monitoring driving environment images of a driver in the vehicle and monitoring whether specific disturbance behaviors aiming at the driver exist or not based on the driving environment images;

the voice acquisition module is used for acquiring voice information in the vehicle when specific interference behaviors exist, and judging the driving environment state of the driver based on the voice information;

And the early warning module is used for starting preset early warning processing corresponding to the dangerous state if the driving environment state is the dangerous state.

In a third aspect, embodiments of the present application provide a computer storage medium having stored thereon a plurality of instructions adapted to be loaded by a processor and to perform the method steps provided in the first aspect of embodiments of the present application.

In a fourth aspect, embodiments of the present application provide a vehicle comprising a memory, a processor and a computer program stored on the memory and executable on the processor, the computer program being adapted to be loaded by the processor and to perform the steps of the method described above.

The technical scheme provided by some embodiments of the present application has the beneficial effects that at least includes:

the embodiment of the application discloses a driving environment monitoring method, which is used for monitoring driving environment images of a driver in a vehicle and monitoring whether specific disturbance behaviors aiming at the driver exist or not based on the driving environment images; when specific interference behaviors exist, voice information in the vehicle is collected, and the driving environment state of the driver is judged based on the voice information; and if the driving environment state is a dangerous state, performing preset early warning processing. Through carrying out real-time supervision to the driving environment in driver and the driver's cabin, through image data and the comprehensive judgement of pronunciation information simultaneously, in time discover the safety problem that the driver appears to this problem early warning fast, guaranteed driver and passenger's safety, improved the accuracy of judgement.

Drawings

In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings that are required to be used in the embodiments will be briefly described below, and it is obvious that the drawings in the following description are only some embodiments of the present application, and other drawings may be obtained according to these drawings without inventive effort for a person skilled in the art.

FIG. 1 is an exemplary system architecture diagram of a driving environment monitoring method provided in an embodiment of the present application;

FIG. 2 is a schematic flow chart of a driving environment monitoring method provided in an embodiment of the present application;

FIG. 3 is a schematic flow chart of a driving environment monitoring method provided in an embodiment of the present application;

fig. 4 is a schematic diagram of a logic framework of a driving environment monitoring method according to an embodiment of the present application;

fig. 5 is a block diagram of a driving environment monitoring device according to an embodiment of the present application;

fig. 6 is a schematic structural diagram of a vehicle according to an embodiment of the present application.

Detailed Description

In order to make the features and advantages of the present application more comprehensible, the following description will be given in detail with reference to the accompanying drawings in which embodiments of the present application are shown, and it is apparent that the described embodiments are merely some but not all embodiments of the present application. All other embodiments, which can be made by those skilled in the art based on the embodiments herein without making any inventive effort, are intended to be within the scope of the present application.

When the following description refers to the accompanying drawings, the same numbers in different drawings refer to the same or similar elements, unless otherwise indicated. The implementations described in the following exemplary examples are not representative of all implementations consistent with the present application. Rather, they are merely examples of apparatus and methods consistent with some aspects of the present application as detailed in the accompanying claims.

In the description of the present application, it should be understood that the terms "first," "second," and the like are used for descriptive purposes only and are not to be construed as indicating or implying relative importance. The specific meaning of the terms in this application will be understood by those of ordinary skill in the art in a specific context. Furthermore, in the description of the present application, unless otherwise indicated, "a plurality" means two or more. "and/or", describes an association relationship of an association object, and indicates that there may be three relationships, for example, a and/or B, and may indicate: a exists alone, A and B exist together, and B exists alone. The character "/" generally indicates that the context-dependent object is an "or" relationship.

In modern society, for some specific professions such as taxi drivers, special car drivers and the like, the drivers may face potential threats from passengers or external environments, such as interference, stress and the like, the driving conditions in the car may affect the driving state of the drivers, and other reasons of the drivers themselves may also cause poor driving states, such as fatigue, sudden diseases and the like, and the driving state of the drivers and the driving conditions in the car directly determine the travel safety of the drivers themselves and passengers. Therefore, the method for monitoring and early warning the driving environment of the driver in real time has important social value and practical significance.

At present, the traditional safety protection measures mainly realize the monitoring of the driving environment through video monitoring, but because of the hysteresis of video acquisition and analysis, the dangerous event can be continuously developed to a certain degree to obtain a more accurate judging result, and the early warning can not be carried out in time at the first time of the dangerous event, meanwhile, the atmosphere and the state of the driving environment are difficult to be completely reflected only by means of image data, the misjudgment condition can possibly occur, and the judging accuracy can not be ensured.

In view of the foregoing, embodiments of the present application provide a driving environment monitoring method, apparatus, computer storage medium, and vehicle. The method aims at judging the states of the driver and the driving environment in the cab in a multi-dimensional way by combining the image data acquired by the camera and the voice information acquired by the voice module in the vehicle with the image recognition and voice recognition technology, and the method can detect and early warn dangerous events in real time, so that the working safety of the driver is greatly improved, the working efficiency and satisfaction of the driver are improved, the personal safety of the driver and passengers is guaranteed, the possibility of misjudgment is reduced, and the judgment accuracy is improved.

Fig. 1 exemplarily shows an exemplary system architecture diagram of a driving environment monitoring method provided in an embodiment of the present application. As shown in fig. 1, the scene mainly includes a vehicle 101, a network 102 and a cloud 103. The network 102 is used as a medium to provide a communication link between the vehicle 101 and the cloud 103. Network 102 may include various types of wireless communication links, such as: including Wireless-Fidelity (Wi-Fi) communication links, microwave communication links, or the like. In general, in order to protect private information and prevent information leakage, a vehicle 101 generally employs an in-vehicle private network provided by a remote service provider (Telematics Service Provider, TSP) for connecting to a cloud 103.

The vehicle 101 may interact with the cloud 103 through the network 102 to receive messages from the cloud 103 or send messages to the cloud 103. The vehicle 101, also referred to as a vehicle's on-board system, has data transceiving and data processing capabilities. The vehicle 101 is provided with an application program and a device for supporting man-machine interaction, and a user can realize man-machine interaction through the application program.

In the embodiment of the application, the vehicle 101 first monitors a driving environment image of a driver inside the vehicle, and monitors whether or not there is a specific disturbance behavior for the driver based on the driving environment image; then, when a specific disturbance behavior exists, the vehicle 101 collects voice information in the vehicle, and judges the driving environment state of the driver based on the voice information; if the driving environment state of the vehicle 101 is a dangerous state, a preset early warning process corresponding to the dangerous state is started.

Cloud 103 is an independent physical server, or a server cluster or a distributed system formed by a plurality of physical servers, or a cloud server providing cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, a distribution network (Content Delivery Network, CDN), basic cloud computing services such as big data and artificial intelligence platforms, and the like. The cloud 103 can provide the necessary computing power support for the vehicle 101, such as providing background services for applications running on the vehicle 101.

It should be appreciated that the number of vehicles 101 and networks 102 in fig. 1 is merely exemplary, and that any number of vehicles 101 and networks 102 may be used, as desired for implementation.

Referring to fig. 2, fig. 2 is a schematic flowchart of a driving environment monitoring method according to an embodiment of the present application. The execution subject of the embodiment of the application may be a vehicle that executes the driving environment monitoring method, may be a processor in the vehicle that executes the driving environment monitoring method, or may be a driving environment monitoring service in the vehicle that executes the driving environment monitoring method. For convenience of description, a specific implementation procedure of the interface display method will be described below taking an example in which the implementation subject is a processor in a vehicle.

As shown in fig. 2, the driving environment monitoring method may include at least:

s202, monitoring a driving environment image of a driver in the vehicle, and monitoring whether specific disturbance behaviors aiming at the driver exist or not based on the driving environment image.

Optionally, the driving environment where the driver is located directly affects the driving state of the driver, so that in order to ensure the safety of the driver and the passengers during the driving process of the vehicle, the driving environment of the driver and whether the disturbance behavior aiming at the driver exists can be monitored so as to quickly and accurately capture the dangerous event occurring in the vehicle. The image can reflect the current shooting environment and the state of the shooting target to a certain extent, so that the driving environment image of the driver in the vehicle can be monitored, and the driving environment image can reflect the driving state and the driving environment state of the driver. Where a specific disturbance behavior refers to any disturbance behavior that prevents the driver from driving normally, which may be, for example, but not limited to, disturbing driving operations, attacking the driver.

Specifically, in order to collect a driving environment image of the inside of the vehicle, an in-vehicle inside camera may be installed in the vehicle, wherein the in-vehicle inside camera is disposed at a position of the sun visor with respect to the left side of the driver, so that the driver and the driving environment in the driver's cabin can be completely photographed. The vehicle-mounted internal camera is used for acquiring the image, so that the stable and accurate acquisition of the target image is facilitated, and no additional equipment is needed. When the vehicle is started, the driving environment image of the driver in the vehicle is continuously acquired through the vehicle-mounted internal camera, and the driving environment image can comprise the driver and the driving environment in the driving cab. Specifically, first the onboard system in the vehicle needs to initialize the camera to ensure that the camera is working properly, which typically includes starting the camera driver and adjusting the camera settings. After the camera initialization is completed, the vehicle may capture the driving environment image through the camera, which may be accomplished by calling a preset function in the camera or using a specific image capture library.

Further, after the driving environment image is acquired, the driving state of the driver can be analyzed according to the driving environment image, that is, whether the driver is subjected to specific disturbance behavior is monitored based on the driving environment image. Under the condition that the driver is monitored to be subjected to specific disturbance behaviors, the situation that dangerous situations possibly exist in the driving environment where the driver is located at the moment is indicated, and further verification and response to the specific disturbance behaviors are needed to ensure the safety of personnel in the vehicle.

S204, when specific interference behaviors exist, voice information in the vehicle is collected, and the driving environment state of the driver is judged based on the voice information.

Optionally, when the driver receives a specific disturbance action, it indicates that an abnormal condition has been primarily judged, however, a single image may have a misjudgment condition, so in order to judge whether the driver is in danger from multiple angles, the accuracy of monitoring the driving environment state is improved, and considering the voice information, the communication and interaction condition between the driver and the passenger in the vehicle can be recorded, so that the atmosphere and the state of the driving environment in the vehicle can be further described, and based on this, the driving environment state of the driver can be further analyzed through the voice information in the vehicle.

Optionally, after determining that the specific interference behavior exists, a voice recognition module in the vehicle can be started immediately, voice information is collected, analyzed and processed, and personnel communication conditions in the vehicle are obtained from the voice information, so that the driving environment state of the driver is judged. Specifically, the receiving device in the vehicle collects the voice information, and the voice recognition module preprocesses the voice information, including but not limited to removing noise and amplifying signals, so as to ensure the accuracy of the judgment result of the environmental state based on the voice information. Therefore, the image detection result is subjected to secondary verification through the voice information, the accurate judgment of the dangerous condition in the vehicle can be realized from multiple dimensions, and the accuracy and the reliability of dangerous event detection are improved.

S206, if the driving environment state is a dangerous state, performing preset early warning processing.

Optionally, if the driving environment state is a dangerous state, it indicates that the driver and the passengers in the vehicle have safety risks at the moment, and the driver and the passengers in the vehicle need to respond to the dangerous state at the moment. Furthermore, for different dangerous states, early warning processing of different response modes can be preset in advance, and adaptive preset early warning processing is adopted according to specific conditions of the dangerous states. For example, when the driver is in an abnormal driving state due to other factors, such as but not limited to that the driving operation is disturbed, attacked, etc., the current in-vehicle information can be recorded in time, and the early warning processing of the associated external response system can be performed to ensure the personal interests of the driver.

Alternatively, if the driving environment state is a safe state, it is explained that the presence of specific disturbance behavior detected by the image may be erroneous judgment for the driver, so that it is considered that no dangerous event occurs at this time, and monitoring of the driving environment image of the driver in the vehicle interior is continued.

The embodiment of the application provides a driving environment monitoring method, which is used for monitoring driving environment images of a driver in a vehicle and monitoring whether specific disturbance behaviors aiming at the driver exist or not based on the driving environment images; when specific interference behaviors exist, voice information in the vehicle is collected, and the driving environment state of the driver is judged based on the voice information; if the driving environment state is a dangerous state, starting a preset early warning process corresponding to the dangerous state. Through carrying out real-time supervision to the driving environment in driver and the driver's cabin, through image data and the comprehensive judgement of pronunciation information simultaneously, in time discover the safety problem that the driver appears to this problem early warning fast, guaranteed driver and passenger's safety, improved the accuracy of judgement.

Referring to fig. 3, fig. 3 is a schematic flowchart of a driving environment monitoring method according to an embodiment of the present application.

As shown in fig. 3, the driving environment monitoring method may include at least:

s302, monitoring driving environment images of a driver in a vehicle, and determining the category of the driving environment images, wherein the category is used for representing external behavior characteristics aiming at the driver in the driving environment.

Specifically, images of the driver and the driving environment in the cab are acquired in real time by the in-vehicle interior camera. When the driving environment image is acquired, the shooting frequency of the camera can be adjusted according to specific application requirements and equipment performance. When the driving environment image is acquired, the camera usually shoots at the frequency of 30 frames per second, for example, but not limited to, so that the image quality is stable, the condition of blurring of a picture is avoided when the driving environment image is acquired, and the equipment can complete the identification and detection process in a short time.

Referring to fig. 4, fig. 4 is a schematic diagram of a logic framework of a driving environment monitoring method according to an embodiment of the present application. As shown in fig. 4, the driving environment image of the driver inside the vehicle is monitored, the image is input into an image detection model, the driving environment image is detected based on the image detection model, and the category of the driving environment image is determined. Specifically, the driving environment image acquired by the camera is input into the model, and the model preprocesses the input image, including but not limited to enhancing contrast, scaling, so that the image is clearer and easy to identify. The model divides the image into a plurality of small blocks, classifies and identifies each small block according to a trained classifier, extracts key features, and judges that an abnormal driving environment image is detected if behaviors or articles conforming to preset features exist. After the driving environment image is acquired, the image detection model detects each frame of image and classifies the images. The image detection model is a violence detection model, and the image can be classified into an abnormal driving environment image and a normal driving environment image. When the driver's limb is restricted directly or indirectly by using a tool or limb by a person other than the driver, the driving environment image is determined to be an abnormal driving environment image. For example, the violence detection model may employ a deep-learned image recognition model such as, but not limited to, yolov5, retina.

S304, when the driving environment image is an abnormal driving environment image, determining an abnormal continuous frame number corresponding to the abnormal driving environment image, wherein the abnormal continuous frame number is the frame number of continuous abnormal driving environment images including the abnormal driving environment image.

Alternatively, when the driving environment image is an abnormal driving environment image, an abnormal continuous frame number corresponding to the abnormal driving environment image is determined. With continued reference to fig. 4, when the image detection model detects an abnormal driving environment image, an abnormal continuous frame number corresponding to the abnormal driving environment image is calculated, where the abnormal continuous frame number is a frame number of continuous abnormal driving environment images including the abnormal driving environment image. For example, when the image detection model detects an abnormal driving environment image, if all the first 14 frames of images of the abnormal driving environment image are determined as the abnormal driving environment image, the abnormal continuous frame number corresponding to the abnormal driving environment image is 15 frames.

S306, if the abnormal continuous frame number is larger than a preset frame number threshold, determining that specific interference behaviors exist.

Optionally, if the number of abnormal continuous frames is greater than a preset frame number threshold, determining that a specific interference behavior exists. Through the scheme, the abnormal continuous frame number is determined, and when the abnormal continuous frame number corresponding to the abnormal driving environment image is larger than the preset frame number threshold value, the specific interference behavior aiming at the driver is determined. For example, when the frame rate of the camera is 30 frames per second, the preset frame number threshold may be set to 15 frames, and if the abnormal continuous frame number is greater than 15 frames, that is, greater than 50% of the frame number per second corresponding to the frame rate of the camera, it may be determined that there is a specific disturbance behavior for the driver.

Alternatively, as shown in fig. 4, if the number of abnormal state frames is greater than the preset number of frame threshold, it is indicated that the state of the driver at this time does not conform to the conventional safe driving state, and it is determined that there is a specific disturbance behavior for the driver.

S308, if the abnormal state frame number is smaller than a preset frame number threshold, determining that specific disturbance behaviors aiming at the driver do not exist, and executing monitoring on driving environment images of the driver in the vehicle.

Alternatively, as shown in fig. 4, if the abnormal state frame number is less than the preset frame number threshold, it may be considered that a dangerous event does not occur, it is determined that there is no specific disturbance behavior for the driver, the driver is in a normal driving state, and the real-time acquisition of image data is continued.

And S310, when specific interference behaviors exist, collecting voice information in the vehicle, and performing text recognition on the voice information to obtain text content corresponding to the voice information.

Specifically, after a specific interference behavior is monitored, a voice recognition module in the vehicle can be started immediately, voice information is collected, feature extraction is carried out on the voice information, key features in voice are extracted, the extracted features are sent into a trained acoustic model and a trained language model, the scores of the acoustic model and the language model are respectively obtained, the 2 scores are synthesized, feature matching is carried out, the most similar result of language recognition is output, and the recognition result is converted into corresponding text content.

S312, analyzing the text content based on the language analysis model, and determining the driving environment state of the driver.

Specifically, semantic analysis and emotion analysis are performed on the obtained text content based on a language analysis model, and the driving environment state of the driver is determined. Specifically, the linguistic analysis model splits the textual content into individual words, terms, or phrases, which are replaced with words below for convenience of description. In the open source language model dataset, each word is pre-processed, including but not limited to part of speech tagging, which refers to determining the part of speech of each word, e.g., verbs, nouns, etc., and removing stop words. And carrying out polarity analysis on the preprocessed words, determining the emotion polarity of each word, and analyzing the emotion of the whole sentence by combining the emotion polarity and the part-of-speech label of each word to determine the emotion tendency of the whole sentence. Training a classifier by using the data set with the tag, classifying the text into sentences with extreme emotion or not, and finally judging the nature of the text content, thereby determining the driving environment state of the driver. The language analysis model may be a large language analysis model such as, but not limited to, chatglm, mos.

In some possible embodiments, the identified text content may be uploaded to a cloud end, and emotion analysis and semantic analysis may be performed on the identified text content based on a large language model of the cloud end. The cloud has stronger calculation power and better expansibility, so that the text content can be analyzed more accurately and comprehensively, and the accuracy is further improved. Meanwhile, the cloud language model can more efficiently share and utilize data resources, and provides quicker service response and processing speed.

Referring to fig. 4, when the driving environment status is a dangerous status, it is indicated that the driver and the passengers in the vehicle have safety risks at the moment, and a preset early warning process can be started; if the driving environment state is a safe state, the driving environment is considered safe at this time, and the monitoring of the driving environment image of the driver in the vehicle is continued.

And S314, if the driving environment state is a dangerous state, storing an abnormal video into a nonvolatile memory, wherein the abnormal video is a video starting from a first frame of abnormal driving environment image.

Optionally, the driving environment state is determined through the scheme, if the driving environment state is a dangerous state, the abnormal video is stored in the nonvolatile memory, the reasons and the passing of things are recorded, and the investigation staff is assisted to know the event. The duration of the abnormal video is a preset duration, and can be any time period from 15 seconds to 3 minutes. It should be noted that, the nonvolatile memory refers to a memory in which stored data is not lost after the device is powered down, and since real-time running data of the vehicle is reserved in a buffer, the buffer data which is not stored in the nonvolatile memory is automatically cleared after the vehicle is powered down, so that in order to ensure the integrity of the abnormal video, the abnormal video needs to be stored in the nonvolatile memory, so that the complete abnormal video can still be read from the vehicle after the vehicle is powered down.

S316, determining positioning information of the vehicle, and uploading the positioning information to the cloud early warning system.

Specifically, positioning information of the vehicle is obtained according to the positioning system, and the positioning information is uploaded to the early warning system of the cloud. The positioning information includes position information of the vehicle and track information, the position information includes longitude and latitude information of the vehicle and specific position information, the track information includes direction information of continuous running of the vehicle, for example, the vehicle is located in longitude 11709.112 and latitude 3403.868, a certain road is located on a certain major road, and the vehicle continues running to the south along the road.

Optionally, the positioning system includes any one of a global positioning system (Global Positioning System, GPS), a beidou satellite navigation system (BeiDou Navigation Satellite System, BDS), a global navigation satellite system (Global Navigation Satellite System, GNSS), a galileo satellite navigation system (GalileoSatellite Navigation System, GSNS), and the like, which is not limited in the embodiments of the present application.

In another possible implementation, an external alarm response system is accessed; determining the positioning information of the vehicle and sending the positioning information to an external alarm response system. And accessing an external alarm response system, and sending the positioning information of the vehicle to the external alarm response system, wherein the external alarm response system can send the positioning information of the vehicle and the time information of occurrence of the event to a preset user terminal. The preset user terminal may be, for example, but not limited to, a public security authority, a bus owner, etc. If the driver signs the privacy agreement, the abnormal video can be synchronously sent to an external alarm response system, and the external alarm response system is sent to a preset user terminal.

According to the driving environment monitoring method, the detection method based on the single-frame image is adopted, the driving environment of a driver can be determined within one second, and the response rate is greatly improved. Meanwhile, when specific disturbance behaviors aiming at drivers occur, corresponding abnormal videos are stored, event reasons and passes are recorded, investigation staff are assisted in knowing the event, and responsibility is conveniently determined. The scheme comprises an early warning system, and the real-time early warning mechanism can provide help for a driver at the first time, so that related personnel can respond and process events more quickly.

Referring to fig. 5, fig. 5 is a block diagram of a driving environment monitoring device according to an embodiment of the present application. As shown in fig. 5: the driving environment monitoring device 500 includes: an image acquisition module 510, a voice acquisition module 520, and an early warning module 530. Wherein:

an image acquisition module 510 for monitoring a driving environment image of a driver inside the vehicle, and monitoring whether a specific disturbance behavior for the driver exists based on the driving environment image;

the voice acquisition module 520 is configured to acquire voice information inside the vehicle when a specific interference behavior exists, and determine a driving environment state in which the driver is located based on the voice information;

The early warning module 530 is configured to start a preset early warning process corresponding to the dangerous state if the driving environment state is the dangerous state.

In some possible embodiments, the image acquisition module 510 is specifically configured to monitor a driving environment image of a driver in a vehicle, and determine a class of the driving environment image, where the class is used to characterize external behavior characteristics of the driver in the driving environment;

in some possible embodiments, the image acquisition module 510 further comprises:

a first determining unit, configured to determine, when the driving environment image is an abnormal driving environment image, an abnormal continuous frame number corresponding to the abnormal driving environment image, where the abnormal continuous frame number is a frame number of continuous abnormal driving environment images including the abnormal driving environment image;

the second determining unit is specifically configured to determine that a specific interference behavior exists if the abnormal continuous frame number is greater than a preset frame number threshold.

In some possible embodiments, the image acquisition module 510 is specifically configured to monitor a driving environment image of the driver in the vehicle, detect the driving environment image based on the image detection model, and determine a class of the driving environment image.

In some possible embodiments, the image acquisition module 510 includes: the comparison module is specifically configured to determine that no specific interference behavior exists if the number of abnormal state frames is smaller than a preset frame number threshold, and perform monitoring of a driving environment image of the driver in the vehicle.

In some possible embodiments, the voice acquisition module 520 includes: the recognition module is specifically used for carrying out character recognition on the voice information to obtain character content corresponding to the voice information;

the text analysis module is specifically used for analyzing text contents based on the language analysis model and determining the driving environment state of the driver.

In some possible embodiments, the early warning module 530 includes:

the storage module is specifically configured to store an abnormal video to the nonvolatile memory if the driving environment state is a dangerous state, where the abnormal video is a video from a first frame of abnormal driving environment image;

the positioning module is specifically used for determining positioning information of the vehicle and uploading the positioning information to the cloud early warning system.

In some possible embodiments, the early warning module 530 includes:

the access module is specifically used for accessing an external alarm response system;

the determining module is specifically used for determining positioning information of the vehicle and sending the positioning information to the external alarm response system.

The embodiment of the application provides a driving environment monitoring device, wherein an image acquisition module is used for monitoring driving environment images of a driver in a vehicle and monitoring whether specific disturbance behaviors aiming at the driver exist or not based on the driving environment images; the voice acquisition module is used for determining that the specific interference behavior exists, acquiring voice information in the vehicle and judging the driving environment state of the driver based on the voice information; and the early warning module is used for starting preset early warning processing corresponding to the dangerous state if the driving environment state is the dangerous state. Through carrying out real-time supervision to the driving environment in driver and the driver's cabin, through image data and the comprehensive judgement of pronunciation information simultaneously, in time discover the safety problem that the driver appears to this problem early warning fast, guaranteed driver and passenger's safety, improved the accuracy of judgement.

It should be noted that, when the driving environment monitoring device provided in the foregoing embodiment executes the driving environment monitoring method, only the division of the foregoing functional modules is used as an example, in practical application, the foregoing functional allocation may be performed by different functional modules according to needs, that is, the internal structure of the apparatus is divided into different functional modules, so as to complete all or part of the functions described above. In addition, the driving environment monitoring device and the driving environment monitoring method provided in the foregoing embodiments belong to the same concept, which embody the detailed implementation process in the method embodiment, and are not repeated here.

The foregoing embodiment numbers of the present application are merely for describing, and do not represent advantages or disadvantages of the embodiments.

Referring to fig. 6, fig. 6 is a schematic structural diagram of a vehicle according to an embodiment of the present application. As shown in fig. 6, a vehicle 600 may include: at least one processor 601, at least one network interface 604, a user interface 603, a memory 605, at least one communication bus 602.

Wherein the communication bus 602 is used to enable connected communications between these components.

The user interface 603 may include a Display screen (Display) and a Camera (Camera), and the optional user interface 603 includes a standard wired interface and a wireless interface.

The network interface 604 may optionally include a standard wired interface, a wireless interface (e.g., WI-FI interface), among others.

Wherein the processor 601 may include one or more processing cores. The processor 601 utilizes various interfaces and lines to connect various portions of the overall vehicle 600, perform various functions of the vehicle 600 and process data by executing or executing instructions, programs, code sets, or instruction sets stored in the memory 605, and invoking data stored in the memory 605. Alternatively, the processor 601 may be implemented in hardware in at least one of digital signal processing (Digital Signal Processing, DSP), field programmable gate array (Field-Programmable Gate Array, FPGA), programmable logic array (Programmable Logic Array, PLA). The processor 601 may integrate one or a combination of several of a central processing unit (Central Processing Unit, CPU), an image processor (Graphics Processing Unit, GPU), and a modem, etc. The CPU mainly processes an operating system, a user interface, an application program and the like; the GPU is used for rendering and drawing the content required to be displayed by the display screen; the modem is used to handle wireless communications. It will be appreciated that the modem may not be integrated into the processor 601 and may be implemented by a single chip.

The Memory 605 may include a random access Memory (Random Access Memory, RAM) or a Read-Only Memory (Read-Only Memory). Optionally, the memory 606 includes a non-transitory computer readable medium (non-transitory computer-readable storage medium). Memory 605 may be used to store instructions, programs, code, sets of codes, or sets of instructions. The memory 605 may include a stored program area and a stored data area, wherein the stored program area may store instructions for implementing an operating system, instructions for at least one function (such as a touch function, a sound playing function, an image playing function, etc.), instructions for implementing the various method embodiments described above, etc.; the storage data area may store data or the like referred to in the above respective method embodiments. The memory 605 may also optionally be at least one storage device located remotely from the processor 601. As shown in fig. 6, an operating system, a network communication module, a user interface module, and a driving environment monitoring application program may be included in the memory 605, which is one type of computer storage medium.

In the vehicle 600 shown in fig. 6, the user interface 603 is mainly used for providing an input interface for a user, and acquiring data input by the user; and the processor 601 may be configured to invoke the driving environment monitoring application stored in the memory 605 and specifically perform the following operations:

In some possible embodiments, the processor 601 performs monitoring of driving environment images of the driver inside the vehicle, and, when monitoring whether there is a specific disturbance action for the driver based on the driving environment images, is specifically configured to perform: monitoring a driving environment image of a driver in a vehicle, and determining a category of the driving environment image, wherein the category is used for representing external behavior characteristics aiming at the driver in the driving environment; when the driving environment image is an abnormal driving environment image, determining an abnormal continuous frame number corresponding to the abnormal driving environment image, wherein the abnormal continuous frame number is the frame number of continuous abnormal driving environment images including the abnormal driving environment image; if the abnormal continuous frame number is larger than the preset frame number threshold value, determining that specific interference behaviors exist.

In some possible embodiments, the processor 601 is configured to monitor driving environment images of a driver in a vehicle, and when determining a class of driving environment images, is further specifically configured to perform: and monitoring a driving environment image of a driver in the vehicle, detecting the driving environment image based on the image detection model, and determining the category of the driving environment image.

In some possible embodiments, after the processor 601 executes the determination of the number of abnormal consecutive frames corresponding to the abnormal driving environment image when the driving environment image is the abnormal driving environment image, the method is further specifically configured to execute: if the abnormal state frame number is smaller than the preset frame number threshold, determining that no specific disturbance behavior exists, and performing monitoring on driving environment images of the driver in the vehicle.

In some possible embodiments, when the processor 601 executes the determination of the driving environment state in which the driver is located based on the voice information, the method is further specifically configured to execute: performing character recognition on the voice information to obtain character content corresponding to the voice information; and analyzing the text content based on the language analysis model, and determining the driving environment state of the driver.

In some possible embodiments, the processor 601 is further specifically configured to perform, when performing the pre-set early warning process if the driving environment state is a dangerous state: if the driving environment state is a dangerous state, storing an abnormal video into a nonvolatile memory, wherein the abnormal video is a video starting from a first frame of abnormal driving environment image; determining positioning information of the vehicle, and uploading the positioning information to a cloud early warning system.

In some possible embodiments, the processor 601 is further specifically configured to perform, when performing the pre-set early warning process if the driving environment state is a dangerous state: accessing an external alarm response system; determining the positioning information of the vehicle and sending the positioning information to an external alarm response system.

The embodiment of the application provides a driving environment monitoring vehicle, which can monitor driving environments of a driver and a cab in real time, ensure life safety of the driver and passengers, timely cope with emergency, enable related personnel to take corresponding measures in time, promote social harmony to a certain extent and reduce social contradiction. Meanwhile, the states of the driver and the driving environment in the cab are judged through the image data and the voice information in a multi-dimensional mode, the possibility of misjudgment is reduced, and the judgment accuracy is improved.

Embodiments of the present application also provide a computer-readable storage medium having instructions stored therein, which when executed on a computer or processor, cause the computer or processor to perform one or more of the steps of the embodiments shown in fig. 2 or 3 described above. The respective constituent modules of the driving environment monitoring device described above may be stored in a computer-readable storage medium if implemented in the form of software functional units and sold or used as independent products.

In the above embodiments, it may be implemented in whole or in part by software, hardware, firmware, or any combination thereof. When implemented in software, may be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the processes or functions in accordance with embodiments of the present application are produced in whole or in part. The computer may be a general purpose computer, a special purpose computer, a computer network, or other programmable apparatus. The computer instructions may be stored in or transmitted across a computer-readable storage medium. The computer instructions may be transmitted from one website, computer, cloud, or data center to another website, computer, cloud, or data center by a wired (e.g., coaxial cable, fiber optic, digital subscriber line (Digital Subscriber Line, DSL)) or wireless (e.g., infrared, wireless, microwave, etc.). Computer readable storage media can be any available media that can be accessed by a computer or a data storage device such as a cloud, data center, or the like that contains an integration of one or more available media. The usable medium may be a magnetic medium (e.g., a floppy Disk, a hard Disk, a magnetic tape), an optical medium (e.g., a digital versatile Disk (Digital Versatile Disc, DVD)), or a semiconductor medium (e.g., a Solid State Disk (SSD)), or the like.

Those skilled in the art will appreciate that implementing all or part of the above-described embodiment methods may be accomplished by way of a computer program, which may be stored in a computer-readable storage medium, instructing relevant hardware, and which, when executed, may comprise the embodiment methods as described above. And the aforementioned storage medium includes: a Read Only Memory (ROM), a random access Memory (Random Access Memory, RAM), a magnetic disk or an optical disk, or the like. The technical features in the present examples and embodiments may be arbitrarily combined without conflict.

The previous description of the disclosed embodiments is provided to enable any person skilled in the art to make or use the present application. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the generic principles defined herein may be applied to other embodiments without departing from the spirit or scope of the application. Thus, the present application is not intended to be limited to the embodiments shown herein but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A driving environment monitoring method, characterized in that the method comprises:

monitoring a driving environment image of a driver inside a vehicle, and monitoring whether specific disturbance behaviors aiming at the driver exist or not based on the driving environment image;

when the specific interference behavior exists, collecting voice information in the vehicle, and judging the driving environment state of the driver based on the voice information;

2. The method of claim 1, wherein the monitoring of the driving environment image of the driver inside the vehicle, based on which monitoring whether there is a specific disturbance behavior for the driver, comprises:

monitoring a driving environment image of a driver in a vehicle, and determining a category of the driving environment image, wherein the category is used for representing external behavior characteristics aiming at the driver in the driving environment;

when the driving environment image is an abnormal driving environment image, determining an abnormal continuous frame number corresponding to the abnormal driving environment image, wherein the abnormal continuous frame number is the frame number of continuous abnormal driving environment images including the abnormal driving environment image;

And if the abnormal continuous frame number is larger than a preset frame number threshold value, determining that the specific interference behavior exists.

3. The method of claim 2, wherein monitoring the driving environment image of the driver inside the vehicle, determining the category of the driving environment image, comprises:

and monitoring driving environment images of a driver in the vehicle, detecting the driving environment images based on an image detection model, and determining the types of the driving environment images.

4. The method of claim 3, wherein when the driving environment image is an abnormal driving environment image, after determining an abnormal continuous frame number corresponding to the abnormal driving environment image, further comprising:

and if the abnormal state frame number is smaller than a preset frame number threshold value, determining that the specific interference behavior does not exist, and executing the monitoring of the driving environment image of the driver in the vehicle.

5. The method of claim 1, wherein said determining a driving environment state in which the driver is located based on the voice information comprises:

performing word recognition on the voice information to obtain word content corresponding to the voice information;

and analyzing the text content based on a language analysis model, and determining the driving environment state of the driver.

6. The method of claim 2, wherein if the driving environment state is a dangerous state, performing a preset early warning process, comprising:

if the driving environment state is a dangerous state, storing an abnormal video into a nonvolatile memory, wherein the abnormal video is a video starting from a first frame of abnormal driving environment image;

determining positioning information of a vehicle, and uploading the positioning information to a cloud early warning system.

7. The method of claim 2, wherein if the driving environment state is a dangerous state, performing a preset early warning process, comprising:

accessing an external alarm response system;

and determining the positioning information of the vehicle, and sending the positioning information to the external alarm response system.

8. A violence warning device, the device comprising:

the image acquisition module is used for monitoring driving environment images of a driver in a vehicle, and monitoring whether specific disturbance behaviors aiming at the driver exist or not based on the driving environment images;

the voice acquisition module is used for acquiring voice information in the vehicle when the specific interference behavior exists, and judging the driving environment state of the driver based on the voice information;

9. A computer storage medium storing a plurality of instructions adapted to be loaded by a processor and to perform the steps of the method according to any one of claims 1 to 7.

10. A vehicle comprising a memory, a processor and a computer program stored on the memory and executable on the processor, the processor implementing the steps of the method according to any one of claims 1 to 7 when the program is executed.