CN111563396A

CN111563396A - Method and device for online identifying abnormal behavior, electronic equipment and readable storage medium

Info

Publication number: CN111563396A
Application number: CN201910075016.1A
Authority: CN
Inventors: 张天明
Original assignee: Beijing Didi Infinity Technology and Development Co Ltd
Current assignee: Beijing Didi Infinity Technology and Development Co Ltd
Priority date: 2019-01-25
Filing date: 2019-01-25
Publication date: 2020-08-21

Abstract

The embodiment of the application provides a method and a device for identifying abnormal behaviors on line, electronic equipment and a readable storage medium, and the method and the device can be used for carrying out face identification on a video image by acquiring the video image shot in the process of travel service so as to determine whether the video image contains a preset target object. And when the preset target object is contained, detecting the video image to determine whether abnormal behaviors occur in the process of travel service. And if the abnormal behavior occurs, sending out prompt information. The real-time video images obtained in the process of travel service can be analyzed to feed back in time when abnormal behaviors occur, and the defect that serious consequences possibly caused by the abnormal behaviors cannot be stopped due to post analysis in the prior art is overcome.

Description

Method and device for online identifying abnormal behavior, electronic equipment and readable storage medium

Technical Field

The application relates to the technical field of computer application, in particular to a method and a device for identifying abnormal behaviors online, electronic equipment and a readable storage medium.

Background

With the rapid development of internet technology, travel APPs (applications) play an important role in people's daily life. However, along with the convenience brought by the travel APP, there are many potential safety hazards. Safety and compliance in the process of a journey are very important links of all large application platforms, at present, all large application platforms generally adopt a pre-education and post-feedback mode to carry out constraint and responsibility pursuit on abnormal behaviors in the process of the journey, however, the pre-education mode often cannot play a strong constraint, the post-feedback mode often has difficulty in obtaining evidence and cannot feed back the abnormal behaviors in time, and serious consequences possibly caused by the abnormal behaviors exist.

Disclosure of Invention

In view of the above, an object of the present application is to provide a method, an apparatus, an electronic device and a readable storage medium for identifying an abnormal behavior online, which can identify and detect a video image obtained in a process of a trip service, so as to feed back a trip including an abnormal behavior of a preset target object in time, thereby avoiding a serious result possibly caused by the abnormal behavior.

According to an aspect of embodiments of the present application, there is provided an electronic device that may include one or more storage media and one or more processors in communication with the storage media. One or more storage media store machine-readable instructions executable by a processor. When the electronic device is operated, the processor is communicated with the storage medium through the bus, and the processor executes the machine readable instructions to execute the method for identifying the abnormal behaviors online.

According to another aspect of the embodiments of the present application, there is provided a method for online identifying abnormal behavior, which is applied to an abnormality detection server, where the abnormality detection server is in communication connection with a monitoring terminal, and the method includes:

acquiring a video image shot by the monitoring terminal in the current travel service process;

performing face recognition on the video image to determine whether the video image contains a preset target object;

when the video image is determined to contain a preset target object, detecting the video image to determine whether abnormal behaviors occur in the current travel service process;

and if the abnormal behavior occurs, sending out prompt information.

In some embodiments of the present application, the abnormality detection server stores a plurality of sample images in advance, and the method may further include:

printing a first label on a sample image containing a female face image in the plurality of sample images, and printing a second label on a sample image containing a male face image;

leading each sample image with the label into a neural network model for training to obtain a gender detection model;

the step of performing face recognition on the video image to determine whether the video image contains a preset target object includes:

importing the video image into a pre-established gender detection model for detection to obtain a detection result;

and determining whether the video image contains a female face image according to the detection result, and determining that the video image contains a preset target object when determining that the video image contains the female face image.

In some embodiments of the present application, the step of importing the video image into a pre-established gender detection model for detection to obtain a detection result may include:

performing frame division processing on the video image to obtain a plurality of image frames contained in the video image;

carrying out face detection on each image frame and determining an image frame containing a face image;

and extracting images in image frames in the image frames, and importing the extracted images into a pre-established gender detection model for detection to obtain a detection result.

In some embodiments of the present application, after the step of performing face detection on each image frame and determining an image frame containing a face image, the method may further include:

acquiring a minimum coordinate value and a maximum coordinate value of each image frame on a corresponding image frame;

and aiming at each image frame, reducing the minimum coordinate value of the image frame by a first preset value, and increasing the maximum coordinate value by a second preset value so as to expand the image frame.

marking a third label on a sample image containing a preset behavior action in the plurality of sample images, and marking a fourth label on a sample image not containing the preset behavior action in the plurality of sample images;

leading each sample image with the label into a neural network model for training to obtain an abnormal behavior detection model;

the step of detecting the video image to determine whether an abnormal behavior occurs in the current travel service process includes:

and importing the video image into the abnormal behavior detection model for detection so as to determine whether abnormal behaviors occur in the current travel service process.

In some embodiments of the present application, the preset target object includes an underage object, and the step of detecting the video image to determine whether an abnormal behavior occurs in the current journey service process may include:

detecting the video image to obtain the number of minor objects contained in the video image and the total number of face images contained in the video image;

and detecting whether the difference value between the total number and the number of the target objects is 1, and if so, determining that abnormal behaviors occur in the current travel service process.

marking different age labels on sample images containing face images in the plurality of sample images;

leading each sample image with the label into a neural network model for training to obtain an age detection model;

performing frame processing and face recognition on the video image to obtain a plurality of images including face images in the video image;

importing each image into the age detection model for detection so as to output an age score corresponding to each image;

and calculating to obtain a comprehensive age score according to age scores corresponding to the multiple images, determining whether the video image contains an immature object or not according to the comprehensive age score, and determining that the video image contains a preset target object when the video image contains the immature object.

In some embodiments of the present application, the abnormality detection server is communicatively connected to a voice collecting device, and the method may further include:

acquiring voice information acquired by the voice acquisition equipment in the current journey service process;

and analyzing the voice information to determine whether the voice information contains abnormal information, and if so, executing the step of sending out the prompt information.

In some embodiments of the present application, the analyzing the voice information to determine whether the voice information contains abnormal information may include:

extracting key words contained in the voice information;

and detecting whether the extracted keywords contain preset sensitive words or not, and if so, determining that the voice information contains abnormal information.

In some embodiments of the present application, the method may further comprise:

detecting whether the monitoring terminal has image acquisition abnormity in the current travel service process, and sending prompt information when the image acquisition abnormity is detected; the step of detecting whether the monitoring terminal has abnormal image acquisition in the current travel service process comprises one of the following steps:

detecting whether a video image acquired by the monitoring terminal within a preset time length is a continuous full white or full black image or not in the current travel service process, and if the video image is the continuous full white or full black image, determining that image acquisition is abnormal;

in the current travel service process, whether a video image acquired by the monitoring terminal within a preset time length is a continuous fixed image or not is detected, and if the video image is the continuous fixed image, the occurrence of image acquisition abnormity is determined.

According to another aspect of the embodiments of the present application, there is provided an apparatus for online identifying abnormal behavior, which is applied to an abnormality detection server, where the abnormality detection server is in communication connection with a monitoring terminal, the apparatus including:

the image acquisition module is used for acquiring a video image shot by the monitoring terminal in the current travel service process;

the image recognition module is used for carrying out face recognition on the video image so as to determine whether the video image contains a preset target object;

the detection module is used for detecting the video image to determine whether abnormal behaviors occur in the current travel service process when the video image is determined to contain a preset target object;

and the information sending module is used for sending out prompt information when abnormal behaviors occur in the current travel service process.

In some embodiments of the present application, the abnormality detection server has a plurality of sample images prestored therein, and the apparatus may further include:

the first marking module is used for printing a first label on a sample image containing a female face image in the plurality of sample images and printing a second label on a sample image containing a male face image;

the first training module is used for leading each sample image which is marked with a label into a neural network model for training so as to obtain a gender detection model;

the image recognition module may be specifically configured to:

In some embodiments of the present application, the image recognition module may obtain the detection result by:

In some embodiments of the present application, the image recognition module may be further specifically configured to:

the second marking module is used for marking a third label on a sample image containing a preset behavior action in the sample images and marking a fourth label on a sample image not containing the preset behavior action in the sample images;

the second training module is used for importing the labeled sample images into a neural network model for training to obtain an abnormal behavior detection model;

the detection module is specifically configured to:

In some embodiments of the present application, the preset target object includes a minor object, and the detection module may be specifically configured to:

the third marking module is used for marking different age labels on sample images containing face images in the plurality of sample images;

the third training module is used for leading each sample image which is marked with a label into the neural network model for training so as to obtain an age detection model;

the image recognition module may be specifically configured to:

In some embodiments of the present application, the abnormality detection server is communicatively connected to a voice collecting device, and the apparatus may further include:

the voice acquisition module is used for acquiring voice information acquired by the voice acquisition equipment in the current journey service process;

the voice recognition module is used for analyzing the voice information to determine whether the voice information contains abnormal information;

the information sending module is further used for sending out prompt information when the voice information contains abnormal information.

In some embodiments of the present application, the speech recognition module may be specifically configured to:

extracting key words contained in the voice information;

In some embodiments of the present application, the apparatus may further comprise:

the acquisition abnormity detection module is used for detecting whether the monitoring terminal has image acquisition abnormity in the current travel service process and sending prompt information when the image acquisition abnormity is detected; the acquisition anomaly detection module may be specifically configured to execute one of:

According to another aspect of embodiments of the present application, a readable storage medium is provided, on which a computer program is stored, which, when being executed by a processor, is capable of performing the above-mentioned steps of the method for identifying abnormal behavior online.

Based on any one of the above aspects, the embodiment of the application can perform face recognition on the video image by acquiring the video image shot in the journey service process to determine whether the video image contains the preset target object. And when the preset target object is contained, detecting the video image to determine whether abnormal behaviors occur in the process of travel service. And if the abnormal behavior occurs, sending out prompt information. The real-time video images obtained in the process of travel service can be analyzed to feed back in time when abnormal behaviors occur, and the defect that serious consequences possibly caused by the abnormal behaviors cannot be stopped due to post analysis in the prior art is overcome.

In order to make the aforementioned objects, features and advantages of the present application more comprehensible, preferred embodiments accompanied with figures are described in detail below.

Drawings

In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings that are required to be used in the embodiments will be briefly described below, it should be understood that the following drawings only illustrate some embodiments of the present application and therefore should not be considered as limiting the scope, and for those skilled in the art, other related drawings can be obtained from the drawings without inventive effort.

FIG. 1 is an interactive schematic block diagram illustrating a system for online identification of abnormal behavior provided by an embodiment of the present application;

FIG. 2 illustrates a schematic diagram of exemplary hardware and software components of an electronic device that may implement the server, the service requester terminal, and the service provider terminal of FIG. 1 provided by an embodiment of the present application;

FIG. 3 is a flow chart of a method for online identification of abnormal behavior provided by an embodiment of the present application;

FIG. 4 is a second flowchart illustrating a method for online identification of abnormal behavior according to an embodiment of the present application;

fig. 5 is a third schematic flowchart illustrating a method for identifying abnormal behavior online according to an embodiment of the present application;

FIG. 6 shows one of the functional block diagrams of the apparatus for online identification of abnormal behavior provided by the embodiments of the present application;

FIG. 7 is a second functional block diagram of an apparatus for online identification of abnormal behavior provided in an embodiment of the present application;

fig. 8 shows a third functional block diagram of an apparatus for online identification of abnormal behavior provided in an embodiment of the present application.

Detailed Description

In order to make the purpose, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application, and it should be understood that the drawings in the present application are for illustrative and descriptive purposes only and are not used to limit the scope of protection of the present application. Additionally, it should be understood that the schematic drawings are not necessarily drawn to scale. The flowcharts used in this application illustrate operations implemented according to some embodiments of the present application. It should be understood that the operations of the flow diagrams may be performed out of order, and steps without logical context may be performed in reverse order or simultaneously. One skilled in the art, under the guidance of this application, may add one or more other operations to, or remove one or more operations from, the flowchart.

In addition, the described embodiments are only a part of the embodiments of the present application, and not all of the embodiments. The components of the embodiments of the present application, generally described and illustrated in the figures herein, can be arranged and designed in a wide variety of different configurations. Thus, the following detailed description of the embodiments of the present application, presented in the accompanying drawings, is not intended to limit the scope of the claimed application, but is merely representative of selected embodiments of the application. All other embodiments, which can be derived by a person skilled in the art from the embodiments of the present application without making any creative effort, shall fall within the protection scope of the present application.

To enable those skilled in the art to utilize the present disclosure, the following embodiments are presented in conjunction with a specific application scenario, "net appointment taxi taking scenario". It will be apparent to those skilled in the art that the general principles defined herein may be applied to other embodiments and applications without departing from the spirit and scope of the application. Although the present application is described primarily in the context of a "net appointment taxi taking scenario," it should be understood that this is only one exemplary embodiment. The application can be applied to any other traffic type. For example, the present application may be applied to different transportation system environments, including terrestrial, marine, or airborne, among others, or any combination thereof. The transportation means of the transportation system may comprise a taxi, a private car, a windmill, a bus, etc., or any combination thereof. The application can also comprise any service system for online taxi taking, for example, a system for sending and/or receiving express delivery, and a service system for business transaction of buyers and sellers. Applications of the system or method of the present application may include web pages, plug-ins for browsers, client terminals, customization systems, internal analysis systems, or artificial intelligence robots, among others, or any combination thereof.

It should be noted that in the embodiments of the present application, the term "comprising" is used to indicate the presence of the features stated hereinafter, but does not exclude the addition of further features.

The term "user" in this application may refer to an individual, entity or tool that requests a service, subscribes to a service, provides a service, or facilitates the provision of a service. For example, the user may be a passenger, a driver, an operator, etc., or any combination thereof.

In order to solve at least one technical problem described in the background of the present application, an embodiment of the present application provides a method, an apparatus, an electronic device, and a readable storage medium for online identifying an abnormal behavior, where a video image captured during a travel service process is acquired, and a face of the video image is identified to determine whether the video image includes a preset target object. And when the preset target object is contained, detecting the video image to determine whether abnormal behaviors occur in the process of travel service. And if the abnormal behavior occurs, sending out prompt information. The real-time video images obtained in the process of travel service can be analyzed to feed back in time when abnormal behaviors occur, and the defect that serious consequences possibly caused by the abnormal behaviors cannot be stopped due to post analysis in the prior art is overcome. The technical solution of the present application is explained below by means of possible implementations.

First embodiment

Fig. 1 is a schematic diagram of an architecture of a system 100 for online identification of abnormal behavior according to an alternative embodiment of the present application. For example, the system 100 for online identification of abnormal behavior may be an online transportation service platform relied upon for transportation services such as taxis, designated driving services, express services, carpooling services, bus services, driver rental services, or regular service, or a combination of any of the above. The system 100 for online identification of abnormal behavior may include a server 110, a network 120, a service requester terminal 130, a service provider terminal 140, and a database 150, and the server 110 may include a processor therein that performs instruction operations. In addition, the system 100 for online identification of abnormal behavior may further include a monitor terminal 160, and the monitor terminal 160 may be a vehicle data recorder with an image capturing function or a video shooting device disposed in a vehicle. The monitor terminal 160 may be communicatively coupled to the server to transmit the captured video images to the server 100. In addition, the server 100 in the system may also be communicatively connected to a management platform, which may be a background management end providing a travel service. The system 100 for online identification of abnormal behavior shown in fig. 1 is only one possible example, and in other possible embodiments, the system 100 for online identification of abnormal behavior may also include only one of the components shown in fig. 1 or may also include other components.

In some embodiments, the server 110 may be a single server or a group of servers. The set of servers can be centralized or distributed (e.g., the servers 110 can be a distributed system). In some embodiments, the server 110 may be local or remote to the terminal. For example, the server 110 may access data stored in the service requester terminal 130, the service provider terminal 140, the monitor terminal 160, and the database 150 via the network 120. As another example, the server 110 may be directly connected to at least one of the monitoring terminal 160, the service requester terminal 130, the service provider terminal 140, and the database 150 to access data stored therein. In some embodiments, the server 110 may be implemented on a cloud platform; by way of example only, the cloud platform may include a private cloud, a public cloud, a hybrid cloud, a community cloud (community cloud), a distributed cloud, an inter-cloud, a multi-cloud, and the like, or any combination thereof. In some embodiments, the server 110 may also be implemented on an electronic device 200 having one or more of the components shown in FIG. 2 in the present application.

In some embodiments, the server 110, the service requester terminal 130 or the service provider terminal 140 may include a processor. The processor may process information and/or data in the travel service process to perform one or more of the functions described herein. For example, during the travel service, the processor may detect whether an abnormality occurs in the video image based on processing the obtained video image during the travel service. A processor may include one or more processing cores (e.g., a single-core processor (S) or a multi-core processor (S)). Merely by way of example, a Processor may include a Central Processing Unit (CPU), an Application Specific Integrated Circuit (ASIC), an Application Specific Instruction Set Processor (ASIP), a Graphics Processing Unit (GPU), a Physical Processing Unit (PPU), a Digital Signal Processor (DSP), a Field Programmable Gate Array (FPGA), a Programmable Logic Device (PLD), a controller, a microcontroller Unit, a Reduced Instruction Set computer (Reduced Instruction Set computer), a microprocessor, or the like, or any combination thereof.

Network 120 may be used for the exchange of information and/or data. In some embodiments, one or more components (e.g., server 110, service requestor terminal 130, service provider terminal 140, monitor terminal 160, and database 150) in the system 100 that identifies abnormal behavior online may send information and/or data to other components. For example, the server 110 may acquire a video image from the monitoring terminal 160 via the network 120. In some embodiments, the network 120 may be any type of wired or wireless network, or combination thereof. Merely by way of example, the Network 130 may include a wired Network, a Wireless Network, a fiber optic Network, a telecommunications Network, an intranet, the internet, a Local Area Network (LAN), a Wide Area Network (WAN), a Wireless Local Area Network (WLAN), a WLAN, a Metropolitan Area Network (MAN), a Wide Area Network (WAN), a Public Switched Telephone Network (PSTN), a bluetooth Network, a ZigBee Network, a Near Field Communication (NFC) Network, or the like, or any combination thereof. In some embodiments, network 120 may include one or more network access points. For example, network 120 may include wired or wireless network access points, such as base stations and/or network switching nodes, through which one or more components of system 100 that identify anomalous behavior online may connect to network 120 to exchange data and/or information.

In some embodiments, "service requester" and "service requester terminal 130" may be used interchangeably, and "service provider" and "service provider terminal 140" may be used interchangeably.

In some embodiments, the service provider terminal 140 may comprise a voice capture enabled device, such as a mobile device, a tablet computer, a laptop computer, or a built-in device in a motor vehicle, or any combination thereof. In some embodiments, the mobile device may include a smart home device, a wearable device, a smart mobile device, a virtual reality device, an augmented reality device, or the like, or any combination thereof. In some embodiments, the wearable device may include a smart bracelet, a smart lace, smart glass, a smart helmet, a smart watch, a smart garment, a smart backpack, a smart accessory, and the like, or any combination thereof. In some embodiments, the smart mobile device may include a smartphone, a Personal Digital Assistant (PDA), a gaming device, a navigation device, or a point of sale (POS) device, or the like, or any combination thereof. In some embodiments, the built-in devices in the motor vehicle may include an on-board computer, an on-board television, and the like.

Database 150 may store data and/or instructions. In some embodiments, the database 150 may store data obtained from the service requester terminal 130 and/or the service provider terminal 140. In some embodiments, database 150 may store data and/or instructions for the exemplary methods described herein. In some embodiments, database 150 may include mass storage, removable storage, volatile Read-write Memory, or Read-Only Memory (ROM), among others, or any combination thereof. By way of example, mass storage may include magnetic disks, optical disks, solid state drives, and the like; removable memory may include flash drives, floppy disks, optical disks, memory cards, zip disks, tapes, and the like; volatile read-write Memory may include Random Access Memory (RAM); the RAM may include Dynamic RAM (DRAM), Double data Rate Synchronous Dynamic RAM (DDR SDRAM); static RAM (SRAM), Thyristor-Based Random Access Memory (T-RAM), Zero-capacitor RAM (Zero-RAM), and the like. By way of example, ROMs may include Mask Read-Only memories (MROMs), Programmable ROMs (PROMs), Erasable Programmable ROMs (PERROMs), Electrically Erasable Programmable ROMs (EEPROMs), compact disk ROMs (CD-ROMs), digital versatile disks (ROMs), and the like. In some embodiments, database 150 may be implemented on a cloud platform. By way of example only, the cloud platform may include a private cloud, a public cloud, a hybrid cloud, a community cloud, a distributed cloud, across clouds, multiple clouds, or the like, or any combination thereof.

In some embodiments, a database 150 may be connected to the network 120 to communicate with one or more components of the system 100 (e.g., the server 110, the service requester terminal 130, the service provider terminal 140, etc.) that identify abnormal behavior online. One or more components in the system 100 that identify anomalous behavior online may access data or instructions stored in the database 150 via the network 120. The database 150 may be directly connected to one or more components (e.g., server 110, service requestor terminal 130, service provider terminal 140, etc.) in the system 100 that identifies abnormal behavior online; or database 150 may be part of server 110.

In some embodiments, one or more components in the system 100 (e.g., the server 110, the service requestor terminal 130, the service provider terminal 140, etc.) that identify anomalous behavior online may have access to the database 150. In some embodiments, one or more components in the system 100 that identify anomalous behavior online may read and/or modify information about a service requestor, a service provider, or the public, or any combination thereof, when certain conditions are met. For example, server 110 may read and/or modify information for one or more users after receiving a service request.

Second embodiment

Fig. 2 illustrates a schematic diagram of exemplary hardware and software components of an electronic device 200 of a server 110, a service requester terminal 130, a service provider terminal 140, which may implement the concepts of the present application, according to some embodiments of the present application. For example, the processor 220 may be used on the electronic device 200 and to perform the functions herein.

The electronic device 200 may be a general purpose computer or a special purpose computer, both of which may be used to implement the method of identifying abnormal behavior online of the present application. Although only a single computer is shown, for convenience, the functions described herein may be implemented in a distributed fashion across multiple similar platforms to balance processing loads.

For example, the electronic device 200 may include a network port 210 connected to a network, one or more processors 220 for executing program instructions, a communication bus 230, and a different form of storage medium 240, such as a disk, ROM, or RAM, or any combination thereof. Illustratively, the computer platform may also include program instructions stored in ROM, RAM, or other types of non-transitory storage media, or any combination thereof. The method of the present application may be implemented in accordance with these program instructions. The electronic device 200 also includes an Input/Output (I/O) interface 250 between the computer and other Input/Output devices (e.g., keyboard, display screen).

For ease of illustration, only one processor is depicted in the electronic device 200. However, it should be noted that the electronic device 200 in the present application may also comprise a plurality of processors, and thus the steps performed by one processor described in the present application may also be performed by a plurality of processors in combination or individually. For example, if the processor of the electronic device 200 executes steps a and B, it should be understood that steps a and B may also be executed by two different processors together or separately in one processor. For example, a first processor performs step a and a second processor performs step B, or the first processor and the second processor perform steps a and B together.

Third embodiment

Fig. 3 is a flowchart illustrating a method for online identification of abnormal behavior according to some embodiments of the present application, where the method for online identification of abnormal behavior provided by the present application may be applied to an abnormality detection server, which may be the server 110 described above. It should be understood that, in other embodiments, the order of some steps in the method for online identifying abnormal behavior described in this embodiment may be interchanged according to actual needs, or some steps may be omitted or deleted. The detailed steps of the method for identifying abnormal behavior online are described as follows.

The application aims at providing a video image based on monitoring equipment carried on the vehicle gathers to whether have special crowds such as women's passenger or the underage passenger on discerning the vehicle, and focus on and carry out anomaly detection to the journey service that contains this type of special crowds. And when detecting the abnormality, sending corresponding abnormal information to the management platform so as to supervise and standardize the abnormal behavior in time and avoid causing serious consequences. And can also send the prompt message to the service provider terminal 140 and the service requester terminal 130 of the travel service, in order to achieve the role of warning to the service provider, and achieve the role of reminding to the service requester.

Step S310, acquiring a video image captured by the monitoring terminal 160 during the current travel service.

Step S320, performing face recognition on the video image to determine whether the video image includes a preset target object, and if the video image includes the preset target object, entering step S330.

In this embodiment, the monitoring terminal 160 disposed on the vehicle may collect a video image during the travel service, for example, the video image may be collected continuously, or the service provider may start to collect a video image during the travel service corresponding to an order after receiving the order. The monitoring terminal 160 may send the video image to the anomaly detection server after acquiring the video image. Optionally, the anomaly detection server may obtain the video images acquired within a preset time period from the monitoring terminal 160 at intervals of the preset time period, where the preset time period may be 10 minutes or 5 minutes, and the like.

The anomaly detection server may perform real-time analysis and processing based on the received video images after obtaining the video images sent by the monitoring terminal 160 at intervals of a preset duration. The anomaly detection server can perform face recognition on the video image to determine whether the video image contains a preset target object.

Considering that the safety of female passengers is more easily threatened when the passengers go out, the safety is easy to cause abnormal events in the journey. Thus, in one possible embodiment, the preset target object may be a female passenger.

The abnormality detection server is pre-stored with a plurality of sample images, and can be trained in advance based on the pre-stored sample images to obtain a gender detection model for detecting the female face image. The first label may be printed on a sample image including a female face image among the plurality of sample images in advance, and the second label may be printed on a sample image including a male face image among the plurality of sample images.

And (4) importing the labeled sample images into a neural network model for training to obtain a gender detection model. The neural network model performs feature extraction, feature self-learning and the like on the basis of the sample image marked with the first label and the sample image marked with the second label, so as to train and obtain a gender detection model capable of classifying the female face image and the male face image. The type of the neural network model, the loss function, the optimization function of the model, and the like may be selected based on the requirement, and the embodiment is not particularly limited. For example, a neural network model with a basic network of ShuffleNet V2 may be used, a loss function may employ Cross EntrophyLoss (cross entropy), and an optimization function of the model may employ SGD (stochastic gradient descent) with Momentum. The specific training process for the neural network model is a common technique in the prior art, and is not described herein again.

After receiving the video image acquired in the current travel service process sent by the monitoring terminal 160, the anomaly detection server performs face recognition on the video image to detect whether a female face image appears in the video image. The collected video images can be imported into the pre-established gender detection model for detection, so that a detection result is obtained. Whether the video image contains the female face image or not can be determined according to the detection result. When the video image is determined to contain the female face image, the video image can be determined to contain the preset target object.

Referring to fig. 4, the video image may be processed to obtain the detection result in the following manner:

step S410, performing frame division processing on the video image to obtain a plurality of image frames included in the video image.

Step S420, performing face detection on each image frame and determining an image frame containing a face image.

Step S430, extracting images in image frames in the image frames, and importing the extracted images into a pre-established gender detection model for detection to obtain a detection result.

In this embodiment, when detecting a video image, the obtained video image may be firstly subjected to frame division to obtain a plurality of image frames included in the video image, and then each image frame is detected respectively. In order to avoid the influence of image background and other interference information on the detection result, the image of the face region can be extracted, so that the image of the face region is emphatically detected.

Optionally, the face detection may be performed on each image frame, an image frame including the face image may be determined, images in the image frame in each image frame may be further extracted, and each extracted image is imported into the pre-established gender detection model for detection, so as to obtain a detection result.

Considering that the detection is performed only with the image of the face region, the result may not be very accurate, and therefore, the face image may be expanded to obtain the image of the region around the face region, such as the hair part, the neck part, and the like, and increasing the image of these parts to perform the detection will improve the detection accuracy.

As an embodiment, after determining the image frame containing the face image in each image frame, the minimum coordinate value and the maximum coordinate value of each image frame on the corresponding image frame may be obtained, for example, the minimum coordinate value may be recorded as (x)_min,y_min) The maximum coordinate value is expressed as (x)_max,y_max). For each image border, the minimum coordinate value of the image border may be decreased by a first preset value, for example, Δ s1, and the maximum coordinate value of the image border may be increased by a second preset value, for example, Δ s2, to expand the image border.

The method specifically comprises the following steps:

x_min＝x_min-Δs1

X_max＝x_max+Δs2

y_min＝y_min-Δs1

y_max＝y_max+Δs2

in the above manner, an image including a face image and an image including, for example, a hair or a neck around the face image in each image frame will be obtained. And importing the obtained image into a pre-established gender detection model for detection so as to determine whether the video image contains a female face image.

In the present embodiment, detection is performed on a per image frame basis, and therefore there is one detection result per image frame. For the accuracy of the detection result, the detection result of each image frame may be integrated to obtain a final detection result. Therefore, the obtained final detection result has better robustness for detection in dynamic scenes such as shooting angle change, posture change and the like.

In addition to safety concerns for female passengers, the present application also considers that underage passengers may be present during travel service, and that the safety of underage passengers is also an important issue. Therefore, as an embodiment, the preset target object further includes an immature target object.

The abnormality detection server prestores a plurality of sample images, and the plurality of sample images comprise face images at various ages. Different age labels can be marked on sample images containing face images in the plurality of sample images in advance, and the age labels can be marked based on actual ages corresponding to the face images in the sample images. And leading each sample image with the label into a neural network model for training to obtain an age detection model.

Optionally, the type of the adopted neural network model may be selected according to requirements, the neural network model performs feature extraction, feature self-learning and the like on the basis of the face images in the imported sample images, and obtains a corresponding age detection model based on the age label training of each sample image, and the specific training process may refer to the prior art and is not described herein again.

The above process is a process in which the abnormality detection server is trained in advance to obtain an age detection model before formal online detection. When the anomaly detection server obtains the video image in the current travel service process sent by the monitoring terminal 160, the obtained video image is imported into an age detection model for detection so as to determine whether an immature object appears in the video image.

Optionally, the anomaly detection server may perform framing processing and face recognition on the obtained video image to obtain multiple images including the face image in the video image. And (3) importing each image into a pre-established age detection model for detection so as to output an age score corresponding to each image.

In order to improve the accuracy of the detection result, in this embodiment, a comprehensive age score may be calculated according to age scores corresponding to the plurality of images, and whether the video image includes an immature subject is determined according to the comprehensive age score. And if the minor objects are contained, determining that the video image contains the preset target objects.

Alternatively, an average value of age scores corresponding to the plurality of images may be calculated, and the average value may be used as the integrated age score. Or the middle value of the age scores corresponding to the multiple images can be obtained to serve as the comprehensive age score. It is also possible to remove the corresponding image with a higher age score, for example, the highest two or three, etc., from the plurality of images, and to remove the corresponding image with a lower age score, for example, the lower two or three, etc., from the plurality of images. And then calculating the average value of the age scores corresponding to the rest images, and taking the average value as the comprehensive age score. It is further checked whether the integrated age score is less than a predetermined value, for example 18 or 16, and if so, it is determined that an underage object is present in the video image.

The processing method of the detection result is only an example, and is not limited to this, and the processing method may be set according to the requirement.

Step S330, detecting the video image to determine whether an abnormal behavior occurs in the current travel service process, and if an abnormal behavior occurs, performing step S340.

Step S340, sending out prompt information.

After determining the presence of a female passenger or an underage passenger in the travel service through the above process, the video images may be further examined to determine whether abnormal behavior is present in the current travel service vehicle. If abnormal behavior occurs, a prompt message may be sent to the management platform and at least one of the service provider terminal 140 and the service requester terminal 130 corresponding to the current travel service. The service provider terminal 140 may be a terminal device held by a driver, and the service requester terminal 130 may be a terminal device held by a passenger.

Referring to fig. 5, in the embodiment, the video image may be detected to determine whether the abnormal behavior occurs by the following steps:

step S510, performing frame division processing and face recognition on the video image to obtain a plurality of images including a face image in the video image.

Step S520, importing each image into the age detection model for detection, so as to output an age score corresponding to each image.

Step S530, calculating to obtain a comprehensive age score according to the age scores corresponding to the multiple images, determining whether the video image contains an immature object according to the comprehensive age score, and determining that the video image contains a preset target object when the video image contains the immature object.

In one embodiment, the abnormality detection server stores a plurality of sample images previously captured from video images acquired during a historical travel service. A third label may be printed in advance on a sample image including a preset behavior action among the plurality of sample images. The preset behavior action may be a behavior action predefined by a manager, and such behavior action may be a behavior action that may threaten the safety of the passenger. In addition, a fourth label can be marked on a sample image which does not contain the preset behavior action in the plurality of sample images.

And leading the labeled sample images into a neural network model for training to obtain an abnormal behavior detection model. When the obtained video image collected by the monitoring terminal 160 in the current travel service process is further analyzed, the anomaly detection server may import the video image into a pre-established anomaly behavior detection model to determine whether an anomaly behavior consistent with a predefined behavior occurs in the current travel service process.

If the abnormal behavior action occurs, the abnormal detection server can send prompt information to the management platform so as to inform a manager at one end of the management platform. In addition, the abnormality detection server may also send a prompt message to the corresponding service provider terminal 140 to serve the purpose of warning. In addition, a prompt message may also be sent to the service requester terminal 130 for prompting purposes.

In one embodiment, the service requester terminal 130 may be registered in association with the abnormality detection service in advance, that is, two or three service requester terminals 130 may be associated with each other, and may be associated with the service requester terminal 130 of, for example, a family or a colleague of the service requester terminal 130. When the subsequent anomaly detection server detects that an abnormal behavior occurs in the travel service process corresponding to a certain service requester terminal 130, the subsequent anomaly detection server may send a prompt message to the service requester terminal 130 and may also send a prompt message to the service requester terminal 130 associated with the service requester terminal 130 through registration. Therefore, the prompt can be automatically sent to the family or the colleague of the object with the abnormal behavior threat, so as to further achieve the purpose of avoiding the subsequent serious consequences.

The above abnormal behavior detection means detects whether or not a behavior consistent with a predefined abnormal behavior occurs. In this embodiment, it is also considered that when the minor subject is taken, the minor subject may go out alone, and this situation also needs to be prompted in time to remind the driver to pay attention to the minor subject, or to further determine whether other abnormal situations occur to cause the minor passenger to appear alone.

Alternatively, after detecting that an underage object appears in the video image, the video image may be detected to obtain the number of the underage objects contained in the video image and the total number of the face images contained in the video image. And further detecting whether the difference value between the total number of the face images contained in the video image and the number of the minor objects is 1, if so, indicating that the current travel service only contains a driver except for minor passengers, and determining that the travel service is the minor passengers to appear alone.

When the abnormal behavior occurs, prompt information can be sent to the management platform to inform the abnormal condition of the travel service. Also, a prompt message may be sent to the service provider terminal 140 to prompt the driver that the order is abnormal, requiring further confirmation. Or the driver is prompted to pay attention to the order, so that the effect of ensuring the safety of the underage passengers is achieved to a certain extent.

In addition, in this embodiment, the anomaly detection server needs to detect whether the monitoring device has an abnormal image acquisition condition in real time, for example, when the driver intentionally shields the camera. If the anomaly detection service detects that the image acquisition of the monitoring terminal 160 is abnormal, prompt information is sent to the management platform to inform the abnormal condition, so that management personnel can intervene to manage.

The detection of whether the image acquisition abnormality occurs at the monitor terminal 160 can be realized by the following method:

in the current travel service process, whether the video image acquired by the monitoring terminal 160 within the preset time length is a continuous full white or full black image is detected, and if the video image is a continuous full white or full black image, it is determined that image acquisition is abnormal.

Or, in the current travel service process, it is detected whether the video image acquired by the monitoring terminal 160 within the preset time duration is a continuous fixed image. And if the image is a continuous fixed and unchangeable image, determining that image acquisition abnormity occurs.

Further, as an embodiment, the abnormality detection server may be further communicatively connected to a voice collecting device, which may be a collecting device mounted on a vehicle, or may be a voice collecting device, such as a microphone, on the service provider terminal 140 or the service requester terminal 130.

In the process of travel service, the voice acquisition equipment can acquire voice information of the driver and the passenger and send the acquired voice information to the abnormality detection server. The anomaly detection server may analyze the received voice information to determine whether the current trip service process contains anomalous information. If the abnormal information is included, the notification information is transmitted to the management platform and the service provider terminal 140 and the service requester terminal 130 corresponding to the current travel service.

Optionally, the anomaly detection server may extract a keyword included in the voice information, detect whether the extracted keyword includes a preset sensitive word, and determine that the voice information includes the anomaly information if the extracted keyword includes the preset sensitive word. And the abnormal information can be judged, and early warning prompt is required. The preset sensitive words can be predefined words which appear when a conflict occurs or a help is asked for.

Therefore, the abnormal detection server can detect by combining the video image and the voice information, the detection accuracy is further improved, and when abnormal behaviors occur, information in the abnormal behaviors can be stored from multiple angles, so that follow-up tracing of responsibility is facilitated.

Further, the method provided by the application also detects whether the environment in the vehicle is good or not in the process of travel service. For example, a plurality of images of good environment in the vehicle and a plurality of images of bad environment in the vehicle may be collected in advance, and the images may be introduced into the neural network model for training to obtain an environment detection model for detecting whether the environment in the vehicle is good or not. During subsequent online detection, the anomaly detection server may analyze the video image received from the monitoring terminal 160 to import the video image into a pre-established environment detection model for detection, and determine whether the environment in the vehicle is good according to the obtained detection result. If the environment in the vehicle is bad, a prompt message may be sent to the service provider terminal 140 corresponding to the current trip service to prompt the driver to perform a clearing.

Fourth embodiment

Fig. 6 shows a functional block diagram of an apparatus 600 for online abnormal behavior identification according to some embodiments of the present application, where the functions implemented by the apparatus 600 for online abnormal behavior identification correspond to the steps executed by the method described above. The apparatus may be understood as the electronic device 200 or the processor 220 of the electronic device 200, or may also be understood as a component that is independent from the electronic device 200 or the processor 220 and implements the functions of the present application under the control of the electronic device 200, as shown in fig. 6, the apparatus 600 for online identifying abnormal behavior may include an image obtaining module 601, an image identifying module 602, a detecting module 603, and an information sending module 604.

An image obtaining module 601, configured to obtain a video image captured by the monitoring terminal 160 in a current travel service process. It is understood that the image obtaining module 601 can be used to execute the above step S310, and for the detailed implementation of the image obtaining module 601, reference can be made to the above contents related to step S310.

An image recognition module 602, configured to perform face recognition on the video image to determine whether the video image includes a preset target object. It is understood that the image recognition module 602 can be used to perform the above step S320, and for the detailed implementation of the image recognition module 602, reference can be made to the above description regarding the step S320.

The detecting module 603 is configured to, when it is determined that the video image includes a preset target object, detect the video image to determine whether an abnormal behavior occurs in a current trip service process. It is understood that the detection module 603 can be used to perform the step S330, and for the detailed implementation of the detection module 603, reference can be made to the above-mentioned contents related to the step S330.

The information sending module 604 is configured to send a prompt message when an abnormal behavior occurs in the current trip service process. It is understood that the information sending module 604 can be used to execute the step S340, and the detailed implementation manner of the information sending module 604 can refer to the content related to the step S340.

Referring to fig. 7, in a possible implementation, the apparatus 600 for online identifying abnormal behavior may further include:

a first labeling module 605, configured to print a first label on a sample image including a female face image among the plurality of sample images, and print a second label on a sample image including a male face image;

the first training module 606 is configured to import each labeled sample image into a neural network model for training to obtain a gender detection model.

The image recognition module 602 may be specifically configured to:

In a possible implementation, the image recognition module 602 may obtain the detection result by:

In a possible implementation manner, the image recognition module 602 may be further configured to:

Referring to fig. 7 again, in a possible embodiment, the abnormality detection server is pre-stored with a plurality of sample images, and the apparatus may further include:

a second labeling module 607, configured to print a third label on a sample image that includes a preset behavior action in the plurality of sample images, and print a fourth label on a sample image that does not include the preset behavior action in the plurality of sample images;

the second training module 608 is configured to import each labeled sample image into the neural network model for training to obtain an abnormal behavior detection model.

The detection module 603 is specifically configured to:

In a possible implementation manner, the preset target object includes a minor object, and the detecting module 603 is specifically configured to:

In a possible implementation manner, the abnormality detection server has a plurality of sample images prestored therein, and the apparatus may further include:

a third labeling module 609, configured to label sample images including face images in the plurality of sample images with different age labels;

the third training module 610 is configured to import each labeled sample image into the neural network model for training to obtain an age detection model;

the image recognition module 602 may be specifically configured to:

Referring to fig. 8, in a possible implementation manner, the abnormality detection server is communicatively connected to a voice collecting device, and the apparatus may further include:

a voice acquiring module 611, configured to acquire voice information acquired by the voice acquiring device in a current trip service process;

a voice recognition module 612, configured to analyze the voice information to determine whether the voice information includes abnormal information;

the information sending module 604 is further configured to send a prompt message when the voice message includes abnormal information.

In a possible implementation manner, the speech recognition module 612 may be specifically configured to:

extracting key words contained in the voice information;

In a possible embodiment, the apparatus may further include:

an acquisition anomaly detection module 613, configured to detect whether an image acquisition anomaly occurs in the monitoring terminal 160 in a current travel service process, and send a prompt message when the image acquisition anomaly is detected; the acquisition anomaly detection module 613 may be specifically configured to execute one of the following:

detecting whether a video image acquired by the monitoring terminal 160 within a preset time length is a continuous full white or full black image in the current travel service process, and if the video image is the continuous full white or full black image, determining that image acquisition is abnormal;

in the current travel service process, whether the video image acquired by the monitoring terminal 160 within the preset time length is a continuous fixed image or not is detected, and if the video image is a continuous fixed image, it is determined that the image acquisition is abnormal.

The modules may be connected or in communication with each other via a wired or wireless connection. The wired connection may include a metal cable, an optical cable, a hybrid cable, etc., or any combination thereof. The wireless connection may comprise a connection over a LAN, WAN, bluetooth, ZigBee, NFC, or the like, or any combination thereof. Two or more modules may be combined into a single module, and any one module may be divided into two or more units.

The embodiment of the present application further provides a readable storage medium, where the readable storage medium stores computer-executable instructions, and the computer-executable instructions can execute the short domain name management method in any method embodiment described above.

It can be clearly understood by those skilled in the art that, for convenience and brevity of description, the specific working processes of the system and the apparatus described above may refer to corresponding processes in the method embodiments, and are not described in detail in this application. In the several embodiments provided in the present application, it should be understood that the disclosed system, apparatus and method may be implemented in other ways. The above-described apparatus embodiments are merely illustrative, and for example, the division of the modules is merely a logical division, and there may be other divisions in actual implementation, and for example, a plurality of modules or components may be combined or integrated into another system, or some features may be omitted, or not executed. In addition, the shown or discussed mutual coupling or direct coupling or communication connection may be an indirect coupling or communication connection of devices or modules through some communication interfaces, and may be in an electrical, mechanical or other form.

The modules described as separate parts may or may not be physically separate, and parts displayed as modules may or may not be physical units, may be located in one place, or may be distributed on a plurality of network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the solution of the embodiment.

In addition, functional units in the embodiments of the present application may be integrated into one processing unit, or each unit may exist alone physically, or two or more units are integrated into one unit.

The functions, if implemented in the form of software functional units and sold or used as a stand-alone product, may be stored in a non-volatile computer-readable storage medium executable by a processor. Based on such understanding, the technical solution of the present application or portions thereof that substantially contribute to the prior art may be embodied in the form of a software product stored in a storage medium and including instructions for causing a computer device (which may be a personal computer, a server, or a network device) to execute all or part of the steps of the method according to the embodiments of the present application. And the aforementioned storage medium includes: various media capable of storing program codes, such as a U disk, a removable hard disk, a ROM, a RAM, a magnetic disk, or an optical disk.

The above description is only for the specific embodiments of the present application, but the scope of the present application is not limited thereto, and any person skilled in the art can easily conceive of the changes or substitutions within the technical scope of the present application, and shall be covered by the scope of the present application. Therefore, the protection scope of the present application shall be subject to the protection scope of the claims.

Claims

1. A method for identifying abnormal behaviors online is applied to an abnormality detection server which is in communication connection with a monitoring terminal, and comprises the following steps:

and if the abnormal behavior occurs, sending out prompt information.

2. The method for on-line identification of abnormal behavior according to claim 1, wherein the abnormality detection server has a plurality of sample images pre-stored therein, the method further comprising:

3. The method for on-line identification of abnormal behaviors as claimed in claim 2, wherein the step of importing the video image into a pre-established gender detection model for detection to obtain a detection result comprises:

4. The method of claim 3, wherein after the steps of performing face detection on each image frame and determining an image frame containing a face image, the method further comprises:

5. The method for on-line identification of abnormal behavior according to claim 1, wherein the abnormality detection server has a plurality of sample images pre-stored therein, the method further comprising:

6. The method of claim 1, wherein the predetermined target object comprises an underage object, and the step of detecting the video image to determine whether an abnormal behavior occurs in the current travel service process comprises:

7. The method for online identification of abnormal behavior according to claim 6, wherein the abnormality detection server has a plurality of sample images pre-stored therein, and the method further comprises:

8. The method for identifying abnormal behavior online according to claim 1, wherein the abnormality detection server is connected to a voice collection device in communication, and the method further comprises:

9. The method of claim 8, wherein the step of analyzing the voice message to determine whether the voice message contains abnormal information comprises:

extracting key words contained in the voice information;

10. The method of online identification of abnormal behavior of claim 1, further comprising:

11. An apparatus for online identifying abnormal behavior, applied to an abnormality detection server, wherein the abnormality detection server is in communication connection with a monitoring terminal, the apparatus comprising:

12. The apparatus for online identification of abnormal behavior according to claim 11, wherein the abnormality detection server has a plurality of sample images pre-stored therein, the apparatus further comprising:

the image recognition module is specifically configured to:

13. The apparatus for online identification of abnormal behavior according to claim 12, wherein the image recognition module obtains the detection result by:

14. The apparatus for online identification of abnormal behavior according to claim 13, wherein the image recognition module is further configured to:

15. The apparatus for online identification of abnormal behavior according to claim 11, wherein the abnormality detection server has a plurality of sample images pre-stored therein, the apparatus further comprising:

the detection module is specifically configured to:

16. The apparatus for online identification of abnormal behavior of claim 11, wherein the preset target object comprises an underage object, and the detection module is specifically configured to:

17. The apparatus for online identification of abnormal behavior according to claim 16, wherein the abnormality detection server has a plurality of sample images pre-stored therein, the apparatus further comprising:

the image recognition module is specifically configured to:

18. The apparatus for online identifying abnormal behavior according to claim 11, wherein the abnormality detection server is communicatively connected to a voice collecting device, the apparatus further comprising:

19. The apparatus according to claim 18, wherein the speech recognition module is specifically configured to:

extracting key words contained in the voice information;

20. The apparatus for online identification of abnormal behavior of claim 11, further comprising:

the acquisition abnormity detection module is used for detecting whether the monitoring terminal has image acquisition abnormity in the current travel service process and sending prompt information when the image acquisition abnormity is detected; the acquisition anomaly detection module is specifically configured to execute one of the following:

21. An electronic device, comprising: a processor, a storage medium and a bus, wherein the storage medium stores machine-readable instructions executable by the processor, when the electronic device runs, the processor communicates with the storage medium through the bus, and the processor executes the machine-readable instructions to execute the steps of the online abnormal behavior recognizing method according to any one of claims 1 to 10.

22. A readable storage medium, having stored thereon a computer program which, when being executed by a processor, is adapted to carry out the steps of the method for online identification of abnormal behavior according to any one of claims 1 to 10.