CN112069043A - Terminal equipment state detection method, model generation method and device - Google Patents
Terminal equipment state detection method, model generation method and device Download PDFInfo
- Publication number
- CN112069043A CN112069043A CN202010774833.9A CN202010774833A CN112069043A CN 112069043 A CN112069043 A CN 112069043A CN 202010774833 A CN202010774833 A CN 202010774833A CN 112069043 A CN112069043 A CN 112069043A
- Authority
- CN
- China
- Prior art keywords
- state
- screen
- image
- detection model
- state detection
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/30—Monitoring
- G06F11/34—Recording or statistical evaluation of computer activity, e.g. of down time, of input/output operation ; Recording or statistical evaluation of user activity, e.g. usability assessment
- G06F11/3466—Performance evaluation by tracing or monitoring
- G06F11/349—Performance evaluation by tracing or monitoring for interfaces, buses
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/30—Monitoring
- G06F11/3003—Monitoring arrangements specially adapted to the computing system or computing system component being monitored
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
- G06N3/045—Combinations of networks
Abstract
The embodiment of the invention provides a terminal equipment state detection method, a model generation method and a device. The terminal equipment state detection method comprises the steps of extracting a screen area image of at least one terminal equipment from a collected monitoring image; inputting a screen area image of the terminal equipment into a preset first state detection model; determining the running state of the terminal equipment according to the first classification information output by the first state detection model; wherein the first classification information includes a screen normal state and a screen abnormal state. Therefore, whether the screen of the terminal equipment in the monitoring image is normally displayed can be detected by adopting the first state detection model, and whether the terminal equipment normally operates can be determined. The terminal equipment is automatically monitored by adopting the monitoring image, and the normal operation of the terminal equipment is ensured.
Description
Technical Field
The present invention relates to the field of image processing technologies, and in particular, to a method for detecting a state of a terminal device, a method for generating a state detection model of a terminal device, a device for detecting a state of a terminal device, and a device for generating a state detection model of a terminal device.
Background
At present, a plurality of terminal devices can be generally arranged in public spaces such as business halls, shopping malls, transportation hubs and the like of various enterprises. The client can handle the business by self through the terminal equipment, and the handling efficiency of the business can be improved to a certain extent. However, since the area where the terminal device is installed is usually unsupervised, it is usually difficult to find the damaged terminal device in time, which affects the use of the terminal device by the client.
Disclosure of Invention
In view of the above problems, embodiments of the present invention are provided to provide a terminal device state detection method, a terminal device state detection model generation method, a terminal device state detection apparatus, and a terminal device state detection model generation apparatus that overcome or at least partially solve the above problems.
In order to solve the above problem, an embodiment of the present invention discloses a method for detecting a state of a terminal device, including:
extracting a screen area image of at least one terminal device from the acquired monitoring image;
inputting a screen area image of the terminal equipment into a preset first state detection model;
determining the running state of the terminal equipment according to the first classification information output by the first state detection model; wherein the first classification information includes a screen normal state and a screen abnormal state.
Optionally, the step of determining the operating state of the terminal device according to the first classification information output by the first state detection model includes:
under the condition that the first classification information output by the first state detection model is in a screen abnormal state, extracting an equipment environment image of the terminal equipment from the monitoring image;
inputting an equipment environment image of the terminal equipment into a preset second state detection model;
determining the running state of the terminal equipment according to second classification information output by the second state detection model; and the second classification information comprises a personnel shielding state and a personnel non-shielding state.
Optionally, the step of extracting, in the monitoring image, an apparatus environment image of the terminal apparatus when the first classification information output by the first state detection model is in a screen abnormal state includes:
under the condition that the first classification information output by the first state detection model is in a screen abnormal state, determining the size of an equipment environment image based on a preset outward expansion ratio and the size of the screen area image;
and extracting the equipment environment image from the monitoring image according to the position of the screen area image in the monitoring image and the size of the equipment environment image.
Optionally, the method further comprises:
and extracting at least one frame of monitoring image from the collected monitoring video by adopting a preset frame extraction frequency.
Optionally, the operation state of the terminal device includes a normal operation state and an abnormal operation state;
the method further comprises the following steps:
and sending alarm information under the condition that the running states of the terminal equipment are all abnormal running states in the continuous monitoring images with the preset number.
The embodiment of the invention also discloses a method for generating the state detection model of the terminal equipment, which comprises the following steps:
acquiring a screen area sample of the terminal equipment; the screen area sample correspondingly has first annotation information, and the first annotation information comprises a screen normal state and a screen abnormal state;
and training a preset first model to be trained by adopting the screen area sample and first marking information corresponding to the screen area sample to generate a first state detection model.
Optionally, the method further comprises:
acquiring an equipment environment sample of terminal equipment; the equipment environment sample correspondingly has second marking information, and the second marking information comprises a personnel shielding state and a personnel non-shielding state;
and training a preset second model to be trained by adopting the equipment environment image sample and second labeling information corresponding to the equipment environment image sample to generate a second state detection model.
The embodiment of the invention also discloses a device for detecting the state of the terminal equipment, which comprises:
the screen extraction module is used for extracting a screen area image of at least one terminal device from the acquired monitoring image;
the screen detection module is used for inputting the screen area image of the terminal equipment into a preset first state detection model;
the state determining module is used for determining the running state of the terminal equipment according to the first classification information output by the first state detection model; wherein the first classification information includes a screen normal state and a screen abnormal state.
Optionally, the state determination module includes:
the environment extraction submodule is used for extracting an equipment environment image of the terminal equipment from the monitoring image under the condition that the first classification information output by the first state detection model is in a screen abnormal state;
the environment detection submodule is used for inputting the equipment environment image of the terminal equipment into a preset second state detection model;
the state determining submodule is used for determining the running state of the terminal equipment according to the second classification information output by the second state detection model; and the second classification information comprises a personnel shielding state and a personnel non-shielding state.
Optionally, the environment extraction sub-module includes:
the size determining unit is used for determining the size of the equipment environment image based on a preset outward expansion ratio and the size of the screen area image under the condition that the first classification information output by the first state detection model is in the abnormal state of the screen;
and the environment extraction unit is used for extracting the equipment environment image from the monitoring image according to the position of the screen area image in the monitoring image and the size of the equipment environment image.
Optionally, the apparatus further comprises:
and the image extraction module is used for extracting at least one frame of monitoring image from the collected monitoring video by adopting a preset frame extraction frequency.
Optionally, the operation state of the terminal device includes a normal operation state and an abnormal operation state;
the device further comprises:
and the alarm module is used for sending alarm information under the condition that the running states of the terminal equipment are determined to be abnormal running states in the continuous monitoring images in the preset number.
The embodiment of the invention also discloses a device for generating the state detection model of the terminal equipment, which comprises the following steps:
the first sample acquisition module is used for acquiring a screen area sample of the terminal equipment; the screen area sample correspondingly has first annotation information, and the first annotation information comprises a screen normal state and a screen abnormal state;
and the first training module is used for training a preset first model to be trained by adopting the screen area sample and first marking information corresponding to the screen area sample to generate a first state detection model.
Optionally, the apparatus further comprises:
the first sample acquisition module is used for acquiring an equipment environment sample of the terminal equipment; the equipment environment sample correspondingly has second marking information, and the second marking information comprises a personnel shielding state and a personnel non-shielding state;
and the second training module is used for training a preset second model to be trained by adopting the equipment environment image sample and second marking information corresponding to the equipment environment image sample to generate a second state detection model.
The embodiment of the invention also discloses a device, which comprises:
one or more processors; and
one or more machine-readable media having instructions stored thereon, which when executed by the one or more processors, cause the apparatus to perform one or more methods as described in embodiments of the invention.
Embodiments of the invention also disclose one or more machine-readable media having instructions stored thereon, which when executed by one or more processors, cause the processors to perform one or more methods as described in embodiments of the invention.
The embodiment of the invention has the following advantages:
according to the terminal equipment state detection method provided by the embodiment of the invention, the screen area image of at least one terminal equipment is extracted from the collected monitoring image, the screen area image of the terminal equipment is input into a preset first state detection model, and the running state of the terminal equipment is determined according to first classification information output by the first state detection model. Therefore, whether the screen of the terminal equipment in the monitoring image is normally displayed can be detected by adopting the first state detection model, and whether the terminal equipment normally operates can be determined. The terminal equipment is automatically monitored by adopting the monitoring image, and the normal operation of the terminal equipment is ensured.
Drawings
Fig. 1 is a flowchart illustrating steps of a method for detecting a state of a terminal device according to an embodiment of the present invention;
fig. 2 is a flowchart illustrating steps of another method for detecting a status of a terminal device according to an embodiment of the present invention;
fig. 3 is a flowchart illustrating steps of a method for generating a state detection model of a terminal device according to an embodiment of the present invention;
fig. 4 is a block diagram of a terminal device state detection apparatus according to an embodiment of the present invention;
fig. 5 is a block diagram of a device for generating a state detection model of a terminal device according to an embodiment of the present invention.
Detailed Description
In order to make the aforementioned objects, features and advantages of the present invention comprehensible, embodiments accompanied with figures are described in further detail below.
Referring to fig. 1, a flowchart illustrating steps of an embodiment of a method for detecting a state of a terminal device according to the present invention is shown, which may specifically include the following steps:
in the embodiment of the present invention, the terminal device may be a terminal device disposed in a public space and configured to provide a service handling function for a person. For example, an automatic teller machine provided in a bank, a self-service payment machine provided in a hospital, a self-service reporter machine provided in an airport, a self-service ticket purchasing machine provided in a railway station, and the like.
In the embodiment of the present invention, the terminal device may be generally provided with a screen so as to display the business related information for the person. Generally, in the case of an abnormality of the terminal device, the screen of the terminal device usually cannot normally show information. For example, in the case of a power failure of the terminal device, the screen of the terminal device may correspond to the case of a black screen. And under the condition that the system of the terminal equipment is abnormal, the screen of the terminal equipment can correspondingly display the abnormal prompt information of the blue background.
Therefore, the area provided with the terminal equipment can be monitored by adopting preset monitoring equipment, and the monitoring equipment can be arranged in the position capable of shooting the screen position of the terminal equipment, so that whether the terminal equipment is in a normal state or not can be determined according to the monitoring image collected by the monitoring equipment.
The monitoring image may include at least one terminal device, and the image of the screen area of the terminal device in the monitoring image may be extracted to obtain the screen area image of the at least one terminal device in the monitoring image.
In a specific implementation, a pre-trained target recognition model can be adopted to recognize the screen area of the terminal device in the monitoring image in a target recognition mode, so that the screen area image in the monitoring image is extracted.
In a specific implementation, since the position of the monitoring device may be unchanged, the area that the monitoring device can shoot may also be unchanged, and therefore, the screen area image of the terminal device may also be obtained by recording the coordinate information of the screen area of the terminal device in the monitoring image in advance, and extracting the image at the coordinate information from the monitoring image after the monitoring image is acquired.
in the embodiment of the present invention, the screen area image of the terminal device may be input into a preset first state detection model to detect the screen display state of the terminal device.
The first state detection model may be a pre-trained model, and may be configured to classify the screen region images to distinguish screen display states of the terminal device. Specifically, the first state detection model may classify the screen region image into a screen normal state in which a screen of the terminal device is normally displayed, and a screen abnormal state in which a screen of the terminal device is abnormally displayed. If the first state detection model classifies the screen region image as a screen normal state, the screen display of the terminal device can be considered to be normal. If the first state detection model classifies the screen region image into the screen abnormal state, the screen display of the terminal device may be considered abnormal.
In a specific implementation, a preset first model to be trained may be trained by using a screen region sample with first labeling information to generate a first state detection model. The first annotation information may include a screen normal state and a screen abnormal state. The preset first model to be trained may be a convolutional neural network model, a perceptron model, a random forest model, a support vector machine model, a K-nearest neighbor algorithm model, or the like, which is not limited in the present invention.
103, determining the running state of the terminal equipment according to the first classification information output by the first state detection model; wherein the first classification information includes a screen normal state and a screen abnormal state.
In an embodiment of the present invention, the first state detection model may output first classification information. The first classification information may include a screen normal state and a screen abnormal state. Whether the screen display state of the terminal equipment is normal can be known according to the first classification information output by the first state detection model, and whether the running state of the terminal equipment is normal can be determined according to the screen display state.
In a specific implementation, when the first classification information output by the first state detection model is in a normal screen state, it may be considered that the screen display of the terminal device is normal, and then the terminal device may be in a normal operating state.
When the first classification information output by the first state detection model is in a screen abnormal state, it may be considered that the screen display of the terminal device is abnormal, and the terminal device may be in an abnormal operation state. At this time, the terminal device may malfunction, which may cause the screen not to be normally displayed.
According to the terminal equipment state detection method provided by the embodiment of the invention, the screen area image of at least one terminal equipment is extracted from the collected monitoring image, the screen area image of the terminal equipment is input into a preset first state detection model, and the running state of the terminal equipment is determined according to first classification information output by the first state detection model. Therefore, whether the screen of the terminal equipment in the monitoring image is normally displayed can be detected by adopting the first state detection model, and whether the terminal equipment normally operates can be determined. The terminal equipment is automatically monitored by adopting the monitoring image, and the normal operation of the terminal equipment is ensured.
Referring to fig. 2, a flowchart illustrating steps of an embodiment of a method for detecting a state of a terminal device according to the present invention is shown, which may specifically include the following steps:
in the embodiment of the present invention, the terminal device may be a terminal device disposed in a public space and configured to provide a service handling function for a person. The terminal device may typically be provided with a screen in order to present the service related information to a person. Generally, in the case of an abnormality of the terminal device, the screen of the terminal device usually cannot normally show information. For example, in the case of a power failure of the terminal device, the screen of the terminal device may correspond to the case of a black screen. And under the condition that the system of the terminal equipment is abnormal, the screen of the terminal equipment can correspondingly display the abnormal prompt information of the blue background.
Therefore, the area provided with the terminal equipment can be monitored by adopting preset monitoring equipment, and the monitoring equipment can be arranged in the position capable of shooting the screen position of the terminal equipment, so that whether the terminal equipment is in a normal state or not can be determined according to the monitoring image collected by the monitoring equipment.
The monitoring image may include at least one terminal device, and the image of the screen area of the terminal device in the monitoring image may be extracted to obtain the screen area image of the at least one terminal device in the monitoring image.
In a specific implementation, a pre-trained target recognition model can be adopted to recognize the screen area of the terminal device in the monitoring image in a target recognition mode, so that the screen area image in the monitoring image is extracted.
In a specific implementation, since the position of the monitoring device may be unchanged, the area that the monitoring device can shoot may also be unchanged, and therefore, the screen area image of the terminal device may also be obtained by recording the coordinate information of the screen area of the terminal device in the monitoring image in advance, and extracting the image at the coordinate information from the monitoring image after the monitoring image is acquired.
In one embodiment of the invention, the method further comprises:
and S11, extracting at least one frame of monitoring image from the collected monitoring video by adopting a preset frame extraction frequency.
In the embodiment of the present invention, the monitoring device may be a video capture device. The monitoring device can acquire the video of the area of the terminal device to obtain the monitoring video. The monitoring video may be composed of a plurality of frames of monitoring images.
For a platform for managing monitoring equipment, because the monitoring video contains more monitoring images, the running state detection of the terminal equipment on all the monitoring images needs to consume more computing resources. Besides monitoring the terminal device, the platform generally needs to process other monitoring tasks, so that the platform may be occupied by other monitoring tasks with more computing resources. Therefore, a frame extracting frequency can be determined based on the processing capacity and the idle resources of the platform for managing the monitoring equipment, and at least one frame of monitoring image is extracted from the collected monitoring video by adopting the preset frame extracting frequency to be used for detecting the running state of the terminal equipment, so that the real-time performance of the monitoring terminal equipment can be ensured to a certain extent, and meanwhile, the computing resources consumed for detecting the running state of the terminal equipment can be reasonably controlled.
The frame extraction frequency may be determined according to actual needs, for example, 4 frames are extracted every 1 second, one monitoring image is extracted every 5 seconds, one monitoring image is extracted every 1 minute, and the like, which is not limited in the present invention.
in the embodiment of the present invention, the screen area image of the terminal device may be input into a preset first state detection model to detect the screen display state of the terminal device.
The first state detection model may be a pre-trained model, and is configured to classify the screen region images to distinguish screen display states of the terminal device. Specifically, the first state detection model may classify the screen region image into a screen normal state in which a screen of the terminal device is normally displayed, and a screen abnormal state in which a screen of the terminal device is abnormally displayed. If the first state detection model classifies the screen region image as a screen normal state, the screen display of the terminal device can be considered to be normal. If the first state detection model classifies the screen region image into the screen abnormal state, the screen display of the terminal device may be considered abnormal.
In a specific implementation, a preset first model to be trained may be trained by using a screen region sample with labeling information to generate a first state detection model. The annotation information may include a screen normal state and a screen abnormal state. The preset classification model may be a convolutional neural network model, a perceptron model, a random forest model, a support vector machine model, a K-nearest neighbor algorithm model, or the like, which is not limited in the present invention.
in an embodiment of the present invention, the first state detection model may output first classification information. The first classification information may include a screen normal state and a screen abnormal state. Whether the screen display state of the terminal equipment is normal can be known according to the first classification information output by the first state detection model.
In a specific implementation, when the first classification information output by the first state detection model is in a normal screen state, it may be considered that the screen display of the terminal device is normal, and then the terminal device may be in a normal operating state.
When the first classification information output by the first state detection model is in a screen abnormal state, it may be considered that the screen display of the terminal device is abnormal, and the terminal device may be in an abnormal operation state. At this time, the terminal device may malfunction, which may cause the screen not to be normally displayed.
In the embodiment of the present invention, when the first classification information output by the first state detection model is a screen abnormal state, there may be a case where the screen of the terminal device is not displayed abnormally, but only the screen of the terminal device is blocked by a person in the monitoring image because the person operates the terminal device, so that the classification information output by the first state detection model is a screen abnormal state.
In order to avoid the situation that the first state detection model judges the operation state of the terminal device incorrectly, if the classification information output by the first state detection model is in the abnormal screen state, it may be further determined whether the abnormal display does occur on the screen of the terminal device.
Therefore, when the first classification information output by the first state detection model is in the abnormal screen state, the device environment image including the terminal device and the surrounding environment of the terminal device can be extracted from the monitoring image, so that whether the terminal device is blocked by people or not can be determined by using the device environment image.
In an embodiment of the present invention, in a case where the first classification information output by the first state detection model is a screen abnormal state, the step of extracting, from the monitoring image, an apparatus environment image of the terminal apparatus includes:
s21, determining the size of the device environment image based on a preset outward expansion ratio and the size of the screen area image under the condition that the first classification information output by the first state detection model is in the abnormal state of the screen;
in the embodiment of the present invention, when the first classification information output by the first state detection model is in the abnormal screen state, the size of the device environment image, which may include the terminal device itself and the surrounding environment of the terminal device, may be determined based on a preset outward expansion ratio and the size of the screen region image.
The external expansion ratio may be a ratio that enables the terminal device itself and the surrounding environment of the terminal device to be included in the device environment image according to actual needs, which is not limited in the present invention.
And S22, extracting the device environment image from the monitoring image according to the position of the screen area image in the monitoring image and the size of the device environment image.
In the embodiment of the present invention, the device environment image of the terminal device may be extracted from the monitoring image according to the position of the screen region image in the monitoring image and the size of the device environment image, so as to further determine whether the terminal device is blocked by a person by using the device environment image.
In a specific implementation, the center coordinates of the screen region image may be used as the center coordinates of the device environment image, and the device environment image is extracted from the monitoring image based on the size of the device environment image, so that the obtained device environment image may include the terminal device itself and the surrounding environment of the terminal device.
in the embodiment of the present invention, the device environment image of the terminal device may be input into a preset second state detection model to detect whether the terminal device is blocked by a person.
The second state detection model may be a pre-trained model, and may be configured to classify the device environment image to distinguish the peripheral environment state of the terminal device. Specifically, the second state detection model may classify the device environment image into a person blocking state in which a person blocks the terminal device, or a person unblocking state in which no person blocks the device but an abnormal person exists in the terminal device environment or the terminal device itself. If the second state detection model classifies the equipment environment image as a person shielding state, the screen of the terminal equipment can be considered to be shielded by the person, and the display of the terminal equipment is normal. If the second state detection model classifies the device image as a person-unobstructed state, it may be considered that the screen of the terminal device is not obstructed by a person, but the surrounding environment of the terminal device is abnormal, or the screen display of the terminal device is abnormal.
In a specific implementation, a preset second model to be trained may be trained by using the device environment sample with the second label information, so as to generate a second state detection model. The second labeling information may include a person shielding state and a person non-shielding state. The preset second model to be trained may be a convolutional neural network model, a perceptron model, a random forest model, a support vector machine model, a K-nearest neighbor algorithm model, or the like, which is not limited in the present invention.
In this embodiment of the present invention, the second state detection model may output second classification information, where the second classification information may include a person-blocked state and a person-unblocked state. Whether the terminal equipment is shielded by personnel can be known according to second classification information output by the second state model, so that whether the terminal equipment is in the abnormal operating state or not is determined.
In a specific implementation, if the second classification information output by the second state detection model is a person blocking state, it may be considered that the terminal device is blocked by the person. In this case, the screen of the terminal device may be displayed normally, and only classified as the screen abnormal state by the first state detection model because the terminal device is blocked by a person. At this time, the terminal device can be considered to be still in a normal operation state.
If the second classification information output by the second state detection model is in a person-unobstructed state, it can be considered that the terminal device is not obstructed by a person. In this case, the terminal device may be classified into the screen abnormal state by the first detection classification model due to the screen display abnormality, and at this time, it may be considered that the screen cannot be normally displayed due to the failure of the terminal device, and the terminal device may be in the abnormal operation state.
If the second classification information output by the second state detection model is in a state that people are not shielded, the terminal device may be classified into a screen abnormal state by the first detection classification model due to the abnormal surrounding environment. At this time, the screen of the terminal device may be in a normal screen state or an abnormal screen state, but due to the fact that abnormal conditions may exist in the surrounding environment of the terminal device, a person cannot normally use the terminal device, and thus the terminal device may be considered to be in an abnormal operation state. For example, if the terminal device is shielded by other objects other than a person, and the person cannot use the terminal device, it may be determined that an abnormal condition exists in the surrounding environment of the terminal device at this time, and the terminal device is in an abnormal operating state.
In one embodiment of the invention, the method further comprises:
and S31, sending alarm information when the running states of the terminal equipment are all abnormal running states in the continuous monitoring images with the preset number.
In the embodiment of the present invention, the operation states of the terminal device may include a normal operation state and an abnormal operation state. The terminal equipment can be considered to be in a normal operation state when the screen of the terminal equipment is in a normal screen state or the terminal equipment is in a personnel shielding state. When the screen of the terminal device is in the screen abnormal state, or the screen of the terminal device is in the screen abnormal state and the terminal device is in the state that the personnel are not shielded, the terminal device can be considered to be in the abnormal operation state.
In the embodiment of the present invention, in the monitoring images of the preset number that are continuous in the time sequence, if it is determined that the operation states of the terminal devices are all abnormal operation states, it may be determined that the terminal devices have a high possibility of actually failing, so that an alarm message may be sent to a maintenance person to prompt the maintenance person to maintain the terminal devices in time.
The preset number may be determined according to actual needs, for example, 3 consecutive monitoring images, 5 consecutive monitoring images, 10 consecutive monitoring images, and the like, which is not limited in the present invention.
According to the terminal equipment state detection method, the screen area image of at least one terminal equipment is extracted from the collected monitoring image, the screen area image of the terminal equipment is input into the preset first state detection model, the equipment environment image of the terminal equipment is extracted from the monitoring image under the condition that the first classification information output by the first state detection model is in the abnormal screen state, the equipment environment image of the terminal equipment is input into the preset second state detection model, and the running state of the terminal equipment is determined according to the second classification information output by the second state detection model. Therefore, whether the screen of the terminal equipment in the monitoring image is normally displayed can be detected by adopting the first state detection model, and whether the terminal equipment normally operates can be determined. Under the condition that the first state detection model detects that the screen of the terminal device is in abnormal display, a second state detection model can be further adopted to determine whether the terminal device is shielded by people so as to further determine whether the terminal device normally operates, and the accuracy of detecting the abnormal operation state of the terminal device is further improved. The terminal equipment is automatically monitored by adopting the monitoring image, and the normal operation of the terminal equipment is ensured.
Referring to fig. 3, a flowchart illustrating steps of an embodiment of a method for generating a terminal device state detection model according to the present invention is shown, which may specifically include the following steps:
in the embodiment of the present invention, in order to train a preset first model to be trained, an image of a screen area of a terminal device may be acquired as a screen area sample. The screen region sample may have manually annotated first annotation information. The first annotation information may include a screen normal state and a screen abnormal state.
The first annotation information is a screen area sample of a normal screen state, and may be an image of a screen area of the terminal device when the terminal device operates normally. The first annotation information is a screen area sample of a screen abnormal state, which may be an image of a screen area of the terminal device when the terminal device operates abnormally.
Optionally, there may be a case where the screen area is displayed normally, but the screen of the terminal device is partially blocked by a person in the screen area sample. Therefore, a certain number of samples with normal screen areas but partially blocked screens of the terminal equipment by people can be properly added in the screen area samples with the normal screen state of the first annotation information. So that the classification accuracy can be improved to some extent.
In the embodiment of the present invention, the preset first model to be trained may be trained by using the screen area sample and the first label information corresponding to the screen area sample, so as to generate the first state detection model.
The first model to be trained may be a convolutional neural network model such as an EfficientNet network, a perceptron model, a random forest model, a support vector machine model, a K-nearest neighbor algorithm model, or the like, which is not limited in this respect.
In a specific implementation, a preset number of screen area samples with the first annotation information in a normal state of the screen and a preset number of screen area samples with the first annotation information in an abnormal state of the screen may be collected. And normalizing the screen area samples into images with the same size and the same number of channels. For example, the screen area samples are normalized to a color image with a size of 300 × 300 and a number of channels of 3.
Optionally, brightness adjustment and definition adjustment may be performed on the screen region samples to obtain more screen region samples under different conditions.
Then, the screen area sample may be used as an input of a model, the first label information may be used as an output of the model, and a preset first model to be trained is trained, so as to generate a first state detection model.
As an example of the present invention, in the case that the first model to be trained is a convolutional neural network model, a deep learning framework, such as a tensrflow framework, may be used to construct the first model to be trained, and the first model to be trained may be trained by using the screen region sample as an input of the model and the first label information as an output of the model. In the training process, the initial learning rate can be set to be 0.001, the learning rate is attenuated by one tenth after each preset turn, model parameters are adjusted by adopting an adaptive moment estimation (Adam) optimizer to enable the model to be converged, and finally a first model to be trained after training is obtained and used as a first state detection model.
In one embodiment of the invention, the method further comprises:
s41, obtaining an equipment environment sample of the terminal equipment; the equipment environment sample correspondingly has second marking information, and the second marking information comprises a personnel shielding state and a personnel non-shielding state;
in the embodiment of the present invention, when the first classification information output by the first state detection model is a screen abnormal state, there may be a case where the screen of the terminal device is not displayed abnormally, but only the screen of the terminal device is blocked by a person in the monitoring image because the person operates the terminal device, so that the classification information output by the first state detection model is a screen abnormal state.
In order to avoid the situation that the first state detection model judges the running state of the terminal device incorrectly, a second state detection model can be trained, when the first classification information output by the first state detection model is in a screen abnormal state, the second state detection model can be adopted to detect the environmental image of the device, and whether the terminal device is partially shielded by people or not is identified, or the terminal device is shielded by people completely
Therefore, the image containing the terminal device and the surrounding environment of the terminal device is obtained and used as the device environment sample of the terminal device to train the second model to be trained, and the second state detection model is obtained.
In the embodiment of the invention, the equipment environment sample can have second labeling information labeled manually. The second labeling information may include a person occlusion state and a person non-occlusion state.
The second labeling information is an equipment environment sample of a personnel shielding state, and can be an equipment environment image in which the terminal equipment normally operates and is partially shielded by personnel or completely shielded by personnel. The second labeling information is an equipment environment sample in a state that the person is not shielded, and can be an equipment environment image in which the terminal equipment abnormally operates and is partially or completely shielded by other objects except the person.
And S42, training a preset second model to be trained by adopting the equipment environment image sample and second labeling information corresponding to the equipment environment image sample, and generating a second state detection model.
In the embodiment of the present invention, the device environment image sample and the second label information corresponding to the device environment image sample may be used to train a preset second model to be trained, so as to generate a second state detection model.
The second model to be trained may be a convolutional neural network model such as an EfficientNet network, a perceptron model, a random forest model, a support vector machine model, a K-nearest neighbor algorithm model, and the like, which is not limited in this respect.
In a specific implementation, the device environment samples with the preset number of second labeling information as the personnel shielding state and the device environment samples with the preset number of second labeling information as the personnel non-shielding state can be collected. And normalizing the device environment samples into images with the same size and the same number of channels. For example, the device environment sample is normalized to a color image with a size of 300 × 300 and a number of channels of 3.
Optionally, brightness adjustment and definition adjustment may be performed on the device environment sample to obtain more device environment samples under different conditions.
Then, the device environment sample may be used as an input of a model, the second labeling information may be used as an output of the model, and a preset second model to be trained is trained, so as to generate a second state detection model.
As an example of the present invention, in a case that the second model to be trained is a convolutional neural network model, a deep learning framework, such as a tensrflow framework, may be used to construct the second model to be trained, and the second model to be trained may be trained by using the device environment sample as an input of the model and the second label information as an output of the model. In the training process, the initial learning rate can be set to be 0.001, the learning rate is attenuated by one tenth after each preset turn, model parameters are adjusted by adopting an adaptive moment estimation (Adam) optimizer to enable the model to be converged, and finally a second model to be trained after training is obtained and used as a second state detection model.
According to the method for generating the state detection model of the terminal equipment, the preset first model to be trained is trained by adopting the screen area sample and the first marking information corresponding to the screen area sample, and the first state detection model is generated. Therefore, whether the screen of the terminal equipment is normally displayed can be detected by adopting the first state detection model, and whether the terminal equipment normally operates can be determined according to the display state of the equipment screen. Meanwhile, a preset second model to be trained is trained by adopting the equipment environment image sample and second labeling information corresponding to the equipment environment image sample, and a second state detection model is generated. Therefore, under the condition that the first state detection model detects that the terminal equipment is in the abnormal screen state, the second state detection model can be adopted to detect whether the terminal equipment is shielded by people, and the accuracy of detecting whether the terminal equipment normally operates is further improved.
It should be noted that, for simplicity of description, the method embodiments are described as a series of acts or combination of acts, but those skilled in the art will recognize that the present invention is not limited by the illustrated order of acts, as some steps may occur in other orders or concurrently in accordance with the embodiments of the present invention. Further, those skilled in the art will appreciate that the embodiments described in the specification are presently preferred and that no particular act is required to implement the invention.
Referring to fig. 4, a block diagram of a structure of an embodiment of a terminal device state detection apparatus according to the present invention is shown, which may specifically include the following modules:
a screen extraction module 401, configured to extract a screen area image of at least one terminal device from the acquired monitoring image;
a screen detection module 402, configured to input a screen area image of the terminal device into a preset first state detection model;
a state determining module 403, configured to determine an operating state of the terminal device according to the first classification information output by the first state detection model; wherein the first classification information includes a screen normal state and a screen abnormal state.
Optionally, the status determining module 403 includes:
the environment extraction submodule is used for extracting an equipment environment image of the terminal equipment from the monitoring image under the condition that the first classification information output by the first state detection model is in a screen abnormal state;
the environment detection submodule is used for inputting the equipment environment image of the terminal equipment into a preset second state detection model;
the state determining submodule is used for determining the running state of the terminal equipment according to the second classification information output by the second state detection model; and the second classification information comprises a personnel shielding state and a personnel non-shielding state.
In one embodiment of the invention, the environment extraction sub-module includes:
the size determining unit is used for determining the size of the equipment environment image based on a preset outward expansion ratio and the size of the screen area image under the condition that the first classification information output by the first state detection model is in the abnormal state of the screen;
and the environment extraction unit is used for extracting the equipment environment image from the monitoring image according to the position of the screen area image in the monitoring image and the size of the equipment environment image.
In one embodiment of the invention, the apparatus further comprises:
and the image extraction module is used for extracting at least one frame of monitoring image from the collected monitoring video by adopting a preset frame extraction frequency.
In an embodiment of the present invention, the operation states of the terminal device include a normal operation state and an abnormal operation state;
the device further comprises:
and the alarm module is used for sending alarm information under the condition that the running states of the terminal equipment are determined to be abnormal running states in the continuous monitoring images in the preset number.
Referring to fig. 5, a block diagram of a structure of an embodiment of a device for generating a terminal device state detection model according to the present invention is shown, and the device specifically includes the following modules:
a first sample obtaining module 501, configured to obtain a screen area sample of a terminal device; the screen area sample correspondingly has first annotation information, and the first annotation information comprises a screen normal state and a screen abnormal state;
the first training module 502 is configured to train a preset first model to be trained by using the screen region sample and the first label information corresponding to the screen region sample, and generate a first state detection model.
In one embodiment of the invention, the apparatus further comprises:
the first sample acquisition module is used for acquiring an equipment environment sample of the terminal equipment; the equipment environment sample correspondingly has second marking information, and the second marking information comprises a personnel shielding state and a personnel non-shielding state;
and the second training module is used for training a preset second model to be trained by adopting the equipment environment image sample and second marking information corresponding to the equipment environment image sample to generate a second state detection model.
For the device embodiment, since it is basically similar to the method embodiment, the description is simple, and for the relevant points, refer to the partial description of the method embodiment.
An embodiment of the present invention further provides an apparatus, including:
one or more processors; and
one or more machine-readable media having instructions stored thereon, which when executed by the one or more processors, cause the apparatus to perform methods as described in embodiments of the invention.
Embodiments of the invention also provide one or more machine-readable media having instructions stored thereon, which when executed by one or more processors, cause the processors to perform the methods described in embodiments of the invention.
The embodiments in the present specification are described in a progressive manner, each embodiment focuses on differences from other embodiments, and the same and similar parts among the embodiments are referred to each other.
As will be appreciated by one skilled in the art, embodiments of the present invention may be provided as a method, apparatus, or computer program product. Accordingly, embodiments of the present invention may take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware aspects. Furthermore, embodiments of the present invention may take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, and the like) having computer-usable program code embodied therein.
Embodiments of the present invention are described with reference to flowchart illustrations and/or block diagrams of methods, terminal devices (systems), and computer program products according to embodiments of the invention. It will be understood that each flow and/or block of the flow diagrams and/or block diagrams, and combinations of flows and/or blocks in the flow diagrams and/or block diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to a processor of a general purpose computer, special purpose computer, embedded processor, or other programmable data processing terminal to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing terminal, create means for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.
These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing terminal to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instruction means which implement the function specified in the flowchart flow or flows and/or block diagram block or blocks.
These computer program instructions may also be loaded onto a computer or other programmable data processing terminal to cause a series of operational steps to be performed on the computer or other programmable terminal to produce a computer implemented process such that the instructions which execute on the computer or other programmable terminal provide steps for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.
While preferred embodiments of the present invention have been described, additional variations and modifications of these embodiments may occur to those skilled in the art once they learn of the basic inventive concepts. Therefore, it is intended that the appended claims be interpreted as including preferred embodiments and all such alterations and modifications as fall within the scope of the embodiments of the invention.
Finally, it should also be noted that, herein, relational terms such as first and second, and the like may be used solely to distinguish one entity or action from another entity or action without necessarily requiring or implying any actual such relationship or order between such entities or actions. Also, the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or terminal that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or terminal. Without further limitation, an element defined by the phrase "comprising an … …" does not exclude the presence of other like elements in a process, method, article, or terminal that comprises the element.
The method for detecting the state of the terminal device, the method for generating the state detection model of the terminal device, the device for detecting the state of the terminal device and the device for generating the state detection model of the terminal device provided by the invention are described in detail, specific examples are applied in the description to explain the principle and the implementation mode of the invention, and the description of the embodiments is only used for helping to understand the method and the core idea of the invention; meanwhile, for a person skilled in the art, according to the idea of the present invention, there may be variations in the specific embodiments and the application scope, and in summary, the content of the present specification should not be construed as a limitation to the present invention.
Claims (10)
1. A method for detecting the state of terminal equipment is characterized by comprising the following steps:
extracting a screen area image of at least one terminal device from the acquired monitoring image;
inputting a screen area image of the terminal equipment into a preset first state detection model;
determining the running state of the terminal equipment according to the first classification information output by the first state detection model; wherein the first classification information includes a screen normal state and a screen abnormal state.
2. The method according to claim 1, wherein the step of determining the operation status of the terminal device according to the first classification information output by the first status detection model comprises:
under the condition that the first classification information output by the first state detection model is in a screen abnormal state, extracting an equipment environment image of the terminal equipment from the monitoring image;
inputting an equipment environment image of the terminal equipment into a preset second state detection model;
determining the running state of the terminal equipment according to second classification information output by the second state detection model; and the second classification information comprises a personnel shielding state and a personnel non-shielding state.
3. The method according to claim 2, wherein the step of extracting, in the monitoring image, an apparatus environment image of the terminal apparatus in the case where the first classification information output by the first state detection model is a screen abnormal state includes:
under the condition that the first classification information output by the first state detection model is in a screen abnormal state, determining the size of an equipment environment image based on a preset outward expansion ratio and the size of the screen area image;
and extracting the equipment environment image from the monitoring image according to the position of the screen area image in the monitoring image and the size of the equipment environment image.
4. The method of claim 1, further comprising:
and extracting at least one frame of monitoring image from the collected monitoring video by adopting a preset frame extraction frequency.
5. The method according to any one of claims 1 to 4, wherein the operation state of the terminal device comprises a normal operation state and an abnormal operation state;
the method further comprises the following steps:
and sending alarm information under the condition that the running states of the terminal equipment are all abnormal running states in the continuous monitoring images with the preset number.
6. A method for generating a state detection model of a terminal device is characterized by comprising the following steps:
acquiring a screen area sample of the terminal equipment; the screen area sample correspondingly has first annotation information, and the first annotation information comprises a screen normal state and a screen abnormal state;
and training a preset first model to be trained by adopting the screen area sample and first marking information corresponding to the screen area sample to generate a first state detection model.
7. A terminal device state detection apparatus, comprising:
the screen extraction module is used for extracting a screen area image of at least one terminal device from the acquired monitoring image;
the screen detection module is used for inputting the screen area image of the terminal equipment into a preset first state detection model;
the state determining module is used for determining the running state of the terminal equipment according to the first classification information output by the first state detection model; wherein the first classification information includes a screen normal state and a screen abnormal state.
8. A generation device of a terminal device state detection model is characterized by comprising:
the first sample acquisition module is used for acquiring a screen area sample of the terminal equipment; the screen area sample correspondingly has first annotation information, and the first annotation information comprises a screen normal state and a screen abnormal state;
and the first training module is used for training a preset first model to be trained by adopting the screen area sample and first marking information corresponding to the screen area sample to generate a first state detection model.
9. An apparatus, comprising:
one or more processors; and
one or more machine-readable media having instructions stored thereon that, when executed by the one or more processors, cause the apparatus to perform the method of one or more of claims 1-5 or 6.
10. One or more machine readable media having instructions stored thereon that, when executed by one or more processors, cause the processors to perform the method of one or more of claims 1-5 or 6.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010774833.9A CN112069043A (en) | 2020-08-04 | 2020-08-04 | Terminal equipment state detection method, model generation method and device |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010774833.9A CN112069043A (en) | 2020-08-04 | 2020-08-04 | Terminal equipment state detection method, model generation method and device |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN112069043A true CN112069043A (en) | 2020-12-11 |
Family
ID=73657493
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202010774833.9A Pending CN112069043A (en) | 2020-08-04 | 2020-08-04 | Terminal equipment state detection method, model generation method and device |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN112069043A (en) |
Cited By (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112712002A (en) * | 2020-12-24 | 2021-04-27 | 深圳力维智联技术有限公司 | CGAN-based environment monitoring method, device, system and storage medium |
| CN113364911A (en) * | 2021-06-11 | 2021-09-07 | 上海兴容信息技术有限公司 | Detection method and system for preset terminal |
| CN114228794A (en) * | 2021-12-17 | 2022-03-25 | 神思电子技术股份有限公司 | Automatic monitoring method and equipment for CTC scheduling |
| CN112712002B (en) * | 2020-12-24 | 2024-05-14 | 深圳力维智联技术有限公司 | CGAN-based environment monitoring method, CGAN-based environment monitoring device, CGAN-based environment monitoring system and storage medium |
-
2020
- 2020-08-04 CN CN202010774833.9A patent/CN112069043A/en active Pending
Cited By (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112712002A (en) * | 2020-12-24 | 2021-04-27 | 深圳力维智联技术有限公司 | CGAN-based environment monitoring method, device, system and storage medium |
| CN112712002B (en) * | 2020-12-24 | 2024-05-14 | 深圳力维智联技术有限公司 | CGAN-based environment monitoring method, CGAN-based environment monitoring device, CGAN-based environment monitoring system and storage medium |
| CN113364911A (en) * | 2021-06-11 | 2021-09-07 | 上海兴容信息技术有限公司 | Detection method and system for preset terminal |
| CN113364911B (en) * | 2021-06-11 | 2023-03-07 | 上海兴容信息技术有限公司 | Detection method and system for preset terminal |
| CN114228794A (en) * | 2021-12-17 | 2022-03-25 | 神思电子技术股份有限公司 | Automatic monitoring method and equipment for CTC scheduling |
| CN114228794B (en) * | 2021-12-17 | 2023-09-22 | 神思电子技术股份有限公司 | Automatic monitoring method and equipment for CTC scheduling |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US11620894B2 (en) | Surveillance system and method for predicting patient falls using motion feature patterns | |
| US9355308B2 (en) | Auditing video analytics through essence generation | |
| US20170155877A1 (en) | System and method for predicting patient falls | |
| US20140247994A1 (en) | Human activity determination from video | |
| CN112669316B (en) | Power production abnormality monitoring method, device, computer equipment and storage medium | |
| WO2013014987A1 (en) | Information identification method, program and system | |
| WO2012038241A1 (en) | Activity determination as function of transaction log | |
| KR102602689B1 (en) | Methods and system for vision-based automatic fault notification and classification of system lighting | |
| CN112069043A (en) | Terminal equipment state detection method, model generation method and device | |
| KR102260123B1 (en) | Apparatus for Sensing Event on Region of Interest and Driving Method Thereof | |
| CN111931862B (en) | Method and system for detecting illegal posted advertisements and electronic equipment | |
| CN106709521A (en) | Fire pre-warning method and fire pre-warning system based on convolution neural network and dynamic tracking | |
| CN111045889A (en) | Closed network equipment state monitoring system, method and device and readable storage medium | |
| US11295167B2 (en) | Automated image curation for machine learning deployments | |
| CN111931719A (en) | High-altitude parabolic detection method and device | |
| CN110716803A (en) | Computer system, resource allocation method and image identification method thereof | |
| EP4105893A1 (en) | Dynamic artifical intelligence camera model update | |
| CN114756455A (en) | Business abnormity positioning method and device, electronic equipment and storage medium | |
| Shuoyan et al. | Abnormal behavior detection based on the motion-changed rules | |
| CN114205667B (en) | Broadcast television broadcast abnormal picture identification and fault analysis system | |
| US11461989B2 (en) | Monitor method and monitor system thereof wherein mask is used to cover image for detecting object | |
| US20240144794A1 (en) | Surveillance system and surveillance device | |
| US20240064353A1 (en) | Media monitoring method, apparatus and system | |
| US20240144683A1 (en) | Systems and methods for video-based fraud detection | |
| US20220343743A1 (en) | Display control apparatus, display control method, and program |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |