CN115759953A

CN115759953A - Visual management system of mechanical equipment

Info

Publication number: CN115759953A
Application number: CN202211259778.5A
Authority: CN
Inventors: 张凯元
Original assignee: Shaanxi Junkai Electronic Technology Co ltd
Current assignee: Shaanxi Junkai Electronic Technology Co ltd
Priority date: 2022-07-14
Filing date: 2022-10-14
Publication date: 2023-03-07

Abstract

The application discloses visual management system of mechanical equipment, it includes administrator registration module, member registration module, authority management module, mechanical equipment information entry module, policeman system interface module, operation worksheet module, orientation module, energy consumption module, car loss statistics module, operation area management module, position monitoring module and operation result module for realize the visual management to mechanical equipment.

Description

Visual management system of mechanical equipment

Technical Field

The present application relates to the field of mechanical equipment management, and more particularly, to a visual management system for mechanical equipment.

Background

With the development and progress of scientific technology, more and more work machines and equipment come up, including but not limited to, sprinkler, fog gun, excavator, loader, bulldozer, crane, crawler or fixed work machine.

However, management of the above-described work machine equipment is a significant technical challenge. With the development of computer technology, the current management mode is changed from human monitoring to unmanned monitoring by using scientific technology, so that not only is the manpower saved, but also the material resources and the financial resources are saved. However, some management functions are difficult to be implemented in the technical aspect and the management effect of other management functions is not good due to the particularity of the management requirements of the work machine equipment.

Therefore, an optimized management system for a work machine device is emphasized.

Disclosure of Invention

The present application is proposed to solve the above-mentioned technical problems. The embodiment of the application provides a visual management system, a visual management system and an electronic device of mechanical equipment, and the visual management system, the visual management system and the visual management system comprise an administrator registration module, a member registration module, an authority management module, a mechanical equipment information input module, a public security system interface module, an operation work order module, a positioning module, an energy consumption module, a vehicle loss counting module, a vehicle expense calculation module, an operation area management module, a position monitoring module and an operation result module, and are used for realizing visual management of the mechanical equipment.

According to an aspect of the present application, there is provided a visualization management system of a mechanical device, comprising:

the administrator registration module is used for registering administrator information and managing the administrator information;

the member registration module is used for registering member information and managing the member information;

the authority management module is used for carrying out authority management;

the mechanical equipment information input module is used for adding information of mechanical equipment, wherein the information of the mechanical equipment comprises the type, the model, the license plate number, the code and a vehicle related certificate image of the mechanical equipment;

the public security system interface module is used for butting the public security system and verifying the authenticity of the uploaded certificate image and the face identification data;

the operation work order module is used for managing the outgoing operation work orders;

the positioning module is used for managing positioning data of the mechanical equipment;

the energy consumption module is used for calculating the actual energy consumption of the mechanical equipment;

the vehicle loss statistical module is used for managing vehicle loss cost of the mechanical equipment, wherein the vehicle loss cost comprises daily maintenance cost and maintenance cost;

the operation area management module is used for managing the operation area of the mechanical equipment through a circled map of an electronic map or an input activity range or a forbidden activity range;

the position monitoring module is used for acquiring the real-time track of the mechanical equipment and monitoring the position of the mechanical equipment;

the operation result module is used for managing the operation result of the mechanical equipment; and

and the vehicle cost calculation module is used for calculating the operation cost of the mechanical vehicle.

In the visual management system for a machine, the work area management module includes:

the operation monitoring image acquisition unit is used for acquiring an operation image of the mechanical equipment in an operation area through a monitoring camera deployed in the operation area;

the vehicle detection unit is used for enabling the operation image to pass through a first convolution neural network model serving as a vehicle detection network so as to obtain a vehicle characteristic map;

the working area boundary line detection unit is used for enabling the working image to pass through a second convolutional neural network model serving as a boundary line detection network so as to obtain a working area characteristic diagram;

a data dense cluster modification weight calculation unit configured to calculate a self-attention-based data dense cluster modification weight between the vehicle feature map and the work area feature map, wherein the self-attention-based data dense cluster modification weight is generated based on a probability value obtained by a classifier of the work area feature map and a probability value obtained by the classifier of an attention feature map obtained by dividing an interaction feature map obtained by multiplying the work area feature map and the vehicle feature map by a position point by a distance between the work area feature map and the vehicle feature map;

the characteristic distribution correction unit is used for weighting the working area characteristic diagram by taking the self-attention-based data dense cluster modification weight as a weight so as to obtain a weighted working area characteristic diagram;

the classification feature map construction unit is used for calculating a difference feature map between the vehicle feature map and the weighted operation region feature map as a classification feature map; and

and the classification result unit is used for enabling the classification characteristic graph to pass through the classifier to obtain a classification result, and the classification result is used for indicating whether the mechanical equipment exceeds a preset operation area or not.

In the visual management system of the mechanical equipment, the first convolutional neural network model and the second convolutional neural network model are target detection networks based on an anchor window, and the target detection networks based on the anchor window comprise Fast R-CNN, fast R-CNN and RetinaNet.

In the visualization management system for a mechanical apparatus described above, the data dense cluster modification weight calculation unit is configured to calculate the self-attention-based data dense cluster modification weight between the vehicle feature map and the work area feature map in the following formula;

wherein the formula is:

wherein softmax ((-)) represents the probability value obtained by the classifier of the feature map, F ₁ Representing said vehicle characteristic map, F ₂ A characteristic diagram showing the working area, d (F) ₁ ，F ₂ ) Indicating the distance between the vehicle characteristic map and the working area characteristic map, as |, indicates dot-by-dot, exp (-) indicates the exponential operation of the characteristic map, which indicates the calculation of natural exponential function values raised to the powers of characteristic values at respective positions in the matrix.

In the visual management system of the above-mentioned mechanical equipment, d (F) ₁ ，F ₂ ) And a Euclidean distance between the vehicle characteristic diagram and the working area characteristic diagram is represented.

In the visual management system for the mechanical equipment, the classification feature map construction unit is configured to calculate an absolute value of a difference in position between the vehicle feature map and the weighted work area feature map to obtain the difference feature map.

In the visual management system for mechanical equipment, the classification result unit includes:

a full-concatenation coding subunit, configured to perform full-concatenation coding on the classification feature map using a plurality of full-concatenation layers of the classifier to convert the classification feature map into a classification feature vector;

a probability value calculation unit, configured to input the classification feature vector into a Softmax classification function of the classifier to obtain a probability value that the classification feature vector belongs to each classification tag, where the classification tag includes that the mechanical device exceeds a predetermined operation area, and the mechanical device does not exceed the predetermined operation area; and

a classification result determination unit for determining the classification result based on the probability value.

In the visual management system for the mechanical equipment, the mechanical equipment comprises a sprinkler, a fog gun truck, an excavator, a loader, a bulldozer and a crane.

In the visual management system for the mechanical equipment, the operation area management module further includes an abnormal state early warning unit, configured to generate a warning prompt in response to that the classification result indicates that the mechanical equipment exceeds a predetermined operation area.

In the visual management system for a mechanical device, the warning indication is whether or not a new work area is added.

According to another aspect of the present application, there is provided a visual management method of a mechanical device, the method including:

collecting a working image of the mechanical equipment in a working area through a monitoring camera deployed in the working area;

enabling the operation image to pass through a first convolution neural network model serving as a vehicle detection network to obtain a vehicle characteristic map;

enabling the operation image to pass through a second convolution neural network model serving as a boundary line detection network to obtain an operation area characteristic diagram;

calculating a self-attention-based data dense cluster modification weight between the vehicle feature map and the work area feature map, wherein the self-attention-based data dense cluster modification weight is generated based on a probability value obtained by a classifier of the work area feature map and a probability value obtained by the classifier of an attention feature map obtained by multiplying an interaction feature map obtained by multiplying the position points of the work area feature map and the vehicle feature map by the distance between the work area feature map and the vehicle feature map;

weighting the working area characteristic diagram by taking the self-attention-based data intensive cluster correction weight as a weight to obtain a weighted working area characteristic diagram;

calculating a difference characteristic diagram between the vehicle characteristic diagram and the weighted operation area characteristic diagram as a classification characteristic diagram; and

and passing the classification characteristic diagram through the classifier to obtain a classification result, wherein the classification result is used for indicating whether the mechanical equipment exceeds a preset operation area or not.

According to still another aspect of the present application, there is provided an electronic apparatus including: a processor; and a memory in which are stored computer program instructions which, when executed by the processor, cause the processor to carry out the visual management method of a mechanical device as described above.

According to yet another aspect of the present application, there is provided a computer readable medium having stored thereon computer program instructions which, when executed by a processor, cause the processor to perform the method of visual management of a mechanical device as described above.

Compared with the prior art, the visual management system of the mechanical equipment comprises an administrator registration module, a member registration module, an authority management module, a mechanical equipment information entry module, a public security system interface module, an operation work order module, a positioning module, an energy consumption module, a vehicle loss statistical module, a vehicle cost calculation module, an operation area management module, a position monitoring module and an operation result module, and is used for realizing visual management of the mechanical equipment.

Drawings

The above and other objects, features and advantages of the present application will become more apparent by describing in more detail embodiments of the present application with reference to the attached drawings. The accompanying drawings are included to provide a further understanding of the embodiments of the application and are incorporated in and constitute a part of this specification, illustrate embodiments of the application and together with the description serve to explain the principles of the application. In the drawings, like reference numbers generally indicate like parts or steps.

Fig. 1 illustrates an application scenario of a visualization management system of a mechanical device according to an embodiment of the present application.

FIG. 2 illustrates a block diagram of a visualization management system of a mechanical device, according to an embodiment of the present application;

FIG. 3 illustrates a block diagram of a work area management module in a visualization management system for a machine according to an embodiment of the present disclosure;

FIG. 4 illustrates a flow chart for determining whether a machine exceeds a predetermined work area in accordance with an embodiment of the present disclosure;

FIG. 5 illustrates a flow chart of a method of visual management of a mechanical device according to an embodiment of the present application;

FIG. 6 illustrates a block diagram of an electronic device in accordance with an embodiment of the present application.

Detailed Description

Hereinafter, example embodiments according to the present application will be described in detail with reference to the accompanying drawings. It should be understood that the described embodiments are only some embodiments of the present application and not all embodiments of the present application, and that the present application is not limited by the example embodiments described herein.

Application overview:

according to the technical scheme, the visual management system of the mechanical equipment comprises a plurality of functional modules to realize visual management of the mechanical equipment. Specifically, the functional modules of the visual management system include, but are not limited to, an administrator registration module, a member registration module, an authority management module, a mechanical device information entry module, a public security system interface module, an operation work order module, a positioning module, an energy consumption module, a vehicle loss statistics module, a vehicle cost calculation module, an operation area management module, a position monitoring module and an operation result module. Of course, it is worth mentioning that, in the technical solution of the present application, an adaptive function module may be developed based on a specific application scenario requirement.

Among the above functional modules, the work area management module is a functional module that is difficult to implement on the technical side. Specifically, in the visual management system for the mechanical equipment, the working area management module is used for judging whether the working area of the mechanical equipment exceeds a predetermined range, wherein the predetermined range can be an electronic map circle area or a manually input allowable moving range or forbidden moving range. And judging whether the working area of the mechanical equipment exceeds a preset range or not by combining GPS positioning.

In recent years, deep learning and neural networks have been widely used in the fields of computer vision, natural language processing, text signal processing, and the like. In addition, deep learning and neural networks also exhibit a level close to or even exceeding that of humans in the fields of image classification, object detection, semantic segmentation, text translation, and the like. The deep learning and the development of the neural network provide new solution ideas and solutions for the operation area management of mechanical equipment.

Specifically, the present inventors considered that determining whether a mechanical device exceeds a predetermined working area is essentially a line press detection problem for a mechanical device, that is, determining whether a mechanical device exceeds a boundary line of a working area, which can be implemented by a convolutional neural network model.

Correspondingly, in the technical scheme of the application, firstly, a monitoring camera deployed in a working area is used for collecting a working image of the mechanical equipment in the working area. Then, extracting the boundary line object of the mechanical equipment object and the working area in the working image through an object detection network to obtain a vehicle feature map and a working area feature map, and particularly, in the technical scheme of the application, using an object detection network based on an anchor window to realize the functional mechanism, wherein the object detection network based on the anchor window can be Fast R-CNN, retinaNet and the like.

Further, the detection of the pressing line of the mechanical device may be represented in a high-dimensional feature space as whether there is an intersection between feature distributions of the vehicle feature map and the working area feature map in the high-dimensional feature space, and the intersection is further classified with a classifier to determine whether the mechanical device exceeds a predetermined working area. In the technical solution of the present application, an intersection between feature distributions of the vehicle feature map and the work area feature map in a high-dimensional feature space is represented by a differential feature map between the vehicle feature map and the work area feature map.

However, since the work area feature map substantially extracts local features of the work area of the work image, which may correspond to a reference window operation on the vehicle feature map and have a data-intensive feature with respect to the vehicle feature map, if the differential feature maps of the vehicle feature map and the work area feature map are directly calculated, a shift of the class probability value may be caused.

Based on this, a vehicle characteristic map F is calculated ₁ And working area feature map F ₂ Self-attention based data-dense cluster modification weights in between, expressed as:

wherein softmax (·) represents the probability value obtained by the classifier through the feature map.

Here, the self-attention-based data-dense cluster modification may enable grid-feature-based spatial interaction of the reference window of the feature map with the feature map as a whole, thereby calculating the similarity between data-dense object instances through a measure of data dissimilarity. Thus, the weight is used to match the working area feature map F ₂ After weighting, calculating the differential feature map of the vehicle feature map and the work area feature map, and improving parameter adaptive variability of the differential feature map to a classification target function by determining adaptive dependence of the data dense cluster, so that classification accuracy is improved.

Based on this, the present application proposes a visual management system of a mechanical device, which includes but is not limited to: the system comprises an administrator registration module, a member registration module, a permission management module, a mechanical equipment information input module, a public security system interface module, an operation work order module, a positioning module, an energy consumption module, a vehicle loss statistics module, an operation area management module, a position monitoring module and an operation result module, and is used for realizing visual management of mechanical equipment. The operation area management module is used for coding a work image of the mechanical equipment in an operation area through a first convolution neural network model serving as a vehicle detection network to obtain a vehicle characteristic diagram, coding the work image through a second convolution neural network model serving as a boundary line detection network to obtain an operation area characteristic diagram, then calculating a self-attention-based data dense cluster correction weight between the vehicle characteristic diagram and the operation area characteristic diagram, weighting the operation area characteristic diagram as a weight to obtain a weighted operation area characteristic diagram, then using a difference characteristic diagram between the vehicle characteristic diagram and the weighted operation area characteristic diagram as a classification characteristic diagram, and classifying through a classifier to obtain a classification result for representing whether the mechanical equipment exceeds a preset operation area.

The visual management system of the mechanical equipment provided by the application realizes visual management of the mechanical equipment in a form of software or software plus hardware, is distributed to client groups such as enterprises, careers, individual organizations or individuals in a software service platform (SaaS) mode, can distribute functions in a user-defined mode according to needs, and runs the visual management system through a mobile phone APP, a small program, a public number or a computer. Taking management of a working area of the mechanical device as an example, as shown in fig. 1, in an application scenario of the present application, first, a monitoring camera (e.g., T in fig. 1) deployed in the working area collects a working image of the mechanical device (not shown in the figure) in the working area (e.g., Q in fig. 1), and then inputs the working image into a server (e.g., S in fig. 1) deployed with a visualization management algorithm of the mechanical device, where the server can process the collected working image with the visualization management algorithm of the mechanical device to output a classification result indicating whether the mechanical device exceeds a predetermined working area.

Having described the general principles of the present application, various non-limiting embodiments of the present application will now be described with reference to the accompanying drawings.

Exemplary System

Fig. 2 illustrates a block diagram of a visualization management system of a mechanical device according to an embodiment of the present application.

As shown in fig. 2, a visualization management system 100 for a mechanical device according to an embodiment of the present application includes: an administrator registration module 101 configured to register administrator information and manage the administrator information; a member registration module 102, configured to register member information and manage the member information; the authority management module 103 is used for carrying out authority management; the mechanical equipment information entry module 104 is used for adding information of mechanical equipment, wherein the information of the mechanical equipment comprises the type, the model, the license plate number, the code and a vehicle-related certificate image of the mechanical equipment; a public security system interface module 105 for interfacing with a public security system and verifying authenticity of the uploaded certificate image and face recognition data; a job order module 106 for managing the outgoing job order; a positioning module 107, configured to manage positioning data of the mechanical device; an energy consumption module 108 for calculating an actual energy consumption of the mechanical device; the vehicle loss statistical module 109 is configured to manage vehicle loss costs of the mechanical equipment, where the vehicle loss costs include daily maintenance costs and maintenance costs; the working area management module 110 is used for managing the working area of the mechanical equipment through a circled map of an electronic map or an input activity range or a forbidden activity range; a position monitoring module 111, configured to obtain a real-time track of the mechanical device and monitor a position of the mechanical device; a work result module 112, configured to manage a work result of the mechanical device; and a vehicle cost calculation module 113 for calculating the operation cost of the machinery.

In the present embodiment, the covered or described working devices, mechanical devices, and the like include but are not limited to: the equipment such as a sprinkler, a fog gun truck, an excavator, a loader, a bulldozer, and a crane, or the equipment such as a crawler, a fixed work machine, and the like, is not limited thereto.

The administrator registration module 101 is configured to register administrator information and manage the administrator information. That is to say, the administrator registration module 101 may be configured to register administrator information, such as an administrator account, for example, upload an id card photo and collect administrator information through face recognition, and select to register an individual user or an administrator such as an organization user by a user, and when selecting to register an organization user, it is necessary to establish organization information, such as information of an organization name, an organization type, an organization industry, an organization introduction, an organization number, and upload a license such as a business license. The administrator registration module 101 manages the administrator information, including but not limited to adding, changing, deleting, etc. the administrator information.

The member registration module 102 is configured to register member information and manage the member information. In the present embodiment, the member includes, but is not limited to, a worker, a driver, and the like operating the machine. The membership information can be collected by uploading a membership card photo (related certificate photos such as a driving qualification card photo or a driving card photo are also uploaded by an operator or a driver), face recognition authentication and the like. The member registration module 102 manages the member information, including but not limited to adding, changing, or deleting the member information.

The right management module 103 is used for performing right management. Specifically, the authority management module 103 may be configured to assign the authority of the member, where the authority includes, but is not limited to, operations authority of turning on, turning off, viewing, exporting, importing, modifying, adding, or deleting functions. Furthermore, the authority management module 103 can set multi-level management authorities, can perform hierarchical management according to enterprise, individual organization and personal set architectures, and can open corresponding authorities for setting, checking, modifying and the like according to post responsibilities at each level, thereby facilitating management and protecting privacy security.

The mechanical device information entry module 104 is configured to add information of a mechanical device, where the information of the mechanical device includes, but is not limited to, a type, a model, a license plate number, a code, and a vehicle-related certificate image of the mechanical device. In this embodiment, the type of mechanical equipment includes, but is not limited to, sprinklers, fog guns, excavators, loaders, dozers, cranes, and the like.

The public security system interface module 105 is used to interface with the public security system and verify the authenticity of the uploaded document image and face recognition data. That is, to prevent others from impersonating, replacing, etc., the police and system interface module 105 may interface with the police system and verify the authenticity of the uploaded credential image (e.g., driver's license photograph, etc.) and the face recognition data.

The job order module 106 is configured to manage outgoing job orders. Specifically, when the machine equipment needs to perform a job or an outgoing job, the job or outgoing job order application is submitted by the job order module 106, and after the approval by an administrator or the like, the machine equipment starts the outgoing job. The contents of the work order include, but are not limited to, the contents of an operator who goes out (needs to upload a relevant certificate), the date, the time, the departure place, the income of the work, the destination, the work duration, the type of the selected work machinery (a sprinkler, a fog gun truck, an excavator, a loader, a bulldozer, a crane and the like), the mode of going to the work place (hauling, driving, if consignment is selected, consignment cost needs to be filled, and if driving is selected, the system automatically calculates the self-driving cost according to the driving track, the actual oil consumption and the vehicle damage cost), and the like.

The positioning module 107 is configured to manage positioning data of the mechanical device. Specifically, the positioning module 107 can acquire information such as a real-time position, a driving path, a driving track, a driving mileage, and time of the mechanical device through technologies such as mobile phone APP acquisition mobile phone GPS positioning, beidou satellite positioning, cellular data positioning, and WiFi positioning. Alternatively, the positioning module 107 may also obtain information such as a real-time position, a driving path, a driving track, a driving mileage, and a time of the mechanical device through a vehicle-owned terminal device or an additional terminal device (e.g., an OBD device, a video capture device, an audio capture device, a face recognition device, an electronic oil-floating device, etc.). It should be noted that the mobile phone APP and the mechanical device own terminal device or the additional terminal device (for example, an OBD device, a video capture device, an audio capture device, a face recognition device, an electronic oil-floating device, etc.) may be used simultaneously, and the mobile phone APP or the mechanical device own terminal device or the additional terminal device may also be used alone to realize the positioning of the mechanical device.

The energy consumption module 108 is configured to calculate an actual energy consumption of the mechanical device. Specifically, the energy consumption module 108 may use a self-built mechanical device information database or a docking third-party interface to call energy consumption data such as fuel oil amount, fuel gas amount, power consumption (new energy mechanical device) of an outgoing mechanical device according to a vehicle type (type, brand, model, etc.), and self-built or docking data such as the third-party today's oil price, today's gas price, charging price, etc., and automatically calculate the actual energy consumption cost of the outgoing mechanical device according to a map travel track distance and an operation duration system. Or, the energy consumption module 108 may also obtain dynamic and static basic vehicle information of the mechanical device, such as a current oil amount, a throttle opening, an oil injection amount, an engine speed, and the like, through a terminal device owned by the mechanical device or an additional terminal device (for example, an OBD device, a video acquisition device, an audio acquisition device, a face recognition device, an electronic oil float, and the like) to calculate the actual energy consumption more accurately.

The vehicle loss statistical module 109 is used for managing the vehicle loss cost of the mechanical equipment, wherein the vehicle loss cost comprises daily maintenance cost and maintenance cost. Specifically, the vehicle loss statistics module 109 may invoke the average vehicle loss cost per kilometer and the average vehicle loss cost per hour of the outgoing mechanical equipment according to the type of the operation vehicle, the equipment (type, brand, model, etc.) through self-building or docking a third party interface, where the vehicle loss includes but is not limited to: daily maintenance costs, etc.

The working area management module 110 is configured to manage the working area of the mechanical device through a circled map of an electronic map or an input activity range or an activity-prohibited range. Specifically, the operating area management module 110 may set an electronic fence, set an allowable activity range or a prohibited activity range through an electronic map circled area or a manual input area, send a warning prompt when the position of the mechanical device is beyond the operating area, and determine an illegal state or temporarily increase a new operating area, where the warning prompt may include whether to add the new operating area, and if the new operating area is not successfully added or not responded after being prompted by voice or the like, determine that the mechanical device is in the illegal state, and if the new operating area is successfully added after being prompted by voice or the like, determine that the new operating area is temporarily increased.

If the mechanical equipment is judged to be in the violation state, the visual management system can push early warning messages to a management responsible person set by the system through modes of pushing messages by the APP, sending short messages, making voice calls, voice prompts and the like (a user can set the messages at the background), information such as the position, the track, the operation state and the like of the mechanical equipment can be checked by the APP at the management end, when the operation is finished, the violation behaviors are marked by red characters in a generated report, the violation behaviors are recorded once for the operation personnel, the bad records are bound in a system database along with identity numbers, and the illegal records cannot be modified without special permission. If the new operation area is judged to be temporarily added, the visual management system can push early warning messages to a management responsible person set by the system through modes of pushing messages by the APP, sending short message prompts and the like (a user can set the information at the background), the APP at the management end can check the position, the track, the operation state and the like of the mechanical equipment, meanwhile, the system automatically generates and submits an application for adding the new operation area, if the application passes, the newly added operation area is marked by yellow characters when a vehicle-using report is generated, and if the application does not pass, the system automatically executes according to illegal behaviors.

The position monitoring module 111 is configured to obtain a real-time track of the mechanical device and monitor a position of the mechanical device. Specifically, the position monitoring module 111 may obtain functions such as a map, a position, and a track through a self-built or docked third-party electronic map and a navigation system (e.g., hectometre, gold, etc.) interface, and when the mechanical device is going out in a driving manner, obtain information such as a violation shooting point, traffic data, a real-time position of the mechanical device, a driving direction, and a current vehicle speed of the map, and record information such as an actual driving track, an actual driving mileage, a rapid acceleration, a rapid deceleration, a rapid turning, and an overspeed of the mechanical device going out.

Further, if the mechanical device belongs to a company and is equipped with a vehicle-owned terminal device or an additional terminal device (e.g., an OBD device, a video acquisition device, an audio acquisition device, a face recognition device, an electronic oil-floating device, etc.), an administrator or an account number assigned with a related authority may view the position and trajectory of the mechanical device in real time through a mobile phone APP or a computer. If the mechanical equipment belongs to a private company or other companies, an administrator or an account number assigned with related authority can view the real-time position and track of the mechanical equipment during outgoing operation (submitted for vehicle use) through a mobile phone APP or a computer.

And a working result module 112, configured to manage a working result of the mechanical device. That is, after the mechanical equipment outing operation is finished, the operation result module 112 may be configured to generate a graph such as a driving trace graph and an energy consumption graph.

And the vehicle cost calculation module 113 is used for generating the operation cost detail list through system analysis and calculation according to the recorded actual mileage, the operation time and energy consumption, the vehicle loss per kilometer, the vehicle loss per hour and the like. The vehicle fee calculating module 113 subtracts the fee of the job and the input labor cost of the job (or adds the labor cost of each worker going out by an administrator or an authority account number to facilitate accurate calculation of profits of the system) from the income of the job, calculates and generates a profit chart of the job, and generates a sub-job report after the system unifies and summarizes the chart. The report can be exported, stored and printed.

Therefore, the functional modules in the visual management system, such as the administrator registration module, the member registration module, the authority management module, the mechanical equipment information entry module, the public security system interface module, the operation work order module, the positioning module, the energy consumption module, the vehicle loss statistics module, the operation area management module, the position monitoring module, the operation result module and the like, are integrally used for realizing the visual management of the mechanical equipment.

In order to facilitate the user such as the administrator or the member to view or manage various items of information in the visual management system, such as one or more of the above administrator information, member information, information of the mechanical device, a work order, positioning data of the mechanical device, actual energy consumption, vehicle damage cost, prohibited activity range in a map, real-time trajectory of the mechanical device, or work result. In an optional embodiment, the visualization management system may further include an interaction module, and the interaction module may be configured to interact with a user to facilitate the user to view or manage various items of information in the visualization management system. Specifically, the interaction module may include a communication module and an interaction interface, and the communication module may be configured to perform network communication with a user terminal or a cloud terminal, such as a cloud server, to transmit information, so that a user may view or manage the above information in real time through a network. The interactive interface, such as a touch screen, may be used to display at least one or more of the above information, and allow a user to perform information interaction operations, such as deleting, modifying, uploading or downloading information, through the touch screen. Further, the visualization management system may further include a memory configured to store the history information, so that a user can view the history information conveniently, or the history information may be transmitted to the cloud for storage through the communication module. Further, the visual management system can receive videos or images collected by a cab camera of the mechanical equipment through the communication module, the videos or images can be used for representing whether the driver has illegal operation or not, the videos or images can be displayed through an interactive interface or transmitted to a user terminal through a network, and therefore the user can conveniently check whether the driver has illegal operation or not in real time.

Further, among the above functional modules, the work area management module 110 is a functional module that is difficult to be realized by a technical end. Specifically, in the visualization management system 100 for a mechanical device, the working area management module 110 is configured to determine whether a working area of the mechanical device exceeds a predetermined range, where the predetermined range may be an electronically delineated area or a manually input allowable range or prohibited range of movement. The traditional technical means is that whether the working area of the mechanical equipment exceeds a preset range is judged through GPS positioning, but the precision of the GPS positioning is low, so that the precision position of the mechanical equipment on a map is difficult to accurately judge, and the judgment accuracy of whether the working area of the mechanical equipment exceeds the preset range is low.

To this end, the inventors consider that determining whether a machine exceeds a predetermined working area is essentially a line press detection problem for a machine, i.e., determining whether a machine exceeds a boundary line of a working area, which can be implemented by a convolutional neural network model.

Specifically, as shown in fig. 3 and 4, the work area management module 110 includes: a working monitoring image obtaining unit 121, configured to collect a working image of the mechanical device in a working area through a monitoring camera deployed in the working area; a vehicle detection unit 122, configured to pass the working image through a first convolutional neural network model as a vehicle detection network to obtain a vehicle feature map; a working area boundary line detection unit 123, configured to pass the working image through a second convolutional neural network model as a boundary line detection network to obtain a working area feature map; a data dense cluster modification weight calculation unit 124 for calculating a self-attention-based data dense cluster modification weight between the vehicle feature map and the work area feature map; a feature distribution correction unit 125 configured to weight the work area feature map with the self-attention-based data dense cluster modification weight as a weight to obtain a weighted work area feature map; a classification feature map construction unit 126, configured to calculate a difference feature map between the vehicle feature map and the weighted work area feature map as a classification feature map; and a classification result unit 127, configured to pass the classification feature map through the classifier to obtain a classification result, where the classification result is used to indicate whether the mechanical equipment exceeds a predetermined working area.

The operation monitoring image acquiring unit 121 is configured to acquire an operation image of the mechanical device in an operation area through a monitoring camera deployed in the operation area. For example, the monitoring camera may be deployed at a fixed position above the working area, and the like, and used for overhead shooting (such as 45 ° downward shooting or vertical downward shooting) to obtain a working image covering or covering the working area of the mechanical equipment, and the like.

The vehicle detection unit 122 is configured to pass the working image through a first convolutional neural network model as a vehicle detection network to obtain a vehicle feature map. The working area boundary line detection unit 123 is configured to pass the working image through a second convolutional neural network model as a boundary line detection network to obtain a working area feature map. That is, in the present application, the mechanical device object and the boundary line object of the work area in the work image are extracted through the object detection network to obtain the vehicle feature map and the work area feature map. Specifically, in order to improve the accuracy of feature distribution of a vehicle feature map and a working area feature map extracted from a working image in a high-dimensional feature space, the first convolutional neural network model and the second convolutional neural network model are both anchor window-based target detection networks, and the anchor window-based target detection networks include, but are not limited to, fast R-CNN, retinaNet and the like.

Specifically, the input data of the first layer of the first convolutional neural network is the working image, and each layer of the first convolutional neural network respectively performs convolution processing, pooling processing along channel dimension and activation processing based on nonlinear activation on the input data in forward pass of the layer to output an activation feature map from the last layer of the first convolutional neural network, wherein the activation feature map output by the last layer of the first convolutional neural network is a vehicle feature map used for representing mechanical equipment. Accordingly, the input data of the first layer of the second convolutional neural network is the working image, and each layer of the second convolutional neural network respectively performs convolution processing, pooling processing along channel dimension and activation processing based on nonlinear activation on the input data in forward pass of the layer to output an activation feature map from the last layer of the second convolutional neural network, wherein the activation feature map output from the last layer of the second convolutional neural network is a working area feature map used for representing a boundary line.

Further, the pressing line detection of the mechanical device may be represented in a high-dimensional feature space as whether there is an intersection between feature distributions of the vehicle feature map and the working area feature map in the high-dimensional feature space, and the intersection is further classified with a classifier to determine whether the mechanical device exceeds a predetermined working area. In the technical solution of the present application, an intersection between feature distributions of the vehicle feature map and the work area feature map in a high-dimensional feature space is represented by a differential feature map between the vehicle feature map and the work area feature map.

Therefore, in the embodiment of the present application, a self-attention-based data dense cluster modification weight between the vehicle feature map and the work area feature map is calculated, the work area feature map is weighted by using the self-attention-based data dense cluster modification weight as a weight to obtain a weighted work area feature map, then a differential feature map between the vehicle feature map and the weighted work area feature map is used as a classification feature map, and classification is performed by a classifier to obtain a classification result indicating whether the mechanical equipment exceeds a predetermined work area.

The data-intensive cluster modification weight calculation unit 124 is configured to calculate a self-attention-based data-intensive cluster modification weight between the vehicle feature map and the work area feature map. Specifically, the data dense cluster modification weight calculation unit 124 is configured to calculate the self-attention based data dense cluster modification weight between the vehicle feature map and the work area feature map in the following formula;

wherein the formula is:

wherein softmax ((-)) represents the probability value obtained by the classifier of the feature map, F ₁ Representing said vehicle characteristic map, F ₂ A characteristic diagram showing the working area, d (F) ₁ ,F ₂ ) Indicates the distance between the vehicle characteristic map and the work area characteristic map, indicates dot-by-dot, exp (-), indicates the exponential operation of the characteristic map, which indicates the calculation of the natural exponent function value raised to the power of the characteristic value at each position in the matrix. Further, said d (F) ₁ ,F ₂ ) And a Euclidean distance between the vehicle characteristic diagram and the working area characteristic diagram is represented.

Here, the self-attention-based data-dense cluster modification may enable grid-feature-based spatial interaction of the reference window of the feature map with the feature map as a whole, thereby calculating the similarity between data-dense object instances through a measure of data dissimilarity. Thus, the weight is used to match the working area characteristic diagram F ₂ After weighting, calculating the differential characteristic diagram of the vehicle characteristic diagram and the working area characteristic diagram, and improving the parameter adaptive variability of the differential characteristic diagram to a classification target function by determining the adaptive dependence of the data dense cluster, thereby improving the classification accuracy.

The feature distribution correction unit 125 is configured to weight the work area feature map with the self-attention-based data dense cluster modification weight as a weight to obtain a weighted work area feature map. Specifically, the characteristic distribution correction unit 125 is configured to weight the work area characteristic map to obtain a weighted work area characteristic map according to a formula, for example, F ₃ ＝w×F ₂ In which F ₃ The weighted work area feature map is obtained.

The classification feature map construction unit 126 is configured to calculate a difference feature map between the vehicle feature map and the weighted work area feature map as a classification feature map. Specifically, the classification feature map construction unit 126 is configured to calculate an absolute value of a position difference between the vehicle feature map and the weighted work area feature map to obtain the difference feature map. That is, the classification feature map construction unit 126 calculates absolute values of the respective position differences in the vehicle feature map and the weighted work area feature map to obtain a differential feature map.

The classification result unit 127 is configured to pass the classification feature map through the classifier to obtain a classification result, where the classification result is used to indicate whether the mechanical equipment exceeds a predetermined operation area.

In this embodiment, the classification result unit 127 is configured to process the classification feature map by using the classifier according to the following formula to generate a classification result, where the formula is: softmax { (W) _n ,B _n ):…:(W ₁ ,B ₁ ) Project (F), where Project (F) represents projecting the classification feature map as a vector, W ₁ To W _n As a weight matrix for all connected layers of each layer, B ₁ To B _n A bias matrix representing the layers of the fully connected layer.

Specifically, the classification result unit 127 includes a full-concatenation coding subunit, configured to perform full-concatenation coding on the classification feature map using a plurality of full-concatenation layers of the classifier to convert the classification feature map into a classification feature vector; a probability value calculation unit, configured to input the classification feature vector into a Softmax classification function of the classifier to obtain a probability value that the classification feature vector belongs to each classification tag, where the classification tag includes that the mechanical device exceeds a predetermined operation area, and the mechanical device does not exceed the predetermined operation area; a classification result determining unit for determining the classification result based on the probability value.

The process of the classification result unit 127 passing the classification feature map through a classifier to obtain a classification result includes: first, the classification feature matrix is full-connection encoded using a plurality of full-connection layers of the classifier to obtain a classification feature vector, and then the classification feature vector is input to a Softmax classification function of the classifier to obtain a first probability that the classification feature vector belongs to the mechanical device exceeding a predetermined working area and a second probability that the mechanical device does not exceed the predetermined working area. Finally, the classification result is generated based on a comparison between the first probability and the second probability. Specifically, when the first probability is greater than the second probability, the classification result is that the mechanical equipment exceeds a predetermined working area; when the first probability is smaller than the second probability, the mechanical equipment does not exceed a preset working area according to the classification result.

The work area management module 110 further includes an abnormal state early warning unit 128, configured to generate an alarm prompt in response to the classification result being that the mechanical equipment exceeds the predetermined work area. Further, the warning is to indicate whether to add a new operation area, and whether to add a new operation area is indicated by voice or the like, which is not limited herein.

In summary, the visual management system of a mechanical device according to an embodiment of the present application is illustrated, which includes, but is not limited to, an administrator registration module, a member registration module, an authority management module, a mechanical device information entry module, a police system interface module, a work order module, a positioning module, an energy consumption module, a vehicle loss statistics module, a vehicle cost calculation module, a work area management module, a location monitoring module, and a work result module, and is configured to perform visual management on the mechanical device, wherein the work area management module is configured to encode a work image of the mechanical device in a work area through a first convolutional neural network model serving as a vehicle detection network to obtain a vehicle feature map, encode the work image through a second convolutional neural network model serving as a boundary detection network to obtain a work area feature map, calculate a self-attention-based data intensive cluster modification weight between the vehicle feature map and the work area feature map, weight the work area feature map as a weight to obtain a weighted work area feature map, and classify the work area feature map as a weight to obtain a classification result of the work area by using a classifier.

As described above, the visual management system 100 for a mechanical device according to the embodiment of the present application may be implemented in various terminal devices, for example, a server for visual management of a mechanical device. In one example, the visualization management system 100 of the mechanical device according to the embodiment of the present application may be integrated into the terminal device as a software module and/or a hardware module. For example, the visualization management system 100 of the machine device may be a software module in the operating system of the terminal device, or may be an application developed for the terminal device; of course, the visual management system 100 of the machine may also be one of many hardware modules of the terminal device.

Alternatively, in another example, the visualization management system 100 of the mechanical device and the terminal device may be separate devices, and the visualization management system 100 of the mechanical device may be connected to the terminal device through a wired and/or wireless network and transmit the interaction information according to the agreed data format.

Exemplary method

Fig. 5 illustrates a flow chart of a method for visualization management of a mechanical device in accordance with an embodiment of the present application. As shown in fig. 5, a visual management method for a mechanical device according to an embodiment of the present application includes:

s110, collecting an operation image of the mechanical equipment in an operation area through a monitoring camera deployed in the operation area;

s120, obtaining a vehicle characteristic diagram by using the operation image through a first convolution neural network model serving as a vehicle detection network;

s130, enabling the operation image to pass through a second convolution neural network model serving as a boundary line detection network to obtain an operation area characteristic diagram;

s140, calculating a self-attention-based data dense cluster correction weight between the vehicle feature map and the operation region feature map, wherein the self-attention-based data dense cluster correction weight is generated based on a probability value obtained by the operation region feature map through a classifier and a probability value obtained by the classifier of an attention feature map obtained by multiplying an interaction feature map obtained by position points of the operation region feature map and the vehicle feature map by dividing the distance between the operation region feature map and the vehicle feature map;

s150, weighting the working area characteristic diagram by taking the data intensive cluster correction weight based on the self-attention as a weight to obtain a weighted working area characteristic diagram;

s160, calculating a difference feature map between the vehicle feature map and the weighted operation area feature map to serve as a classification feature map; and

s170, enabling the classification characteristic graph to pass through the classifier to obtain a classification result, wherein the classification result is used for indicating whether the mechanical equipment exceeds a preset operation area or not.

In one example, in the visual management method for the mechanical equipment, the first convolutional neural network model and the second convolutional neural network model are anchor window-based target detection networks, and the anchor window-based target detection networks comprise Fast R-CNN, fast R-CNN and RetinaNet.

In one example, in the visualization management method for a mechanical apparatus described above, the self-attention-based data-dense cluster modification weight between the vehicle feature map and the work area feature map is calculated in the following formula;

wherein the formula is:

wherein softmax ((-)) represents the probability value obtained by the classifier of the feature map, F ₁ Representing said vehicle characteristic map, F ₂ A characteristic diagram showing the working area, d (F) ₁ ,F ₂ ) Indicates a distance between the vehicle characteristic map and the operating region characteristic map, indicates dot-by-dot, exp (-) indicates an exponential operation of the characteristic map, the exponential operation of the characteristic map indicates a calculation toThe eigenvalues of each position in the matrix are the natural exponential function values of the power.

In one example, in the visual management method of a mechanical device, the d (F) ₁ ,F ₂ ) And a Euclidean distance representing a distance between the vehicle characteristic diagram and the working area characteristic diagram.

In one example, in the visual management method for a mechanical device, the classification feature map is processed by the classifier according to the following formula to generate a classification result, where the formula is: softmax { (W) _n ,B _n ):…:(W ₁ ,B ₁ ) L Project (F) }, where Project (F) denotes the projection of the classification feature map as a vector, W ₁ To W _n As a weight matrix for all connected layers of each layer, B ₁ To B _n A bias matrix representing the layers of the fully connected layer.

Here, it may be understood by those skilled in the art that the specific functions and steps in the visualization management method of a mechanical device described above have been described in detail in the above description of the visualization management system of a mechanical device with reference to fig. 2 to 4, and thus, a repeated description thereof will be omitted.

Exemplary electronic device

Next, an electronic apparatus according to an embodiment of the present application is described with reference to fig. 6.

FIG. 6 illustrates a block diagram of an electronic device in accordance with an embodiment of the application.

As shown in fig. 6, the electronic device 10 includes one or more processors 11 and memory 12.

The processor 11 may be a Central Processing Unit (CPU) or other form of processing unit having data processing capabilities and/or instruction execution capabilities, and may control other components in the electronic device 10 to perform desired functions.

Memory 12 may include one or more computer program products that may include various forms of computer-readable storage media, such as volatile memory and/or non-volatile memory. The volatile memory may include, for example, random Access Memory (RAM), cache memory (cache), and/or the like. The non-volatile memory may include, for example, read Only Memory (ROM), hard disk, flash memory, etc. One or more computer program instructions may be stored on the computer-readable storage medium and executed by processor 11 to implement the visualization management system of the mechanical device of the various embodiments of the present application described above and/or other desired functionality. Various contents such as weights may also be stored in the computer-readable storage medium.

In one example, the electronic device 10 may further include: an input device 13 and an output device 14, which are interconnected by a bus system and/or other form of connection mechanism (not shown).

The input device 13 may include, for example, a keyboard, a mouse, and the like.

The output device 14 can output various information including classification results or warning prompts to the outside. The output devices 14 may include, for example, a display, speakers, a printer, and a communication network and its connected remote output devices, among others.

Exemplary computer program product and computer-readable storage Medium

In addition to the above-described methods and apparatus, embodiments of the present application may also be a computer program product comprising computer program instructions that, when executed by a processor, cause the processor to perform the steps in the method for visual management of a mechanical device according to various embodiments of the present application described in the "exemplary methods" section of this specification above.

The computer program product may be written with program code for performing the operations of embodiments of the present application in any combination of one or more programming languages, including an object oriented programming language such as Java, C + + or the like and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The program code may execute entirely on the user's computing device, partly on the user's device, as a stand-alone software package, partly on the user's computing device and partly on a remote computing device, or entirely on the remote computing device or server.

Furthermore, embodiments of the present application may also be a computer-readable storage medium having stored thereon computer program instructions that, when executed by a processor, cause the processor to perform the steps in the method for visual management of a mechanical device according to various embodiments of the present application described in the "exemplary methods" section above of this specification.

The computer-readable storage medium may take any combination of one or more readable media. The readable medium may be a readable signal medium or a readable storage medium. A readable storage medium may include, for example, but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or a combination of any of the foregoing. More specific examples (a non-exhaustive list) of the readable storage medium include: an electrical connection having one or more wires, a portable diskette, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

The foregoing describes the general principles of the present application in conjunction with specific embodiments, however, it is noted that the advantages, effects, etc. mentioned in the present application are merely examples and are not limiting, and they should not be considered essential to the various embodiments of the present application. Furthermore, the foregoing disclosure of specific details is for the purpose of illustration and description and is not intended to be limiting, since the foregoing disclosure is not intended to be exhaustive or to limit the disclosure to the precise details disclosed.

The block diagrams of devices, apparatuses, systems referred to in this application are only given as illustrative examples and are not intended to require or imply that the connections, arrangements, configurations, etc. must be made in the manner shown in the block diagrams. These devices, apparatuses, devices, systems may be connected, arranged, configured in any manner, as will be appreciated by one skilled in the art. Words such as "including," "comprising," "having," and the like are open-ended words that mean "including, but not limited to," and are used interchangeably therewith. The words "or" and "as used herein mean, and are used interchangeably with, the word" and/or, "unless the context clearly dictates otherwise. The word "such as" is used herein to mean, and is used interchangeably with, the phrase "such as but not limited to".

It should also be noted that in the devices, apparatuses, and methods of the present application, the components or steps may be decomposed and/or recombined. These decompositions and/or recombinations are to be considered as equivalents of the present application.

The previous description of the disclosed aspects is provided to enable any person skilled in the art to make or use the present application. Various modifications to these aspects will be readily apparent to those skilled in the art, and the generic principles defined herein may be applied to other aspects without departing from the scope of the application. Thus, the present application is not intended to be limited to the aspects shown herein but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

The foregoing description has been presented for purposes of illustration and description. Furthermore, the description is not intended to limit embodiments of the application to the form disclosed herein. While a number of example aspects and embodiments have been discussed above, those of skill in the art will recognize certain variations, modifications, alterations, additions and sub-combinations thereof.

Claims

1. A visual management system for a mechanical device, comprising:

the system comprises an administrator registration module, an administrator information management module and an administrator information management module, wherein the administrator registration module is used for registering administrator information and managing the administrator information;

the authority management module is used for carrying out authority management;

the position monitoring module is used for acquiring a real-time track of the mechanical equipment and monitoring the position of the mechanical equipment;

2. The visual management system of a mechanical device of claim 1, wherein the work area management module comprises:

the vehicle detection unit is used for enabling the operation image to pass through a first convolutional neural network model serving as a vehicle detection network so as to obtain a vehicle characteristic diagram;

3. The visual management system of a mechanical device of claim 2, wherein the first and second convolutional neural network models are anchor window-based target detection networks including Fast R-CNN, and RetinaNet.

4. The visualization management system of the machine equipment according to claim 3, wherein the data-dense cluster modification weight calculation unit is configured to calculate the self-attention-based data-dense cluster modification weight between the vehicle feature map and the work area feature map with the following formula;

wherein the formula is:

wherein softmax ((-)) represents the probability value obtained by the classifier of the feature map, F ₁ Showing the vehicle characteristic map, F ₂ A characteristic diagram showing the working area, d (F) ₁ ,F ₂ ) Indicates the distance between the vehicle characteristic map and the work area characteristic map, indicates dot-by-dot, exp (-), indicates the exponential operation of the characteristic map, which indicates the calculation of the natural exponent function value raised to the power of the characteristic value at each position in the matrix.

5. Visual management system of a mechanical device according to claim 4, wherein said d (F) ₁ ,F ₂ ) And a Euclidean distance between the vehicle characteristic diagram and the working area characteristic diagram is represented.

6. The visual management system of a mechanical device according to claim 5, wherein the classification feature map construction unit is configured to calculate an absolute value of a difference in position between the vehicle feature map and the weighted work area feature map to obtain the difference feature map.

7. The visual management system of a mechanical device of claim 6, wherein the classification result unit comprises:

the probability value calculation unit is used for inputting the classification feature vector into a Softmax classification function of the classifier to obtain a probability value of the classification feature vector belonging to each classification label, wherein the classification label comprises that mechanical equipment exceeds a preset operation area, and mechanical equipment does not exceed the preset operation area; and

8. The visual management system of mechanical equipment of claim 7, wherein the mechanical equipment comprises a sprinkler, a fog gun truck, an excavator, a loader, a dozer, a crane, a crawler, or a fixed work machine.

9. The visual management system of the mechanical equipment according to claim 8, wherein the work area management module further comprises an abnormal state early warning unit for generating an alarm prompt in response to the classification result indicating that the mechanical equipment exceeds a predetermined work area.

10. The visual management system of a mechanical device according to claim 9, wherein the alert is whether to add a new work area.