CN112667728A - Visual single-machine data acquisition method in wharf efficiency analysis - Google Patents

Visual single-machine data acquisition method in wharf efficiency analysis Download PDF

Info

Publication number
CN112667728A
CN112667728A CN202110011595.0A CN202110011595A CN112667728A CN 112667728 A CN112667728 A CN 112667728A CN 202110011595 A CN202110011595 A CN 202110011595A CN 112667728 A CN112667728 A CN 112667728A
Authority
CN
China
Prior art keywords
management
data
module
wharf
rule
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Granted
Application number
CN202110011595.0A
Other languages
Chinese (zh)
Other versions
CN112667728B (en
Inventor
梁海玉
黄清波
郜长庆
黄雷
原好
范文静
熊会
钱晶晶
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Shanghai Zhenghua Heavy Industries Co Ltd
Original Assignee
Shanghai Zhenghua Heavy Industries Co Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Shanghai Zhenghua Heavy Industries Co Ltd filed Critical Shanghai Zhenghua Heavy Industries Co Ltd
Priority to CN202110011595.0A priority Critical patent/CN112667728B/en
Publication of CN112667728A publication Critical patent/CN112667728A/en
Application granted granted Critical
Publication of CN112667728B publication Critical patent/CN112667728B/en
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Images

Landscapes

  • Management, Administration, Business Operations System, And Electronic Commerce (AREA)

Abstract

The invention discloses a visual single-machine data acquisition method in wharf efficiency analysis, which belongs to the technical field of single-machine data acquisition, and the visual single-machine data acquisition method adopts the latest frontier ETL technology to realize data acquisition of various wharf single-machine equipment; by introducing an expression engine technology, diversified data types and formats are expressed by a uniform expression; a task scheduling technology is introduced to realize acquisition task scheduling; then, an acquisition system is used for converting and processing the irregular data source into standard, clean and effective data; the visual front-end configuration interface is provided by adopting a front-end and back-end separation technology, the application of a web end is supported, a large amount of manpower and material resources are saved, the usability of the system is greatly improved, the data of the whole acquisition system is more accurate and more efficient, a friendly visual interface is provided, and the user experience is improved.

Description

Visual single-machine data acquisition method in wharf efficiency analysis
Technical Field
The invention relates to the technical field of single-machine data acquisition, in particular to a visual single-machine data acquisition method in wharf efficiency analysis.
Background
Along with the development of the container automation wharf business, a user pays more and more attention to the efficiency of wharf equipment operation, a wharf efficiency analysis system is basically a standard distribution system of a wharf, the analysis of single-machine operation efficiency is more important, the core of efficiency analysis is equipment operation data analysis, and the collection and processing of data are often the key of successful projects.
Because the services of the equipment and the wharf in the prior art have great difference, the equipment also has great difference, the wharf equipment mainly comprises a double-trolley shore bridge, a single-trolley shore bridge, a tire crane, a rail crane and the like at present, the difference of the wharf services is mainly embodied that each wharf has certain self definition on the management of the operation flow, and the complexity of acquiring the operation data of the equipment is increased by integrating the aspects, such as: how to realize the persistence of the collected data of the equipment operation; how to realize the uniform collection rule of the equipment operation data; how to realize the dispatching management of the multi-task acquisition plan; how to realize the application of the scheduling configuration of the acquisition plan at a web end; how to implement user account rights management; how to realize the technical problem of the running state monitoring of the acquisition system.
Disclosure of Invention
The invention aims to provide a visual single-machine data acquisition method in wharf efficiency analysis, which solves the technical problems in the prior art that the complexity of equipment operation data acquisition, the persistence of the equipment operation acquired data cannot be realized, the acquisition rule of the equipment operation data cannot be unified, the scheduling management of a multi-task acquisition plan cannot be realized, the application of an acquisition system at a web end, the management of user account authority, the monitoring of the running state of the acquisition system cannot be realized, and the like.
In order to achieve the purpose, the invention adopts the following technical scheme:
according to one aspect of the invention, a visual single-machine data acquisition method in wharf efficiency analysis is provided, which comprises the following steps,
step 1, uniformly extracting data from an InfluxDB time sequence database by a TBI ETL platform to realize data acquisition of various wharf single-machine equipment; adopting a ZPMC OPC technology, and pushing and persisting single-machine data into the InfluxDB through a middleware to complete single-machine data persistence;
step 2, an expression engine technology is introduced to realize that diversified data types and formats are expressed by a uniform expression; and a task scheduling technology is introduced to realize acquisition task scheduling. (ii) a
Step 3, summarizing and summarizing all the data extracted in the step 1 according to extraction requirements, defining the generation of each piece of data as a standard rule, and converting an irregular data source into standard, clean and effective data; and step 4, providing a visual front-end configuration interface by adopting a front-end and back-end separation technology, realizing the application of the scheduling configuration of the acquisition plan on a web end, and not influencing the running process of a back-end scheduling task.
According to the visual single-machine data acquisition method in the wharf efficiency analysis in the above aspect of the present invention, the TBI ETL platform in step 1 includes:
the source library management module is used for configuring an InfluxDB database address and information of equipment and a TAG point in a TBI OPD project and executing the extraction module to use; and/or
The target database management module is used for realizing the basic information configuration of a target database, namely the data extracted from the InfluxDB is processed and finally stored into tables in which database, namely the connection character string of the target database and the table information in the target database are configured; and/or
The management of the scheduling module, mainly dispose the scheduling information of different execution frequencies, for carrying out the extraction module and using, carry on the scheduling management; and/or
The extraction execution module is used for managing, so that a set of platform meets the functions of all projects; and/or
The log monitoring module is used for realizing management of log system information extraction; and/or
And the system information management module is used for maintaining the core basic information of the TBI ETL platform.
According to the visual single-machine data acquisition method in wharf efficiency analysis in the above aspect of the invention, the source library management module includes HOST management, data source management, data table management, TAG management and equipip management.
According to the visual single-machine data acquisition method in the wharf efficiency analysis, the target library management module comprises data source management of the target library, data table management of the target library and list information management of the target library.
According to the visual single-machine data acquisition method in wharf efficiency analysis in the above aspect of the invention, the extraction execution module management comprises an extraction rule configuration module; a configuration for providing a series of expressions to fulfill various acquisition requirements; and/or an extraction rule analysis module for realizing the function that the data is generated into the target database from the InfluxDB database through processing.
According to the visualized single-machine data acquisition method in the wharf efficiency analysis, the monitoring log module comprises extraction history management and/or TAG point final record management.
According to the visual single-machine data acquisition method in wharf efficiency analysis in the aspect of the invention, the system information management module comprises user management, menu management, role management and dictionary management.
According to the visualized single-machine data acquisition method in the wharf efficiency analysis in the above aspect of the invention, the step 2 includes the following steps:
s21, expressing the extraction rule in the step 1 by an expression engine based on google aviator to realize expression of the data extraction rule;
and S22, a Spring is used for integrating the Quartz, so that the development of a timing task and the realization of a scheduling module are realized, and the reliable and efficient normal execution of data extraction is realized.
According to the visualized single-machine data acquisition method in wharf efficiency analysis in the above aspect of the invention, the rule in step 3 includes
A Start Rule (Start Rule), i.e., a service data generation Rule; and/or
An initial start Rule (StartInit Rule), namely an update Rule when generating service data; and/or
Update Rule (Update Rule), i.e. service data Update Rule; and/or
An End Rule (End Rule), i.e. a service data End Rule; and/or
The initial end Rule (EndInit Rule) is an update Rule at the end of the service data.
According to the visualized single-machine data acquisition method in the wharf efficiency analysis in the above aspect of the invention, the step 4 includes the following steps:
s41, the system deploys a B/S framework to complete configuration management of the system;
s42, providing a Rest API interface at the back end to facilitate interaction of the front end and the back end;
s43, a front-end and rear-end separation technology is adopted, a friendly visual configuration interface and a friendly monitoring interface are provided, and the operation condition of the system is monitored in real time;
s44, adopting SpringSecurity + JWT authority management technology to perform refined authority management and ensure the safety of the system;
and S45, introducing Logback log management, finely controlling the application range of the log, and monitoring the running condition of the system.
By adopting the technical scheme, the invention has the following advantages:
compared with the traditional wharf efficiency analysis technology, the visual single-machine data acquisition method for wharf efficiency analysis saves a large amount of manpower and material resources, and greatly improves the usability of the system; introducing the technology of an expression engine, and regularly combing scattered data to form standardization; the data of the whole acquisition system is more accurate and efficient, a B/S framework is adopted, a front-end and back-end separation technology is realized, a friendly visual interface is provided, and user experience is improved.
Drawings
Other features, objects and advantages of the present invention will become more apparent upon reading of the following detailed description of non-limiting embodiments thereof, with reference to the accompanying drawings.
FIG. 1 is an architectural scenario screenshot of a visual stand-alone data collection of the present invention;
FIG. 2 is a data rule description screen shot of the present invention;
FIG. 3 is a visualization standalone data acquisition system deployment architecture screenshot of the present invention;
FIG. 4 is a data source management screen shot of the present invention;
FIG. 5 is a collection task scheduling management screen shot of the present invention;
FIG. 6 is a capture configuration management screen shot of the present invention;
FIG. 7 is a collection monitoring management screen shot of the present invention;
fig. 8 is a system information management screen shot of the present invention.
Detailed Description
The technical solutions of the present invention are described in detail below with reference to the accompanying drawings, and the detailed features and advantages of the present invention are described in detail in the detailed description, which is sufficient for anyone skilled in the art to understand the technical contents of the present invention and implement the present invention, and the related objects and advantages of the present invention can be easily understood by those skilled in the art from the description, the claims and the accompanying drawings disclosed in the present specification.
Examples
FIG. 1 illustrates an architectural scenario screenshot of a visual stand-alone data collection; FIG. 2 shows a screenshot of a data rule specification; the invention provides a visual single-machine data acquisition method in wharf efficiency analysis, which specifically comprises the following steps of flow,
step 1: a TBI ETL platform in a data acquisition service layer uniformly extracts data from an InfluxDB time sequence database, so that data acquisition of various wharf single-machine equipment in single-machine equipment data is realized; in an embodiment, due to the introduction of the infilxdb time sequence database (unifies the data source), all the single-machine signal historical data are unified and persisted to the infilxdb time sequence database, which ensures the security of the data, and the TBI ETL platform uniformly obtains the data from the infilxdb time sequence database, which ensures the unification of the data source formats, as shown in fig. 1, and realizes the persistence of the data.
The TBI ETL platform comprises 6 sub-modules which are respectively a source library management module, a target library management module, a scheduling module management module, an extraction execution module management module, a monitoring log module and a system information management module.
Wherein: the source library management module mainly comprises HOST management, data source management, data table management, TAG management and EQUIP management; in one embodiment, the source library management module is used for configuring the address of the InfluxDB database, and in another embodiment, the source library management module is used for information of equipment and a TAG point in TBI OPD engineering and used by the extraction module.
The target library management module comprises data source management of a target library, data table management of the target library and list information management of the target library; in one embodiment, the target library management module is mainly used for realizing basic information configuration of the target library, that is, data extracted from the infiluxdb time sequence database is processed and then finally stored in tables in which database, and connection character strings of the target database and table information in the target database are configured.
In one embodiment, the role of the scheduling module management is mainly to configure scheduling information of different execution frequencies for the execution extraction module to use for scheduling management.
The extraction execution module management mainly comprises an extraction rule configuration module and an extraction rule analysis module. The extraction rule configuration module can realize the configuration of various acquisition requirements through a series of expression methods, the extraction rule analysis module is used for analyzing the configuration rules, the platform-based expression rule analysis module can correctly complete the acquisition and processing work of data, and the function that the data is generated into a target database from an InfluxDB database through processing is realized. In one embodiment, the extraction execution module management provides extraction category management and provides extraction device management, in another embodiment, the extraction execution module management provides association device rule management, sort rule management, and base rule management functions, and in yet another embodiment, the extraction execution module management provides extraction target table management. The extraction execution module realizes the function that one set of platform meets the use of all projects.
The monitoring log module mainly realizes management of log system information extraction, including extraction history management and final record management of the TAG point.
The system information management module mainly maintains the core basic information of the system, including user management, menu management, role management and dictionary management.
In one embodiment, step 2: adopting a ZPMC OPC technology, pushing and persisting single-machine data into InfluxDB through a middleware to complete single-machine data persistence, and then acquiring the data in Influxdb into a target database through a rule to realize persistence of extracted data as shown in figure 1;
wherein: step 2 specifically includes the following embodiments;
in one embodiment, S21: the expression engine based on the google aviator is used for realizing a regular, lightweight and high-performance Java expression execution engine, expressing a data extraction rule by an expression and realizing a unified data acquisition rule;
in another embodiment, S22: the Spring is used for integrating the Quartz to easily complete timing tasks, the realization of a scheduling module can ensure that data extraction is reliable and efficient, different time scheduling settings can be realized by different extractions, the execution, pause, stop and other controls are convenient to perform, and the scheduling management of a multi-task acquisition plan is realized.
And step 3: the method includes the steps that a data generation Rule is defined in a standardized mode, all data extraction requirements are summarized, in one embodiment, generation of one data record is defined into 5 standard rules, and the rules specifically include a Start Rule (Start Rule), namely a business data generation Rule; and/or an initial start Rule (Startlnit Rule), i.e. a Rule updated at the time of generation of traffic data; and/or Update Rule (Update Rule), i.e. business data Update Rule; and/or an End Rule (End Rule), i.e. a service data End Rule; and/or initial end Rule (Endlnit Rule), i.e. update Rule at the end of business data, as shown in fig. 2 in particular, the standardization of data generation rules provides a basis for standardized development of systems.
FIG. 3 illustrates a visualization stand-alone data collection system deployment architecture screenshot; FIG. 4 illustrates a data source management screen shot; FIG. 5 illustrates an acquire task schedule management screen shot; FIG. 6 illustrates an acquisition configuration management screen shot; FIG. 7 illustrates an acquisition monitoring management screen shot; FIG. 8 shows a system information management screenshot;
and 4, step 4: and a front-end and back-end separation technology is adopted to provide a visual front-end configuration interface and realize the application of the visual single-computer data acquisition system on a web end.
The step 4 comprises the following embodiments:
in one embodiment, S41: the system deploys a B/S architecture, and configuration management of the system is completed as shown in FIG. 3;
in another embodiment, S42: the back end provides a Rest API interface, which is convenient for the interaction of the front end and the back end;
in yet another embodiment, S43: a front-end and rear-end separation technology is adopted, a friendly visual configuration interface and a monitoring interface are provided, and the operation condition of the system is monitored in real time;
in yet another embodiment, S44: by adopting the SpringSecurity + JWT right management technology, refined right management is realized, the safety of the system is ensured, and the user account right management is realized;
in the next embodiment, S45: logback log management is introduced, the application range of the log is finely controlled, the running condition of the system is monitored, and the running state monitoring of the acquisition system is realized.
As an embodiment of the invention, a set of data acquisition platforms applicable to various types of wharf projects is realized by combining implementation experience of existing KPI projects and performing repeated debugging, wherein the differences of the projects can be customized by visual configuration, and the application examples are specifically shown in fig. 4, fig. 5, fig. 6, fig. 7 and fig. 8; meanwhile, the expandability and configurability of the platform can further improve the overall service level of the wharf. At present, similar product cases are rarely found internationally, and the innovation fills a gap in the field in China.
Finally, it should be noted that while the present invention has been described with reference to the specific embodiments thereof, it should be understood by those skilled in the art that the above embodiments are only for illustrating the present invention and are not to be construed as limiting the present invention, and various equivalent changes and substitutions may be made therein without departing from the spirit of the present invention, and therefore, it is intended that all changes and modifications to the above embodiments within the spirit and scope of the present invention be covered by the appended claims.

Claims (9)

1. A visual single-machine data acquisition method in wharf efficiency analysis is characterized by comprising the following steps:
step 1, uniformly extracting data from an InfluxDB time sequence database by a TBI ETL platform to realize data acquisition of various wharf single-machine equipment;
step 2, an expression engine technology is introduced to realize that diversified data types and formats are expressed by a uniform expression; and a task scheduling technology is introduced to realize acquisition task scheduling.
Step 3, summarizing and summarizing all the data extracted in the step 1 according to extraction requirements, defining the generation of each piece of data as a standard rule, and converting an irregular data source into standard, clean and effective data;
and step 4, providing a visual front-end configuration interface by adopting a front-end and back-end separation technology, realizing the application of the scheduling configuration of the acquisition plan on a web end, and not influencing the running process of a back-end scheduling task. 2. The visual stand-alone data collection method for wharf efficiency analysis of claim 1, wherein the TBI ETL platform in step 1 comprises
The source library management module: the system comprises a configuration module, a configuration module and a configuration module, wherein the configuration module is used for configuring an InfluxDB database address and information of equipment and a TAG point in TBIOPD engineering and is used for executing an extraction module; and/or
The target database management module is used for realizing the basic information configuration of the target database, namely the data extracted from the InfluxDB is processed and finally stored into which tables in which database, namely the connection character string of the target database and the table information in the target database are configured; and/or
The management of the scheduling module is mainly to configure scheduling information of different execution frequencies for the execution extraction module to use for scheduling management; and/or
The extraction execution module is used for realizing the function that a set of platform meets the use of all projects; and/or
A monitoring log module: the management of extracting log system information is realized; and/or
A system information management module: the method is used for maintaining the core basic information of the TBI ETL platform.
2. The visual stand-alone data collection method for wharf efficiency analysis of claim 1, wherein the source library management module comprises HOST management, data source management, data table management, TAG management and equipip management.
3. The visual stand-alone data collection method for wharf efficiency analysis of claim 1, wherein the target library management module comprises data source management of the target library, data table management of the target library and table list information management of the target library.
4. The visual stand-alone data collection method for wharf efficiency analysis of claim 1, wherein the extraction execution module management comprises an extraction rule configuration module; a configuration for providing a series of expressions to fulfill various acquisition requirements; and/or an extraction rule analysis module for realizing the function that the data is generated into the target database from the InfluxDB database through processing.
5. The visual stand-alone data collection method for wharf efficiency analysis of claim 1, wherein the monitoring log module comprises extraction history management and/or TAG point last record management.
6. The visual stand-alone data collection method for wharf efficiency analysis of claim 1, wherein the system information management module comprises user management, menu management, role management and dictionary management.
7. The visual stand-alone data collection method for wharf efficiency analysis of claim 1, wherein the step 2 comprises the following steps:
s21, expressing the extraction rule in the step 1 by an expression engine based on google aviator to realize expression of the data extraction rule;
and S22, a Spring is used for integrating the Quartz, so that the development of a timing task and the realization of a scheduling module are realized, and the reliable and efficient normal execution of data extraction is realized.
8. The visual stand-alone data collection method for wharf efficiency analysis of claim 1, wherein the rule in step 3 comprises
Start rule: generating rules of the business data; and/or
Initial start rules: updating rules when the business data are generated; and/or
And (3) updating the rule: updating rules of the business data; and/or
And (4) finishing the rule: a service data end rule; and/or
Initial end rule: and updating the rule when the service data is finished.
9. The visual stand-alone data collection method for wharf efficiency analysis of claim 1, wherein the step 4 comprises the following steps:
s41, the system deploys a B/S framework to complete configuration management of the system;
s42, providing a Rest API interface at the back end to facilitate interaction of the front end and the back end;
s43, a front-end and rear-end separation technology is adopted, a friendly visual configuration interface and a friendly monitoring interface are provided, and the operation condition of the system is monitored in real time;
s44, adopting SpringSecurity + JWT authority management technology to perform refined authority management and ensure the safety of the system;
and S45, introducing Logback log management, finely controlling the application range of the log, and monitoring the running condition of the system.
CN202110011595.0A 2021-01-06 2021-01-06 Visual single machine data acquisition method in wharf efficiency analysis Active CN112667728B (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202110011595.0A CN112667728B (en) 2021-01-06 2021-01-06 Visual single machine data acquisition method in wharf efficiency analysis

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202110011595.0A CN112667728B (en) 2021-01-06 2021-01-06 Visual single machine data acquisition method in wharf efficiency analysis

Publications (2)

Publication Number Publication Date
CN112667728A true CN112667728A (en) 2021-04-16
CN112667728B CN112667728B (en) 2023-11-21

Family

ID=75413091

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202110011595.0A Active CN112667728B (en) 2021-01-06 2021-01-06 Visual single machine data acquisition method in wharf efficiency analysis

Country Status (1)

Country Link
CN (1) CN112667728B (en)

Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2017166644A1 (en) * 2016-03-31 2017-10-05 乐视控股(北京)有限公司 Data acquisition method and system
KR101907656B1 (en) * 2017-08-29 2018-10-12 주식회사 아모티 Trouble diagnosis and management system for electrical vehicle charger
CN109669983A (en) * 2018-12-27 2019-04-23 杭州火树科技有限公司 Visualize multi-data source ETL tool
CN111984709A (en) * 2019-05-23 2020-11-24 云南青年学园科技有限公司 Visual big data middle station-resource calling and algorithm

Patent Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2017166644A1 (en) * 2016-03-31 2017-10-05 乐视控股(北京)有限公司 Data acquisition method and system
KR101907656B1 (en) * 2017-08-29 2018-10-12 주식회사 아모티 Trouble diagnosis and management system for electrical vehicle charger
CN109669983A (en) * 2018-12-27 2019-04-23 杭州火树科技有限公司 Visualize multi-data source ETL tool
CN111984709A (en) * 2019-05-23 2020-11-24 云南青年学园科技有限公司 Visual big data middle station-resource calling and algorithm

Also Published As

Publication number Publication date
CN112667728B (en) 2023-11-21

Similar Documents

Publication Publication Date Title
EP3798846B1 (en) Operation and maintenance system and method
CN106104533B (en) Handle the data set in large data repository
US20190179732A1 (en) System and method for automatically generating software testing scripts from test cases
CN107506451A (en) abnormal information monitoring method and device for data interaction
CN110647387B (en) Education cloud big data task scheduling method and system
CN104778124A (en) Automatic testing method for software application
AU2019201821B2 (en) Generating an execution script for configuration of a system
CN110019203B (en) Automatic server inspection system and method based on knowledge base
CN107748782A (en) Query statement processing method and processing device
CN110727557A (en) Log collector control method, platform, electronic device and storage medium
JP6256115B2 (en) Operation search program, operation search method, and operation search device
CN105487971A (en) Method and system for recording and reproducing operation steps in software testing
CN111061743A (en) Data processing method and device and electronic equipment
CN111198902A (en) Metadata management method and device, storage medium and electronic equipment
CN107423035B (en) Product data management system in software development process
CN112667728A (en) Visual single-machine data acquisition method in wharf efficiency analysis
CN112541746A (en) Full-stack automatic arrangement method and system
CN109508244B (en) Data processing method and computer readable medium
CN111913706B (en) Topology construction method of dispatching automation system, storage medium and computing equipment
CN114911677A (en) Monitoring method and device for containers in cluster and computer readable storage medium
CN104572835B (en) A kind of Process Meta-Model construction method and device
CN112115234A (en) Question bank analysis method and device
CN113901227A (en) Relation analysis method and system of power grid equipment based on knowledge graph
CN115600037A (en) Data acquisition method, device and equipment based on front-end buried point and storage medium
US10909079B1 (en) Data-driven reduction of log message data

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
GR01 Patent grant
GR01 Patent grant