CN115202653A

CN115202653A - Flight data fusion compound disk system based on Django

Info

Publication number: CN115202653A
Application number: CN202210758181.9A
Authority: CN
Inventors: 殷华杰; 李虎; 王凯; 马玺; 陈龙龙; 刘佩惠
Original assignee: China Aeronautical Radio Electronics Research Institute
Current assignee: China Aeronautical Radio Electronics Research Institute
Priority date: 2022-06-29
Filing date: 2022-06-29
Publication date: 2022-10-18

Abstract

The invention discloses a Django-based flight data fusion multi-disk system.A configuration management module is deployed on an application server, and comprises a template layer, a model layer and a view layer, wherein the model layer is used for defining a data structure of flight data and managing the flight data; the template layer is used for generating various HTTP interfaces, the view layer is used for calling required flight data from the model layer, filling the flight data into the HTTP interfaces, sending the flight data to the client browser for a user to perform newly adding, analyzing, cleaning and converting on the flight data, and calling the model layer to complete corresponding logic operation after receiving an operation request of the client browser. The system is developed based on the Django webpage framework, and an MTV layered development mode is adopted, so that the system development efficiency is improved; users can easily operate through the Web page, and the full life cycle processes of analyzing, storing, converting, applying and the like of structured/unstructured data are realized.

Description

Flight data fusion compound disk system based on Django

Technical Field

The invention relates to the technical field of data engineering, in particular to a Django-based flight data fusion compound disk system.

Background

Django is a Python-based open source code Web application framework, and adopts an MTV framework mode similar to an MVC design mode, namely a model M, a view V and a template T. The model M is used for defining a data structure and providing a database table management mechanism; the template T is used for defining an HTML format, namely a front-end UI interface, and data of the model M is extracted through the view V to fill an HTML file; the view V is used for processing the business logic, receiving the HTTP request sent by the template T, realizing the business logic, acquiring data required by the client from the model M, transmitting the data to the template T in a certain format, and then Django sends the template T to the browser in an HTTP response format, so that layering of the data model, the business logic and the front-end logic is completed. And the interface is optimized by combining the django theme template simplex UI, the configuration is simplified, and the expandability, the code reuse rate and the development efficiency are improved. Due to the rapid development of Django in recent years, the application is more and more extensive, and by using the architecture, a high-quality, easy-to-maintain and database-driven application program can be conveniently and quickly created.

The flight data fusion multi-disk system is a process of storing, analyzing and replaying data after unloading batch data to ground storage equipment. The traditional method adopts a table form to store and playback data, is difficult to retrieve data such as unstructured photos and the like, manually searches imported files through client software in the analysis process, and has low data management and application efficiency. With the data processing scale of the system becoming larger and larger, the software complexity becomes higher and higher, and the security requirement also becomes higher and higher. The existing system for fusing and copying software has the problems of low development efficiency, low system reliability, low retrieval efficiency, incomplete and discrete data processing operation flow, low automation degree, low safety and the like, and the data application requirement of a copying task cannot be guaranteed.

Disclosure of Invention

The invention aims to provide a flight data fusion multi-disk system based on Django to solve the problems that in the existing data fusion multi-disk technology, the data processing flow is complex to operate, the automation degree is low, the safety is low and the like.

The invention aims to be realized by the following technical scheme:

a flight data fusion compound disk system based on Django adopts an application server, a database server and a disk array to form a server end in a hardware environment in a deployment system, a plurality of clients form a client end, the database server and the disk array are used as storage modules for the application server, the application server and the client end are crosslinked through a switch, and the client provides a unified portal through a WEB browser and interacts with the application server;

the application server is used as a data center for deploying a multi-disk application system, the multi-disk application system comprises a data management module, the data management module comprises a template layer, a model layer and a view layer, and the model layer is used for defining a data structure of flight data and managing the flight data; the template layer is used for generating various HTTP interfaces, the view layer is used for calling required flight data from the model layer, filling the flight data into the HTTP interfaces, sending the flight data to the client browser for a user to perform addition, analysis, cleaning and conversion of the flight data, and calling the model layer to complete corresponding logic operation after receiving an operation request sent by the client browser in an HTTP request mode.

Preferably, after the user inputs the flight task metadata of a certain batch of original data on the HTTP interface and submits a new request, the view layer receives the new request, calls the model layer to store the input flight task metadata into the database server, associates the input flight task metadata with the original data folder of the corresponding batch, fills the execution result into the flight task information interface, and sends the execution result to the client browser for display.

Preferably, when a user selects original data and submits an analysis and cleaning conversion instruction, the view layer receives the analysis and cleaning conversion request, a background execution task is established, the model layer is called to analyze the original data to be processed to obtain a standard data file, the standard data file is stored in a standard data directory of the database server, and the standard data file is imported into a corresponding service data table in the database server; cleaning the standard data file according to a cleaning rule to generate a clean data file; converting the clean data file into a theme data file according to the requirement of the multi-disk task, and supporting the retrieval and modification of theme data through a model layer; and finally generating a theme data file and storing the theme data file in a theme data directory.

Preferably, the hard disk application system further comprises a configuration management module, configured to configure an NAS server directory path for storing the original data, an NAS server node IP, an application server IP, and a database backup path on the disk array.

Preferably, the multi-disk application system further comprises a metadata management module for managing database information including data processing status, chinese and English comparison information, and a file attribute table;

the metadata management module generates data processing states in all stages of a flight data analysis, cleaning and conversion process of the data management module and stores the generated data into a corresponding standard data sheet, a clean data sheet and a theme data sheet, and the data management module calls all data sheets in the metadata management module through a view layer to fill all the data processing states into an HTTP interface and then feeds the data processing states back to a front-end browser;

the Chinese and English comparison information table carries out automatic Chinese conversion on the meanings of table names and field names of the standard data table, the clean data table and the subject data table, and the Chinese names are displayed on an interface;

the file attribute table stores attribute information of all files stored in the NAS server directory, and the file attribute tables can be automatically updated after the data management module performs data directory operation.

Preferably, the management of the standard data table, the clean data table, the theme data table, the Chinese and English comparison information table and the file attribute table in the metadata management module is realized through a simpleUI theme template.

Preferably, the multi-disk application system further comprises a multi-disk application module, which is implemented by a WebGIS technology based on a map engine, the data management module sends a message to notify the multi-disk playback module after the theme data file is stored in the theme data directory, the multi-disk playback module synchronizes the theme data of the current batch into a data table of the multi-disk playback module after receiving the notification message issued by the data management module, the user creates a multi-disk playback task, selects the designated batch of data for playback, displays a flight task timeline on the map through a visual interface, and plays back the flight task of the current batch through multi-disk playback.

The invention has the beneficial effects that:

the system is developed based on the Django webpage framework, and an MTV layered development mode is adopted, so that the system development efficiency is improved; and a unified operation portal is provided, a user can easily operate through a Web page, the full life cycle processes of analysis, storage, conversion, application and the like of structured/unstructured data are realized, the management of task information, data processing states and file attributes is realized based on a metadata technology, the data security is guaranteed through authorized access, and abundant maintenance tools such as logs and configuration are provided, the full-process full-element automatic operation and maintenance management and control capability is formed, the purpose of flight data replication evaluation is achieved, and the operation of the user is more humanized and simplified.

In addition to the technical problems addressed by the present invention, the technical features constituting the technical solutions, and the advantages brought by the technical features of the technical solutions described above. In order to make the objects, technical solutions and advantages of the present invention clearer, other technical problems, other technical features included in the technical solutions and advantages brought by the technical features which can be solved by the present invention will be described more clearly and completely with reference to the accompanying drawings in the embodiments of the present invention.

Drawings

FIG. 1 is a configuration diagram of a flight data fusion system deployment according to the present invention.

FIG. 2 is a functional architecture diagram of a flight data fusion system according to the present invention.

FIG. 3 is a flow chart of a main line of a flight data fusion multi-disk system according to the present invention.

Fig. 4 is a schematic diagram of a system configuration management interface in an embodiment of the present invention.

FIG. 5 is a diagram of a metadata management interface in an embodiment of the invention.

FIG. 6 is a diagram of a data management interface in an embodiment of the invention.

FIG. 7 is a diagram of a data processing state interface in an embodiment of the present invention.

FIG. 8 is a schematic diagram of various data browsing interfaces generated in an embodiment of the present invention.

Detailed Description

The present invention will be described in further detail with reference to the accompanying drawings and examples.

In the flight data fusion multi-disk system based on Django shown in this embodiment, as shown in fig. 1, a hardware environment in a deployment system adopts an application server, a database server, and a disk array to form a server side, a plurality of clients form a client side, the database server and the disk array serve as a storage module for the application server to use, the application server and the client side are cross-linked through a switch, and the client provides a unified portal through a WEB browser to interact with the application server.

Referring to fig. 2, the application server is used as a data center for deploying a multi-disk application system, and includes a configuration management module, a data management module, a metadata management module, and a multi-disk playback module. The data management module converts unstructured original data files stored in the disk array into structured flight data, stores the structured flight data into the database server, and responds to a request for accessing the flight data in the database server sent by a client. The unstructured original data in the disk array may be files in a data card copied to the disk array by a user, or may be any other method, which is not limited in this embodiment. Each block is explained in detail below.

The data management module comprises a template layer, a model layer and a view layer, wherein the model layer is used for defining a data structure of flight data and managing the flight data; the template layer is used for generating various HTTP interfaces, the view layer is used for calling required flight data from the model layer, filling the flight data into the HTTP interfaces, sending the flight data to the client browser for a user to perform operations such as flight data addition, analysis, cleaning and conversion, and calling the model layer to complete corresponding logic operations after receiving an operation request sent by the client browser in an HTTP request mode. See in particular fig. 3.

As shown in fig. 5, after a user inputs a flight task metadata of a certain batch of original data in a flight data task information interface and submits a new request, after receiving the new request, a view layer calls a model layer to store the input flight task metadata in a database server, associates the input flight task metadata with an original data folder of a corresponding batch, fills an execution result in the flight task information interface, sends the flight task metadata to a client browser for display, selects a new row by the user, clicks "jump to a selected data directory", and jumps to a current operation directory of the data directory interface, as shown in fig. 6. The flight task metadata is used for describing original data, and fields can be customized according to business requirements, such as the take-off and landing time, duration, machine number, operator, subject name, task area, record file path and other information of the current flight task.

Referring to fig. 6, a directory structure obtained from a disk array is displayed on a data directory interface, a user selects a left original data check box to be processed, after the original data check box is clicked for analysis, cleaning and conversion, a background execution task is established after the view layer receives an analysis, cleaning and conversion request, a model layer is called to analyze and process original data to be processed to obtain standard data files in standard formats such as CSV (common server virtualization) and the like, the standard data files are stored in a standard data directory of a database server, and the standard data files are imported into a corresponding service data table in the database server; cleaning the standard data file according to a cleaning rule (including non-null value, repeated value, abnormal value and useless value verification, wherein the specific rule is determined according to a specific data format) to generate a clean data table; converting the clean data file into a theme data file according to the requirement of the multi-disk task, and supporting the functions of retrieval, modification and the like of the theme data through a model layer; finally, generating a subject data standard CSV file and storing the subject data standard CSV file in a subject data directory; and after the completion, sending a message to inform the disk replication playback module, filling the execution result into a flight task information interface, and sending the flight task information interface to a client browser for display. The data directory interface also supports the functions of downloading, deleting, copying, cutting, renaming and directory retrieval, after a user clicks a check box on the left side of the directory to select, the user clicks the function button, a directory operation command is sent to the Django server through an HTTP request, the view layer completes the operation after receiving the user request, an execution result is filled into an HTML interface generated by the template layer and sent to a browser, the result is displayed on the interface, and operation information such as downloading, deleting, copying, cutting, renaming and the like is recorded in a file attribute table of the metadata management module. Wherein, the download information includes the record of download times, time, personnel, etc.

Referring to fig. 4, the configuration management module is configured to configure an NAS server directory path for storing original data, an NAS server node IP, an application server IP, a database backup path on a disk array, and the like.

The NAS server is used for mapping the disk array to the drive letter of the application server in a mode of adding a network position; when the system is used for the first time, a user needs to map the NAS server directory on the disk array to the drive letter of the Django application server in a network position mode, and the mapping mode comprises the following steps: shared folders, FTP file servers, etc., shared folder paths such as "\ \ server \ share", FTP file server paths such as FTP://192.168.10.1. And then, the user configures the path into the directory path of the NAS server in a configuration management interface.

Configuring an NAS server node IP and an application server IP for cross-linking with external applications;

configuring a database backup path for carrying out operation and maintenance operations such as database backup and recovery, wherein a user carries out full backup of a database server through one-key backup, data is backed up to the configured database backup path in an SQL (structured query language) file mode, and the SQL file is stored in a folder named in a format of year, month, day, hour, minute and second; and selecting the corresponding sql file by the user through 'one-key recovery', and realizing the one-key recovery of the database server. Other configurations can be added according to requirements, the Django server comprises a config directory, and configuration information can be easily written into the conf file for a user to read.

The metadata management module is used for managing database information including data processing states, chinese and English comparison information and a file attribute table.

The metadata management module generates data processing states in all stages of the flight data analysis, cleaning and conversion process of the data management module and stores the generated data into corresponding data tables (standard data tables, clean data tables and theme data tables). The data processing state comprises the description information of the flight data such as batch, generation state, generation time, generation path and the like. And the data management module calls each data table in the metadata management module through the view layer to fill each data processing state into the HTTP interface and feed back the data processing state to the front-end browser.

The Chinese-English comparison information table carries out automatic Chinese conversion on the meanings of the table names and the field names of the standard data table, the clean data table and the theme data table, and displays the Chinese names on an interface.

The file attribute table stores attribute information such as path, size, creation time, modification/upload time, owner, creation/upload person, download times/person/time and the like of all files stored in the NAS server directory, and the file attribute tables can be automatically updated after the data directory operation is performed by the data management module.

The management of the standard data sheet, the clean data sheet, the theme data sheet, the Chinese and English comparison information sheet and the file attribute sheet can be directly realized through a simpleUI theme template without developing a template and a view, because the simpleUI is internally provided with a universal template and a view, the display parameters are only required to be configured in generated admin. The development amount of front-end logic is greatly saved.

As shown in fig. 7, the data management module executes an analysis and cleaning conversion process, and after data analysis, the metadata management module automatically updates the generation state, generation time, and generation path information of the standard data table; after data is cleaned, the generation state, the generation time and the generation path information of the clean data table are updated; after the data conversion, the generation state, the generation time and the generation path information of the theme data table are updated. The data status page shows the processing status of the batch data, a drop-down list with check boxes is used for the user to configure fields to be displayed, and default fields, all fields and configuration default fields are supported. Referring to fig. 8, when the generated standard data table, clean data table and subject data table are browsed, the table name and field name are automatically converted into chinese through the chinese and english information table of the metadata management module, and the chinese name is displayed on the interface.

And the copy application module is realized by adopting a WebGIS technology based on a map engine. The multi-disc playback module is software with independent function, and is not realized by django, and the invention is not specifically described. In the traditional method, data required by a compound disk playback module can be used after low-efficiency manual compilation; after the flight mission information is input, the method can realize the processes of analysis, cleaning, conversion and multi-disk application of the original data by one key, and connect the whole process in series; after receiving the notification message issued by the data management module, the multi-disk playback module synchronizes the subject data of the current batch to the data table of the multi-disk playback module, a user creates a multi-disk playback task, selects the data of the specified batch for playback, displays a flight task timeline on a map through a visual interface, and plays back information such as coordinate tracks, states, events and the like in the flight tasks of the current batch.

The invention provides a Django-based flight data fusion compound disk system, and particularly provides a method and a way for realizing the technical scheme, for example, an operating system can support Linux and Windows versions, and an application server can be realized through Java. The described embodiments are only some, but not all embodiments of the invention. The components of embodiments of the present invention generally described and illustrated in the figures herein may be arranged and designed in a wide variety of different configurations. Thus, the detailed description of the embodiments of the invention presented in the drawings is not intended to limit the scope of the invention as claimed, but is merely representative of selected embodiments of the invention. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.

Claims

1. The utility model provides a flight data fusion disklike system based on Django, hardware environment adopts an application server, a database server, a disk array to constitute server end in deployment system, and many clients constitute the customer end, and database server and disk array supply application server to use as storage module, and application server and customer end cross-link through the switch, and the customer end provides unified portal through the WEB browser, and is interactive with application server, its characterized in that:

the application server is used as a data center for deploying a multi-disk application system, the multi-disk application system comprises a data management module, the data management module comprises a template layer, a model layer and a view layer, and the model layer is used for defining a data structure of flight data and managing the flight data; the template layer is used for generating various HTTP interfaces, the view layer is used for calling required flight data from the model layer, filling the flight data into the HTTP interfaces, sending the flight data to the client browser for a user to perform newly adding, analyzing, cleaning and converting on the flight data, and calling the model layer to complete corresponding logic operation after receiving an operation request sent by the client browser in an HTTP request mode.

2. The system of claim 1, wherein when a user inputs mission metadata of a certain batch of raw data on an HTTP interface and submits a new request, the view layer receives the new request, invokes a model layer to store the input mission metadata in a database server, associates the input mission metadata with a corresponding batch of raw data folders, fills execution results in a mission information interface, and sends the execution results to a client browser for display.

3. The Django-based flight data fusion replication system according to claim 1, wherein when a user selects original data and submits an analysis, cleaning and conversion instruction, the view layer receives the analysis, cleaning and conversion request, establishes a background execution task, calls the model layer to analyze and process the original data to be processed to obtain a standard data file, stores the standard data file in a standard data directory of a database server, and imports the standard data file into a corresponding service data table in the database server; cleaning the standard data file according to a cleaning rule to generate a clean data file; converting the clean data file into a theme data file according to the requirement of the multi-disk task, and supporting the retrieval and modification of theme data through a model layer; and finally generating a theme data file and storing the theme data file in a theme data directory.

4. The Django-based flight data fusion multi-disk system as claimed in claim 1, wherein the multi-disk application system further comprises a configuration management module for configuring a NAS server directory path, a NAS server node IP, an application server IP and a database backup path on the disk array for storing original data.

5. The Django-based flight data fusion disk replication system as claimed in claim 1, wherein the disk replication application system further comprises a metadata management module for managing database information including data processing status, chinese and English comparison information, and a file attribute table;

the Chinese-English comparison information table carries out automatic Chinese conversion on the meanings of the table names and the field names of the standard data table, the clean data table and the theme data table, and displays the Chinese names on an interface;

6. The Django-based flight data fusion copy system as claimed in claim 5, wherein the management of the standard data table, the clean data table, the subject data table, the Chinese and English comparison information table and the file attribute table in the metadata management module is realized through a simpleUI subject template.

7. The Django-based flight data fusion copy system as claimed in claim 1, wherein the copy application system further comprises a copy application module, and is implemented by adopting a WebGIS technology based on a map engine, the data management module sends a message to notify the copy playback module after storing a theme data file in a theme data directory is completed, the copy playback module synchronizes the theme data of the current batch into a data table of the copy playback module after receiving the notification message issued by the data management module, a user creates a copy playback task, selects a designated batch of data for playback, displays a flight task timeline on a map through a visual interface, and plays back the flight tasks of the current batch through a copy.