CN114610923A - Big data processing method, device, equipment and medium - Google Patents

Big data processing method, device, equipment and medium Download PDF

Info

Publication number
CN114610923A
CN114610923A CN202210287079.5A CN202210287079A CN114610923A CN 114610923 A CN114610923 A CN 114610923A CN 202210287079 A CN202210287079 A CN 202210287079A CN 114610923 A CN114610923 A CN 114610923A
Authority
CN
China
Prior art keywords
data
spatial
processing
data processing
model
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Pending
Application number
CN202210287079.5A
Other languages
Chinese (zh)
Inventor
孙伟利
陈桂红
李娜
殷世康
刘丽正
郑传生
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Chengxin Desai Beijing Technology Co ltd
Beijing Big Data Center
Original Assignee
Chengxin Desai Beijing Technology Co ltd
Beijing Big Data Center
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Chengxin Desai Beijing Technology Co ltd, Beijing Big Data Center filed Critical Chengxin Desai Beijing Technology Co ltd
Priority to CN202210287079.5A priority Critical patent/CN114610923A/en
Publication of CN114610923A publication Critical patent/CN114610923A/en
Pending legal-status Critical Current

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/50—Information retrieval; Database structures therefor; File system structures therefor of still image data
    • G06F16/53—Querying
    • G06F16/532—Query formulation, e.g. graphical querying
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
    • G06F16/29—Geographical information databases
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/50—Information retrieval; Database structures therefor; File system structures therefor of still image data
    • G06F16/53—Querying
    • G06F16/538—Presentation of query results
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/50—Information retrieval; Database structures therefor; File system structures therefor of still image data
    • G06F16/56—Information retrieval; Database structures therefor; File system structures therefor of still image data having vectorial format
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/50—Information retrieval; Database structures therefor; File system structures therefor of still image data
    • G06F16/58—Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually
    • G06F16/587—Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually using geographical or spatial information, e.g. location

Abstract

The invention discloses a big data processing method, a big data processing device, big data processing equipment and a big data processing medium. The method comprises the following steps: determining a data processing model based on model configuration operation of a user on a visual interface, and acquiring to-be-processed spatial data corresponding to the data processing model from a local database; processing the spatial data to be processed through the data processing model to obtain a processing result; and determining the spatial chart data corresponding to the processing result based on the open geographic spatial information alliance OGC standard service and/or the vector slice map service and the display parameters configured on the visual interface by the user, and outputting the spatial chart data on the visual interface. By the technical scheme provided by the embodiment of the invention, the spatial big data can be automatically processed, and the flexible customization of the data analysis processing process and the visual display content is further realized.

Description

Big data processing method, device, equipment and medium
Technical Field
The embodiments of the present invention relate to a big data processing technology, and in particular, to a big data processing method, apparatus, device, and medium.
Background
With the rapid development of technologies such as internet, internet of things, cloud computing and the like, mass data are generated in various fields, and the data have the characteristics of rich sources, various types, rapid updating frequency, huge data volume and the like.
Disclosure of Invention
The invention provides a big data processing method, a big data processing device, big data processing equipment and a big data processing medium, which can automatically process spatial big data and further realize flexible customization of a data analysis processing process and visual display contents.
In a first aspect, an embodiment of the present invention provides a big data processing method, including:
determining a data processing model based on model configuration operation of a user on a visual interface, and acquiring to-be-processed spatial data corresponding to the data processing model from a local database;
processing the spatial data to be processed through the data processing model to obtain a processing result;
and determining the spatial chart data corresponding to the processing result based on the open geographic spatial information alliance OGC standard service and/or the vector slice map service and the display parameters configured on the visual interface by the user, and outputting the spatial chart data on the visual interface.
In a second aspect, an embodiment of the present invention further provides a big data processing apparatus, including:
the acquisition module is used for determining a data processing model based on model configuration operation of a user on a visual interface and acquiring to-be-processed spatial data corresponding to the data processing model from a local database;
the obtaining module is used for processing the spatial data to be processed through the data processing model to obtain a processing result;
and the output module is used for determining the spatial chart data corresponding to the processing result based on open geographic spatial information alliance OGC standard service and/or vector slice map service and display parameters configured on the visual interface by a user, and outputting the spatial chart data on the visual interface.
In a third aspect, an embodiment of the present invention further provides an electronic device, where the electronic device includes:
one or more processors;
a memory for storing one or more programs;
when executed by the one or more processors, cause the one or more processors to implement a big data processing method as provided in any embodiment of the invention.
In a fourth aspect, an embodiment of the present invention further provides a computer-readable storage medium on which a computer program is stored. Wherein the program when executed by a processor implements a big data processing method as provided by any of the embodiments of the invention.
According to the scheme provided by the embodiment of the invention, based on model configuration operation of a user on a visual interface, a data processing model is determined, the to-be-processed spatial data corresponding to the data processing model is obtained from a local database, the to-be-processed spatial data is processed through the data processing model to obtain a processing result, and based on open geographic space information alliance OGC standard service and/or vector slice map service and display parameters configured on the visual interface by the user, the spatial chart data corresponding to the processing result is determined, and the spatial chart data is output on the visual interface. By the method, the spatial big data can be automatically processed, and the flexible customization of the data analysis processing process and the visual display content is further realized.
Drawings
Fig. 1A is a flowchart of a big data processing method according to an embodiment of the present invention;
FIG. 1B is a schematic diagram of determining a data processing model according to an embodiment of the present invention;
fig. 1C is a schematic diagram of a visualization interface provided in the first embodiment of the present invention;
fig. 2 is a flowchart of a big data processing method according to a second embodiment of the present invention;
FIG. 3 is a block diagram of a big data processing system according to a third embodiment of the present invention;
fig. 4 is a block diagram of a big data processing apparatus according to a fourth embodiment of the present invention;
fig. 5 is a schematic structural diagram of an electronic device according to a fifth embodiment of the present invention.
Detailed Description
The present invention will be described in further detail with reference to the accompanying drawings and examples. It is to be understood that the specific embodiments described herein are merely illustrative of the invention and are not limiting of the invention. It should be further noted that, for the convenience of description, only some of the structures related to the present invention are shown in the drawings, not all of the structures.
Example one
Fig. 1A is a flowchart of a big data processing method according to an embodiment of the present invention, fig. 1B is a schematic diagram of a data processing model according to an embodiment of the present invention, and fig. 1C is a schematic diagram of a visualization interface according to an embodiment of the present invention. As shown in fig. 1A, the big data processing method provided in this embodiment specifically includes:
s101, determining a data processing model based on model configuration operation of a user on a visual interface, and acquiring to-be-processed space data corresponding to the data processing model from a local database.
The model configuration operation refers to an operation of configuring a data processing model, and specifically may include an operation of configuring an existing data processing model to be automatically started at a fixed time, and may also include an operation of configuring and constructing a new data processing model by dragging a data processing model component. A data processing model refers to a model component that can perform processing operations on data. The local database is a database that is configured locally to the big data processing system. The spatial data is data including spatial information, and specifically, may be data including position coordinate information of data that can be added to a map for labeling. Non-spatial data refers to data that does not include spatial information, such as data that includes only name information for a certain address.
Optionally, the data processing model may include basic data processing model components, such as one or more of common functions, data filtering, table merging, table association, address matching, compartment statistics, case transfer, time ordering, type transfer, obtaining a center point, finding an area, finding a distance, and including analysis. The data processing model may also include already built data processing model components for data analysis, such as density analysis, mesh analysis, cluster analysis, buffer analysis, traditional density analysis, traditional buffer analysis, overlay analysis, trajectory reconstruction analysis, attribute summary statistical analysis, vector crop analysis, single object space query, topology inspection, and OD (dynamic debug) analysis.
Optionally, the data processing model may be generated according to a code input by a user on the visual interface, specifically, the user may directly open a related script program file in the data processing system, autonomously write an input script source code, and integrate the source code to generate the data processing model, that is, determine the data processing model.
Optionally, determining the data processing model based on a model configuration operation of the user on the visualization interface includes: based on the timed starting configuration operation of a user on the existing model on a visual interface, acquiring a data processing model meeting the timed starting requirement at the current moment; and/or determining the connection relation between the dragging component and the selectable component based on the dragging configuration operation of the user on the selectable component on the visual interface, and generating the data processing model according to the connection relation between the dragging component and the dragging component.
The timing starting configuration operation refers to an operation of the user for autonomously configuring how to start the existing data processing model at a fixed time according to a timing starting requirement. The configuration operation may include configuring connection relationships and configuration parameters. The optional component is a minimum unit model component with a data processing function, and at least one optional component is connected and combined to determine a data processing model.
Optionally, there may be many ways to determine the data processing model, and one possible implementation is: based on the configuration operation of the user on the existing model at the visual interface, the existing data processing model is started at regular time, namely the data processing model is determined. Another possible implementation is: the method comprises the steps of firstly determining dragged optional components based on dragging configuration operations of the optional components for constructing the data processing model by a user, and further constructing a new data model, namely determining the data processing model based on configuration connection relations and configuration parameters corresponding to the configuration operations. Another possible implementation is: and simultaneously, determining at least two data processing models, namely determining the data processing models, based on the timed starting configuration operation of the user on the existing models and the dragging configuration operation of the selectable components.
For example, referring to fig. 1B, a user may drag data 1 and data 2 to a working area of a visualization interface in a local database, further drag components 1 and 2 in selectable components to the working area of the visualization interface, and finally connect the components with the data, that is, determine a connection relationship between the dragged component and the dragged component, so that a new data processing model may be generated, and in the process of generating the data processing model, configuration operations may be cancelled or cleared at any time, or the model may be saved or executed after the new data processing model is generated.
Optionally, the relevant parameters of each data processing model or optional component may be adjusted according to the configuration operation of the user, for example, in response to the double-click operation of the zone statistics optional component, a configuration page of the zone statistics optional component is started, and then in response to the parameter setting operation of the user on the configuration page of the zone statistics optional component, the relevant information of the component is adjusted, such as information of an adjustment statistic type (spatial statistics and non-spatial statistics), a statistic field (a field in the table, such as a name, a region, an age field, etc.), and a zone type (whether the spatial data belongs to a county-level or city-level region).
Optionally, after the data processing model is determined, to-be-processed spatial data corresponding to the data processing model may be acquired from the local database according to the type of the determined data processing model.
It should be noted that the local Database may store data from a plurality of data sources, the data sources may be external existing databases, such as commonly used relational databases like Oracle, PostgreSql, MySQL, and the like, and the data sources may also be data from data transmission, data synchronization, file import, and other manners, and the types of the data include, but are not limited to, common data formats like csv (Comma-Separated Values), xlsx, accdb (access Database), shp, and the like.
Optionally, after determining a connection relationship between the dragging component and the selectable component based on a dragging configuration operation of the user on the selectable component on the visual interface, and generating the data processing model according to the connection relationship, the method further includes: and saving the data processing model, and configuring timing starting parameters for the data processing model.
The timing start parameter is a parameter configuring information such as a start cycle or start duration of the data processing model, and the execution cycle may include every cycle, every week, and every month. Optionally, the big data processing system may further generate a related information table, a timing task table, and a model structure string of the data processing model according to the stored data processing model, so as to store different types of model information.
Optionally, after the constructed data processing model is stored, the newly constructed data processing model may be correspondingly added to a model management module of the big data processing system, and then the data processing model may be directly used in a dragging or timing manner. That is, the model management module of a big data processing system may include some models that are self-contained in the system itself, and may also include models that are generated by user-defined constructs.
The advantage of setting in this way is that the user can flexibly customize the data processing model in the visual interface by storing the data processing model customized by the user and setting the relevant parameters.
And S102, processing the spatial data to be processed through the data processing model to obtain a processing result.
Optionally, after determining the data processing model and acquiring the corresponding to-be-processed spatial data, the preset execution button or shortcut key may be clicked in the big data processing system, that is, the to-be-processed spatial data may be automatically processed, so as to obtain a processing result; and after the data processing model is determined and the corresponding to-be-processed spatial data is acquired, automatically executing the processing of the to-be-processed spatial data to obtain a processing result when the processing condition is met based on the timing processing parameter set by the user.
Optionally, there may be many ways to process the spatial data to be processed, and specifically, the processing the spatial data to be processed through the data processing model includes: and performing at least one of grid-to-vector processing, fusion processing, clustering processing, region analysis processing, address matching processing, density analysis processing and grid analysis processing on the spatial data to be processed through the data processing model.
The Clustering process may include a Clustering process Based on DBSCAN (Density-Based Clustering of Applications with Noise) or K-Means (K-Means Clustering algorithm). The area analysis processing may include common processing modes such as buffer analysis, partition statistics, area inclusion analysis, area intersection, area union, inclusion relation analysis, and overlay analysis.
Optionally, the spatial data to be processed may be subjected to data fusion, shearing, splicing, merging (union set), intersection (intersection processing), and the like through the data processing model.
S103, determining the spatial chart data corresponding to the processing result based on the open geographic spatial information alliance OGC standard service and/or the vector slice map service and the display parameters configured on the visual interface by the user, and outputting the spatial chart data on the visual interface.
The Open Geospatial information Consortium (OGC) standard Service is a Service that provides a framework for integrating Geospatial information into a world information Service, and specifically, the OGC Service may include WMS (Web Map Service), WFS (Web Feature Service), and WMTS (OpenGIS Web Tile Service). The vector slice is a dynamic interactive map display mode controlled by a new technology, and the vector slice map can provide a customized personalized map at a personal mobile end or a browser end. The space diagram data refers to space diagram data displayed by depending on a map.
Optionally, the determining, based on the OGC standard service and/or the vector slice map service, and the display parameter configured by the user on the visualization interface, the spatial chart data corresponding to the processing result includes: analyzing a processing result based on OGC standard service and/or vector slice map service; and generating the spatial chart data corresponding to the processing result according to the analysis result and the display parameters configured on the visual interface by the user.
The display parameters refer to parameters for displaying data on a visual interface, such as color grades, contour lines, contour planes, points, lines, planes and other configuration parameters. The spatial diagram data may include three diagram library data of maps, statistics and diagrams, and specifically, the spatial diagram data may include a plurality of thematic diagram categories, such as a thermodynamic diagram, a compound diagram, a scatter diagram, a pie diagram, a histogram, a ring diagram, a grid diagram, a migration diagram, a pie diagram, a semi-circle diagram, a funnel diagram, a pyramid diagram, a data grid diagram, a grid diagram and the like.
Optionally, after the processing result is analyzed based on the OGC standard service and/or the vector slice map service, the spatial chart data corresponding to the processing result may be generated based on the OGC standard service and/or a standard interface of the vector slice map service, such as MVT (Model-View-Template) or PBF, in combination with a display parameter configured by the user on the visual interface.
Optionally, the display parameters configured by the user on the visual interface may be received, specifically, the user customizes a plurality of chart display areas in the visual panel, and the size and the position of the chart are dragged to control the layout, so that the configuration of the visual display interface can be completed, and the display parameters configured by the user can be obtained.
Optionally, after generating the spatial diagram data corresponding to the processing result, the diagram in the diagram data may be automatically adapted to adjust the layout size.
Optionally, after the visualization interface outputs the spatial chart data, the visualization display interface may be issued and called online, or may be loaded into a local system.
For example, referring to fig. 1C, after the space chart data is output by the visualization interface, the space chart data of the relevant processing result may be displayed in the visualization interface, specifically, the space chart data such as the personnel distribution map, the service visit rate, the user visit rate, and the like corresponding to different geographic areas may be displayed, and optionally, the system information such as the relevant information of the weather condition, the work progress, the visit rate, the service success rate, the service failure rate, the disk usage, and the like may also be displayed.
According to the scheme provided by the embodiment of the invention, based on model configuration operation of a user on a visual interface, a data processing model is determined, the to-be-processed spatial data corresponding to the data processing model is obtained from a local database, the to-be-processed spatial data is processed through the data processing model to obtain a processing result, and based on open geographic space information alliance OGC standard service and/or vector slice map service and display parameters configured on the visual interface by the user, the spatial chart data corresponding to the processing result is determined, and the spatial chart data is output on the visual interface. By the method, the spatial big data can be automatically processed, and the flexible customization of the data analysis processing process and the visual display content is further realized.
Example two
Fig. 2 is a flowchart of a big data processing method according to a second embodiment of the present invention, and the present embodiment further describes in detail how to add spatial data and a plotting result to a local database when a data update event is detected, based on the above embodiment, as shown in fig. 2, the big data processing method according to the present embodiment specifically includes:
s201, if a data updating event is detected, acquiring data to be updated.
The data to be updated refers to multi-source heterogeneous data and can include spatial data and non-spatial data.
Optionally, the data to be updated may be data obtained by means of importing, synchronizing, or transmitting, and the like.
Optionally, the data update event is detected, and includes at least one of the following: detecting a relational database access event; detecting that at least one of newly added data, deleted data and changed data exists in the relational database; and detecting that the current moment meets the automatic data import period.
Wherein a relational database is a database of external sources relative to a local database.
Optionally, if a relational database access event is detected, it is considered that a data update event is detected, and data of multiple accessed relational databases may be imported into the local database, specifically, data of the relational databases may be completely imported into the local database, or data meeting requirements may be imported into the local database according to a preset rule, that is, data to be updated is obtained.
Optionally, a trigger may be preset, and configured to determine whether a new data, a delete data, or a change data exists in the relational database, and if it is detected that at least one of the new data, the delete data, and the change data exists in the relational database, it is determined that a data update event is detected, and corresponding data is obtained, that is, data to be updated is obtained.
Optionally, if it is detected that the current time meets the automatic data importing period, it may be considered that a data updating event is detected, and the data to be updated is acquired. For example, the relational database includes an a folder, the local database also includes an a folder, that is, the relational database and the local database have the same name folder, and when it is detected that the current time satisfies the automatic data importing period, the a folder of the relational database is imported into the a folder of the local database, that is, a data updating event is detected, and the data to be updated is acquired.
S202, performing spatialization processing and cleaning processing on the data to be updated to obtain spatial data to be added.
The spatialization processing is a processing method for making data contain spatial information. The cleaning treatment refers to a treatment mode of sorting the acquired data to be treated so as to screen out the data meeting the requirements.
Optionally, after the data to be updated is obtained, the non-spatial data in the data to be updated may be spatially processed to become spatial data, for example, the school a and the park B are non-spatial data, and the geographic position coordinates (longitude and latitude coordinates) of the school a and the park B are determined by means of searching and the like, so that the non-spatial data may be converted into spatial data and added to the corresponding spatial position.
Optionally, after the spatialization processing, the acquired data to be updated may be sorted, the illegal or missing data is cleaned, or the missing data is supplemented in a preset manner, or the duplicate data is subjected to deduplication processing, so as to screen out data meeting the requirement, and further, according to the screened data to be updated, the data in the corresponding local database is triggered to be automatically updated once, so as to obtain the spatial data to be added.
And S203, performing online plotting on the base map according to the spatial data to be added to obtain a plotting result.
The base map refers to a base map on which spatial data can be plotted, and may be a map base map, for example.
Specifically, the spatial data to be added may be grouped, 100 pieces of data are accessed by default each time, further, the position coordinate information in the spatial data to be added is plotted on a base map, and the spatial data to be added is plotted online to obtain a plotting result.
Optionally, after the plotting result is obtained, the plotting result may be displayed on a visual interface based on a base map, so that the user performs audit correction on the plotted data according to the matching similarity, and if the plotting correction data fed back by the user is received, the plotting result may be corrected based on the plotting correction data.
And S204, storing the spatial data to be added and the plotting result in a local database.
Optionally, after the spatial data to be added and the plotting result are obtained, the spatial data to be added and the plotting result may be stored in the local database, so that the data is updated and stored in the local database. The updated data can be directly called when the data processing is carried out subsequently.
Optionally, the obtained original data to be updated may be directly updated to the local library, so as to complete the update of the data.
S205, determining a data processing model based on model configuration operation of a user on the visual interface, and acquiring to-be-processed space data corresponding to the data processing model from a local database.
And S206, processing the spatial data to be processed through the data processing model to obtain a processing result.
And S207, determining the spatial chart data corresponding to the processing result based on the open geographic spatial information alliance OGC standard service and/or the vector slice map service and the display parameters configured on the visual interface by the user, and outputting the spatial chart data on the visual interface.
It should be noted that, in the embodiments of the present invention, the processes of determining spatial data to be added and a plotting result and storing the spatial data and the plotting result in the local database and the processes of performing data processing and visual display based on model configuration operations of the user in S205 to S207 are two independent processes, and there is no order.
According to the scheme provided by the embodiment of the invention, if a data updating event is detected, the data to be updated is obtained, the data to be updated is further subjected to spatialization treatment and cleaning treatment to obtain the spatial data to be added, the base map is subjected to online plotting according to the spatial data to be added to obtain the plotting result, and finally the spatial data to be added and the plotting result are stored in the local database, so that after the model configuration operation of a user on a visual interface is detected, the data processing can be carried out and the visual display can be carried out. By the mode, the real-time performance and the richness of the data in the local database can be guaranteed, so that the spatial big data can be better subjected to automatic processing and visual display.
EXAMPLE III
Fig. 3 is a schematic structural diagram of a big data processing system according to a third embodiment of the present invention, and this embodiment provides a preferred example of how the big data processing system implements data processing and visual display on the basis of the above embodiment.
As shown in fig. 3, the big data processing system provided by the embodiment may include: the system comprises five modules of home page, data access, data processing, visualization and background management.
The system home page may include display of various data information such as real-time data and computer state, and specifically may include information display of the number of data models, the number of SQL scripts, the total amount of data (in units of bars), the number of data updates (in units of bars), network state, memory state, and data access condition.
The data access module of the system can display information of a local database and data sources, can also gather multi-source heterogeneous data sources together, and the gathering mode comprises a relational database, data transmission, data synchronization, file import and the like, and can also execute the operations of S201-S205 in the embodiment of the invention. For data sources of different sources, the data access module can perform functions of database addition, data synchronization, data transmission and the like, so that data of the data sources are converged.
Optionally, after the data access module of the system performs data aggregation, the background may automatically generate: the system comprises a space library intermediate information table, a database connection information table, a transmission monitoring table and a data synchronization table. Wherein, the fields of the database intermediate information table include: one or more of a self-increment sequence, a database original table name, an adding time, an updating time, a data table data type, a data table alias, an updating time and an affiliated user. The database connection information table is used for storing external database connection information, and may specifically include: one or more fields of a self-increment sequence, a database connection name, a port number, a database name, a user name, a password, a database type, an insertion time, an update time, a state and a belonging user. The transmission monitoring table may be used to store data transmission information, and its fields include: one or more of a self-increment sequence, a transmission table name, a progress, a starting time, an ending time and an affiliated user. The data synchronization table may be used to store data synchronization information, and its fields may include: one or more of a self-increment sequence, a target table name, a source database ID, a source total table number, an interval time, a timing start time, a start time, an end time, a day of week, a synchronization progress, a remark, a time consumption matching field and a belonging user.
Optionally, the user may set a timing task for the aggregated data to be automatically synchronized, so as to implement real-time data update. The spatial address matching can be carried out on the place name address data while the data are gathered, and the problem of conversion from non-spatial data to spatial data is solved.
The data processing module of the system may include a data modeling unit and an SQL (Structured Query Language) unit, and the specific data modeling is a process of determining a data processing model described in this embodiment, and specifically, the data modeling may be implemented by dragging an optional component such as a common function or data filtering, and the data processing model that is modeled is used to process data, or the SQL script may be written by using a common SQL statement and an aggregation function, and the processing of data is implemented by executing a code.
Optionally, the data processing module of the system may automatically generate a model information table, a timing task table, and a model structure string according to the stored data processing model, store different types of model information, and facilitate subsequent use. Specifically, the fields of the model information table include: the model structure JSON string comprises a model unique code, a model name, model description information, a model execution state, a model execution time, a model execution type, a model execution progress, a model execution time, a model creation time, a model modification time, a user to which the model belongs, and a model structure JSON (JavaScript Object Notation) string.
The visualization module of the system may include a topic configuration unit for setting visualization display, specifically may configure spatial chart data to be displayed in a map, statistics or chart manner, and may further include a topic visualization function, i.e., a display function of a large-screen signboard, for example, may visually display a topic catalog such as an artificial topic, a business topic, and the like.
The visualization module supports creation of general statistical charts such as pie charts, bar charts, pyramid charts, radar charts and the like; the visual expression method of thematic maps such as a scatter diagram, a grid diagram, a grade color diagram, a thermodynamic diagram and a density diagram is supported, the visual expression method comprises the functions of spatial big data thematic service setting, thematic map element attribute configuration, statistical chart making and the like, a customization tool is provided for the online display of spatial big data, and the automatic rapid generation and timely publishing and displaying of various thematic maps and statistical charts in various industries can be realized. The operation steps of the user in the visualization module can be as follows: 1) creating special topic service, and selecting a data source to be configured with a chart; 2) selecting the type of the chart to be configured; 3) setting chart display parameters and display styles; 4) generating a visual chart; 5) and configuring a visual display interface and publishing data.
The background management module of the system can monitor and manage the model, specifically, after a data processing model is established by user definition, the model is stored, and related timing configuration time is set; the background management module of the system can also comprise the management of the script, namely, the maintenance management of the code independently written by the user; the background management module can also comprise synchronous monitoring, specifically, timing synchronization of data can be set, automatic synchronization is carried out when the data are detected to change, or the timing automatic synchronization is carried out at preset time intervals; the background management module may further include management of a system log, and specifically may include management of a data access log and a data processing log.
The scheme provided by the embodiment of the invention provides a big data processing system for realizing data processing and visual display, which can automatically process spatial big data and further realize flexible customization of a data analysis processing process and visual display contents.
Example four
Fig. 4 is a block diagram of a big data processing apparatus according to a fourth embodiment of the present invention, where the big data processing apparatus according to the fourth embodiment of the present invention is capable of executing a big data processing method according to any embodiment of the present invention, and has functional modules and beneficial effects corresponding to the execution method.
The big data processing apparatus may include: an obtaining module 401, a obtaining module 402 and an output module 403.
The acquiring module 401 is configured to determine a data processing model based on model configuration operation of a user on a visual interface, and acquire to-be-processed spatial data corresponding to the data processing model from a local database;
an obtaining module 402, configured to process the spatial data to be processed through the data processing model to obtain a processing result;
an output module 403, configured to determine, based on an open geospatial information federation OGC standard service and/or a vector slicing map service, and a display parameter configured by a user on the visualization interface, spatial chart data corresponding to the processing result, and output the spatial chart data on the visualization interface.
According to the scheme provided by the embodiment of the invention, based on model configuration operation of a user on a visual interface, a data processing model is determined, the to-be-processed spatial data corresponding to the data processing model is obtained from a local database, the to-be-processed spatial data is processed through the data processing model to obtain a processing result, and based on open geographic space information alliance OGC standard service and/or vector slice map service and display parameters configured on the visual interface by the user, the spatial chart data corresponding to the processing result is determined, and the spatial chart data is output on the visual interface. By the method, the spatial big data can be automatically processed, and the flexible customization of the data analysis processing process and the visual display content is further realized.
Further, the output module 403 is specifically configured to:
analyzing the processing result based on OGC standard service and/or vector slice map service;
and generating the spatial chart data corresponding to the processing result according to the analysis result and the display parameters configured on the visual interface by the user.
Further, the obtaining module 401 may include:
the model acquisition unit is used for acquiring a data processing model meeting the timed starting requirement at the current moment based on the timed starting configuration operation of a user on the existing model on a visual interface; and/or the presence of a gas in the gas,
the model generation unit is used for determining the connection relation between the dragging component and the dragging component based on the dragging configuration operation of the user on the selectable component on the visual interface, and generating a data processing model according to the connection relation between the dragging component and the dragging component.
Further, the obtaining module 401 may be further configured to:
after a data processing model is generated, the data processing model is saved, and timing starting parameters are configured for the data processing model.
Further, the obtaining module 402 is specifically configured to:
and performing at least one of grid-to-vector processing, fusion processing, clustering processing, area analysis processing, address matching processing, density analysis processing and grid analysis processing on the spatial data to be processed through the data processing model.
Further, the above apparatus further comprises:
the updating data acquisition module is used for acquiring data to be updated if a data updating event is detected;
the data obtaining module is used for performing spatialization processing and cleaning processing on the data to be updated to obtain spatial data to be added;
the result obtaining module is used for carrying out online plotting on the base map according to the spatial data to be added to obtain a plotting result;
and the storage module is used for storing the spatial data to be added and the plotting result in a local database.
Further, the detecting of the data update event includes at least one of:
detecting a relational database access event;
detecting that at least one of newly added data, deleted data and changed data exists in the relational database;
and detecting that the current moment meets the automatic data import period.
EXAMPLE five
Fig. 5 is a schematic structural diagram of an electronic device according to a fifth embodiment of the present invention, and fig. 5 shows a block diagram of an exemplary device suitable for implementing the embodiment of the present invention. The device shown in fig. 5 is only an example and should not bring any limitation to the function and the scope of use of the embodiments of the present invention.
As shown in FIG. 5, electronic device 12 is embodied in the form of a general purpose computing device. The components of the electronic device 12 may include, but are not limited to: one or more processors or processing units 16, a system memory 28, and a bus 18 that couples various system components including the system memory 28 and the processing unit 16.
Bus 18 represents one or more of any of several types of bus structures, including a memory bus or memory controller, a peripheral bus, an accelerated graphics port, and a processor or local bus using any of a variety of bus architectures. By way of example, such architectures include, but are not limited to, Industry Standard Architecture (ISA) bus, micro-channel architecture (MAC) bus, enhanced ISA bus, Video Electronics Standards Association (VESA) local bus, and Peripheral Component Interconnect (PCI) bus.
Electronic device 12 typically includes a variety of computer system readable media. Such media may be any available media that is accessible by electronic device 12 and includes both volatile and nonvolatile media, removable and non-removable media.
The system memory 28 may include computer system readable media in the form of volatile memory, such as Random Access Memory (RAM)30 and/or cache memory (cache 32). The electronic device 12 may further include other removable/non-removable, volatile/nonvolatile computer system storage media. By way of example only, storage system 34 may be used to read from and write to non-removable, nonvolatile magnetic media (not shown in FIG. 5, and commonly referred to as a "hard drive"). Although not shown in FIG. 5, a magnetic disk drive for reading from and writing to a removable, nonvolatile magnetic disk (e.g., a "floppy disk") and an optical disk drive for reading from or writing to a removable, nonvolatile optical disk (e.g., a CD-ROM, DVD-ROM, or other optical media) may be provided. In these cases, each drive may be connected to bus 18 by one or more data media interfaces. System memory 28 may include at least one program product having a set (e.g., at least one) of program modules that are configured to carry out the functions of embodiments of the invention.
A program/utility 40 having a set (at least one) of program modules 42 may be stored, for example, in system memory 28, such program modules 42 including, but not limited to, an operating system, one or more application programs, other program modules, and program data, each of which examples or some combination thereof may comprise an implementation of a network environment. Program modules 42 generally carry out the functions and/or methodologies of embodiments described herein.
Electronic device 12 may also communicate with one or more external devices 14 (e.g., keyboard, pointing device, display 24, etc.), with one or more devices that enable a user to interact with electronic device 12, and/or with any devices (e.g., network card, modem, etc.) that enable electronic device 12 to communicate with one or more other computing devices. Such communication may be through an input/output (I/O) interface 22. Also, the electronic device 12 may communicate with one or more networks (e.g., a Local Area Network (LAN), a Wide Area Network (WAN), and/or a public network, such as the Internet) via the network adapter 20. As shown, the network adapter 20 communicates with other modules of the electronic device 12 via the bus 18. It should be understood that although not shown in FIG. 5, other hardware and/or software modules may be used in conjunction with electronic device 12, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems, among others.
The processing unit 16 executes various functional applications and data processing by executing programs stored in the system memory 28, for example, implementing a big data processing method provided by an embodiment of the present invention.
EXAMPLE six
The sixth embodiment of the present invention further provides a computer-readable storage medium, on which a computer program (or referred to as computer-executable instructions) is stored, where the computer program is used for executing the big data processing method provided by the sixth embodiment of the present invention when the computer program is executed by a processor.
Computer storage media for embodiments of the invention may employ any combination of one or more computer-readable media. The computer readable medium may be a computer readable signal medium or a computer readable storage medium. A computer readable storage medium may be, for example, but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination of the foregoing. More specific examples (a non-exhaustive list) of the computer readable storage medium would include the following: an electrical connection having one or more wires, a portable computer diskette, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing. In the context of this document, a computer readable storage medium may be any tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device.
A computer readable signal medium may include a propagated data signal with computer readable program code embodied therein, for example, in baseband or as part of a carrier wave. Such a propagated data signal may take many forms, including, but not limited to, electro-magnetic, optical, or any suitable combination thereof. A computer readable signal medium may also be any computer readable medium that is not a computer readable storage medium and that can communicate, propagate, or transport a program for use by or in connection with an instruction execution system, apparatus, or device.
Program code embodied on a computer readable medium may be transmitted using any appropriate medium, including but not limited to wireless, wireline, optical fiber cable, RF, etc., or any suitable combination of the foregoing.
Computer program code for carrying out operations for embodiments of the present invention may be written in any combination of one or more programming languages, including an object oriented programming language such as Java, Smalltalk, C + +, and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The program code may execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network, including a Local Area Network (LAN) or a Wide Area Network (WAN), or the connection may be made to an external computer (for example, through the Internet using an Internet service provider).
It is to be noted that the foregoing is only illustrative of the preferred embodiments of the present invention and the technical principles employed. It will be understood by those skilled in the art that the present invention is not limited to the particular embodiments described herein, but is capable of various obvious changes, rearrangements and substitutions as will now become apparent to those skilled in the art without departing from the scope of the invention. Therefore, although the embodiments of the present invention have been described in more detail through the above embodiments, the embodiments of the present invention are not limited to the above embodiments, and many other equivalent embodiments may be included without departing from the spirit of the present invention, and the scope of the present invention is determined by the scope of the appended claims.

Claims (10)

1. A big data processing method is characterized by comprising the following steps:
determining a data processing model based on model configuration operation of a user on a visual interface, and acquiring to-be-processed spatial data corresponding to the data processing model from a local database;
processing the spatial data to be processed through the data processing model to obtain a processing result;
and determining the spatial chart data corresponding to the processing result based on the open geographic spatial information alliance OGC standard service and/or the vector slice map service and the display parameters configured on the visual interface by the user, and outputting the spatial chart data on the visual interface.
2. The method according to claim 1, wherein the determining the spatial chart data corresponding to the processing result based on OGC standard service and/or vector slice map service and the display parameters configured by the user on the visual interface comprises:
analyzing the processing result based on OGC standard service and/or vector slice map service;
and generating the spatial chart data corresponding to the processing result according to the analysis result and the display parameters configured on the visual interface by the user.
3. The method of claim 1, wherein determining the data processing model based on the model configuration operation of the user at the visualization interface comprises:
based on the timed starting configuration operation of a user on the existing model on a visual interface, acquiring a data processing model meeting the timed starting requirement at the current moment; and/or the presence of a gas in the gas,
determining a connection relation between a dragging component and the dragging component based on dragging configuration operation of a user on the selectable component on a visual interface, and generating a data processing model according to the connection relation between the dragging component and the dragging component.
4. The method of claim 3, after generating the data processing model, further comprising:
and storing the data processing model, and configuring a timing starting parameter for the data processing model.
5. The method according to claim 1, wherein the processing the spatial data to be processed by the data processing model comprises:
and performing at least one of grid-to-vector processing, fusion processing, clustering processing, area analysis processing, address matching processing, density analysis processing and grid analysis processing on the spatial data to be processed through the data processing model.
6. The method according to any one of claims 1-5, further comprising:
if a data updating event is detected, acquiring data to be updated;
carrying out spatialization processing and cleaning processing on the data to be updated to obtain spatial data to be added;
according to the spatial data to be added, performing online plotting on the base map to obtain a plotting result;
and storing the spatial data to be added and the plotting result in a local database.
7. The method of claim 6, wherein the detecting of the data update event comprises at least one of:
detecting a relational database access event;
detecting that at least one of newly added data, deleted data and changed data exists in the relational database;
and detecting that the current moment meets the automatic data import period.
8. A big data processing apparatus, comprising:
the acquisition module is used for determining a data processing model based on model configuration operation of a user on a visual interface and acquiring to-be-processed spatial data corresponding to the data processing model from a local database;
the obtaining module is used for processing the spatial data to be processed through the data processing model to obtain a processing result;
and the output module is used for determining the spatial chart data corresponding to the processing result based on the open geographic spatial information alliance OGC standard service and/or the vector slice map service and the display parameters configured on the visual interface by the user, and outputting the spatial chart data on the visual interface.
9. An electronic device, comprising:
one or more processors;
a memory for storing one or more programs;
when executed by the one or more processors, cause the one or more processors to implement a big data processing method as claimed in any of claims 1-7.
10. A computer-readable storage medium, on which a computer program is stored, which, when being executed by a processor, carries out a big data processing method according to any one of claims 1 to 7.
CN202210287079.5A 2022-03-22 2022-03-22 Big data processing method, device, equipment and medium Pending CN114610923A (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202210287079.5A CN114610923A (en) 2022-03-22 2022-03-22 Big data processing method, device, equipment and medium

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202210287079.5A CN114610923A (en) 2022-03-22 2022-03-22 Big data processing method, device, equipment and medium

Publications (1)

Publication Number Publication Date
CN114610923A true CN114610923A (en) 2022-06-10

Family

ID=81864102

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202210287079.5A Pending CN114610923A (en) 2022-03-22 2022-03-22 Big data processing method, device, equipment and medium

Country Status (1)

Country Link
CN (1) CN114610923A (en)

Cited By (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN116069202A (en) * 2023-03-09 2023-05-05 苏州傲林科技有限公司 Operating condition radar chart processing method and device
WO2024041134A1 (en) * 2022-08-22 2024-02-29 腾讯科技（深圳）有限公司 Data stream signage generation method and apparatus, and computer device and storage medium

Cited By (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2024041134A1 (en) * 2022-08-22 2024-02-29 腾讯科技（深圳）有限公司 Data stream signage generation method and apparatus, and computer device and storage medium
CN116069202A (en) * 2023-03-09 2023-05-05 苏州傲林科技有限公司 Operating condition radar chart processing method and device

Similar Documents

Publication Publication Date Title
US10635276B2 (en) Cohort-based presentation of user interaction data
EP3557443A1 (en) Resource dependency system and graphical user interface
US10140739B1 (en) User interface for automatically generating charts
US20180052572A1 (en) Modifying style layer properties of a digital map
US9934299B2 (en) Systems and methods for interest-driven data visualization systems utilizing visualization image data and trellised visualizations
US10579678B2 (en) Dynamic hierarchy generation based on graph data
CN114610923A (en) Big data processing method, device, equipment and medium
CN103984755A (en) Multidimensional model based oil and gas resource data key system implementation method and system
WO2015060893A1 (en) Systems and methods for interest-driven data visualization systems utilizing visualization image data and trellised visualizations
WO2017206484A1 (en) Geographic data presentation method and apparatus
US11809446B2 (en) Visualizing time metric database
US20230015068A1 (en) Unified data model and interface for databases storing disparate types of data
CN103744676A (en) Method for displaying relevance of multi-level information
US11416514B2 (en) Interactive dynamic geo-spatial application with enriched map tiles
CN109766373A (en) Electric network data methods of exhibiting and computer readable storage medium
CN113010696A (en) Engineering field knowledge graph construction method based on metadata model
US10394844B2 (en) Integrating co-deployed databases for data analytics
CN109360106A (en) Portrait construction method, system, medium and computer system
US20160154781A1 (en) Semantic Modeling of Geographic Information in Business Intelligence
CN111859187A (en) POI query method, device, equipment and medium based on distributed graph database
CN115640300A (en) Big data management method, system, electronic equipment and storage medium
Merritt et al. Big Earth Data: a comprehensive analysis of visualization analytics issues
CN109033157B (en) Complex data retrieval method and system based on user-defined search condition tree
CN110502595A (en) A method of picking up longitude and latitude information from planning map
CN106599241A (en) Big data visual management method for GIS software

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination