CN113641497B - System for realizing distributed high concurrency data summarization based on dimension reduction segmentation technology - Google Patents
System for realizing distributed high concurrency data summarization based on dimension reduction segmentation technology Download PDFInfo
- Publication number
- CN113641497B CN113641497B CN202110935417.7A CN202110935417A CN113641497B CN 113641497 B CN113641497 B CN 113641497B CN 202110935417 A CN202110935417 A CN 202110935417A CN 113641497 B CN113641497 B CN 113641497B
- Authority
- CN
- China
- Prior art keywords
- data
- module
- computing node
- computing
- model
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 230000009467 reduction Effects 0.000 title claims abstract description 13
- 238000005516 engineering process Methods 0.000 title claims description 8
- 230000011218 segmentation Effects 0.000 title claims description 8
- 238000012545 processing Methods 0.000 claims abstract description 36
- 238000000034 method Methods 0.000 claims abstract description 16
- 238000004364 calculation method Methods 0.000 claims description 19
- 238000012821 model calculation Methods 0.000 claims description 18
- 238000007405 data analysis Methods 0.000 claims description 9
- 238000009877 rendering Methods 0.000 claims description 7
- 238000007619 statistical method Methods 0.000 claims description 6
- 230000002085 persistent effect Effects 0.000 claims description 4
- 238000012546 transfer Methods 0.000 claims description 4
- 238000006243 chemical reaction Methods 0.000 claims description 3
- 230000002776 aggregation Effects 0.000 claims description 2
- 238000004220 aggregation Methods 0.000 claims description 2
- 230000002045 lasting effect Effects 0.000 claims 1
- 239000003344 environmental pollutant Substances 0.000 description 9
- 231100000719 pollutant Toxicity 0.000 description 9
- 238000004088 simulation Methods 0.000 description 9
- 230000005540 biological transmission Effects 0.000 description 8
- 238000009792 diffusion process Methods 0.000 description 8
- 230000008569 process Effects 0.000 description 7
- 238000004458 analytical method Methods 0.000 description 6
- 230000006870 function Effects 0.000 description 5
- 238000011160 research Methods 0.000 description 4
- 230000000903 blocking effect Effects 0.000 description 3
- 230000008859 change Effects 0.000 description 3
- 230000007613 environmental effect Effects 0.000 description 3
- 239000012634 fragment Substances 0.000 description 3
- 239000000779 smoke Substances 0.000 description 3
- 239000002689 soil Substances 0.000 description 3
- 238000003745 diagnosis Methods 0.000 description 2
- 230000003068 static effect Effects 0.000 description 2
- 230000009286 beneficial effect Effects 0.000 description 1
- 238000004891 communication Methods 0.000 description 1
- 239000000356 contaminant Substances 0.000 description 1
- 230000007547 defect Effects 0.000 description 1
- 238000011161 development Methods 0.000 description 1
- 230000018109 developmental process Effects 0.000 description 1
- 238000010586 diagram Methods 0.000 description 1
- 238000009499 grossing Methods 0.000 description 1
- 230000005012 migration Effects 0.000 description 1
- 238000013508 migration Methods 0.000 description 1
- 230000002688 persistence Effects 0.000 description 1
- 238000007781 pre-processing Methods 0.000 description 1
- 238000003908 quality control method Methods 0.000 description 1
- 238000004062 sedimentation Methods 0.000 description 1
- 239000000126 substance Substances 0.000 description 1
- 230000002123 temporal effect Effects 0.000 description 1
- 238000012876 topography Methods 0.000 description 1
- XLYOFNOQVPJJNP-UHFFFAOYSA-N water Substances O XLYOFNOQVPJJNP-UHFFFAOYSA-N 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/50—Allocation of resources, e.g. of the central processing unit [CPU]
- G06F9/5061—Partitioning or combining of resources
- G06F9/5072—Grid computing
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F30/00—Computer-aided design [CAD]
- G06F30/20—Design optimisation, verification or simulation
- G06F30/23—Design optimisation, verification or simulation using finite element methods [FEM] or finite difference methods [FDM]
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/50—Allocation of resources, e.g. of the central processing unit [CPU]
- G06F9/5005—Allocation of resources, e.g. of the central processing unit [CPU] to service a request
- G06F9/5011—Allocation of resources, e.g. of the central processing unit [CPU] to service a request the resources being hardware resources other than CPUs, Servers and Terminals
- G06F9/5016—Allocation of resources, e.g. of the central processing unit [CPU] to service a request the resources being hardware resources other than CPUs, Servers and Terminals the resource being the memory
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/50—Allocation of resources, e.g. of the central processing unit [CPU]
- G06F9/5005—Allocation of resources, e.g. of the central processing unit [CPU] to service a request
- G06F9/5011—Allocation of resources, e.g. of the central processing unit [CPU] to service a request the resources being hardware resources other than CPUs, Servers and Terminals
- G06F9/5022—Mechanisms to release resources
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/50—Allocation of resources, e.g. of the central processing unit [CPU]
- G06F9/5005—Allocation of resources, e.g. of the central processing unit [CPU] to service a request
- G06F9/5027—Allocation of resources, e.g. of the central processing unit [CPU] to service a request the resource being a machine, e.g. CPUs, Servers, Terminals
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L47/00—Traffic control in data switching networks
- H04L47/10—Flow control; Congestion control
- H04L47/41—Flow control; Congestion control by acting on aggregated flows or links
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F2111/00—Details relating to CAD techniques
- G06F2111/02—CAD in a network environment, e.g. collaborative CAD or distributed simulation
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F2113/00—Details relating to the application field
- G06F2113/08—Fluids
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F2209/00—Indexing scheme relating to G06F9/00
- G06F2209/50—Indexing scheme relating to G06F9/50
- G06F2209/5017—Task decomposition
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02D—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE
- Y02D10/00—Energy efficient computing, e.g. low power processors, power management or thermal management
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Software Systems (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Mathematical Physics (AREA)
- Computer Hardware Design (AREA)
- Evolutionary Computation (AREA)
- Geometry (AREA)
- Computer Networks & Wireless Communication (AREA)
- Signal Processing (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The method is realized based on the high concurrency data summarizing system, and comprises a data processing system and a computing node system; slicing the data according to the latitude of time, space and the like of the data, and processing the data concurrently, so that the more the data slices are, the more the data overall summarizing efficiency can be greatly improved along with the increase of the computing node servers; after the node server gathers all the data of the server, the data is subjected to dimension reduction, and the method greatly improves the data processing efficiency, improves the network utilization efficiency, reduces the network pressure and simultaneously reduces the requirement on server hardware.
Description
Technical Field
The invention relates to the technical field of atmospheric environment and computer processing, in particular to a system for realizing distributed high concurrency data summarization based on a dimension-reduction segmentation technology.
Background
In the distributed complex calpff model calculation process, a large amount of model calculation result data is generated. The data result calculated by the calpff model is multidimensional. For example, calculating a calpff case requires a time latitude, a coordinate latitude, and finally a contaminant latitude.
In general, the time period of one calpff case calculation is about one week, but can reach one year under certain specific calculation scenes; the calculated coordinate latitude is the number of model grids, which is generally ten thousand to tens of thousands, and can reach millions in special scenes; the calculated pollutant latitude, namely the number of pollutants, is generally 5, and hundreds of pollutants can be achieved in a special scene. Since the size of these data increases linearly with the complexity of the model and the time of computation, it can range from a few hundred MB to hundreds of GB. In this way, the summary and statistical analysis of the result data of the model calculation place high demands, especially for the transmission and summary of several hundred GB data in a distributed environment.
In the prior art, data transmission can be performed through various different protocols, each protocol is adapted to different scenes, and there are schemes suitable for large data transmission like TCP protocol, FTP protocol, etc. Although these protocols can solve the single problem of big data transmission, in the calculation scene of the distributed complex calpff model, multiple situations such as IO performance, calculation performance, memory use and the like need to be considered, and factors of hardware cost and calculation efficiency need to be comprehensively considered. Therefore, a single solution cannot effectively solve all the problems. Thus, the present solution combines the above problems, providing a solution that can be adapted to the above usage scenario, meeting the above requirements.
Disclosure of Invention
The invention provides a system for realizing distributed high concurrency data summarization based on a dimension-reduction segmentation technology, which aims to solve the problems that the prior art cannot solve the multiple conditions of IO, calculation efficiency, memory use and the like in a distributed complex CALPUFF model calculation scene when adopting a communication protocol for data transmission.
The system for realizing distributed high concurrency data summarization based on the dimension reduction segmentation technology comprises a data processing system and a computing node system;
The computing node system consists of a plurality of computing node servers, wherein the CALPUFF model computing cases are randomly distributed to the corresponding computing node servers, and each computing node server comprises a WRF computing model computing module, a CALMET model computing module and a CALPUFF model computing module; each computing node server performs slicing according to the result of model calculation, then performs local summarization, and then transmits the data which are locally summarized by each computing node server to a data processing system for final summarization;
the data processing system comprises a data summarizing module, a data analyzing module, a data restoring module and a picture rendering module;
the data summarizing module is used for summarizing the local summarized data transmitted by each computing node server in real time;
The data analysis module is used for carrying out secondary data analysis on the final calculation result of the CALMET and CALPUFF cases;
the data transfer module completes the persistent storage of various data; the data to be stored in a persistent manner comprises result data calculated by the CALMET and CALPUFF models and data subjected to secondary statistical analysis;
the picture rendering module renders the data into a picture with a formulated format; the data content to be rendered includes the final computed result data of CALMET and calpff cases.
The invention has the beneficial effects that:
1. The data processing efficiency is high. The invention adopts the data slicing and the grading summarization mode, thereby greatly improving the efficiency of data summarization. The number of the data fragments is determined according to the complexity of the CALPUFF model case, and under the condition that the number of the fragments is fixed, the more the computing node servers are, the shorter the time required for the data fragments to complete the primary summary is. Therefore, the time for data primary summarization can be greatly reduced by adding the computing node server. The data secondary summary is based on the concurrency setting of the data processing server, and since the data processing service is a single server, the concurrency number cannot be infinitely increased, but in general, the data secondary summary can be set to 70% of the number of the CPU of the server, for example, a server with 20 CPUs, and 14 ready-made data secondary summaries can be simultaneously performed. Therefore, the efficiency is greatly improved compared with the original mode. And the analysis processing time of the calculation result data of the complex CALPUFF model case is greatly reduced by integrating the concurrent quantity of the first stage and the second stage.
2. The data transmission efficiency is greatly improved. The calculation result of the complex calpff model can reach the data magnitude of hundreds of GB, so that large data transmission on the network needs to consume a great deal of time, and network blocking is easy to cause, so that the use of other systems in the network is affected. The size of the data can be greatly reduced in a data dimension reduction mode.
3. And the IO is greatly reduced. Because the invention adopts the data slicing mode, the number of each data slicing is not large, the data slicing can be directly loaded into the memory at one time, rather than decomposing the calculation result of the CALPUFF model into a plurality of small files through CALPOST, and then analyzing the small files one by one.
4. The peak value of the memory is greatly reduced. Because the scheme adopts the data slicing mode, in the data primary summarizing process of each computing node, the whole data slicing can be read into the memory, and the memory is cleaned after the whole data slicing is rapidly processed, so that the amount of the memory used by the system cannot be increased along with the increase of the slicing amount. The data volume of each data slice is not large, so that the requirement of the system on the use of the memory is greatly reduced.
5. The method of the invention greatly improves the data processing efficiency, improves the network utilization efficiency, reduces the network pressure, and simultaneously reduces the requirement on the hardware of the server.
Drawings
Fig. 1 is a schematic diagram of a system for implementing distributed high concurrency data aggregation based on a dimension reduction segmentation technique according to the present invention.
Detailed Description
With reference to fig. 1, the system for implementing distributed high concurrency data summarization based on the dimension reduction segmentation technology is described in this embodiment, and the system is implemented by a computing node system and a data processing system. The computation results of the complex calpff model are sliced and distributed into several compute node servers. The computing node servers firstly carry out primary summarization on the slice data of the sub-cases, and then transmit the data to the data processing server through a network. The data processing server performs secondary summarization on the primary summarized data so as to obtain a final data result, and then develops subsequent processing.
The computing node system comprises a plurality of computing node servers and is used for integrating complex computing flows, completing computing tasks of all links, locally summarizing computing results and then sending the computing results to the data processing servers for final summarizing. Each computing node server comprises a WRF computing model computing module, a CALMET model computing module and a CALPUFF model computing module. The respective modules will be described below.
The WRF (WEATHER RESEARCH Forecast) model is a new generation of mesoscale forecasting models and assimilation systems that are jointly engaged in development studies by many scientists at universities in the united states research division. The WRF mode is a fully compressible non-static mode, adopts an Arakawa C grid, combines an advanced numerical method and a data assimilation technology, adopts an improved multiple physical process parameter scheme, has multiple nesting and easy positioning capabilities at different geographic positions, integrates numerical weather forecast, atmospheric simulation and data assimilation into a mode system, and can better improve the simulation and forecast of medium-scale weather from meters to thousands of kilometers.
The WRF Preprocessing System (WPS) is a module consisting of three programs whose function is to prepare the input field for real data simulation. The respective uses of the three procedures were: geogrid determining a mode region (including longitude and latitude range, center point coordinates, grid nesting, grid number in horizontal direction and resolution size of the region) and interpolating topographic data (including topographic elevation, land utilization type, vegetation coverage, soil category and the like) of the static topographic data to the grid points; ungrib extracting a meteorological element field from GRIB format data; metgird interpolates the extracted meteorological element field levels to grid points determined by geogrid.
The WRF-ARW/NMM system is a core module of the WRF mode, has the advantages of high efficiency, easiness in mastering, capability of parallel operation and the like, the data is required to be processed by a real program before WRF programs are operated, meteorological element data interpolated by metgrid programs are identified by the real program of the WRF mode, required boundary layer files are generated after vertical interpolation to an eta layer of the WRF mode, and boundary condition initialization is performed.
The WRF model data comprises meteorological data and basic geographic data
The weather data refer to weather re-analysis data FNL data adopted for simulating historical weather, are global grid point data which are jointly manufactured and released by a national environmental forecast center (NCEP) and a national atmospheric research center (NCAR) in the NCEP official network, and adopt the most advanced global data assimilation system and perfect database at present to carry out quality control and assimilation treatment on observation data of various data sources (ground, ships, radiosonde, satellites and the like), so that a complete set of analysis data is obtained, and the method has the characteristics of more time, high density, strong continuity, high resolution, abundant content and the like, and can effectively make up the defects of conventional observation data in the aspect of disastrous weather analysis.
The FNL data has a spatial resolution of 1 DEG x1 DEG and a temporal resolution of 6 hours. Four times 0, 6, 12, 18 of world time per day were subjected to a global data analysis. The data content comprises the content of air pressure, temperature, relative humidity, rainfall and the like. The data format is divided into GRIB and GRIB, wherein GRIB data time is 1999.07.30 to 2007.12.06, GRIB2 data time is 2007.12.06 and is continuously updated. The weather data format provided this time is GRIB < 2 >.
The WRF mode base geographic data includes: terrain elevation, land use type, other underlying surface information, and the like. Terrain data GTOP030, resolution 30", land use type data USGS, MODIS, USGS contains 24 types of land types, MODIS contains 20 land types, satellite land cover product data, resolution is up to 30". Other underlying data includes vegetation type, soil moisture, soil texture, and the like.
The WRF mode operation setting comprises basic data input, control file setting and physical parameter setting, wherein the basic data comprise data such as topography, land utilization type and the like, weather forecast grid point data GFS, historical weather analysis data FNL and ground and high altitude observation data. The simulation settings comprise basic simulation settings such as nested grid range, map projection, simulation time and the like, the WRF mode provides various physical parameterization schemes, and proper schemes are selected, so that simulation and forecast of the medium-scale weather can be improved better.
The CALMET model is a diagnostic wind farm calculation model developed by SIGMA RESEARCH Corporation (now a subsidiary of EARTH TECH, inc) recommended by the united states EPA. The CALMET is a meteorological module for describing an hour wind field and a temperature field in a three-dimensional grid simulation domain by utilizing a mass conservation continuous equation, the core part of the meteorological module comprises a diagnosis wind field and a microclimate field mode, the diagnosis wind field module carries out terrain dynamics, slope flow and terrain blocking effect adjustment on an initial guessed wind field (a mesoscale mode output wind field, conventionally monitored ground and high-altitude image data) to generate a first step wind field, and the first step wind field is used for guiding observation data and generating a final wind field through interpolation, smoothing, vertical velocity calculation, divergence minimization and the like. The CALMET module takes the dynamic influence of terrain, oblique airflow and blocking effects into account in detail in the three-dimensional wind field simulation process.
The calpff model is an air quality diffusion model recommended by the united states environmental protection agency EPA (Environmental Protection Agency) and developed by SIGMA RESEARCH Corporation (now a subsidiary of EARTH TECH, inc), and consists of a CALMET weather module, a calpff smoke mass diffusion module and a CALPOST post-treatment module, and is a gaussian smoke mass diffusion model for simulating unstable multi-layer and multi-species pollution (such as SO 2、NOx and the like), and the migration diffusion process, the dry-wet sedimentation process and the basic chemical conversion process of different pollutants under the weather conditions of time and space change are considered. Considering the influence of complex terrain, water transmission, coastal boundary influence and subsidence influence of buildings, the concentration and the subsidence amount at preset points can be estimated by simulating pollutants discharged from a source in a advection diffusion mode.
Calpff mode computation is mainly divided into two parts: the system comprises a CALMET weather processing module and a CALPUFF smoke mass diffusion module, wherein the CALMET module is used for generating weather field files required by a CALPUFF main module. The calpff module is the main module of the model and is a multi-level Gaussian (Gaussian) diffusion model for simulating or predicting a variety of pollutants under unsteady and unsteady conditions. The concentration diffusion calculation can be carried out through the module, but related data of a pollution emission source is required to be input externally, SO that the mass concentration distribution condition of pollutants (such as SO 2, NOx and the like) under meteorological factors which change along with time and space positions is obtained.
The data processing system is a data processing server and mainly aims at processing and analyzing the calculation result. Model calculations result in a large number of data files, and the results of a complex case often contain billions of data. Data processing systems need real-time, efficient processing to analyze these data results and to spool the data into a data persistence layer. Then, the result data needs to be subjected to secondary analysis, and the result data needs to be rendered into pictures and other functions. The system comprises a data summarizing module, a data analyzing module, a data transfer module and a picture rendering module;
The main function of the data summarizing module is summarizing data. A complex case can produce a large number of sub-cases that can be randomly assigned to the model computing systems of the various nodes. After all distributed calculation is completed, the model calculation system of each node performs local summarization, and then needs to send the final summarization to the data processing system. Therefore, the main task of the data summarizing module is to accept the locally summarized data from the model computing system of each computing node, and then summarize the received data in real time.
The data analysis module is mainly used for carrying out secondary data analysis on the final calculation result of the CALMET and CALPUFF cases. Such as the contribution rate of the pollution source, the maximum floor concentration, wind speed and direction statistical analysis, etc.
The data transfer module mainly has the function of completing the persistent storage of various data. The data that needs to be persisted includes the result data of CALMET and calpff model calculations, as well as data of secondary statistical analysis.
The main function of the picture rendering module is to render data into pictures in a formulated format. The data content to be rendered includes the final computed result data of CALMET and calpff cases.
According to the high concurrency data summarizing method, CALPUFF model calculation cases are randomly scattered on different calculation node servers, each node performs local summarization according to model calculation results, and after data summarization of a certain node is finished, the data of the node are summarized on a data processing server to perform final summarization. The data dimension reduction refers to changing an original multidimensional data structure into one-dimensional data according to a specific coding mode, only retaining the final concentration value of pollutants in each grid, and removing the limit values of other dimensions.
According to the embodiment, the data can be processed in a concurrent manner in a slicing manner, the data is sliced according to the time dimension, the space dimension and the like of the data, and the data can be processed in a concurrent manner, so that the more the data slices are along with the increase of the computing node servers, the efficiency of the overall data summarization can be greatly improved. After the node server gathers all the data of the server, the data is subjected to dimension reduction, namely, the data streams in all the dimensions are spliced into one-dimensional data streams according to the format of the formulated data streams in each dimension, and then the one-dimensional data streams are transmitted to the data processing server. When the data processing server receives the one-dimensional data stream, the data is restored into multidimensional data through the agreed data format. The data processing server receives one-dimensional data of a plurality of computing node services, and performs data conversion and secondary summarization in real time every time data of one computing node is received, so that the data of all the computing node servers can be summarized secondarily. After all the data are summarized, the data processing server can perform subsequent data analysis and statistics and picture rendering.
The embodiment provides a way to segment and dimension down data to transmit and aggregate data. In the data slicing, in the present embodiment, in the process of calculating a complex calpff model in a distributed manner, the data calculated by the sub-cases are randomly dispersed to different calculation node servers, each node performs local summary according to the result of model calculation, and when the summary of data of a certain node is completed, the data of the node is summarized to the data processing server, so as to perform final summary. The data dimension reduction is to change the original multidimensional data structure into one-dimensional data according to a specific coding mode, only the final concentration value of pollutants in each grid is reserved, and the limit values of other dimensions are removed.
Claims (2)
1. A system for realizing distributed high concurrency data summarization based on a dimension reduction segmentation technology is characterized in that: the system for summarizing the data comprises a data processing system and a computing node system;
The computing node system consists of a plurality of computing node servers, wherein the CALPUFF model computing cases are randomly distributed to the corresponding computing node servers, and each computing node server comprises a WRF computing model computing module, a CALMET model computing module and a CALPUFF model computing module; each computing node server performs slicing according to the result of model calculation, then performs local summarization, and then transmits the data which are locally summarized by each computing node server to a data processing system for final summarization;
After each computing node server performs local summary, the data is subjected to dimension reduction, namely: splicing the data streams of all dimensions into one-dimensional data streams according to the format of the formulated data stream of each dimension, and then transmitting the one-dimensional data streams to a data processing system; when the data processing system receives the one-dimensional data stream, the data is restored into multidimensional data through the appointed data format;
The data processing system receives one-dimensional data of a plurality of computing node servers, and performs data conversion and secondary summarization in real time;
the data processing system comprises a data summarizing module, a data analyzing module, a data restoring module and a picture rendering module;
the data summarizing module is used for summarizing the local summarized data transmitted by each computing node server in real time;
the data analysis module is used for carrying out secondary data analysis on the result of final case calculation output by the CALMET model calculation module and the CALPUFF model calculation module;
The data transfer module completes the persistent storage of various data; the data to be stored in a lasting way comprises result data of the case and data of secondary statistical analysis output by a CALMET model calculation module and a CALPUFF model calculation module;
The picture rendering module renders the data into a picture with a formulated format; the data content to be rendered comprises result data which is finally calculated by the case and is output by the CALMET model calculation module and the CALPUFF model calculation module.
2. The system for implementing distributed high concurrency data aggregation based on the dimension reduction splitting technique of claim 1, wherein: the secondary data analysis comprises contribution rate to pollution sources, maximum landing concentration and wind speed and wind direction statistical analysis.
Applications Claiming Priority (2)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN2021108854946 | 2021-08-03 | ||
| CN202110885494 | 2021-08-03 |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN113641497A CN113641497A (en) | 2021-11-12 |
| CN113641497B true CN113641497B (en) | 2024-07-05 |
Family
ID=78421911
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202110935417.7A Active CN113641497B (en) | 2021-08-03 | 2021-08-16 | System for realizing distributed high concurrency data summarization based on dimension reduction segmentation technology |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN113641497B (en) |
Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110348746A (en) * | 2019-07-15 | 2019-10-18 | 北京三易思创科技有限公司 | Air quality impact evaluation method and apparatus based on single pollution sources |
| CN110472782A (en) * | 2019-08-01 | 2019-11-19 | 软通动力信息技术有限公司 | A kind of data determination method, device, equipment and storage medium |
Family Cites Families (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| KR20120101952A (en) * | 2011-03-07 | 2012-09-17 | 부경대학교 산학협력단 | Processing system for temperature field of diffusion model using weather diagnostic model and processing method therefor |
| CN106443833B (en) * | 2015-08-06 | 2019-02-19 | 中国电力科学研究院 | A Numerical Weather Forecast Method Based on Cloud Computing |
| CN107038244A (en) * | 2017-04-24 | 2017-08-11 | 北京北信源软件股份有限公司 | A kind of data digging method and device, a kind of computer-readable recording medium and storage control |
-
2021
- 2021-08-16 CN CN202110935417.7A patent/CN113641497B/en active Active
Patent Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110348746A (en) * | 2019-07-15 | 2019-10-18 | 北京三易思创科技有限公司 | Air quality impact evaluation method and apparatus based on single pollution sources |
| CN110472782A (en) * | 2019-08-01 | 2019-11-19 | 软通动力信息技术有限公司 | A kind of data determination method, device, equipment and storage medium |
Also Published As
| Publication number | Publication date |
|---|---|
| CN113641497A (en) | 2021-11-12 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| Emerton et al. | Developing a global operational seasonal hydro-meteorological forecasting system: GloFAS-Seasonal v1. 0 | |
| Oki et al. | Global assessment of current water resources using total runoff integrating pathways | |
| Zadra et al. | The next Canadian regional climate model | |
| Noh et al. | Applying sequential Monte Carlo methods into a distributed hydrologic model: lagged particle filtering approach with regularization | |
| Mazzoleni et al. | Exploring the influence of citizen involvement on the assimilation of crowdsourced observations: a modelling study based on the 2013 flood event in the Bacchiglione catchment (Italy) | |
| Finardi et al. | A deterministic air quality forecasting system for Torino urban area, Italy | |
| Giebel et al. | Shortterm forecasting using advanced physical modelling-the results of the anemos project | |
| CN113902981B (en) | Forest fire spreading prediction method and device, electronic equipment and storage medium | |
| CN112100922A (en) | Wind resource prediction method based on WRF and CNN convolutional neural network | |
| CN115481558A (en) | A method, system and device for inversion of air pollution emissions based on machine learning | |
| CN118260715A (en) | A method for processing meteorological data | |
| Villa et al. | A stochastic model of future extreme temperature events for infrastructure analysis | |
| Freitas et al. | The influence of the Arctic Oscillation on North Atlantic wind and wave climate by the end of the 21st century | |
| Chowdhury et al. | Evaluation and ranking of different gridded precipitation datasets for Satluj River basin using compromise programming and f-TOPSIS | |
| Gazulis et al. | Exploring the DNA of our regions: Classification of outputs from the SLEUTH model | |
| Quesada et al. | Cold waves still matter: characteristics and associated climatic signals in Europe | |
| Brun et al. | A high performance computing framework for continental-scale forest fire spread prediction | |
| CN113641497B (en) | System for realizing distributed high concurrency data summarization based on dimension reduction segmentation technology | |
| CN118332964B (en) | Prediction method and system for ocean near-inertial flow | |
| CN116384284B (en) | Red tide gridding forecasting method and system | |
| Just et al. | XIS-PM2. 5: A daily spatiotemporal machine-learning model for PM2. 5 in the contiguous United States | |
| CN111177982A (en) | Emergency decision command system and method | |
| CN113641744B (en) | Calculation method for realizing CALPUFF model by distributed high concurrency scheduling | |
| CN116882741A (en) | Method for dynamically and quantitatively evaluating super-standard flood disasters | |
| Roshny et al. | An assessment of a very severe cyclonic storm in the Arabian sea using the COSMO model |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |