CN109597795B

CN109597795B - High-efficiency processing system for roadbed compaction construction data

Info

Publication number: CN109597795B
Application number: CN201811485278.7A
Authority: CN
Inventors: 沈礼伟; 黄诗丰; 杨红; 刘理想; 朱亚林
Original assignee: Nanjing Tianchen Lida Electronic Technology Co ltd
Current assignee: Nanjing Tianchen Lida Electronic Technology Co ltd
Priority date: 2018-12-06
Filing date: 2018-12-06
Publication date: 2020-10-16
Anticipated expiration: 2038-12-06
Also published as: WO2020114155A1; CN109597795A

Abstract

The invention provides a high-efficiency processing system for roadbed compaction construction data, which comprises a data splitting unit, a data storage queue unit, a storage task scheduling thread pool unit, an original data storage unit, a data calculation task queue unit and a calculation task execution unit. The high-concurrency rapid storage of real-time data is realized by designing a reasonable data fragmentation rule and a storage task scheduling strategy based on a thread pool. Through the fragment storage of the data, the high-concurrency reading, calculation and processing of the original data are realized.

Description

High-efficiency processing system for roadbed compaction construction data

Technical Field

The invention relates to a high-concurrency storage method for original data of roadbed compaction construction; the invention relates to a high-concurrency calculation processing method for roadbed compaction construction data.

Background

During roadbed compaction construction, construction vehicles generate a large amount of construction data. How to rapidly store a large amount of real-time construction data is a technical difficulty, and how to rapidly analyze and calculate a large amount of original construction data is another technical difficulty. If the storage or calculation processing delay is too long, the real-time performance of the system is reduced, and the larger the data volume is, the lower the real-time performance is, and finally the significance of real-time monitoring is lost.

Disclosure of Invention

In order to solve the defects, the invention provides an efficient processing system for roadbed compaction construction data, and aims to solve the problems of high-concurrency rapid storage and high-concurrency rapid calculation processing of a large amount of construction original data. The high-concurrency rapid storage of real-time data is realized by designing a reasonable data fragmentation rule and a storage task scheduling strategy based on a thread pool. Through the fragment storage of the data, the high-concurrency reading, calculation and processing of the original data are realized.

The invention provides a roadbed compaction construction data efficient processing system which comprises a data splitting unit, a data storage queue unit, a storage task scheduling thread pool unit, an original data storage unit, a data calculation task queue unit and a calculation task execution unit.

In the system, the data splitting unit splits the original construction data according to a designed data splitting strategy, and the data belonging to the same segment are stored in the same database file.

In the system, after the data storage queue unit splits the original construction data into data packets, a data storage queue is created in units of the split data packets, and the queue is scheduled and managed by the data storage thread pool.

In the system, the storage task scheduling thread pool unit schedules the storage task according to the storage task queue condition by a policy.

In the system, the original data storage unit stores the original data in different database files according to different data packets.

In the system, the data calculation task queue unit is based on database files stored in segments, a calculation task is created from an original data storage file, the calculation task is placed in a queue, and the queue is managed by a calculation task scheduling thread pool.

In the system, the computing task execution unit schedules the data computing task according to the policy by scheduling the thread pool according to the computing task, and the computing task can be concurrently executed with a higher concurrency number because the data to be computed are in different data files.

The invention provides a high-efficiency processing system for roadbed compaction construction data, which has the following beneficial effects: the high-concurrency rapid storage of real-time data is realized by designing a reasonable data fragmentation rule and a storage task scheduling strategy based on a thread pool. High-concurrency reading, calculation and processing of original data are realized through the fragment storage of the data; a data splitting strategy is established to realize the block storage of data, a storage task scheduling strategy is established according to the split data, and the high-concurrency rapid storage of a large amount of real-time data is realized. According to the split data, a scheduling strategy of a calculation task is established, high-concurrency rapid calculation processing of a large amount of data is achieved, and rapid storage of calculation results is achieved.

Drawings

The invention and its features, aspects and advantages will become more apparent from reading the following detailed description of non-limiting embodiments with reference to the accompanying drawings. Like reference symbols in the various drawings indicate like elements. The drawings are not necessarily to scale, emphasis instead being placed upon illustrating the principles of the invention.

Fig. 1 and fig. 2 are a flow chart of the construction data track and gridding processing and a flow chart of the construction data extraction and picture generation, respectively.

Detailed Description

In the following description, numerous specific details are set forth in order to provide a more thorough understanding of the present invention. It will be apparent, however, to one skilled in the art, that the present invention may be practiced without one or more of these specific details. In other instances, well-known features have not been described in order to avoid obscuring the invention.

In the following description, for the purposes of explanation, numerous specific details are set forth in order to provide a thorough understanding of the present invention. The following detailed description of the preferred embodiments of the invention, however, the invention is capable of other embodiments in addition to those detailed.

The high-concurrency data storage and the high-concurrency data calculation processing are core technical functions of the roadbed compaction construction informatization system and are the foundation of construction real-time monitoring.

Referring to fig. 1 and 2, the roadbed compaction construction data efficient processing system provided by the invention comprises a data splitting unit, a data storage queue unit, a storage task scheduling thread pool unit, an original data storage unit, a data calculation task queue unit and a calculation task execution unit.

According to the invention, the data splitting unit splits the original construction data according to a designed data splitting strategy, and the data belonging to the same segment are stored in the same database file.

In a preferred but nonlimiting embodiment of the present invention, after the data storage queue unit splits the original construction data into data packets, a data storage queue is created in units of the split data packets, and the queue is scheduled and managed by the data storage thread pool.

In a preferred but nonlimiting embodiment of the present invention, the storage task scheduling thread pool unit schedules the storage tasks according to the storage task queue conditions and a certain policy, so that the storage tasks perform data storage with as large a concurrency number as possible, and the storage tasks can be concurrently executed with a larger concurrency number because the original data is stored in different database files in a packetized manner.

In a preferred but non-limiting embodiment of the present invention, the original data storage unit stores in different database files according to different data packets, which is the basis for concurrent storage of original data and also for concurrent processing of data.

In a preferred but nonlimiting embodiment of the present invention, the data calculation task queue unit is based on database files stored in segments, a calculation task is created from an original data storage file, the calculation task is placed in a queue, and the queue is managed by a calculation task scheduling thread pool.

In a preferred but nonlimiting embodiment of the present invention, the computing task execution unit schedules the data computing task with a certain policy according to the computing task scheduling thread pool, and since the data to be computed is in different data files, the computing task can be concurrently executed with a higher concurrency number.

According to the invention, the data splitting strategy is established to realize the block storage of the data, and the storage task scheduling strategy is established according to the split data to realize the high-concurrency and quick storage of a large amount of real-time data. According to the split data, a scheduling strategy of a calculation task is established, high-concurrency rapid calculation processing of a large amount of data is achieved, and rapid storage of calculation results is achieved.

In the invention, the block storage of the data is the basis of concurrent storage and concurrent computation. In the construction process, a large amount of data are transmitted to the server side, and the server side needs to receive and store real-time data. If the storage is processed in a single-entry manner, the storage entry becomes a bottleneck of the storage, and the larger the data volume is, the higher the storage delay is. Through the engineering attribute of the construction data, a set of data partitioning strategy is established, and the data belonging to the same partition is stored in the same database file. And establishing a data storage queue according to the database files stored in blocks, and putting the storage queue into a data storage thread pool for management. And the data storage thread pool reasonably schedules the storage tasks according to the set scheduling strategy, so that the high-concurrency execution of the data storage tasks is realized. And establishing a data calculation processing task queue by taking the database files stored in blocks as a basis. The block storage of the data provides a data basis for the concurrent execution of the calculation, and the data calculation task scheduling thread pool reasonably schedules the calculation tasks according to the set scheduling strategy, so that the high concurrent execution of the data calculation tasks is realized. The calculation result of each block is still stored in the database file of the block, and the storage operation is executed when the calculation task is completed, so that the concurrent execution of the storage operation of the calculation result is realized.

The invention realizes high-concurrency and rapid storage of real-time data by designing a reasonable data fragmentation rule and a storage task scheduling strategy based on a thread pool. High-concurrency reading, calculation and processing of original data are realized through the fragment storage of the data; a data splitting strategy is established to realize the block storage of data, a storage task scheduling strategy is established according to the split data, and the high-concurrency rapid storage of a large amount of real-time data is realized. According to the split data, a scheduling strategy of a calculation task is established, high-concurrency rapid calculation processing of a large amount of data is achieved, and rapid storage of calculation results is achieved.

The above description is of the preferred embodiment of the invention. It is to be understood that the invention is not limited to the particular embodiments described above, in that devices and structures not described in detail are understood to be implemented in a manner common in the art; those skilled in the art can make many possible variations and modifications to the disclosed embodiments, or modify equivalent embodiments to equivalent variations, without departing from the spirit of the invention, using the methods and techniques disclosed above. Therefore, any simple modification, equivalent change and modification made to the above embodiments according to the technical essence of the present invention are still within the scope of the protection of the technical solution of the present invention, unless the contents of the technical solution of the present invention are departed.

Claims

1. A roadbed compaction construction data high-efficiency processing system is characterized by comprising a data splitting unit, a data storage queue unit, a storage task scheduling thread pool unit, an original data storage unit, a data calculation task queue unit and a calculation task execution unit; the data splitting unit splits the original construction data according to a designed data splitting strategy, and the data belonging to the same fragment are stored in the same database file; the data storage queue unit divides original construction data into data packets, and creates a data storage queue by taking the divided data packets as units, wherein the queue is scheduled and managed by a data storage thread pool; the storage task scheduling thread pool unit schedules the storage task according to the storage task queue condition by a strategy; the original data storage unit is stored in different database files according to different data packets; the data calculation task queue unit is based on database files stored in a fragmentation mode, a calculation task is established by an original data storage file, the calculation task is placed in a queue, and the queue is managed by a calculation task scheduling thread pool; the computing task execution unit schedules the data computing tasks according to the computing task scheduling thread pool and the strategies, and the computing tasks can be executed concurrently with high concurrency number because the data to be computed are in different data files.