CN111722887A - A method and system for processing and displaying massive scatter data - Google Patents

Abstract

The invention discloses a method and system for processing and displaying massive scattered data. It includes the following steps: converting the external original data into a designed intermediate cache format, and using a worker thread; after the user initiates a request, configure, request and calculate the data in the intermediate cache format; complete the configuration, request and After the calculation, the processing results are fed back to the user in stages. The invention does not depend on graphics cards, especially graphics cards of specific brands; it does not need to introduce additional parallel computing APIs (CUDA and OpenCL) with higher thresholds, which reduces the cost of learning and maintenance; This solution can complete the processing and display of a large amount of data in a short time, and improve the user's fluency experience.

Description

A method and system for processing and displaying massive scatter data

technical field

The invention relates to the field of computer technology, and in particular, to a method and system for processing and displaying massive scatter data.

Background technique

At present, the processing idea for discrete data is "multi-line start"-using its "context-independent" feature to process simultaneously on multiple pipelines. The more mature solution is to use CUDA, but it relies on NVIDIA's graphics card, which means that the program itself is more dependent on hardware, and at the same time tests the performance of the graphics card; in addition, OpenCL is also competent, but compared to CUDA , although it is out of the constraints of a specific brand of graphics card, the difficulty of use is further increased. in:

Context-free: means that there are no association constraints between data individuals, and their respective processing processes and results will not affect other data.

CUDA: CUDA (Compute Unified Device Architecture), is a computing platform launched by the graphics card manufacturer NVIDIA. CUDA is a general-purpose parallel computing architecture introduced by NVIDIA that enables GPUs to solve complex computing problems. It contains the CUDA instruction set architecture (ISA) and the parallel computing engine inside the GPU. Developers can use C, the most widely used high-level programming language, to write programs for the CUDA architecture. Programs written can run with ultra-high performance on processors that support CUDA.

OpenCL: the full name of Open Computing Language, open computing language. OpenCL is a framework for writing programs for heterogeneous platforms, which can consist of CPUs, GPUs, or other types of processors. OpenCL consists of a language (based on C99) for writing kernels (functions that run on OpenCL devices) and a set of APIs for defining and controlling the platform. OpenCL provides a parallel computing mechanism based on task partitioning and data partitioning.

The worker threads of the thread pool can avoid accumulating work on the main thread and cause interface blocking; multi-threaded parallel computing can greatly improve efficiency. When the interface is drawn, the large amount of data is only an appearance, and the main thing behind it is the huge overhead generated by the execution method - that is, the more scattered points, the more instructions that need to be executed for a single drawing point, so the data is as much as possible. Passing the breath to the target is also an effective means of reducing overhead. Therefore, it is urgent to propose a method and system for processing and displaying massive scatter data, so as to solve the problem of time-consuming and stuttering in processing when the scatter data in the prior art is very large. The processing of discrete data needs to rely on graphics cards, and additional parallel computing APIs (CUDA and OpenCL) with high thresholds need to be introduced, which reduces the cost of learning and maintenance.

SUMMARY OF THE INVENTION

In view of the above-mentioned defects of the prior art, the technical problem to be solved by the present invention is to provide a method and system for processing and displaying massive scattered data, so as to solve the problem that the processing of discrete data in the prior art needs to rely on a graphics card and requires The additional introduction of high-threshold parallel computing APIs (CUDA and OpenCL) reduces the cost of learning and maintenance.

In order to achieve the above purpose, the present invention provides a method for processing and displaying massive scatter data, comprising the following steps:

Convert external raw data into a designed intermediate cache format, and need to use worker threads;

After the user initiates the request, configure, request and calculate the data in the intermediate cache format;

After completing the configuration, request and calculation, the processing results are fed back to the user in stages.

Preferably, the cache is designed by using a multi-level hash table or a hash table with sequential containers.

Preferably, the worker thread is a worker thread or a thread pool.

Preferably, the configuration, request and calculation are specifically:

Configuration: According to the designed format, the configuration items used for request calculation are organized, and the content of the configuration items includes the location and size of the sampling, and the scaling ratio in the horizontal and vertical directions;

request: pass the configuration item to the processing unit and make it run the processing;

Calculation: Divide the cached data according to the configuration items, create a thread pool, distribute the cut blocks, and use multiple worker threads to asynchronously complete the processing of each block.

Preferably, the configuration process also includes the display requirements for any zoom ratio of a certain sampling area, and the processing of the scattered point data has the following formula:

X _aisplay =Scale _x *(X _splash -X _topleft )

Y _display =Scale _y ，*(Y _splash -Y _topleft )

Where display refers to display, scale is the zoom ratio, splash is the scatter point; topleft is the upper left corner coordinate of the sampling area.

Preferably, after the user obtains the processing result, the result is displayed on the GUI interface: if the output is selected as a block data set, that is, including the coordinates of the upper left corner and the scatter set, the function of drawing points is used; if the obtained 8-bit image is , use the function to draw a picture.

A system for processing and displaying massive scattered data, including:

The data conversion module is used to convert the external original data into the designed intermediate cache format, and requires the use of worker threads;

a data request processing module, used for configuring, requesting and calculating the data in the intermediate cache format after the user initiates a request;

The data feedback module is used to feed back the processing results to the user in stages after completing the configuration, request and calculation.

Preferably, the data conversion module is used to convert the external original data into a designed intermediate cache format: the cache is designed by using a multi-level hash table or a hash table with a sequential container.

Preferably, the worker thread used by the data conversion module is a worker thread or a thread pool.

Preferably, after the user initiates the request, the data request processing module configures, requests and calculates the data in the intermediate cache format, specifically:

Configuration: According to the designed format, the configuration items used for request calculation are organized, and the content of the configuration items includes the location and size of the sampling, and the scaling ratio in the horizontal and vertical directions;

request: pass the configuration item to the processing unit and make it run the processing;

Calculation: Divide the cached data according to the configuration items, create a thread pool, distribute the cut blocks, and use multiple worker threads to asynchronously complete the processing of each block.

Preferably, the configuration process of the data request processing module further includes the display requirement of any zoom ratio for a certain sampling area, and the processing of the scattered data has the following formula:

X _display =Scale _x *(X _splash -X _topleft )

Y _display =Scale ₃ *(Y _splash -Y _topleft )

Among them, display refers to the display, scale is the zoom ratio, splash is the scatter point; topleft is the coordinate of the upper left corner of the sampling area.

Preferably, it also includes an interface refresh module, which is used to display the results on the GUI interface after the user obtains the processing results: if the output is selected as a block data set, that is, including the coordinates of the upper left corner and the scatter set, then use the drawing point's function; if you get an 8-bit picture, use the function of drawing the picture.

The beneficial effects of the present invention are:

The invention does not depend on graphics cards, especially graphics cards of a specific brand; it does not need to introduce additional parallel computing APIs (CUDA and OpenCL) with high thresholds, which reduces the cost of learning and maintenance; This solution can complete the processing and display of a large amount of data in a short time, and improve the user's fluency experience.

The concept, specific structure and technical effects of the present invention will be further described below in conjunction with the accompanying drawings, so as to fully understand the purpose, characteristics and effects of the present invention.

Description of drawings

Figure 1 is a flow chart of the method of the present invention.

Fig. 2 is a system principle block diagram of the present invention.

FIG. 3 is a schematic diagram showing that in the 8-bit picture of the present invention, each pixel value is recorded only by an index value pointing to the color table.

Detailed ways

As shown in Figure 1, the present invention provides a method for processing and displaying massive scattered data, comprising the following steps:

Step 1: Convert the external original data into a designed intermediate cache format, and need to use a worker thread (a worker thread is a worker thread or a thread pool), there are three considerations;

1) On the premise that the original data remains unchanged, the converted format is convenient for repeated segmentation and sampling for different requests;

2) The cached data needs to provide support for fast pick-up detection, so that the user can quickly determine which scatter points should be picked up at the mouse position;

3) The use of worker threads avoids blocking the main thread due to a large number of data conversions. (If the main thread is blocked, the program will not be able to respond to user behavior in time at this time, affecting the experience).

Among them, the cache is designed by using a multi-level hash table or a hash table with a sequential container. Such as hash<x, hash<y, vector<pointf>>> and hash<x, hash<y, set<pointf>>>, it can also be hash<point, vector<pointf>> or hash<point, set< pointf>>. Where hash refers to a hash table with key and value pairs, vector is a sequential container, set is a hash table with only key, x and y refer to the integer value of the horizontal and vertical coordinates of the scattered points, point consists of x and y, and pointf is The original data of the scatter, whose horizontal and vertical coordinates are floating point numbers. Because multiple floating-point coordinates may have the same integer coordinates after being approximated by an integer, after x, y or point is mapped, it points to a container (vector or set) instead of a single pointf. The characteristics of the hash table make data insertion and matching search very fast, which is in line with the needs of processing a large number of scattered data.

Step 2: After the user initiates the request, configure, request and calculate the data in the intermediate cache format. The configuration, request and calculation are as follows:

Configuration: According to the designed format, the configuration items used for request calculation are organized. The content of the configuration items includes the location and size of the sampling, and the scaling ratio in the horizontal and vertical directions:

request: pass the configuration item to the processing unit and make it run the processing;

Calculation: Divide the cached data according to the configuration items, create a thread pool, distribute the cut blocks, and use multiple worker threads to asynchronously complete the processing of each block.

At the same time, the configuration process also includes the display requirements for any zoom ratio of a certain sampling area, and the processing of the scattered data is as follows:

X _display =Scale _x *(X _splash -X _topleft )

Y _display =Scale _y *(Y _splash -Y _topleft )

Where display refers to display, scale is the zoom ratio, splash is the scatter point; topleft is the upper left corner coordinate of the sampling area. Depending on the actual processing method, there are two definitions:

1) A request to share a unified upper left corner, the upper left corner coordinate is the upper left corner of the sampling area;

2) Each block calculates the upper left corner of its own local area, and calculates and processes it independently in units of blocks. The obtained result (x, y) is stored in the set as a point, and its purpose is to exclude duplicate coordinate points as a key value, achieve a similar effect of cropping and culling, and reduce unnecessary subsequent processing actions. (After the set is collected, you can choose to convert it to a regular container, such as a vector, and then feed it back to the next step)

Step 3: After completing the configuration, request and calculation, the processing results are fed back to the user in stages. The block information includes the starting point (that is, the coordinates of the upper left corner, which helps to be compatible with the two processing methods mentioned in the previous step), the processed scattered point collection (you can choose to use a container such as set or vector to hold it). Before this, the block data can be processed again: because of the display of scatter data, each point is a single pixel point, and at least in the same batch, the color is consistent. According to this general characteristic, an 8-bit image can be selected to replace the scatter set, and the result of the pixel map can be directly fed back. As shown in Figure 3, in an 8-bit picture, each pixel value is only recorded with an index value pointing to the color table, rather than a separate R, G, B color value, so the picture storage is simple and fast, and can be directly written in the form of an array. (Specifically, consider using Qt's QImage, and select QImage::Format_Indexed8 for format).

Finally, after the user obtains the processing results, the results are displayed on the GUI interface: if the output is selected as a block data set, that is, including the upper left corner coordinates and a scatter set, the function of drawing points is used; if the obtained 8-bit image, Then use the function to draw a picture.

As shown in Figure 2, the present invention also provides a system for processing and displaying massive scattered data, including:

The data conversion module is used to convert the external original data into the designed intermediate cache format, and requires the use of worker threads;

The data request processing module is used to configure, request and calculate the data in the intermediate cache format after the user initiates the request;

The data feedback module is used to feed back the processing results to the user in stages after completing the configuration, request and calculation.

The specific implementation method thereof has been described above, and will not be repeated here.

To sum up, the present invention does not depend on graphics cards, especially graphics cards of specific brands; it does not need to introduce additional parallel computing APIs (CUDA and OpenCL) with high thresholds, which reduces the cost of learning and maintenance; compared with the conventional single worker thread Or a single-threaded linear processing method, this solution can complete the processing and display of a large amount of data in a short time, and improve the user's fluency experience.

The preferred embodiments of the present invention have been described in detail above. It should be understood that those skilled in the art can make numerous modifications and changes according to the concept of the present invention without creative efforts. Therefore, all technical solutions that can be obtained by those skilled in the art through logical analysis, reasoning or limited experiments on the basis of the prior art according to the concept of the present invention shall fall within the protection scope determined by the claims.

Claims (10)

1. a method for processing and displaying massive scatter data, is characterized in that, comprises the following steps: Convert the external original data into a designed intermediate cache format, and need to use worker threads; after the user initiates a request, configure, request and calculate the data in the intermediate cache format; After completing the configuration, request and calculation, the processing results are fed back to the user in stages; The configuration, request and calculation are specifically: Configuration: According to the designed format, the configuration items used for request calculation are organized, and the content of the configuration items includes the location and size of the sampling, and the scaling ratio in the horizontal and vertical directions; request: pass the configuration item to the processing unit and make it run the processing; Calculation: Divide the cached data according to the configuration items, create a thread pool, distribute the cut blocks, and use multiple worker threads to asynchronously complete the processing of each block. 2 . The method for processing and displaying massive scatter data according to claim 1 , wherein the cache is designed by using a multi-level hash table or a hash table with sequential containers. 3 . 3 . The method for processing and displaying massive scattered data according to claim 1 , wherein the working thread is a working thread or a thread pool. 4 . 4. The method for processing and displaying massive scatter data as claimed in claim 1, wherein the configuration process further comprises a display requirement for an arbitrary zoom ratio of a certain sampling area, the scatter data of which The processing has the following formula: X _display =Scale _x *(X _splash -X _topleft ) Y _display =Scale _y *(Y _splash -Y _topleft ) Where display refers to display, scale is the zoom ratio, splash is the scatter point; topleft is the upper left corner coordinate of the sampling area. 5. a kind of method for processing and displaying massive scattered data as claimed in claim 1 is characterized in that: after the user obtains the processing result, the result is displayed on the GUI interface: if the selection output is a block data set, that is, If the upper left corner coordinate and scatter set are included, the function of drawing points is used; if the obtained 8-bit picture is obtained, the function of drawing pictures is used. 6. A system for the processing and display of massive scatter data, comprising: The data conversion module is used to convert the external original data into the designed intermediate cache format, and requires the use of worker threads; a data request processing module, used for configuring, requesting and calculating the data in the intermediate cache format after the user initiates a request; The data feedback module is used to feed back the processing results to the user in stages after completing the configuration, request and calculation; The data request processing module configures, requests and calculates the data in the intermediate cache format after the user initiates the request, specifically: Configuration: According to the designed format, the configuration items used for request calculation are organized, and the content of the configuration items includes the location and size of the sampling, and the scaling ratio in the horizontal and vertical directions; request: pass the configuration item to the processing unit and make it run the processing; Calculation: Divide the cached data according to the configuration items, create a thread pool, distribute the cut blocks, and use multiple worker threads to asynchronously complete the processing of each block. 7. a kind of system to the processing and display of massive scattered point data as claimed in claim 6, is characterized in that, described data conversion module, is used for external original data to be converted into the intermediate cache format that is designed: The cache described above is designed to use a multi-level hash table or hash table with sequential containers. 8 . The system for processing and displaying massive scattered data according to claim 7 , wherein the working thread used by the data conversion module is a working thread or a thread pool. 9 . 9. A system for processing and displaying massive scatter data as claimed in claim 8, wherein the configuration process of the data request processing module further comprises a display requirement for an arbitrary zoom ratio of a certain sampling area, The processing of the scattered data has the following formula: X _display =Scale _x *(X _splash -X _topleft ) Y _display =Scale _y *(Y _splash -Y _topleft ) Among them, display refers to the display, scale is the zoom ratio, splash is the scatter point; topleft is the coordinate of the upper left corner of the sampling area. 10. A system for processing and displaying massive scatter data as claimed in claim 6, characterized in that it further comprises an interface refresh module for displaying the results on the GUI interface after the user obtains the processing results : If the output is selected as a block data set, that is, including the upper left corner coordinates and a scatter set, the function of drawing points is used; if the obtained 8-bit picture is obtained, the function of drawing pictures is used.

Images