CN110059153B - Data exchange method for real-time data processing system - Google Patents
Data exchange method for real-time data processing system Download PDFInfo
- Publication number
- CN110059153B CN110059153B CN201910246356.6A CN201910246356A CN110059153B CN 110059153 B CN110059153 B CN 110059153B CN 201910246356 A CN201910246356 A CN 201910246356A CN 110059153 B CN110059153 B CN 110059153B
- Authority
- CN
- China
- Prior art keywords
- data
- area
- real
- dictionary
- head address
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/30—Information retrieval; Database structures therefor; File system structures therefor of unstructured textual data
- G06F16/31—Indexing; Data structures therefor; Storage structures
- G06F16/316—Indexing structures
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/30—Information retrieval; Database structures therefor; File system structures therefor of unstructured textual data
- G06F16/36—Creation of semantic tools, e.g. ontology or thesauri
- G06F16/374—Thesaurus
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Data Mining & Analysis (AREA)
- Databases & Information Systems (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Software Systems (AREA)
- Computational Linguistics (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention discloses a data exchange method for a real-time data processing system, which comprises the following steps of writing a corresponding data dictionary; defining a data structure in a continuous storage area, and estimating the data quantity required to be processed when the data structure is used according to a data dictionary; searching according to the received original data and the data structure; after the head address corresponding to the original data is found, converting is carried out according to the data types defined in the data dictionary, and the actual value of the data is obtained. In real-time data processing, the data structure is employed in which the processed data and the associated definitions of the data are all stored. By using the data access method provided by the structure, the software modification workload caused by frequent data exchange format or content modification can be reduced, and the efficiency of data processing, exchange and use can be improved. Meanwhile, the structure can be shared in a mode of sharing the memory by the computer, so that real-time data exchange among processes is realized.
Description
Technical Field
The present invention relates to a data structure and a data exchange method for data exchange of a plurality of two-dimensional table classes of a real-time data processing system.
Background
In some data processing fields, data is often classified in the form of a two-dimensional table, for example, a certain type of data is determined by X-Y, where X is a table number and Y is what number of data is in the table. In data processing, a two-dimensional array or a piece of continuous memory is typically employed to store real-time data. The processing method has two defects, namely when one-dimensional or two-dimensional arrays are adopted to store data, when the data type is changed, the arrays need to be maintained at a code level; and secondly, the data in the array storage mode can only represent the value of the data or simple some meaning description.
Disclosure of Invention
In order to solve the problem of data exchange in real-time data processing, the invention establishes a two-dimensional linear structure of the stored data structure, can access the value of the data and various attribute information through simple calculation and positioning search, supports various data types, and is easy to realize in different operating systems.
In view of this, the technical scheme adopted by the invention is as follows: a data exchange method for a real-time data processing system, comprising the steps of:
(1) Writing a corresponding data dictionary according to the application scene; and defining the table number and the serial number of each data in the data dictionary, wherein the table comprises code numbers, lengths and types, and other information can be increased or decreased according to actual requirements.
(2) In a continuous storage area, defining a data structure, estimating the data quantity required to be processed when using the data structure according to a data dictionary, and opening up a proper memory space according to the data quantity. Typically 1.5 times the theoretical data capacity is employed.
(3) And searching according to the received original data and the data structure.
(4) After the head address corresponding to the original data is found, converting is carried out according to the data types defined in the data dictionary, and the actual value of the data is obtained.
The data structure includes the following 6 parts:
the number of the tables is used for storing the number of the tables stored in the head address information area;
the list head address information area is used for storing the head address of each list in the original data area and the data display area, and realizes the positioning of the list in the data access process;
the code number is used for storing the data code number and the number of the mapping tables in the X-Y mapping area;
the data code number and X-Y mapping area is used for storing the corresponding relation between the data code number and the data table number and the serial number;
the received original data area is used for storing the received original data area which is not processed later;
the data description field is used to store the data types defined in the data dictionary.
The invention defines a plurality of data structure combinations in a continuous storage area of the computer, and realizes the matching storage of related information such as data values, data meanings and the like. Meanwhile, a series of data access interfaces are provided, hidden dangers such as data damage and structural damage caused by direct operation of the memory are avoided, and usability and data safety are greatly improved.
In real-time data processing, the data structure of the invention can be used to store all the processed data and the relevant definition of the data in the data structure. By using the data access method provided by the structure, for example, the data is accessed through code numbers, the data exchange format or frequent change of the content can be reduced, the workload of software modification brought by a data application layer is reduced, and the efficiency of data processing, exchange and use is improved. Meanwhile, the structure can be shared in a mode of sharing the memory by the computer, so that real-time data exchange among processes is realized.
Drawings
FIG. 1 is a schematic diagram of a data dictionary structure;
FIG. 2 is a schematic diagram of a data distribution diagram;
FIG. 3 is a data dictionary read flow diagram;
FIG. 4 find a specified raw data flow chart;
FIG. 5 is a flow chart for locating specified data attributes.
Detailed Description
For the purpose of making the objects, technical solutions and advantages of the present invention more apparent, the embodiments of the present invention will be described in further detail with reference to the accompanying drawings.
The invention adopts the scheme that a continuous storage area is divided into 6 parts, and the storage area is shown in figure 2. Wherein:
table number: the number of tables stored in the header address information area.
Header address information area: the area stores the first address of each table in the original data area and the data area, and realizes the positioning (X positioning) of the table in the data access process.
Number of codes: the number of mapping tables in the data code and X-Y mapping area.
Data code and X-Y mapping region: the region stores the corresponding relation between the data code number and the data table number and the serial number.
Received original data area: the area is a received raw data area that has not been post-processed. The received data may be continuous or discontinuous, as determined by information about the data presentation area.
The data illustrate the area: the data name, code name, data offset, length, type, unit, theoretical value, and the like of each data are stored. The content of the part is consistent with the sequence of the data dictionary in fig. 1, and can be increased or decreased according to the actual application condition. Wherein the data offset stores the offset length (Y position) of the data in the received data field within the correspondence table.
The structure of each region is defined in the language C as follows:
header address mapping area:
data code and X-Y mapping region:
the data illustrate the area:
the data dictionary structure corresponding to the data is shown in fig. 1, the table number and the serial number of each data are defined in the data dictionary, the code number, the length and the type are required items, and other information can be increased or decreased according to actual requirements or designed.
The initialization of the data structure and the loading flow of the data dictionary are as shown in fig. 3:
and the size of each region in the data structure is estimated and allocated according to the capacity which is actually required to be processed and the related structure body, and the dictionary is read and analyzed according to the rows. A flag is set at the end of each data table or in front of a new data table, and the flag is judged when the dictionary is read. If the new data table is the new data table, the original data area head address of the new table is recalculated and the data area head address is stated to be the new table. If the data table is not new, the relevant information is added to the old table.
Data lookup utilization
1. Looking up the head address of the original data area of the data table
And searching the corresponding head address of the original data area of the table in the head address mapping area of the table.
2. Look-up data table illustrates data area head address
The corresponding table data is searched in the table head address mapping area to indicate the head address of the area.
3. Searching for the original data of each data
As shown in FIG. 4, if the data code number is searched, the code number is converted into an X-Y form through the content of the mapping area corresponding to the code number and the X-Y, and the original data code address is obtained after the corresponding table code address and the data offset address are added. Specifically, finding the data description area head address of the corresponding data table according to the X information; finding out detailed attributes of data in the corresponding data table according to the Y information; finding the offset address of the data in the original data area in the detailed attribute; finding the initial data area head address according to the X information; the data area head address plus the offset address yields the specific data address.
4. Searching for detailed attributes of each data
As shown in fig. 5, the location is made by specifying the head address of the area and the offset by the corresponding data, the attribute length of each data is fixed, and the offset is derived by the Y attribute length. Specifically, finding the head address of the data table data corresponding to the X information; and finding out the detailed attribute of the data in the corresponding data table according to the Y information.
5. Data usage
After the head address of the corresponding original data is found, converting the 16-system original data according to the data types defined in the dictionary to obtain the actual value of the data.
Claims (4)
1. A data exchange method for a real-time data processing system, comprising the steps of:
(1) Writing a corresponding data dictionary according to the application scene;
(2) Defining a data structure in a continuous storage area, estimating the data quantity required to be processed when the data structure is used according to a data dictionary, and opening up a proper memory space according to the related data quantity;
the data structure comprises a table number, a table header address information area, a code number, a data code number and X-Y mapping area, a received original data area and a data interpretation area;
the number of the tables is used for storing the number of the tables stored in the head address information area;
the table head address information area is used for storing the head address of each table in the original data area and the data interpretation area, and positioning the table is realized in the data access process;
the code number is used for storing the data code number and the number of mapping tables in the X-Y mapping area;
the data code number and X-Y mapping area is used for storing the corresponding relation between the data code number and the data table number and the serial number;
the received original data area is used for storing the received original data area which is not processed later;
the data description area is used for storing data types defined in the data dictionary;
(3) Searching according to the received original data and the data structure;
(4) After the head address corresponding to the original data is found, converting is carried out according to the data types defined in the data dictionary, and the actual value of the data is obtained.
2. A data exchange method for a real-time data processing system according to claim 1, wherein: and defining the table number and the serial number of each data in the data dictionary, wherein the table comprises a code number, a length and a type.
3. A data exchange method for a real-time data processing system according to claim 1, wherein: the estimate uses 1.5 times the theoretical data capacity when using the data structure for the amount of data to be processed.
4. A data exchange method for a real time data processing system according to any of claims 1-3, characterized in that: the step (3) comprises searching the original data of each data, and comprises the following specific steps:
finding a data display area head address according to the X information; finding detailed attributes of the corresponding data according to the Y information; finding the offset address of the data in the original data area in the detailed attribute; finding the initial data area head address according to the X information; the data area head address plus the offset address yields the specific data address.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910246356.6A CN110059153B (en) | 2019-03-29 | 2019-03-29 | Data exchange method for real-time data processing system |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910246356.6A CN110059153B (en) | 2019-03-29 | 2019-03-29 | Data exchange method for real-time data processing system |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN110059153A CN110059153A (en) | 2019-07-26 |
| CN110059153B true CN110059153B (en) | 2023-09-26 |
Family
ID=67317823
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201910246356.6A Active CN110059153B (en) | 2019-03-29 | 2019-03-29 | Data exchange method for real-time data processing system |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN110059153B (en) |
Citations (11)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN1652112A (en) * | 2005-03-02 | 2005-08-10 | 北京北方烽火科技有限公司 | Implementing method of data dictionary under embedded environment |
| CN102081644A (en) * | 2010-11-25 | 2011-06-01 | 浙江鸿程计算机系统有限公司 | Data storage method for storing data and data meanings separately |
| CN102332030A (en) * | 2011-10-17 | 2012-01-25 | 中国科学院计算技术研究所 | Data storing, managing and inquiring method and system for distributed key-value storage system |
| CN102722540A (en) * | 2012-05-23 | 2012-10-10 | 中兴通讯股份有限公司 | Method and device for processing data in real-time memory database system |
| CN103150397A (en) * | 2013-03-26 | 2013-06-12 | 北京经纬恒润科技有限公司 | Establishing method for data index, and data searching method and system |
| CN103970855A (en) * | 2014-05-06 | 2014-08-06 | 车智互联（北京）科技有限公司 | Device and method for processing data |
| CN104462280A (en) * | 2014-11-27 | 2015-03-25 | 国网上海市电力公司 | Data information storage and fast inquiry method used for embedded system |
| CN104915383A (en) * | 2015-05-19 | 2015-09-16 | 中国电子科技集团公司第二十八研究所 | General method for automatic object adapting in database access |
| EP2924594A1 (en) * | 2014-03-25 | 2015-09-30 | Sap Se | Data encoding and corresponding data structure in a column-store database |
| CN106980665A (en) * | 2017-03-21 | 2017-07-25 | 广东神马搜索科技有限公司 | Data dictionary implementation method, device and data dictionary management system |
| CN108228606A (en) * | 2016-12-14 | 2018-06-29 | 北京国双科技有限公司 | The wiring method and device of data |
-
2019
- 2019-03-29 CN CN201910246356.6A patent/CN110059153B/en active Active
Patent Citations (11)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN1652112A (en) * | 2005-03-02 | 2005-08-10 | 北京北方烽火科技有限公司 | Implementing method of data dictionary under embedded environment |
| CN102081644A (en) * | 2010-11-25 | 2011-06-01 | 浙江鸿程计算机系统有限公司 | Data storage method for storing data and data meanings separately |
| CN102332030A (en) * | 2011-10-17 | 2012-01-25 | 中国科学院计算技术研究所 | Data storing, managing and inquiring method and system for distributed key-value storage system |
| CN102722540A (en) * | 2012-05-23 | 2012-10-10 | 中兴通讯股份有限公司 | Method and device for processing data in real-time memory database system |
| CN103150397A (en) * | 2013-03-26 | 2013-06-12 | 北京经纬恒润科技有限公司 | Establishing method for data index, and data searching method and system |
| EP2924594A1 (en) * | 2014-03-25 | 2015-09-30 | Sap Se | Data encoding and corresponding data structure in a column-store database |
| CN103970855A (en) * | 2014-05-06 | 2014-08-06 | 车智互联（北京）科技有限公司 | Device and method for processing data |
| CN104462280A (en) * | 2014-11-27 | 2015-03-25 | 国网上海市电力公司 | Data information storage and fast inquiry method used for embedded system |
| CN104915383A (en) * | 2015-05-19 | 2015-09-16 | 中国电子科技集团公司第二十八研究所 | General method for automatic object adapting in database access |
| CN108228606A (en) * | 2016-12-14 | 2018-06-29 | 北京国双科技有限公司 | The wiring method and device of data |
| CN106980665A (en) * | 2017-03-21 | 2017-07-25 | 广东神马搜索科技有限公司 | Data dictionary implementation method, device and data dictionary management system |
Also Published As
| Publication number | Publication date |
|---|---|
| CN110059153A (en) | 2019-07-26 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN105117417B (en) | A kind of memory database Trie tree indexing means for reading optimization | |
| CN102622434B (en) | Data storage method, data searching method and device | |
| CN106991102A (en) | The processing method and processing system of key-value pair in inverted index | |
| US20050076018A1 (en) | Sorting result buffer | |
| CN104158744A (en) | Method for building table and searching for network processor | |
| CN100392676C (en) | Colour space quantized discribing code structure | |
| CN110059021A (en) | A kind of algorithm for reducing write-in magnifying power and promoting random writing performance | |
| CN110597851B (en) | Big data based data processing and report display method | |
| CN110928900B (en) | Multi-table data query method, device, terminal and computer storage medium | |
| CN100511152C (en) | Method for managing priority queue | |
| CN110059153B (en) | Data exchange method for real-time data processing system | |
| US8566342B2 (en) | In-memory data optimization system | |
| CN104090954A (en) | Connecting method and system of read-only tables | |
| CN104462080A (en) | Index structure creating method and system with group statistics for search results | |
| CN101459599B (en) | Method and system for implementing concurrent execution of cache data access and loading | |
| CN110297836B (en) | User label storage method and retrieval method based on compressed bitmap mode | |
| CN109857366B (en) | Insertion ordering method, system, equipment and storage medium based on external memory | |
| CN113157692B (en) | Relational memory database system | |
| CN112035380B (en) | Data processing method, device and equipment and readable storage medium | |
| CN109241098B (en) | Query optimization method for distributed database | |
| CN113573125B (en) | Subtitle rendering method and system based on SVG word stock | |
| CN101271466B (en) | Electronic dictionary work retrieval method based on self-adapting dictionary tree | |
| CN114398373A (en) | File data storage and reading method and device applied to database storage | |
| CN110083812B (en) | Picture layout method and device of electronic document and electronic equipment | |
| CN1326159A (en) | Index method for feature vector data space |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |