TW201626254A - Big data real-time storage and real-time access in NoSQL - Google Patents

Abstract

A big data real-time storage and real-time access in NoSQL is disclosed, which utilizes the attribute in current NoSQL to store multiple lines in a big data sheet, wherein each line is structured with a primary key and columns in any number. The disclosed records only one data storage record in a data storage record sheet for the relevant records in each batch big data storage process, wherein a data storage record contains the primary key of the first data in the batch, the primary key of the last data, and the total data number in the batch. The data sheet is to be used in data access, which only requires access in time sequence and analyze the data storage record to run a batched big data accessing process and coordinate with an auto primary keys ranking of the NoSQL, the data access time can then be shortened. Furthermore, the disclosed provides a distributed multi-tasking access data mechanism based on a real-time operation process efficacy, and a real-time access program module that enables multiple user ends program to be accessed at the same time when data receiving interface is in use, thereby allowing each user end to access non-overlapped data segment, and increasing the instantaneity of data access.

Description

Instant storage and reading method for huge amount of data for non-associated database

The invention belongs to the technical field of G06F 17/30 information retrieval and database structure, in particular to a method for instant storage and reading of huge amount of data for non-associated database. The invention is a control method, as a method for storing and reading a large amount of data, and can provide a distributed multiplexed reading data mechanism according to the real-time computing processing performance requirement.

Traditional relational databases, in the face of large amounts of data writing and large amounts of data storage, suffer from insufficient write performance and the difficulty of expansion or high expansion costs. If the In-memory Database technology is used, although the write performance requirement can be met, there is a limit that the data cannot be stored in a large amount, and once the system is turned off (the machine), the data is lost and cannot be saved.

A non-associative database, or NoSQL database, which is different from the traditional relational database, has the feature of horizontal expansion capability. As long as a new server node is added, the capacity of the database system can be continuously expanded to meet long-term mass storage. The requirements, and can meet the performance requirements of a large number of data writing, as well as the need for a large amount of historical data analysis. The Key-Value data mode adopted by NoSQL technology simplifies the structure of each data to only one Key value corresponding to a Value value. There is no correlation between each data, so it can be arbitrarily cut or adjusted, or distributed to Make a copy in a different server.

However, NoSQL technology is suitable for the key value as the operation target of writing and querying. It cannot directly satisfy the application mode of writing and reading data by many people at the same time, such as a large number of sensors. Data is written, and a large number of sensor data are read and analyzed, and the analysis module must be able to read the unrepeated data blocks and parse them. In view of the shortcomings derived from the above-mentioned conventional methods, the inventor of the present invention has improved and innovated, and after years of painstaking research, he finally successfully developed and stored this huge amount of data for non-associated database for instant storage and reading. method.

The object of the present invention is to provide a method for instant storage and reading of huge amounts of data for a non-associated database, which fully utilizes the advantages of the NoSQL database through software, and solves the application of bulk data writing and bulk data reading and analysis. demand. With the NoSQL feature, the present invention can be dynamically expanded at a low cost, meeting long-term storage history data, and providing a need for subsequent analysis applications. At the same time, a control method is implemented, so that the data can be read by multiple client programs at the same time, and each of the non-repeated data segments is read and analyzed.

The method for instant storage and reading of huge amounts of data for a non-associated database that achieves the above object is to use the attributes of the non-associative database itself, and the structure of each row also has a primary key value Key and an arbitrary number of columns. Field, based on this attribute, the present invention provides a method for instantaneous storage and reading of huge amounts of data for a non-associated database, and stores a large amount of data storage operations in a batch in a data table. The data storage record includes the primary key value of the first data of the batch, the primary key value of the last data, and the total number of stored data in the batch. The storage record data table can be regarded as a batch data index by another viewpoint. Record data sheet, which is used for real-time huge data reading. When reading data, it only needs to read and parse the data storage records in chronological order and then combine them into a batch of data query conditions. Reading jobs can increase the amount of data query and read efficiency.

110‧‧‧ Instant Storage Module

120‧‧‧ data receiving interface

130‧‧‧Unrelated database

140‧‧‧ Instant Reader Module

150‧‧‧ data reading interface

160‧‧‧ Data Storage Record Control Module

210~240‧‧‧Step process

310~340‧‧‧Step process

1 is a schematic diagram of a computer system architecture for an instant storage and reading method of a huge amount of data for a non-associated database; FIG. 2 is a flow chart of the method for storing a huge amount of data for a non-associated database; And Figure 3 is a flow chart of the method for instantly reading huge amounts of data for a non-associated database.

Please refer to FIG. 1 for a schematic diagram of the system architecture of the present invention, which mainly includes three modules: an instant storage program module 110, a data storage record control module 160 and an instant reading program module 140, respectively providing a data receiving interface 120 and reading data. The interface 150 is configured to allow the data provider to store the real-time data into the non-associated database 130 through the data receiving interface 120, so that the data recipient can read the real-time data from the unrelated database 130 through the data reading interface 150. take.

The instant storage program module 110 mainly provides a data receiving interface 120 for processing and receiving a large number of structured data objects (JAVA objects), and when the receiving data is completed, the data type conversion is started one by one, and the data is structured. The data object is converted into unstructured data, and the primary key value (ROW KEY) of the data is used according to the time when the data occurs, and finally the corresponding data table in the non-associated database 130 is written to complete the non-association. The huge amount of data in the database 130 is stored immediately.

The data storage record control module 160 is responsible for storing a large number of data storage operation related records for each batch, and storing only one data storage record in a data storage record data table, wherein one data storage record contains the first data of the batch. The primary key value, the primary key value of the last data, and the total amount of the batch data. This data table can be used to read the data. When reading the data, only the data storage record needs to be read and parsed according to the chronological order. Then combined into a batch of data query conditions for bulk data reading operations.

The instant-reading program module 140 is configured to provide a data reading interface 150 for processing a large amount of non-structural data stored in the non-associated database 130, and querying the query conditions based on the data storage record control module 160. To perform the data query operation of the corresponding data table, obtain the data type and convert the data type into pieces, and convert the unstructured data into the structured data object (JAVA object) to complete the huge amount of data of the unrelated database 130. Read.

From another point of view, the data storage record data table can be regarded as a batch of data index record data table. Therefore, the data storage record control module can be regarded as the core of the invention, through the control and data table of the primary key value of the batch data. The index function itself has only one execution of the query operation, and the batch data can be completely and accurately read by the non-associated database 130, which can achieve the huge amount of data instant access and provide data backup and guarantee. The information is not lost.

2 is a flow chart of a method for storing huge amounts of data in a non-associated database according to an embodiment of the present invention. The following is a detailed step of how to write a batch of large amounts of data into a non-associative database. . In this embodiment, the non-associated database has a specific data table for storing the batch data storage related records.

Referring to FIG. 2, first step 210 is to convert the structured data object into unstructured data. Before that, the data provider must use the data receiving interface 120 provided by the instant storage program module 110 to design according to the structured data field. a structured data object program, completes the data encapsulation operation, and passes the data to step 210, and then extracts the field data through the object piece by piece, and then uses the data storage function provided by the non-relevant database to structure Data is converted to unstructured data in units of conduct.

Next, in step 220, the primary key value required for the unstructured data is generated, and in step 220, the primary key element is stored in the configuration file by the instant storage program module 110. The prime contains the data generation time and the primary key field of the original structured data. In addition, the unstructured data converted by the step 210 can form an unstructured data having a primary key value, and waits to be stored in the non-associated database 130 by the data storage record control module 160 in a row format. .

In step 230, the data storage record control module 160 determines each piece of data of each batch of data, firstly determines whether it is the first piece of data, and records the primary key value for the first piece of data. , is the primary key value of the first data, and then judge whether the structured data object of the batch has been converted into unstructured data, if there is any data, repeat step 220; if there is no data, it means After the pen data is the last data, and the primary key value is recorded, it is the primary key value of the last data, and step 240 is performed.

Finally, in step 240, the data storage record control module 160 stores a large amount of data storage operations generated by the instant storage program module 110 for each batch, and stores only one data storage record in the data storage record data of the non-related data storage. In the table, the data storage record table itself is also a non-associated data table. Each data storage record contains the first primary key, the last primary key of the batch, the total number of data written in the batch, and the status value. The primary key element is the pen. Data storage record data generation time.

For example, when there is a batch of 1000 data to be stored, and the primary key value element is set to the date (year, month, day) plus the serial number, the first primary key value is generated by the method of the present invention. 201311010001 and the last primary key value is 201311011000, the status value is preset to "0", and in step 240, a new data storage record is added (the data generation time, 20131101000, 201311011000, 1000, 0).

When a large amount of real-time data is written into a non-associative database, it also needs to be read out immediately for subsequent related applications, such as providing immediate arithmetic processing. Figure 3 is in accordance with this A flow chart of a method for instantly reading a huge amount of data using a non-associated database is shown in an embodiment of the present invention. First, as shown in step 310, the data storage record control module 160 reads a data storage in the current time sequence. Record data sheet, the reading rule is up to two minutes within three minutes, and the time that has not been read is recorded as the earliest data storage record, up to two strokes. The detailed method is to first query the latest data in the data storage record data sheet. The time of the data, then compare the time of each record to filter out the effective record within three minutes, and pass the status field value of the valid record through the built-in function of NoSql database (such as incrementColumnValue() in HBase The action of automatically incrementing after the LOCK field is performed, and then determining whether the status field value is "1" (indicating that it has not been read yet), and if it is "1", storing the queried data. The record is passed to step 320, otherwise the data is ignored. The method of filtering effective records can achieve a control method, which allows multiple client programs to simultaneously read, and each reads non-repeated data segments to achieve data dispersion processing purposes.

In step 320, the batch large-scale data query condition analysis operation is performed and a query condition object is generated. The data storage record control module 160 performs the data storage record for batch data query condition analysis operation and generates a query condition object, and then transmits the data. The step 330 is used to read the program.

After obtaining a query condition object in step 330, the data query function provided by a non-associative database is combined with the query condition object to perform batch batch data query operation.

For example, if the content of the data storage record queried in step 310 is (data generation time, 20131101000, 201311011000, 1000, 1), after the step 320 is parsed, the first primary key of the batch is obtained. The value is 20131101000 and the last primary key value is 201311011000, and then through step 330, the query condition of combining a non-associated database is (STARTROW=20131101000, STOPROW=201311011000), and then batch batch data query operation is performed.

Finally, in step 340, the batch of unstructured data is converted into structured data objects, and the data provider is provided by the data provider according to one of the structured data objects (JAVA object) programs, and is provided by the non-associated database. After the reading function is taken out one by one, the unstructured data field data of the behavior unit is converted into a structured data object.

The instant storage and reading method for the huge amount of data provided by the invention for the non-associated database has the following advantages when compared with other conventional technologies:

1. Easy-to-use interface to convert unstructured data through structured data objects, allowing users to use structured data objects to store huge amounts of data in unrelated databases without the need for non-relevant database-related skills. With the read job.

2. Provide decentralized multiplexed access data mechanism, which can provide distributed multiplexed access data mechanism according to real-time computing processing performance requirements. The instant reading program module can display the number of data receiving interfaces used in the current, and allow multiple client programs. Simultaneous reading, each reading non-repeated data segments to increase the immediacy of data reading.

3. All data can be completely preserved, and the application scenarios of historical data analysis can be taken into consideration. At the same time, the data receiving end can obtain the latest and unprocessed data, and solve the huge problem realized by the memory database (In-memory Database). The short-term access mechanism of the quantity data, the shortcoming of the data is lost after the system is shut down.

The detailed description above is a detailed description of one of the possible embodiments of the present invention, but the embodiment is not intended to limit the scope of the invention. The equivalent implementation or change of God shall be included in the scope of the patent in this case.

To sum up, this case is not only innovative in terms of technical thinking, but also has many of the above-mentioned functions that are not in the traditional methods of the past. It has fully complied with the statutory invention patent requirements of novelty and progressiveness, and applied for it according to law. Approved this invention patent application, in order to invent invention, to the sense of virtue.

110‧‧‧ Instant Storage Module

120‧‧‧ data receiving interface

130‧‧‧Unrelated database

140‧‧‧ Instant Reader Module

150‧‧‧ data reading interface

160‧‧‧ Data Storage Record Control Module

Claims (9)

A method for instant storage and reading of huge amounts of data for a non-associative database, the steps of which are: a. converting structured objects to unstructured data (210); b. generating primary key values (220); Generate batch data storage records and related indexes (230); d. store a large amount of data in each batch, store only one data storage record in a data storage record data table, and store batch data in non-associated In the database (240); e. reading a data storage record data table (310); f parsing a batch of large data query conditions and generating a query condition (320); g. using a non-relevant database The data query function is combined with the query condition object to perform batch data query (330); h. batch large amount of unstructured data conversion structured data object (340). An instant storage and reading method for a huge amount of data for a non-associated database, as described in claim 1, wherein the structured data object conversion unstructured data (210) is an instant storage program module ( 110) The instant storage program module (110) provides a data receiving interface (120) for receiving the structured original data object provided by the data provider and converting the original data object into unstructured data. An instant data storage and reading method for a non-relevant database according to claim 1, wherein the generating data primary key value (220) is executed by the instant storage program module (110). , the first key element is stored in the set file, when the structured data object is connected After receiving, the primary key element is used to generate the primary key used to generate the structured data object, converted into unstructured data, and then stored in the line format by the data storage record control module (160). Database (130). An instant data storage and reading method for a non-relevant database according to claim 3, wherein the primary key element includes a data generation time and a primary key field of the original structured data. An instant storage and reading method for a huge amount of data for a non-associated database, as described in claim 1, wherein the batch data storage record and related index (230) are data storage record control modules. (160) Execution, the data is judged by a large amount of data in each batch. First, it is judged whether it is the first data. If the primary data is recorded for the first data, it is the primary key of the first data. Value, and then judge whether the materialized data object of the batch has been converted into unstructured data, and if there is still data, the structural data object is converted into unstructured data (210)); if there is no data Indicates that the data is the last data and records the primary key value, which is the primary key value of the last data. An instant storage and reading method for a huge amount of data for a non-associated database, as described in claim 1, wherein each batch of large-scale data storage operations is stored in a data storage record with only one data storage record. The data sheet (240) is executed by the data storage record control module (160). The data storage operations generated by the batch of instant storage module (110) are stored in a data storage record. In a non-relevant database data storage record data table, the data storage record includes a first primary key, a last primary key, a total number of data written in the batch, and a status value, wherein the primary key element stores the recorded data generation time by using the data. An instant storage and reading method for a huge amount of data for a non-associated database, as described in claim 1, wherein reading a data storage record data table (310) is an instant reading program module ( 140) Executing with the data storage record control module (160), the data storage record control module (160) first reads a data storage record data table in the current time sequence, and the reading rule is within a set period of time. The earliest data storage record is recorded at the time when it has not been read, up to two times. An instant storage and reading method for a huge amount of data for a non-associated database, as described in claim 1, wherein the batch of large data query conditions is parsed and a query condition (320) is generated by the data storage record. The control module (160) executes the batch data query condition analysis operation according to the data storage record and generates a query condition object to provide an instant reading program module (140) for batch large data query operation. An instant reading method for a huge amount of data for a non-associated database, as described in claim 1, wherein a large number of unstructured data conversion structured data objects (340) are instant reading programs. Executed by the module (140), the instant reading program module includes a data reading interface (150), which is responsible for providing the data requester to perform data encapsulation operations, and the reading function provided by the non-relevant database is written one by one. After being taken out, the unstructured data field data in the unit of action is converted into a structured data object.