CN110659295A - Method, apparatus and medium for recording valid data based on HAWQ - Google Patents
Method, apparatus and medium for recording valid data based on HAWQ Download PDFInfo
- Publication number
- CN110659295A CN110659295A CN201910911896.1A CN201910911896A CN110659295A CN 110659295 A CN110659295 A CN 110659295A CN 201910911896 A CN201910911896 A CN 201910911896A CN 110659295 A CN110659295 A CN 110659295A
- Authority
- CN
- China
- Prior art keywords
- hawq
- field
- data
- reference table
- record
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/22—Indexing; Data structures therefor; Storage structures
- G06F16/2282—Tablespace storage structures; Management thereof
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/21—Design, administration or maintenance of databases
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02D—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE
- Y02D10/00—Energy efficient computing, e.g. low power processors, power management or thermal management
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Databases & Information Systems (AREA)
- Data Mining & Analysis (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Software Systems (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The embodiment of the invention discloses a method, a device and a medium for recording effective data based on HAWQ, which call a preset script program to generate an HAWQ reference table and an HAWQ view; and setting corresponding field identification for the source data according to the operation type corresponding to the received source data. The source data and its corresponding field identification are added as a record to the HAWQ reference table. And setting corresponding field identification for the source data, and effectively recording which record is added, which record is modified and which record is deleted. Because the HAWQ reference table can only add data and does not support the modification and deletion of the data, the HAWQ reference table has a lot of invalid data, in order to improve the calling efficiency of the data, valid data can be screened out from the HAWQ reference table according to the key value and the field identification corresponding to each record, and the valid data is stored in the HAWQ view, so that the problem that the HAWQ reference table cannot delete and modify the history records is solved.
Description
Technical Field
The present invention relates to the field of data storage technology, and in particular, to a method, an apparatus, and a computer-readable storage medium for recording valid data based on HAWQ.
Background
The HAWQ is a Hadoop native SQL query engine, and combines the key technical advantages of the MPP database and the expandability and convenience of Hadoop. The HAWQ reads data locally from the HDFS and writes data to the HDFS. Because data in the HDFS file can only be added, and modification or deletion is not allowed at present, data in the HAWQ reference table only allows the mode of addition, and does not allow the mode of modification or deletion.
But in real-world applications, some data records need to be modified or deleted many times. Because the data in the HAWQ reference table is only allowed to be added, when a certain item of data in the HAWQ reference table needs to be modified, only new data can be added in the HAWQ table, and for a user, it is not clear whether the newly added data belongs to independent new data, or belongs to modified data or deleted data of original data, which is not beneficial to query and call of the data.
It can be seen that how to solve the problem that the HAWQ reference table cannot delete and modify the history is a problem that needs to be solved by those skilled in the art.
Disclosure of Invention
An object of an embodiment of the present invention is to provide a method, apparatus, and computer-readable storage medium for recording valid data based on HAWQ, which can solve the problem that a HAWQ reference table cannot delete and modify a history.
To solve the above technical problem, an embodiment of the present invention provides a method for recording valid data based on HAWQ, including:
calling a preset script program to generate an HAWQ reference table and an HAWQ view;
setting corresponding field identification for the source data according to the operation type corresponding to the received source data;
adding the source data and the corresponding field identification thereof as a record to the HAWQ reference table;
and screening out valid data from the HAWQ reference table according to the key value and the field identification corresponding to each record, and storing the valid data into the HAWQ view so as to directly call the valid data from the HAWQ view.
Optionally, the field identification includes a version field and a deletion field.
Optionally, the setting, according to the operation type corresponding to the received source data, a corresponding field identifier for the source data includes:
when the operation type corresponding to the source data is data adding operation or data modifying operation, adding 1 to the field value of the version field of the source data, and updating the deleted field of the source data into an invalid deleted identifier;
and when the operation type corresponding to the source data is data deletion, adding 1 to the field value of the version field of the source data, and updating the deletion field of the source data into an effective deletion identifier.
Optionally, the screening out valid data from the HAWQ reference table according to the key value and the field identifier corresponding to each record includes:
querying all target records corresponding to the target key values from the HAWQ reference table; each record corresponding to the same object has the same key value; the target key value is any one of all key values;
according to the field value of each target record, selecting the latest target record corresponding to the target key value from all the target records;
when the deletion field of the latest target record is a valid deletion identifier, querying all records corresponding to the next key value from the HAWQ reference table;
and when the deletion field of the latest target record is an invalid deletion identifier, taking the latest target record as valid data of the target key value until all records in the HAWQ reference table are traversed, and ending the operation.
Optionally, the selecting, according to the field value of each target record, the latest target record corresponding to the target key value from all the target records includes:
according to the field values of the target records, performing descending arrangement on all the target records;
and taking the first arranged target record as the latest target record corresponding to the target key value.
Optionally, the invoking a preset script program, and the generating the HAWQ reference table and the HAWQ view includes:
calling the HAWQ reference table to generate a script program according to the requirement of the number of items, and generating the HAWQ reference table in batches;
and calling the HAWQ view generation script program according to the requirement of the number of the items, and generating the HAWQ reference table in batches.
The embodiment of the invention also provides a device for recording effective data based on HAWQ, which comprises a generating unit, a setting unit, an adding unit and a screening unit;
the generating unit is used for calling a preset script program to generate an HAWQ reference table and an HAWQ view;
the setting unit is used for setting corresponding field identification for the source data according to the operation type corresponding to the received source data;
the adding unit is used for adding the source data and the corresponding field identification thereof as a record to the HAWQ reference table;
and the screening unit is used for screening out effective data from the HAWQ reference table according to the key value and the field identification corresponding to each record, and storing the effective data into the HAWQ view so as to directly call the effective data from the HAWQ view.
Optionally, the field identification includes a version field and a deletion field.
Optionally, the setting unit includes a first updating subunit and a second updating subunit;
the first updating subunit is configured to, when the operation type corresponding to the source data is a data addition operation or a data modification operation, add 1 to a field value of a version field of the source data, and update a deletion field of the source data to an invalid deletion identifier;
and the second updating subunit is configured to, when the operation type corresponding to the source data is data deletion, add 1 to the field value of the version field of the source data, and update the deletion field of the source data to an effective deletion identifier.
Optionally, the screening unit includes a query subunit, a selection subunit, and a serving subunit;
the query subunit is configured to query all target records corresponding to the target key values from the HAWQ reference table; each record corresponding to the same object has the same key value; the target key value is any one of all key values;
the selecting subunit is configured to select, according to a field value of each target record, a latest target record corresponding to the target key value from all target records;
when the deletion field of the latest target record is a valid deletion identifier, triggering the query subunit to query all records corresponding to the next key value from the HAWQ reference table;
and the serving as a subunit, configured to, when the deletion field of the latest target record is an invalid deletion identifier, take the latest target record as valid data of the target key value, and end the operation until all records in the HAWQ reference table are traversed.
Optionally, the selecting subunit is specifically configured to perform descending order arrangement on all the target records according to the field value of the target record; and taking the first arranged target record as the latest target record corresponding to the target key value.
Optionally, the generating unit includes a first calling subunit and a second calling subunit;
the first calling subunit is used for calling the HAWQ reference table to generate a script program according to the requirement of the number of items and generating the HAWQ reference table in batches;
and the second calling subunit is used for calling the HAWQ view generation script program according to the requirement of the number of the items and generating the HAWQ reference table in batches.
An embodiment of the present invention further provides a device for recording valid data based on HAWQ, including:
a memory for storing a computer program;
a processor for executing the computer program to implement the steps of the method of recording valid data based on HAWQ as in any one of the above.
An embodiment of the present invention also provides a computer-readable storage medium, on which a computer program is stored, which, when being executed by a processor, implements the steps of the method for recording valid data based on HAWQ as described in any one of the above.
According to the technical scheme, a preset script program is called to generate an HAWQ reference table and an HAWQ view; and setting corresponding field identification for the source data according to the operation type corresponding to the received source data. The source data and its corresponding field identification are added as a record to the HAWQ reference table. The operation type can be various, including addition, modification and deletion of data, and the field identification reflects the operation type of the source data. And setting a corresponding field identifier for each source data, thereby effectively recording which record is added, which record is modified and which record is deleted under the characteristic that the HAWQ reference table only allows the addition of the record. Each record has a corresponding key value, and the records corresponding to the same object have the same key value. In order to improve the calling efficiency of the data, effective data can be screened out from the HAWQ reference table according to the key value and the field identification corresponding to each record, and the effective data is stored in the HAWQ view, so that the effective data can be called directly from the HAWQ view, and the problem that the HAWQ reference table cannot be deleted and the history record cannot be modified is solved.
Drawings
In order to illustrate the embodiments of the present invention more clearly, the drawings that are needed in the embodiments will be briefly described below, and it is obvious that the drawings in the following description are only some embodiments of the present invention, and that other drawings can be obtained by those skilled in the art without inventive effort.
Fig. 1 is a flowchart of a method for recording valid data based on HAWQ according to an embodiment of the present invention;
fig. 2 is a schematic structural diagram of an apparatus for recording valid data based on HAWQ according to an embodiment of the present invention;
fig. 3 is a schematic hardware structure diagram of an apparatus for recording valid data based on HAWQ according to an embodiment of the present invention.
Detailed Description
The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments. All other embodiments obtained by a person of ordinary skill in the art based on the embodiments of the present invention without any creative work belong to the protection scope of the present invention.
In order that those skilled in the art will better understand the disclosure, the invention will be described in further detail with reference to the accompanying drawings and specific embodiments.
Next, a method for recording valid data based on HAWQ according to an embodiment of the present invention will be described in detail. Fig. 1 is a flowchart of a method for recording valid data based on HAWQ according to an embodiment of the present invention, where the method includes:
s101: and calling a preset script program to generate a HAWQ reference table and a HAWQ view.
In the embodiment of the present invention, the HAWQ reference table generation script program and the HAWQ view generation script program may be set in advance.
In practical application, according to the requirement of the number of items, calling the HAWQ reference table to generate a script program, and generating the HAWQ reference table in batches; and calling the HAWQ view generation script program according to the requirement of the number of the items, and generating the HAWQ reference table in batches.
The HAWQ reference table is used to record the received source data, including adding source data, modifying source data, deleting source data, and the like.
Since the HAWQ reference table only allows for add mode, there may be a large amount of duplicate or invalid data in the HAWQ reference table, and in an embodiment of the invention, the HAWQ view is used to record valid data after the HAWQ reference table excludes duplicate and invalid data.
S102: and setting corresponding field identification for the source data according to the operation type corresponding to the received source data.
The operation types corresponding to the source data may include add, modify, and delete operations. In order to distinguish between different operation types, a corresponding field identification may be set for each operation type.
The specific form of the field identifier may be various, for example, different operation types may be represented by different forms of character strings.
In the embodiment of the present invention, the field identifier may be represented in the form of a version field and a deletion field.
The version field is used to indicate version information of the source data, and in the embodiment of the present invention, a field value of the version field may be indicated by a number.
The delete field is used to indicate whether the source data performs a delete operation, and the delete field may include both invalid delete identification and valid delete identification.
In a specific implementation, when the operation type corresponding to the source data is a data adding operation or a data modifying operation, the field value of the version field of the source data is added with 1, and the deleted field of the source data is updated to be an invalid deleted identifier. And when the operation type corresponding to the source data is data deletion, adding 1 to the field value of the version field of the source data, and updating the deletion field of the source data into an effective deletion identifier.
In the initial state, a field value of the version field may be set to 0. When new source data needs to be added, the field value record of the version field corresponding to the source data at this time is 1.
Taking one source data as an example, by querying the source data already recorded in the HAWQ reference table, the current latest field value of the source data can be obtained, and when the source data needs to be modified, 1 can be added to the current latest field value to serve as the field value of the source data to be modified.
For example, when 2 records have been recorded in the HAWQ reference table for the same source data, the field value of the version field corresponding to the source data is 2, and when the source data to be modified is obtained when the modification operation of the source data is received, the field value of the version field corresponding to the source data to be modified is 3.
S103: the source data and its corresponding field identification are added as a record to the HAWQ reference table.
Each source data has its corresponding field identification, and in the embodiment of the present invention, when the source data is added to the HAWQ reference table, its corresponding field identification may be added to the HAWQ reference table together.
According to the field identification contained in each record, which record is added, which record is modified and which record is deleted can be clearly known.
S104: and screening effective data from the HAWQ reference table according to the key value and the field identification corresponding to each record, and storing the effective data into the HAWQ view so as to directly call the effective data from the HAWQ view.
For the same source data, multiple records may exist in the HAWQ reference table, so that redundancy of data exists in the HAWQ reference table, and valid data in the HAWQ reference table can be extracted for facilitating call of subsequent data.
The plurality of records corresponding to the same source data have the same key value. Therefore, in the embodiment of the present invention, all records corresponding to the source data can be extracted from the HAWQ reference table depending on the key value.
The processing mode of each source datum is similar, and in the embodiment of the present invention, an example of any one of all key values, that is, a target key value, is used as an introduction. In a specific implementation, all target records corresponding to the target key value may be searched from the HAWQ reference table, and the latest target record corresponding to the target key value may be selected from all target records according to the field value of each target record.
In the embodiment of the present invention, the record corresponding to the target key value may be referred to as a target record. The field value corresponding to each record reflects the version of the record, and the field values are sequentially added with 1 in combination with the adding, modifying or deleting operation of the source data. Therefore, the target record with the largest value of the field value in all the target records is the latest target record corresponding to the target key value.
In a specific implementation, all the target records can be sorted in a descending order according to the field values of the target records; and taking the first aligned target record as the latest target record corresponding to the target key value.
When the deletion field of the latest target record is a valid deletion identifier, it indicates that the latest operation corresponding to the source data is a deletion operation, all records in the HAWQ reference table related to the source data are invalid data, and at this time, the next source data can be processed, that is, all records corresponding to the next key value are queried from the HAWQ reference table.
When the deletion field of the latest target record is an invalid deletion identifier, it indicates that the latest operation corresponding to the source data is an adding or modifying operation, and at this time, the latest target record can be used as valid data of the target key value, and the valid data is added to the HAWQ view until all records in the HAWQ reference table are traversed, and the operation is ended.
According to the technical scheme, a preset script program is called to generate an HAWQ reference table and an HAWQ view; and setting corresponding field identification for the source data according to the operation type corresponding to the received source data. The source data and its corresponding field identification are added as a record to the HAWQ reference table. The operation type can be various, including addition, modification and deletion of data, and the field identification reflects the operation type of the source data. And setting a corresponding field identifier for each source data, thereby effectively recording which record is added, which record is modified and which record is deleted under the characteristic that the HAWQ reference table only allows the addition of the record. Each record has a corresponding key value, and the records corresponding to the same object have the same key value. In order to improve the calling efficiency of the data, effective data can be screened out from the HAWQ reference table according to the key value and the field identification corresponding to each record, and the effective data is stored in the HAWQ view, so that the effective data can be called directly from the HAWQ view, and the problem that the HAWQ reference table cannot be deleted and the history record cannot be modified is solved.
Fig. 2 is a schematic structural diagram of an apparatus for recording valid data based on HAWQ according to an embodiment of the present invention, which includes a generating unit 21, a setting unit 22, an adding unit 23, and a screening unit 24;
a generating unit 21, configured to invoke a preset script program, and generate a HAWQ reference table and a HAWQ view;
a setting unit 22, configured to set a corresponding field identifier for the source data according to the operation type corresponding to the received source data;
an adding unit 23, configured to add the source data and the corresponding field identifier as a record to the HAWQ reference table;
and the screening unit 24 is used for screening the valid data from the HAWQ reference table according to the key value and the field identification corresponding to each record, and storing the valid data into the HAWQ view so as to directly call the valid data from the HAWQ view.
Optionally, the field identification includes a version field and a delete field.
Optionally, the setting unit includes a first updating subunit and a second updating subunit;
the first updating subunit is used for adding 1 to the field value of the version field of the source data and updating the deleted field of the source data into an invalid deleted identifier when the operation type corresponding to the source data is data adding operation or data modifying operation;
and the second updating subunit is used for adding 1 to the field value of the version field of the source data and updating the deleted field of the source data into an effective deleted identifier when the operation type corresponding to the source data is data deletion.
Optionally, the screening unit includes a query subunit, a selection subunit, and a serving subunit;
the query subunit is used for querying all target records corresponding to the target key values from the HAWQ reference table; each record corresponding to the same object has the same key value; the target key value is any one of all key values;
the selecting subunit is used for selecting the latest target record corresponding to the target key value from all the target records according to the field value of each target record;
when the deletion field of the latest target record is a valid deletion identifier, triggering the query subunit to query all records corresponding to the next key value from the HAWQ reference table;
and the sub-unit is used for taking the latest target record as the valid data of the target key value when the deletion field of the latest target record is the invalid deletion identification, and ending the operation until all records in the HAWQ reference table are traversed.
Optionally, the selecting subunit is specifically configured to perform descending order arrangement on all the target records according to the field value of the target record; and taking the first aligned target record as the latest target record corresponding to the target key value.
Optionally, the generating unit includes a first calling subunit and a second calling subunit;
the first calling subunit is used for calling the HAWQ reference table to generate a script program according to the requirement of the number of items and generating the HAWQ reference table in batches;
and the second calling subunit is used for calling the HAWQ view generation script program according to the requirement of the number of the items and generating the HAWQ reference table in batches.
The description of the features in the embodiment corresponding to fig. 2 may refer to the related description of the embodiment corresponding to fig. 1, and is not repeated here.
According to the technical scheme, a preset script program is called to generate an HAWQ reference table and an HAWQ view; and setting corresponding field identification for the source data according to the operation type corresponding to the received source data. The source data and its corresponding field identification are added as a record to the HAWQ reference table. The operation type can be various, including addition, modification and deletion of data, and the field identification reflects the operation type of the source data. And setting a corresponding field identifier for each source data, thereby effectively recording which record is added, which record is modified and which record is deleted under the characteristic that the HAWQ reference table only allows the addition of the record. Each record has a corresponding key value, and the records corresponding to the same object have the same key value. In order to improve the calling efficiency of the data, effective data can be screened out from the HAWQ reference table according to the key value and the field identification corresponding to each record, and the effective data is stored in the HAWQ view, so that the effective data can be called directly from the HAWQ view, and the problem that the HAWQ reference table cannot be deleted and the history record cannot be modified is solved.
Fig. 3 is a schematic hardware structure diagram of an apparatus 30 for recording valid data based on HAWQ according to an embodiment of the present invention, including:
a memory 31 for storing a computer program;
a processor 32 for executing a computer program to implement the steps of any of the above-described methods of recording valid data based on HAWQ.
An embodiment of the present invention further provides a computer-readable storage medium, on which a computer program is stored, and the computer program, when executed by a processor, implements the steps of any one of the above-mentioned methods for recording valid data based on HAWQ.
A method, an apparatus, and a computer-readable storage medium for recording valid data based on HAWQ according to embodiments of the present invention are described in detail above. The embodiments are described in a progressive manner in the specification, each embodiment focuses on differences from other embodiments, and the same and similar parts among the embodiments are referred to each other. The device disclosed by the embodiment corresponds to the method disclosed by the embodiment, so that the description is simple, and the relevant points can be referred to the method part for description. It should be noted that, for those skilled in the art, it is possible to make various improvements and modifications to the present invention without departing from the principle of the present invention, and those improvements and modifications also fall within the scope of the claims of the present invention.
Those of skill would further appreciate that the various illustrative elements and algorithm steps described in connection with the embodiments disclosed herein may be implemented as electronic hardware, computer software, or combinations of both, and that the various illustrative components and steps have been described above generally in terms of their functionality in order to clearly illustrate this interchangeability of hardware and software. Whether such functionality is implemented as hardware or software depends upon the particular application and design constraints imposed on the implementation. Skilled artisans may implement the described functionality in varying ways for each particular application, but such implementation decisions should not be interpreted as causing a departure from the scope of the present invention.
The steps of a method or algorithm described in connection with the embodiments disclosed herein may be embodied directly in hardware, in a software module executed by a processor, or in a combination of the two. A software module may reside in Random Access Memory (RAM), memory, Read Only Memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art.
Claims (10)
1. A method of recording valid data based on HAWQ, comprising:
calling a preset script program to generate an HAWQ reference table and an HAWQ view;
setting corresponding field identification for the source data according to the operation type corresponding to the received source data;
adding the source data and the corresponding field identification thereof as a record to the HAWQ reference table;
and screening out valid data from the HAWQ reference table according to the key value and the field identification corresponding to each record, and storing the valid data into the HAWQ view so as to directly call the valid data from the HAWQ view.
2. The method of claim 1, wherein the field identification comprises a version field and a delete field.
3. The method according to claim 2, wherein the setting of the corresponding field identifier for the source data according to the operation type corresponding to the received source data comprises:
when the operation type corresponding to the source data is data adding operation or data modifying operation, adding 1 to the field value of the version field of the source data, and updating the deleted field of the source data into an invalid deleted identifier;
and when the operation type corresponding to the source data is data deletion operation, adding 1 to the field value of the version field of the source data, and updating the deletion field of the source data into an effective deletion identifier.
4. The method of claim 3, wherein the filtering out valid data from the HAWQ reference table according to the key value and the field identification corresponding to each record comprises:
querying all target records corresponding to the target key values from the HAWQ reference table; each record corresponding to the same object has the same key value; the target key value is any one of all key values;
according to the field value of each target record, selecting the latest target record corresponding to the target key value from all the target records;
when the deletion field of the latest target record is a valid deletion identifier, querying all records corresponding to the next key value from the HAWQ reference table;
and when the deletion field of the latest target record is an invalid deletion identifier, taking the latest target record as valid data of the target key value until all records in the HAWQ reference table are traversed, and ending the operation.
5. The method according to claim 4, wherein the extracting the latest target record corresponding to the target key value from all the target records according to the field value of each target record comprises:
according to the field values of the target records, performing descending arrangement on all the target records;
and taking the first arranged target record as the latest target record corresponding to the target key value.
6. The method of any one of claims 1-5, wherein the invoking a pre-defined script program to generate the HAWQ reference table and the HAWQ view comprises:
calling the HAWQ reference table to generate a script program according to the requirement of the number of items, and generating the HAWQ reference table in batches;
and calling the HAWQ view generation script program according to the requirement of the number of the items, and generating the HAWQ reference table in batches.
7. An apparatus for recording valid data based on HAWQ, comprising a generating unit, a setting unit, an adding unit and a screening unit;
the generating unit is used for calling a preset script program to generate an HAWQ reference table and an HAWQ view;
the setting unit is used for setting corresponding field identification for the source data according to the operation type corresponding to the received source data;
the adding unit is used for adding the source data and the corresponding field identification thereof as a record to the HAWQ reference table;
and the screening unit is used for screening out effective data from the HAWQ reference table according to the key value and the field identification corresponding to each record, and storing the effective data into the HAWQ view so as to directly call the effective data from the HAWQ view.
8. The apparatus of claim 7, wherein the field identification comprises a version field and a delete field.
9. An apparatus for recording valid data based on HAWQ, comprising:
a memory for storing a computer program;
a processor for executing the computer program to implement the steps of the method of recording valid data based on HAWQ according to any one of claims 1 to 6.
10. A computer-readable storage medium, having stored thereon a computer program which, when being executed by a processor, carries out the steps of the method of recording valid data based on HAWQ according to any one of claims 1 to 6.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910911896.1A CN110659295B (en) | 2019-09-25 | 2019-09-25 | Method, apparatus and medium for recording valid data based on HAWQ |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910911896.1A CN110659295B (en) | 2019-09-25 | 2019-09-25 | Method, apparatus and medium for recording valid data based on HAWQ |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN110659295A true CN110659295A (en) | 2020-01-07 |
| CN110659295B CN110659295B (en) | 2023-03-24 |
Family
ID=69039117
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201910911896.1A Active CN110659295B (en) | 2019-09-25 | 2019-09-25 | Method, apparatus and medium for recording valid data based on HAWQ |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN110659295B (en) |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111797108A (en) * | 2020-07-17 | 2020-10-20 | 西安数据如金信息科技有限公司 | Method and device for updating analysis database |
| CN115544048A (en) * | 2022-11-29 | 2022-12-30 | 福建慧政通信息科技有限公司 | Method and terminal for monitoring data change |
Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20090300069A1 (en) * | 2008-05-29 | 2009-12-03 | O'sullivan Michael Patrick | Method and system for the logical deletion of relational database records |
| CN104317963A (en) * | 2014-11-14 | 2015-01-28 | 中国建设银行股份有限公司 | Method and device for processing data |
| US20160364655A1 (en) * | 2016-04-07 | 2016-12-15 | Shahzad Muhammad | System to generate Logical Design for MPP Clusters using self-learning model |
| CN108427684A (en) * | 2017-02-14 | 2018-08-21 | 华为技术有限公司 | Data query method, apparatus and computing device |
| CN108932286A (en) * | 2018-05-23 | 2018-12-04 | 阿里巴巴集团控股有限公司 | A kind of date storage method, data query method and device |
-
2019
- 2019-09-25 CN CN201910911896.1A patent/CN110659295B/en active Active
Patent Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20090300069A1 (en) * | 2008-05-29 | 2009-12-03 | O'sullivan Michael Patrick | Method and system for the logical deletion of relational database records |
| CN104317963A (en) * | 2014-11-14 | 2015-01-28 | 中国建设银行股份有限公司 | Method and device for processing data |
| US20160364655A1 (en) * | 2016-04-07 | 2016-12-15 | Shahzad Muhammad | System to generate Logical Design for MPP Clusters using self-learning model |
| CN108427684A (en) * | 2017-02-14 | 2018-08-21 | 华为技术有限公司 | Data query method, apparatus and computing device |
| CN108932286A (en) * | 2018-05-23 | 2018-12-04 | 阿里巴巴集团控股有限公司 | A kind of date storage method, data query method and device |
Non-Patent Citations (2)
| Title |
|---|
| LEI C: "HAWQ: a massively parallel processing SQL engine in hadoop", 《ACM》 * |
| 徐国天: "不依赖日志文件的Oracle数据库综合检验工具开发", 《网络安全技术与应用》 * |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111797108A (en) * | 2020-07-17 | 2020-10-20 | 西安数据如金信息科技有限公司 | Method and device for updating analysis database |
| CN115544048A (en) * | 2022-11-29 | 2022-12-30 | 福建慧政通信息科技有限公司 | Method and terminal for monitoring data change |
Also Published As
| Publication number | Publication date |
|---|---|
| CN110659295B (en) | 2023-03-24 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN107832406B (en) | Method, device, equipment and storage medium for removing duplicate entries of mass log data | |
| US7117294B1 (en) | Method and system for archiving and compacting data in a data storage array | |
| CN108875077B (en) | Column storage method and device of database, server and storage medium | |
| CN108536745B (en) | Shell-based data table extraction method, terminal, equipment and storage medium | |
| US20220179642A1 (en) | Software code change method and apparatus | |
| CN107145538B (en) | Table data query method, device and system | |
| CN110659295B (en) | Method, apparatus and medium for recording valid data based on HAWQ | |
| CN105488050A (en) | Database multi-index method, apparatus and system | |
| CN111752925B (en) | Visual data management method | |
| CN109086456B (en) | Data indexing method and device | |
| CN111914020A (en) | Data synchronization method and device and data query method and device | |
| CN114281779A (en) | Data synchronization method and device, computer equipment and storage medium | |
| US20090313212A1 (en) | Relational database with compound identifier | |
| CN111046106A (en) | Cache data synchronization method, device, equipment and medium | |
| CN108038253B (en) | Log query processing method and device | |
| CN106503186A (en) | A kind of data managing method, client and system | |
| CN110362540B (en) | Data storage and visitor number acquisition method and device | |
| CN109660587B (en) | Data pushing method and device based on random number, storage medium and server | |
| CN109101191B (en) | Data storage method, data storage device, storage medium and electronic equipment | |
| CN113268288B (en) | Shared file configuration method and device | |
| CN105893445A (en) | Data processing method, server and terminal device | |
| CN106372121B (en) | Server and data processing method | |
| JP6646754B2 (en) | Job scheduling system, job scheduling method, and job scheduling device | |
| CN109740762A (en) | Feature selection approach, device, storage medium and electronic equipment | |
| CN105740131B (en) | Software user behavior rollback processing method and device |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |