CN112612818B - Data processing method and device, computing equipment and storage medium - Google Patents
Data processing method and device, computing equipment and storage medium Download PDFInfo
- Publication number
- CN112612818B CN112612818B CN202011542519.4A CN202011542519A CN112612818B CN 112612818 B CN112612818 B CN 112612818B CN 202011542519 A CN202011542519 A CN 202011542519A CN 112612818 B CN112612818 B CN 112612818B
- Authority
- CN
- China
- Prior art keywords
- data
- written
- field
- time axis
- time
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 238000003672 processing method Methods 0.000 title claims abstract description 37
- 238000000034 method Methods 0.000 claims abstract description 24
- 238000004590 computer program Methods 0.000 claims description 9
- 238000006243 chemical reaction Methods 0.000 claims description 5
- 238000010586 diagram Methods 0.000 description 5
- 238000004891 communication Methods 0.000 description 4
- 230000004048 modification Effects 0.000 description 2
- 238000012986 modification Methods 0.000 description 2
- 230000003287 optical effect Effects 0.000 description 2
- 238000004458 analytical method Methods 0.000 description 1
- 230000007547 defect Effects 0.000 description 1
- 238000005516 engineering process Methods 0.000 description 1
- 238000002955 isolation Methods 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
- 230000001502 supplementing effect Effects 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/245—Query processing
- G06F16/2457—Query processing with adaptation to user needs
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/248—Presentation of query results
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/27—Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/28—Databases characterised by their database models, e.g. relational or object models
- G06F16/284—Relational databases
- G06F16/285—Clustering or classification
- G06F16/287—Visualization; Browsing
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Databases & Information Systems (AREA)
- Data Mining & Analysis (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Computational Linguistics (AREA)
- Computing Systems (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention provides a data processing method and device, computing equipment and a storage medium, wherein the method comprises the following steps: acquiring original data, and converting the original data into data to be written, wherein the data to be written comprises a first service table name field and a first time range field; inquiring a time axis configuration table corresponding to the service table according to the first service table name field, and acquiring historical data of a second service table name field corresponding to the first service table name field in the time axis configuration table, wherein the historical data comprises the second service table name field and a second time range field; the method comprises the steps of determining the time axis relation between data to be written and historical data according to a first time range field of the data to be written and a second time range field of the historical data, adding the data to be written into a business table based on the time axis relation between the data to be written and the historical data, obtaining the business table with the time axis relation, and facilitating query and use of a business party without manual operation.
Description
Technical Field
The present invention relates to the field of data processing technologies, and in particular, to a data processing method and apparatus, a computing device, and a storage medium.
Background
In the business side project, a business side needs to process a large amount of data, the same type of data is required to have a time axis attribute, the accurate data of each object at each time point is accurately recorded, and future effective data can be dynamically displayed.
In the prior art, data acquired by a service party does not have a time axis attribute, needs manual increase, and is time-consuming and labor-consuming.
Disclosure of Invention
The invention provides a data processing method and device, computing equipment and a storage medium, which are used for solving the technical defects in the prior art.
The invention provides a data processing method, which comprises the following steps:
acquiring original data, and converting the original data into data to be written, wherein the data to be written comprises a first service table name field and a first time range field;
inquiring a time axis configuration table corresponding to a service table according to the first service table name field, and acquiring historical data of a second service table name field corresponding to the first service table name field in the time axis configuration table, wherein the historical data comprises the second service table name field and a second time range field;
and determining the time axis relation between the data to be written and the historical data according to the first time range field of the data to be written and the second time range field of the historical data, and adding the data to be written into the business table based on the time axis relation between the data to be written and the historical data.
According to a data processing method provided by the present invention, converting the original data into data to be written, includes:
judging whether the write command is a distributed write command or a relational write command;
if the data are distributed write commands, the original data are analyzed by a time axis component and are respectively converted into first data to be written and second data to be written;
and if the data is the relational writing command, analyzing and converting the original data into first data to be written through a time axis component.
According to a data processing method provided by the present invention, determining a time axis relationship between the data to be written and the history data according to a first time range field of the data to be written and a second time range field of the history data, and adding the data to be written to the service table based on the time axis relationship between the data to be written and the history data, the method includes:
judging whether the time axis relation between the data to be written and the historical data conflicts or not according to the first time range field of the data to be written and the second time range field of the historical data;
if so, updating a first time range field of the data to be written and a second time range field of the historical data to eliminate the conflict of the time axis relationship between the data to be written and the historical data, and then adding the data to be written to the business table based on the updated time axis relationship between the data to be written and the historical data;
if not, adding the data to be written into the business table directly based on the time axis relation between the data to be written and the historical data.
According to a data processing method provided by the present invention, updating a first time range field of the data to be written and a second time range field of the history data to eliminate a conflict between the time axis relationship between the data to be written and the history data, and adding the data to be written to the service table based on the updated time axis relationship between the data to be written and the history data, includes:
determining the time sequence of the data to be written and the historical data based on the first starting time field of the data to be written and the second starting time field of the historical data; wherein the first time range field comprises a first start time field and a first end time field, and the second time range field comprises a second start time field and a second end time field;
modifying a second start time field or a second end time field of the historical data so that a first time range field of the data to be written and a second time range field of the historical data do not coincide, and then adding the data to be written to the business table based on the updated time axis relationship between the data to be written and the historical data.
According to a data processing method provided by the invention, the method further comprises the following steps: and under the condition of deleting or modifying the target historical data in the service table, deleting or modifying the target historical data, and changing a second time range field of the historical data which conflicts with the time axis relation of the target historical data according to a time axis configuration table corresponding to the service table to generate an updated service table.
According to a data processing method provided by the present invention, adding the data to be written to the service table based on a time axis relationship between the data to be written and the history data includes:
if the data to be written is a distributed writing command, based on the time axis relation between the data to be written and the historical data, adding the first data to be written into a business table of a relational database, and adding the second data to be written into the business table of the distributed database;
and if the data to be written is a relational writing command, adding the first data to be written into a business table of the relational database based on the time axis relation between the data to be written and the historical data.
According to a data processing method provided by the invention, the method further comprises the following steps:
under the condition of receiving the query command, determining a third service table name field corresponding to the query command, and judging whether the queried database is a distributed database or a relational database;
if the query is successful, returning the historical data in the second format; if the query fails, querying and returning the historical data in the first format in the service table of the relational database according to the third service table name field;
and if so, inquiring and returning the historical data in the first format in the service table of the relational database according to the third service table name field corresponding to the inquiry command.
The present invention provides a data processing apparatus, comprising:
the conversion module is used for acquiring original data and converting the original data into data to be written, wherein the data to be written comprises a first service table name field and a first time range field;
the historical data acquisition module is used for inquiring a time axis configuration table corresponding to a service table according to the first service table name field and acquiring historical data of a second service table name field corresponding to the first service table name field in the time axis configuration table, wherein the historical data comprises the second service table name field and a second time range field;
and the relation determining module is used for determining the time axis relation between the data to be written and the historical data according to the first time range field of the data to be written and the second time range field of the historical data, and adding the data to be written into the business table based on the time axis relation between the data to be written and the historical data.
The invention also provides an electronic device comprising a memory, a processor and a computer program stored on the memory and executable on the processor, wherein the processor implements the steps of any of the data processing methods described above when executing the program.
The invention also provides a non-transitory computer-readable storage medium having stored thereon a computer program which, when executed by a processor, carries out the steps of the data processing method as described in any of the above.
According to the data processing method and device provided by the invention, original data are converted into data to be written, historical data are inquired in the time axis configuration table according to the name field of the first service table of the data to be written so as to determine the time axis relation between the data to be written and the historical data, and the data to be written is written into the service table based on the time axis relation between the data to be written and the historical data, so that the service table with the time axis relation can be obtained, manual operation is not needed, and inquiry and use of a service party are facilitated.
Drawings
In order to more clearly illustrate the technical solutions of the present invention or the prior art, the drawings needed for the description of the embodiments or the prior art will be briefly described below, and it is obvious that the drawings in the following description are some embodiments of the present invention, and those skilled in the art can also obtain other drawings according to the drawings without creative efforts.
FIG. 1 is a schematic flow chart of a data processing method provided by the present invention;
FIG. 2 is a second schematic flow chart of a data processing method according to the present invention;
FIG. 3 is a diagram of written data for a new employee provided by the present invention;
FIG. 4 is a schematic diagram of employee delete transaction data provided by the present invention;
FIG. 5 is a schematic illustration of an employee's modified business data provided by the present invention;
FIG. 6 is a third schematic flow chart of a data processing method provided by the present invention;
FIG. 7 is a schematic diagram of a data processing apparatus according to the present invention;
FIG. 8 is a schematic structural diagram of an electronic device provided by the present invention;
Detailed Description
In order to make the objects, technical solutions and advantages of the present invention clearer, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings, and it is obvious that the described embodiments are some, but not all embodiments of the present invention. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.
First, the noun terms related to the present embodiment are explained.
A service table: the service data needed by the service party is stored in the form of a service table. Some of the service data in the service table need to have a time axis sequence to associate the relationship between the service data.
Time axis configuration table: and the configuration manager is used for processing the service data needing to be accessed to the time axis to obtain a configuration table which is used for determining the relation between the data to be written of the original data and the historical data.
Configuring an association table: and the table related to the time axis configuration table is stored in a server of a service party, and the configuration related table and the time axis configuration table are in a many-to-one relationship. For example, the timeline configuration table is an employee profile table, and the configuration association table may include an employee age table, an employee scholarship table, an employee work experience table, and the like.
Sql (Structured Query Language): a database query and programming language for accessing data and querying, updating and managing relational databases.
ES (Elastic Search, distributed Search): the full-text search engine with distributed multi-user capability is provided, and the retrieval is fast and accurate.
The embodiment of the invention discloses a data processing method, as shown in fig. 1, comprising the following steps:
101. the method comprises the steps of obtaining original data and converting the original data into data to be written, wherein the data to be written comprises a first service table name field and a first time range field.
For example, in this embodiment, the data to be written includes two types: first data to be written for a relational database, such as a mysql database; second data to be written for a distributed database, such as an es database.
Specifically, in use, the written database is determined by setting a write switch.
If the write switch is configured with a relational database, correspondingly generating a distributed write command;
if the write switch is configured with a distributed database, a relational write command is correspondingly generated.
Step 101 comprises: judging whether the write command is a distributed write command or a relational write command; if the command is a distributed write command, analyzing the original data through a time axis component to respectively convert the original data into first data to be written and second data to be written; if the command is a relational write command, the original data is analyzed and converted into first data to be written through a time axis component.
As can be seen from the above steps, whether the command is a distributed write command or a relational write command, data is stored in the relational database mysql. The bottom layer storage adopts a mysql database and an es database, the switching from the mysql database to the es database is realized through configuration, and the service data can have the time axis capability quickly.
In order to prevent the problems of network problems, es service downtime and the like, retries are carried out through a ribbon, the retries still fail after reaching the configuration times, error alarm is printed, and manual intervention is carried out.
102. And inquiring a time axis configuration table corresponding to a service table according to the first service table name field, and acquiring historical data of a second service table name field corresponding to the first service table name field in the time axis configuration table, wherein the historical data comprises the second service table name field and a second time range field.
The time axis configuration table is generated in advance according to the service table. Correspondingly, the method further comprises: and generating a corresponding time axis configuration table according to the historical data in each service table through a configuration manager, wherein a second time range field of the historical data in the service table comprises a second starting time field and a second ending time field.
Table 1 below shows an example of a time axis configuration table.
TABLE 1
| ID | Tenant_id | api_name | description | state | Es_read_state | Es_write_state |
| 1 | ajrenown | Order | Order form | 1 | 0 | 0 |
| 2 | urixjjwr | Order_detail | Order detail table | 1 | 0 | 0 |
| 3 | wongnnsl | Coupon | Coupon table | 1 | 0 | 0 |
Wherein, Tenant _ ID is a Tenant ID, and realizes data isolation of each service party; the api _ name field is a second service table name field; the state is the state of the service data, and 0 is invalid and 1 is valid; es _ read _ state is an Es read state, and 0 is closed 1 and is opened; es _ write _ state is the Es write state, and 0 is OFF 1 is ON.
It should be noted that, in this embodiment, through the first service table name field and the second service table name field that correspond to each other, the history data that belongs to the same type as the data to be written can be found. For example, the data to be written is a piece of data to be written a1 of employee a, the first business table name field of the data to be written may be an employee file, the data to be written relates to employee data of a future period, and the corresponding first time range field is 20/11/20-21/11/20. Then, according to a first service table name field of the data to be written, searching historical data having a second service table name field corresponding to the first service table name field, and finally searching historical data a2 and A3 of the employee a, wherein corresponding second time range fields are 20/11/1-20/12/11 and 21/01/01-21/12/31 respectively, that is, the data to be written, a1, and the historical data a2 and A3 generate a conflict relationship of a time axis.
103. And determining the time axis relation between the data to be written and the historical data according to the first time range field of the data to be written and the second time range field of the historical data, and adding the data to be written into the business table based on the time axis relation between the data to be written and the historical data.
Specifically, the adding the data to be written to the service table based on the time axis relationship between the data to be written and the historical data in step 103 includes:
if the data to be written is a distributed writing command, based on the time axis relation between the data to be written and the historical data, adding the first data to be written into a business table of a relational database, and adding the second data to be written into the business table of the distributed database;
and if the data is a relational writing command, adding the first data to be written into a business table of a relational database based on the time axis relation between the data to be written and the historical data.
For example, the data to be written is a piece of preferential order data A, the first business table name field of the data to be written can be a preferential order, the data to be written only relates to the preferential measures of the last 3 days, and the corresponding first time range field is 20/11/10-20/11/12. Then, according to the "preferential order" of the first service table name field of the data to be written, the historical data with the second service table name field corresponding to the first service table name field is searched, and finally the historical data B and C are searched, wherein the corresponding second time range fields are 20/11/1-20/11/11 and 20/11/12-20/11/13 respectively, that is, the conflict relationship between the data a to be written and the historical data B and C is generated by a time axis, the time ranges of the historical data B and C need to be adjusted to eliminate the conflict relationship between the data a to be written and the time axes of the historical data B and C, and then the data a can be added into the service table to form the service table related to the time axis.
According to the data processing method provided by the invention, the original data are converted into the data to be written, the historical data are inquired in the time axis configuration table according to the name field of the first service table of the data to be written, so that the time axis relation between the data to be written and the historical data is determined, and the data to be written is written into the service table based on the time axis relation between the data to be written and the historical data, so that the service table with the time axis relation can be obtained, manual operation is not needed, and the inquiry and the use of a service party are facilitated.
The embodiment of the invention also discloses a data processing method, as shown in fig. 2, comprising the following steps 201-205:
201. the method comprises the steps of obtaining original data and converting the original data into data to be written, wherein the data to be written comprises a first service table name field and a first time range field.
Specifically, in use, the written database is determined by setting a write switch.
If the write switch is configured with the relational database mysql, correspondingly generating a distributed write command; if the write switch is configured with the distributed database es, a relational write command is correspondingly generated.
In the embodiment, the business side project has the capacity of a time axis, mysql + es is stored in the bottom layer, the project is built by using the springboot, the draid is used as a database connection pool, mybatis is used for operating the relational database mysql, and the data can be flexibly switched from the relational database mysql to the distributed database es through configuration.
Specifically, in the conversion process, step 201 includes steps S211 to S213:
step S211, determining that the write command is a distributed write command or a relational write command, if the write command is a distributed write command, performing step S212, and if the write command is a relational write command, performing step S213.
Step S212, if the command is a distributed write command, the original data is analyzed by a time axis component and is respectively converted into first data to be written and second data to be written;
if the data is the distributed writing command, based on the time axis relation between the original data and the historical data, the first data to be written is added to a business table of the relational database mysql, and the second data to be written is added to the business table of the distributed database es.
Step S213, if the command is a relational write command, converting the original data into the first data to be written through time axis component analysis.
And if the data is the relational write command, adding the first data to be written into a business table of the relational database mysql based on the time axis relation between the data to be written and the historical data. In order to prevent the problems of network problems, downtime of distributed database es service and the like, retries are carried out through a ribbon, the retries still fail after reaching the configuration times, error alarm is printed, and manual intervention is carried out.
Through steps S211 to S213, the original data can be converted into the data to be written in the corresponding format, so that the data to be written in the subsequent steps can be written in the service table.
202. And inquiring a time axis configuration table corresponding to a service table according to the first service table name field, and acquiring historical data of a second service table name field corresponding to the first service table name field in the time axis configuration table, wherein the historical data comprises the second service table name field and a second time range field.
The time axis configuration table is generated in advance according to the service table. Correspondingly, the method further comprises: and generating a corresponding time axis configuration table according to the historical data in each service table through a configuration manager.
In this embodiment, the history data of the time axis configuration table respectively has corresponding second service table name fields corresponding to different types of services. For example, the data to be written M1 includes a first service table name H2, and second service table name fields of the historical data H1, H2, H3, H4, and H5 are H1, H2, H3, H2, and H1, respectively, and the historical data H2 and H4 belonging to the same type as the data to be written M1 can be found through the first service table name field and the second service table name field corresponding to each other.
203. And judging whether the time axis relation between the data to be written and the historical data conflicts or not according to the first time range field of the data to be written and the second time range field of the historical data, if so, executing step 204, and if not, executing step 205.
The historical data is stored data in a time axis configuration table. In order to add the data to be written into the service table, it is necessary to determine whether the data to be written conflicts with the stored historical data according to the time axis, and write the data to be written into the service table after the conflicts of the time axis are eliminated.
Specifically, the data at each time point on the time axis realizes time axis data (accurate to seconds) by adding a start time field effDate field and an end time effnddate field in the service table. Through the two fields, the time interval of the data to be written on the time axis can be determined.
Then, whether the time axis relation between the data to be written and the history data conflicts or not is judged through a first starting time field and a first ending time field of the data to be written, and a second starting time field and a second ending time field of the history data.
For example, if the first start time field of the data to be written M1 is 11/15, the first end time field is 11/30, the second start time field of the history data H2 is 11/11, and the second end time field is 11/16, the data to be written and the history data H4 collide with each other in a time axis relationship; the second start time field of the history data H4 is 11/10, and the second end time field is 11/14, so that the data to be written and the history data H4 do not generate a conflict of the time axis relationship.
204. Updating the first time range field of the data to be written and the second time range field of the historical data to eliminate the conflict between the data to be written and the time axis relation of the historical data, and then adding the data to be written to the business table based on the updated time axis relation between the data to be written and the historical data.
Specifically, step 204 includes: determining the time sequence of the data to be written and the historical data based on a first starting time field of the data to be written and a second starting time field of the historical data; wherein the first time range field comprises a first start time field and a first end time field, and the second time range field comprises a second start time field and a second end time field; and modifying a second starting time field or a second ending time field of the historical data to enable a first time range field of the data to be written and a second time range field of the historical data not to coincide, and then adding the data to be written into the business table based on the updated time axis relation between the data to be written and the historical data.
For example, the start time of the data M1 to be written is 11/15, the end time is 11/30, the start time of the history data H2 is 11/11, and the end time is 11/16, then the data M1 to be written and the history data H2 generate a conflict of time axis relationship, then the end time of the history data H2 is modified to be 11/14, so that the data M1 to be written and the history data H2 eliminate the conflict of time axis relationship; the start time of the history data H4 is 11/25, and the end time is 12/14, then the start time of the modified history data H4 is 12/1, so that the data to be written M1 and the history data H4 eliminate the conflict of the time axis relationship.
205. And adding the data to be written into the business table directly based on the time axis relation between the data to be written and the historical data.
For example, the start time of the data M1 to be written is 11/15, the end time is 11/30, the start time of the history data H4 is 11/10, and the end time is 11/14, if the data M1 to be written and the history data H4 do not generate a conflict of the time axis relationship, the data M1 to be written is directly added to the service table.
According to the data processing method provided by the invention, original data are converted into data to be written, historical data are inquired in a time axis configuration table according to a first service table name field of the data to be written so as to determine the time axis relation between the data to be written and the historical data, then the relation of time axis conflict between the data to be written and the historical data is eliminated based on the time axis relation between the data to be written and the historical data, and then the data to be written is written into a service table, so that the service table with the time axis relation can be obtained, manual operation is not needed, and inquiry and use of a service party are facilitated.
In addition, the relational database mysql and the distributed database es are adopted for bottom storage in the embodiment, and switching between the relational database mysql and the distributed database es in the reading and writing process is realized through configuration, so that the service data can have the time axis capability quickly.
Specifically, the data processing method of the present embodiment is described by taking a new employee data as an example.
Referring to fig. 3, fig. 3 illustrates a process of writing data for a new employee.
(1) The employee 6 enters the job, and the newly added number 6 of the employee is to be written with data including age, academic calendar, work experience and the like.
(2) Employee 8 leaves, newly-added 8 of employee wait to write in data, including reason of leaving.
(3) And supplementing and recording the number 4 data to be written of the employee, wherein the number 4 data comprises interview data, post requirement data and the like.
In the step (1), the first step of the method,
adding a time range field of data to be written of the employee number 6 in the service table: effDate ═ 6, effntdate ═ infinity;
in step (2), newly adding employee number 8 to-be-written data, and because the time axis relationship conflict between number 8 data and original number 6 data, the end time field of number 6 data needs to be changed:
time range field of data No. 6: effDate 6, effnddate 7;
time range field of data No. 8: effDate ═ 8, effntdate ∞.
In the step (3), newly adding the data to be written of the employee number 4, and if the data to be written of the employee number 4 does not conflict with the original data of the employee number 6 in the time axis relationship, then:
time range field of data No. 4: effDate 4, effnddate 5;
time range field of data No. 6: effDate 6, effnddate 7;
time range field of data No. 8: effDate ═ 8, effntdate ∞.
Optionally, in addition to the new added data, the data processing method of this embodiment further includes: delete, modify, and query data.
Optionally, the method further comprises:
and under the condition of deleting or modifying the target historical data in the service table, deleting or modifying the target historical data, and changing a second time range field of the historical data which conflicts with the time axis relation of the target historical data according to a time axis configuration table corresponding to the service table to generate an updated service table.
For a second time range field of the history data that does not conflict with the time axis relationship of the target history data, no modification is required.
Wherein, the history data in the modified service table comprises the sequence of the modified history data.
Referring to fig. 4, fig. 4 shows a process of deleting business data of an employee. The employee has data number 4, data number 6, and data number 8.
(1) And deleting the number 4 data of the employee, wherein no conflict exists between the number 6 data and the number 8 data in the time axis relation, and then:
time range field of data No. 6: effDate 6, effnddate 7;
time range field of data No. 8: effDate ═ 8, effntdate ∞.
(2) Deleting the data No. 6 of the employee, generating the conflict of the time axis relation for the data No. 4, and not generating the conflict of the time axis relation for the data No. 8, then:
time range field of data No. 4: effDate 4, effnddate 7;
time range field of data No. 8: effDate ═ 8, effntdate ∞.
(3) And deleting the employee No. 8 data, wherein the conflict of the time axis relation does not occur to the data No. 4, and the conflict of the time axis relation occurs to the data No. 6, then:
time range field of data No. 4: effDate 4, effnddate 5
Time range field of data No. 6: effDate ═ 6, effEndDate ∞
Referring to FIG. 5, FIG. 5 illustrates a process for an employee to modify business data. The employee has data number 4, data number 6, and data number 8:
time range field of data No. 4: effDate 4, effnddate 5;
time range field of data No. 6: effDate 6, effnddate 7;
time range field of data No. 8: effDate ═ 8, effntdate ∞.
Changing the effective time of employee number 4 to number 3, and not generating time axis relation conflict for data of numbers 6 and 8, then:
time range field of data No. 3: effDate 3, effnddate 5;
time range field of data No. 6: effDate 6, effnddate 7;
time range field of data No. 8: effDate ═ 8, effntdate ∞.
Changing the effective time of employee number 4 to number 7, generating time axis relation conflict for number 6 data, and not time axis relation conflict for number 8 data, then:
time range field of data No. 6: effDate 6, effnddate 6;
time range field of data No. 7: effDate ═ 7, effnddate ═ 7;
time range field of data No. 8: effDate ═ 8, effntdate ∞.
Changing the effective time of employee number 4 to number 10, not generating time axis relation conflict for number 6 data, and generating time axis relation conflict for number 8 data, then:
time range field of data No. 6: effDate 6, effnddate 7;
time range field of data No. 8: effDate 8, effnddate 9;
time range field of data No. 10: effDate ═ 10 and effntdate ∞.
Referring to fig. 6, the method of this embodiment further includes a process of querying data, including the following steps 601-604:
601. and under the condition of receiving the query command, determining a third service table name field corresponding to the query command, judging whether the queried database is a distributed database or a relational database, if so, executing step 602, and if so, executing step 604.
Specifically, the query command is determined by the es read switch. If the es reading switch is turned on, the inquired database is a distributed database; and if the es reading switch is closed, the inquired database is a relational database.
602. And determining that the primary index condition and the secondary index condition are inquired in a service table of the distributed database, judging whether the inquiry is successful, if so, executing step 603, and if not, executing step 604.
Specifically, determining a main index condition to be queried according to a third service table name field apiname, then analyzing a reference splicing query condition, and executing query of the es main index condition;
and according to the relationship _ api _ name, the relationship _ field and the field value as the secondary index condition, analyzing the reference splicing query condition, and executing the query of the es-time index condition.
603. The historical data in the second format is returned.
In this embodiment, the first format is a mysql format, and the second format is a distributed format es.
604. And inquiring and returning the historical data in the first format in the service table of the relational database according to the third service table name field.
As can be seen from the steps 601-604, when the corresponding historical data cannot be queried in the distributed database es, the relational database mysql is transferred to perform the dimension reduction query, so that the queried historical data can be returned.
The data processing device provided by the invention is described below, and the data processing device described below and the data processing method described above can be referred to correspondingly.
The embodiment of the present invention also discloses a data processing apparatus, referring to fig. 7, including:
a conversion module 701, configured to obtain original data, and convert the original data into data to be written, where the data to be written includes a first service table name field and a first time range field;
a historical data obtaining module 702, configured to query a time axis configuration table corresponding to a service table according to the first service table name field, and obtain historical data in the time axis configuration table, where the time axis configuration table has a second service table name field corresponding to the first service table name field, where the historical data includes the second service table name field and a second time range field;
a relation determining module 703, configured to determine, according to the first time range field of the data to be written and the second time range field of the historical data, a time axis relation between the data to be written and the historical data, and add the data to be written to the service table based on the time axis relation between the data to be written and the historical data.
Optionally, the conversion module 701 is specifically configured to:
judging whether the write command is a distributed write command or a relational write command;
if the data are distributed write commands, the original data are analyzed by a time axis component and are respectively converted into first data to be written and second data to be written;
and if the data is the relational writing command, analyzing and converting the original data into first data to be written through a time axis component.
Optionally, the apparatus further comprises: and the configuration module is used for generating a corresponding time axis configuration table according to the historical data in each service table through the configuration manager, wherein the historical data in the service table comprises a service starting field and a service ending field.
Optionally, the relationship determining module 703 is specifically configured to:
judging whether the time axis relation between the data to be written and the historical data conflicts or not according to the first time range field of the data to be written and the second time range field of the historical data;
if so, updating a first time range field of the data to be written and a second time range field of the historical data to eliminate the conflict of the time axis relationship between the data to be written and the historical data, and then adding the data to be written to the business table based on the updated time axis relationship between the data to be written and the historical data;
if not, adding the data to be written into the business table directly based on the time axis relation between the data to be written and the historical data.
Optionally, the relationship determining module 703 is specifically configured to:
determining the time sequence of the data to be written and the historical data based on the first starting time field of the data to be written and the second starting time field of the historical data; wherein the first time range field comprises a first start time field and a first end time field, and the second time range field comprises a second start time field and a second end time field;
modifying a second start time field or a second end time field of the historical data so that a first time range field of the data to be written and a second time range field of the historical data do not coincide, and then adding the data to be written to the business table based on the updated time axis relationship between the data to be written and the historical data.
Optionally, the apparatus further comprises: and the deleting module is used for deleting or modifying the target historical data under the condition of deleting or modifying the target historical data in the service table, and changing a second time range field of the historical data which conflicts with the time axis relation of the target historical data according to the time axis configuration table corresponding to the service table so as to generate an updated service table.
Optionally, the apparatus further comprises: a relationship determination module 703 configured to: if the data to be written is a distributed writing command, based on the time axis relation between the data to be written and the historical data, adding the first data to be written into a business table of a relational database, and adding the second data to be written into the business table of the distributed database;
and if the data to be written is a relational writing command, adding the first data to be written into a business table of the relational database based on the time axis relation between the data to be written and the historical data.
Optionally, the apparatus further comprises: the query module is used for determining a third service table name field corresponding to the query command under the condition of receiving the query command, and judging whether the queried database is a distributed database or a relational database;
if the query is successful, returning the historical data in the second format; if the query fails, querying and returning the historical data in the first format in the service table of the relational database according to the third service table name field;
and if so, inquiring and returning the historical data in the first format in the service table of the relational database according to the third service table name field corresponding to the inquiry command.
According to the data processing device provided by the invention, the original data is converted into the data to be written, the historical data is inquired in the time axis configuration table according to the first service table name field of the data to be written so as to determine the time axis relation between the data to be written and the historical data, and the data to be written is written into the service table based on the time axis relation between the data to be written and the historical data, so that the service table with the time axis relation can be obtained, manual operation is not needed, and the inquiry and the use of a service party are facilitated.
Fig. 8 illustrates a physical structure diagram of an electronic device, and as shown in fig. 8, the electronic device may include: a processor (processor)810, a communication Interface 820, a memory 830 and a communication bus 840, wherein the processor 810, the communication Interface 820 and the memory 830 communicate with each other via the communication bus 840. The processor 810 may call logical instructions in the memory 830 to perform a data processing method, including:
acquiring original data, and converting the original data into data to be written, wherein the data to be written comprises a first service table name field and a first time range field;
inquiring a time axis configuration table corresponding to a service table according to the first service table name field, and acquiring historical data of a second service table name field corresponding to the first service table name field in the time axis configuration table, wherein the historical data comprises the second service table name field and a second time range field;
and determining the time axis relation between the data to be written and the historical data according to the first time range field of the data to be written and the second time range field of the historical data, and adding the data to be written into the business table based on the time axis relation between the data to be written and the historical data.
In addition, the logic instructions in the memory 830 may be implemented in software functional units and stored in a computer readable storage medium when the logic instructions are sold or used as independent products. Based on such understanding, the technical solution of the present invention may be embodied in the form of a software product, which is stored in a storage medium and includes instructions for causing a computer device (which may be a personal computer, a server, or a network device) to execute all or part of the steps of the method according to the embodiments of the present invention. And the aforementioned storage medium includes: a U-disk, a removable hard disk, a Read-Only Memory (ROM), a Random Access Memory (RAM), a magnetic disk or an optical disk, and other various media capable of storing program codes.
In another aspect, the present invention also provides a computer program product comprising a computer program stored on a non-transitory computer-readable storage medium, the computer program comprising program instructions which, when executed by a computer, enable the computer to perform the data processing method provided by the above methods, including:
acquiring original data, and converting the original data into data to be written, wherein the data to be written comprises a first service table name field and a first time range field;
inquiring a time axis configuration table corresponding to a service table according to the first service table name field, and acquiring historical data of a second service table name field corresponding to the first service table name field in the time axis configuration table, wherein the historical data comprises the second service table name field and a second time range field;
and determining the time axis relation between the data to be written and the historical data according to the first time range field of the data to be written and the second time range field of the historical data, and adding the data to be written into the business table based on the time axis relation between the data to be written and the historical data.
In yet another aspect, the present invention also provides a non-transitory computer-readable storage medium, on which a computer program is stored, the computer program being implemented by a processor to perform the data processing method provided above, including:
acquiring original data, and converting the original data into data to be written, wherein the data to be written comprises a first service table name field and a first time range field;
inquiring a time axis configuration table corresponding to a service table according to the first service table name field, and acquiring historical data of a second service table name field corresponding to the first service table name field in the time axis configuration table, wherein the historical data comprises the second service table name field and a second time range field;
and determining the time axis relation between the data to be written and the historical data according to the first time range field of the data to be written and the second time range field of the historical data, and adding the data to be written into the business table based on the time axis relation between the data to be written and the historical data.
The above-described embodiments of the apparatus are merely illustrative, and the units described as separate parts may or may not be physically separate, and parts displayed as units may or may not be physical units, may be located in one place, or may be distributed on a plurality of network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of the present embodiment. One of ordinary skill in the art can understand and implement it without inventive effort.
Through the above description of the embodiments, those skilled in the art will clearly understand that each embodiment can be implemented by software plus a necessary general hardware platform, and certainly can also be implemented by hardware. With this understanding in mind, the above-described technical solutions may be embodied in the form of a software product, which can be stored in a computer-readable storage medium such as ROM/RAM, magnetic disk, optical disk, etc., and includes instructions for causing a computer device (which may be a personal computer, a server, or a network device, etc.) to execute the methods described in the embodiments or some parts of the embodiments.
Finally, it should be noted that: the above examples are only intended to illustrate the technical solution of the present invention, but not to limit it; although the present invention has been described in detail with reference to the foregoing embodiments, it will be understood by those of ordinary skill in the art that: the technical solutions described in the foregoing embodiments may still be modified, or some technical features may be equivalently replaced; and such modifications or substitutions do not depart from the spirit and scope of the corresponding technical solutions of the embodiments of the present invention.
Claims (10)
1. A data processing method, comprising:
acquiring original data, and converting the original data into data to be written, wherein the data to be written comprises a first service table name field and a first time range field;
inquiring a time axis configuration table corresponding to a service table according to the first service table name field, and acquiring historical data of a second service table name field corresponding to the first service table name field in the time axis configuration table, wherein the historical data comprises the second service table name field and a second time range field;
and determining a time axis relation between the data to be written and the historical data according to the first time range field of the data to be written and the second time range field of the historical data, and adding the data to be written to the business table based on the time axis relation between the data to be written and the historical data to obtain the business table with the time axis relation.
2. The data processing method of claim 1, wherein converting the original data into data to be written comprises:
judging whether the write command is a distributed write command or a relational write command;
if the data are distributed write commands, the original data are analyzed by a time axis component and are respectively converted into first data to be written and second data to be written;
and if the data is the relational writing command, analyzing and converting the original data into first data to be written through a time axis component.
3. The data processing method according to claim 1, wherein determining a timeline relationship between the data to be written and the history data according to a first time range field of the data to be written and a second time range field of the history data, and adding the data to be written to the business table based on the timeline relationship between the data to be written and the history data comprises:
judging whether the time axis relation between the data to be written and the historical data conflicts or not according to the first time range field of the data to be written and the second time range field of the historical data;
if so, updating a first time range field of the data to be written and a second time range field of the historical data to eliminate the conflict of the time axis relationship between the data to be written and the historical data, and then adding the data to be written to the business table based on the updated time axis relationship between the data to be written and the historical data;
if not, adding the data to be written into the business table directly based on the time axis relation between the data to be written and the historical data.
4. The data processing method according to claim 3, wherein updating the first time range field of the data to be written and the second time range field of the history data to eliminate the conflict between the data to be written and the time axis relationship of the history data, and then adding the data to be written to the business table based on the updated time axis relationship between the data to be written and the history data comprises:
determining the time sequence of the data to be written and the historical data based on the first starting time field of the data to be written and the second starting time field of the historical data; wherein the first time range field comprises a first start time field and a first end time field, and the second time range field comprises a second start time field and a second end time field;
modifying a second start time field or a second end time field of the historical data so that a first time range field of the data to be written and a second time range field of the historical data do not coincide, and then adding the data to be written to the business table based on the updated time axis relationship between the data to be written and the historical data.
5. The data processing method of claim 3, wherein the method further comprises:
and under the condition of deleting or modifying the target historical data in the service table, deleting or modifying the target historical data, and changing a second time range field of the historical data which conflicts with the time axis relation of the target historical data according to a time axis configuration table corresponding to the service table to generate an updated service table.
6. The data processing method of claim 2, wherein adding the data to be written to the business table based on a timeline relationship of the data to be written and the historical data comprises:
if the data to be written is a distributed writing command, based on the time axis relation between the data to be written and the historical data, adding the first data to be written into a business table of a relational database, and adding the second data to be written into the business table of the distributed database;
and if the data to be written is a relational writing command, adding the first data to be written into a business table of the relational database based on the time axis relation between the data to be written and the historical data.
7. The data processing method of claim 6, wherein the method further comprises:
under the condition of receiving the query command, determining a third service table name field corresponding to the query command, and judging whether the queried database is a distributed database or a relational database;
if the query is successful, returning the historical data in the second format; if the query fails, querying and returning the historical data in the first format in the service table of the relational database according to the third service table name field;
and if so, inquiring and returning the historical data in the first format in the service table of the relational database according to the third service table name field corresponding to the inquiry command.
8. A data processing apparatus, comprising:
the conversion module is used for acquiring original data and converting the original data into data to be written, wherein the data to be written comprises a first service table name field and a first time range field;
the historical data acquisition module is used for inquiring a time axis configuration table corresponding to a service table according to the first service table name field and acquiring historical data of a second service table name field corresponding to the first service table name field in the time axis configuration table, wherein the historical data comprises the second service table name field and a second time range field;
and the relation determining module is used for determining the time axis relation between the data to be written and the historical data according to the first time range field of the data to be written and the second time range field of the historical data, and adding the data to be written to the business table based on the time axis relation between the data to be written and the historical data to obtain the business table with the time axis relation.
9. An electronic device comprising a memory, a processor and a computer program stored on the memory and executable on the processor, characterized in that the steps of the data processing method according to any of claims 1 to 7 are implemented when the program is executed by the processor.
10. A non-transitory computer-readable storage medium, on which a computer program is stored, which, when being executed by a processor, carries out the steps of the data processing method according to any one of claims 1 to 7.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202011542519.4A CN112612818B (en) | 2020-12-21 | 2020-12-21 | Data processing method and device, computing equipment and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202011542519.4A CN112612818B (en) | 2020-12-21 | 2020-12-21 | Data processing method and device, computing equipment and storage medium |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN112612818A CN112612818A (en) | 2021-04-06 |
| CN112612818B true CN112612818B (en) | 2022-04-15 |
Family
ID=75244538
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202011542519.4A Active CN112612818B (en) | 2020-12-21 | 2020-12-21 | Data processing method and device, computing equipment and storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN112612818B (en) |
Families Citing this family (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113468213A (en) * | 2021-07-26 | 2021-10-01 | 中兴飞流信息科技有限公司 | Dynamic data source switching method and system based on multi-data source configuration |
| CN118193544B (en) * | 2024-05-20 | 2024-07-30 | 浪潮通用软件有限公司 | Automatic labeling method, equipment and medium for business data |
Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN105046601A (en) * | 2015-07-09 | 2015-11-11 | 传成文化传媒（上海）有限公司 | User data processing method and system |
| CN105095255A (en) * | 2014-05-07 | 2015-11-25 | 中兴通讯股份有限公司 | Data index creating method and device |
| CN108388958A (en) * | 2018-01-31 | 2018-08-10 | 中国地质大学（武汉） | A kind of method and device of two-dimensional attitude maneuvering satellite mission planning technical research |
Family Cites Families (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102243664B (en) * | 2011-08-22 | 2013-04-03 | 西北大学 | Data storage and query method for compound fields |
| CN104112008A (en) * | 2014-07-16 | 2014-10-22 | 深圳市国泰安信息技术有限公司 | Multi-table data association inquiry optimizing method and device |
| CN106294792B (en) * | 2016-08-15 | 2019-05-31 | 上海携程商务有限公司 | The method for building up of correlation inquiry system and establish system |
| CN106407360B (en) * | 2016-09-07 | 2020-07-24 | 广州视源电子科技股份有限公司 | Data processing method and device |
| CN111915046B (en) * | 2019-05-09 | 2024-02-20 | 北京百度网讯科技有限公司 | Method and device for outputting information |
| CN110442595B (en) * | 2019-07-26 | 2022-07-22 | 南京南瑞继保电气有限公司 | Method and device for constructing universal SQL report data set |
-
2020
- 2020-12-21 CN CN202011542519.4A patent/CN112612818B/en active Active
Patent Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN105095255A (en) * | 2014-05-07 | 2015-11-25 | 中兴通讯股份有限公司 | Data index creating method and device |
| CN105046601A (en) * | 2015-07-09 | 2015-11-11 | 传成文化传媒（上海）有限公司 | User data processing method and system |
| CN108388958A (en) * | 2018-01-31 | 2018-08-10 | 中国地质大学（武汉） | A kind of method and device of two-dimensional attitude maneuvering satellite mission planning technical research |
Also Published As
| Publication number | Publication date |
|---|---|
| CN112612818A (en) | 2021-04-06 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN111522816B (en) | Data processing method, device, terminal and medium based on database engine | |
| CN109977110B (en) | Data cleaning method, device and equipment | |
| US8103704B2 (en) | Method for database consolidation and database separation | |
| AU695765B2 (en) | Method and apparatus for data storage and retrieval | |
| US9390176B2 (en) | System and method for recursively traversing the internet and other sources to identify, gather, curate, adjudicate, and qualify business identity and related data | |
| US20070192374A1 (en) | Virtual repository management to provide functionality | |
| CN112612818B (en) | Data processing method and device, computing equipment and storage medium | |
| US8442959B2 (en) | Methods and systems for automated processing of fallout orders | |
| CN110597844B (en) | Unified access method for heterogeneous database data and related equipment | |
| KR20200094074A (en) | Method, apparatus, device and storage medium for managing index | |
| CN109753517A (en) | A kind of method, apparatus, computer storage medium and the terminal of information inquiry | |
| US8965879B2 (en) | Unique join data caching method | |
| CN117112590A (en) | Method for generating structural query language and data query equipment | |
| CN116628018A (en) | Data query method, device, server and storage medium | |
| CN106844497A (en) | The check device and method of a kind of database code | |
| CN115809268B (en) | Adaptive query method and device based on fragment index | |
| CN116467310A (en) | Lock-free marking method for invalid index, storage medium and computer equipment | |
| CN108388610B (en) | Data ETL processing method and device | |
| US20230066110A1 (en) | Creating virtualized data assets using existing definitions of etl/elt jobs | |
| CN111061759A (en) | Data query method and device | |
| JP2017010376A (en) | Mart-less verification support system and mart-less verification support method | |
| US20240220876A1 (en) | Artificial intelligence (ai) based data product provisioning | |
| CN117093597B (en) | Data processing method and device | |
| JP2013171495A (en) | Data management device, data management method and data management program | |
| US20240086387A1 (en) | Delta transition table for database triggers |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| TA01 | Transfer of patent application right |
Effective date of registration: 20210708 Address after: 100085 Floor 101 102-1, No. 35 Building, No. 2 Hospital, Xierqi West Road, Haidian District, Beijing Applicant after: Seashell Housing (Beijing) Technology Co.,Ltd. Address before: Unit 05, room 112, 1st floor, office building, Nangang Industrial Zone, economic and Technological Development Zone, Binhai New Area, Tianjin 300457 Applicant before: BEIKE TECHNOLOGY Co.,Ltd. |
|
| TA01 | Transfer of patent application right | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |