CN111767340B

CN111767340B - Data processing method, device, electronic equipment and medium

Info

Publication number: CN111767340B
Application number: CN202010482031.0A
Authority: CN
Inventors: 唐阳光; 杨诗平; 黄景超; 毛超丹
Original assignee: Industrial and Commercial Bank of China Ltd ICBC
Current assignee: Industrial and Commercial Bank of China Ltd ICBC
Priority date: 2020-05-29
Filing date: 2020-05-29
Publication date: 2024-01-05
Anticipated expiration: 2040-05-29
Also published as: CN111767340A

Abstract

The present disclosure provides a data storage method, comprising: acquiring a data record stored by a Kaff card system; determining whether the data record meets an analysis condition; under the condition that the data record meets the analysis condition, carrying out content analysis on the data record to obtain data information comprising a target field; and storing the data information in a target database. The disclosure also provides a data storage device, an electronic device and a medium.

Description

Data processing method, device, electronic equipment and medium

Technical Field

The present disclosure relates to the field of computer technology, and more particularly, to a data processing method, apparatus, electronic device, and medium.

Background

There are many problems associated with unified storage of data in a centralized database, such as: the database table structure cannot be changed according to the application requirement, and the field increase and decrease is very difficult.

Disclosure of Invention

In view of this, the present disclosure provides a data processing method, apparatus, electronic device, and medium.

One aspect of the present disclosure provides a data storage method, including: acquiring a data record stored by a Kaff card system; determining whether the data record meets an analysis condition; under the condition that the data record meets the analysis condition, carrying out content analysis on the data record to obtain data information comprising a target field; and storing the data information in a target database.

According to an embodiment of the present disclosure, storing the data information to a target database includes: under the condition that a plurality of target databases exist, acquiring address information and fragmentation rules of the target databases; determining a target database to which each piece of data sub-information in the data information belongs respectively based on a slicing rule; and storing each data sub-information in the data information into a respective target database.

According to an embodiment of the present disclosure, determining whether the data record satisfies a resolution condition includes: acquiring a pre-configured analysis condition, wherein the analysis condition indicates reference information triggering analysis of the data record; comparing the related information of the data record with the reference information; and if the related information accords with the reference information, determining that the data record meets the analysis condition.

According to an embodiment of the present disclosure, performing content parsing on the data record to obtain data information including a target field includes: acquiring content rules; based on the content rule, determining a corresponding relation between a source field and a target field in the data record; calculating an initial value of the source field based on the content rule to determine a target value of a target field corresponding to the source field; and acquiring a record type of the source field from the Kaff card system, and generating an SQL sentence based on the record type and a key value pair formed by the target field and the target value, wherein the record type comprises an update data type and an insertion data type.

According to an embodiment of the present disclosure, the method further comprises: acquiring a preset hash field; performing hash calculation on a plurality of SQL sentences executed by the same target database according to the hash field to determine queues for respectively storing the SQL sentences; and storing the SQL sentences into respective queues respectively so that the SQL sentences can be executed by the target database simultaneously.

According to an embodiment of the present disclosure, the method further comprises: acquiring configuration parameter values; and under the condition that the number of SQL sentences in the queue reaches the configuration parameter value, controlling the target database to sequentially execute the SQL sentences according to the sequence of the SQL sentences in the queue.

According to an embodiment of the present disclosure, the method further comprises: receiving feedback information from the target database for executing the SQL statement; under the condition that the feedback information indicates SQL execution abnormality, determining an abnormality reason according to the feedback information; and determining to process the abnormal operation according to the abnormal reason.

According to an embodiment of the present disclosure, the method further comprises: after the abnormal handling operation is executed, storing the abnormal SQL for executing the abnormality in an abnormal data table under the condition that the feedback information still indicates that the SQL is abnormal to execute; acquiring abnormal SQL from the abnormal data table at intervals of a preset time period; and executing the abnormal SQL.

Another aspect of the present disclosure provides a data storage device comprising: the acquisition module is used for acquiring the data record stored by the Kaff card system; the determining module is used for determining whether the data record meets analysis conditions or not; the analysis module is used for carrying out content analysis on the data record to obtain data information comprising a target field under the condition that the data record meets analysis conditions; and the storage module is used for storing the data information into a target database.

According to an embodiment of the present disclosure, the storage module includes: the first acquisition sub-module is used for acquiring address information and a fragmentation rule of the target database under the condition that a plurality of target databases exist; the first determining submodule is used for determining a target database to which each piece of data sub-information in the data information belongs respectively based on a slicing rule; and the storage sub-module is used for storing each piece of data sub-information in the data information into a respective target database.

According to an embodiment of the present disclosure, the determining module includes: the second acquisition sub-module is used for acquiring a pre-configured analysis condition, wherein the analysis condition indicates reference information for triggering analysis of the data record; a second determining sub-module for comparing the related information of the data record with the reference information; and the third determining submodule is used for determining that the data record meets the analysis condition if the related information accords with the reference information.

According to an embodiment of the present disclosure, the parsing module includes: a third obtaining sub-module, configured to obtain a content rule; a fourth determining submodule, configured to determine a correspondence between a source field and a target field in the data record based on the content rule; a calculation sub-module for calculating an initial value of the source field based on the content rule to determine a target value of a target field corresponding to the source field; and the generation sub-module is used for acquiring the record type of the source field from the Kaff card system and generating an SQL sentence based on the record type and a key value pair formed by the target field and the target value, wherein the record type comprises an update data type and an insertion data type.

Another aspect of the present disclosure provides an electronic device, comprising: one or more processors; and a storage device for storing one or more programs, wherein the one or more programs, when executed by the one or more processors, cause the one or more processors to perform the method described above.

Another aspect of the present disclosure provides a computer-readable storage medium storing computer-executable instructions that, when executed, are configured to implement a method as described above.

Another aspect of the present disclosure provides a computer program comprising computer executable instructions which when executed are for implementing a method as described above.

Drawings

The above and other objects, features and advantages of the present disclosure will become more apparent from the following description of embodiments thereof with reference to the accompanying drawings in which:

fig. 1 schematically illustrates an application scenario of a data storage method according to an embodiment of the present disclosure;

FIG. 2 schematically illustrates a flow chart of a data storage method according to an embodiment of the present disclosure;

FIG. 3 schematically illustrates a flow chart of a method of storing data information to a target database in accordance with an embodiment of the present disclosure;

FIG. 4 schematically illustrates a method flow diagram for determining whether the data record satisfies a resolution condition, in accordance with an embodiment of the disclosure;

FIG. 5 schematically illustrates a flow chart of a method of content parsing a data record to obtain data information including a target field, in accordance with an embodiment of the present disclosure;

FIG. 6 schematically illustrates a flow chart of a data storage method according to another embodiment of the present disclosure;

FIG. 7 schematically illustrates a block diagram of a data storage device according to an embodiment of the present disclosure; and

Fig. 8 schematically illustrates a block diagram of an electronic device according to an embodiment of the disclosure.

Detailed Description

Hereinafter, embodiments of the present disclosure will be described with reference to the accompanying drawings. It should be understood that the description is only exemplary and is not intended to limit the scope of the present disclosure. In the following detailed description, for purposes of explanation, numerous specific details are set forth in order to provide a thorough understanding of the embodiments of the present disclosure. It may be evident, however, that one or more embodiments may be practiced without these specific details. In addition, in the following description, descriptions of well-known structures and techniques are omitted so as not to unnecessarily obscure the concepts of the present disclosure.

The terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the disclosure. The terms "comprises," "comprising," and/or the like, as used herein, specify the presence of stated features, steps, operations, and/or components, but do not preclude the presence or addition of one or more other features, steps, operations, or components.

All terms (including technical and scientific terms) used herein have the same meaning as commonly understood by one of ordinary skill in the art unless otherwise defined. It should be noted that the terms used herein should be construed to have meanings consistent with the context of the present specification and should not be construed in an idealized or overly formal manner.

Where expressions like at least one of "A, B and C, etc. are used, the expressions should generally be interpreted in accordance with the meaning as commonly understood by those skilled in the art (e.g.," a system having at least one of A, B and C "shall include, but not be limited to, a system having a alone, B alone, C alone, a and B together, a and C together, B and C together, and/or A, B, C together, etc.). Where a formulation similar to at least one of "A, B or C, etc." is used, in general such a formulation should be interpreted in accordance with the ordinary understanding of one skilled in the art (e.g. "a system with at least one of A, B or C" would include but not be limited to systems with a alone, B alone, C alone, a and B together, a and C together, B and C together, and/or A, B, C together, etc.).

Embodiments of the present disclosure provide a data storage method. The data storage method can be used for storing data in the Kaff card system into a target database. The method may include obtaining a data record stored by the card system; determining whether the data record meets an analysis condition; under the condition that the data record meets the analysis condition, carrying out content analysis on the data record to obtain data information comprising a target field; and storing the data information in a target database.

Fig. 1 schematically illustrates an application scenario 100 of a data storage method according to an embodiment of the present disclosure. It should be noted that fig. 1 illustrates only an example of an application scenario in which the embodiments of the present disclosure may be applied to help those skilled in the art understand the technical content of the present disclosure, but it does not mean that the embodiments of the present disclosure may not be applied to other devices, systems, environments, or scenarios.

As shown in fig. 1, the application scenario 100 may include a source database 101, a kaff card system 102, a data processing device 103, and a target database 104.

The source database 101 may be, for example, DB2, ORACLE, myAQL, etc.

In order to at least partly avoid the problems with storing data using a centralized database, for example, a business database may be built up for the application that belongs to itself to store data using its business database. According to embodiments of the present disclosure, for example, data in the source database 101 may be synchronized in real time into the card system 102, and then data from the source database 101 in the card system 102 is transferred to the service database, i.e., the target database 104. The data processing device 103 is configured to process the data record in the kav card, so that the data record in the kav card can be correctly transferred to the target database 104.

It should be noted that the data storage method provided by the embodiments of the present disclosure may be generally executed by the data processing apparatus 103.

It should be understood that the number of source databases and target databases in fig. 1 is merely illustrative. There may be any number of source and target databases, as desired for implementation.

Fig. 2 schematically illustrates a flow chart of a data storage method according to an embodiment of the present disclosure.

As shown in fig. 2, the method may include operations S201 to S204.

In operation S201, a data record stored in the card system is acquired.

For example, a request to acquire a data record may be sent to the card system and a data record sent from the card system to the data processing apparatus 103 in response to the request may be received, or the card system may send a data record to the data processing apparatus 103 in real time and the data processing apparatus 103 may receive a data record from the card system in real time.

According to embodiments of the present disclosure, the data records stored in the card system may be generated from data from a database of DB2, ORACLE, myAQL, etc.

In operation S202, it is determined whether the data record satisfies the parsing condition.

According to embodiments of the present disclosure, the parsing conditions may be preconfigured by those skilled in the art. The parsing condition may be, for example, that a value of a certain parsing field in the data record is equal to a reference value, or that a type of a certain parsing field in the data record is a specific type, or the like.

The following table schematically illustrates an example of parsing fields in which parsing conditions may be configured according to an embodiment of the present disclosure. As shown in table one below, the parsing conditions that can be configured by one skilled in the art may include the topic name, primary key field, trigger condition type, trigger condition content, trigger sequence, rule ID, HASH calculation field name, and shard calculation field of the card. According to embodiments of the present disclosure, one skilled in the art may configure a topic name, a primary key field, a trigger condition type, trigger condition contents, a trigger sequence, a rule ID, a HASH calculation field name, and a fragment calculation field of a kafu card, respectively, thereby generating one parsing condition.

List one

topIC_NAME	Topic name of Kafka
		KEY_COLUMN	Main key field
TRIGGER_RULE_TYPE	Trigger condition type
		TRIGGER_RULE_CONTENT	Content of trigger condition
TRIGGER_ORDER	Trigger sequence
		RULE_ID	Rule ID
HASH_COLUMN	HASH calculation field name
		SHARDING_COLUMN	Fragment calculation field

The parsing field trigger_run_type may include, for example:

START WITH content

End_with ENDs WITH a certain content

ContAIN CONTAINs a content

The value of EQUAL is EQUAL to a certain value

INDEX_OF_CHAR_EQUAL has a certain value EQUAL to

REG_EXP satisfies regular expression

ALL records of ALL are parsed according to this

SKIP recording when skip_start_with STARTs WITH a certain content

SKIP recording when skip_end_with content ENDs

SKIP this recording if skip_CONTAIN CONTAINs a content

SKIP the record if SKIP_EQUAL value is EQUAL to a certain value

SKIP the recording if the SKIP_INDEX_OF_CHAR_EQUAL value is EQUAL to

SKIP_REG_EXP satisfying regular expression SKIPs the record

……

According to embodiments of the present disclosure, the trigger_run_content may be stored in a json format, for example, with a key being the source table field name and a value being the matching CONTENT. For example, trigger_run_type is configured to: "START_WITH", the TRIGGER_RULE_CONTENT is configured to: { "CI_ID": "0001" }, it is determined whether the value corresponding to the field ci_id in the data record starts with 0001. If the value corresponding to the field ci_id in the data record starts with 0001, it is determined that the data record satisfies the analysis condition.

In operation S203, in the case where it is determined that the data record satisfies the parsing condition, content parsing is performed on the data record to obtain data information including a target field.

According to embodiments of the present disclosure, the data records may be parsed, for example, according to content rules. Table two below schematically shows examples of content rule fields according to embodiments of the present disclosure.

Watch II

SOURCE_COLUMN_NAME	Source field name
		TARGET_COLUMN_NAME	Target field name
CONTENT_TRANSFORM_TYPE	Field parsing type
		CONTENT_TRANSFORM_CONTENT	Field detail parsing content
RULE_ID	Rule ID (equivalent to foreign key)

The content_transmission_type may contain, for example, the following TYPEs:

sub_string segmentation character STRING

The CALC_PLUS primitive value is added with a certain value

CALC_MULTIPLY primary value multiplied by a value

The RANGE OF returns a certain value

Range OF CALC RANGE value calculation

Date_transfer DATE format conversion

CONSTANT returns a CONSTANT

SPLIT AND splice fields

DICTIONARY data word typical return

SKIP this field is not duplicated

Default DEFAULT (return to original field value)

The CONTENT and format of the field content_transmission_content permission to be filled in are different rules according to the value of the field content_transmission_type. Examples of the values of the field content_transfer_type and the CONTENT, format and meaning of which the field content_transfer_content allows to be filled are schematically shown below.

The content_transfer_type is configured as sub_string, and if the value of the content_transfer_content is configured as [0, 10], then 11 characters 1 to 11 can be truncated for return.

The content_transmission_type is configured as CALC_PLUS, if the value of the content_transmission_content is configured as-100, the original value can be converted into a numerical value and then added with (-100) to return,

the content_transmission_type is configured as CALC_MULTIPLY, and if the value of the content_transmission_content is configured as 1/10, the original value may be multiplied by (1/10) to return.

Content_transfer_type is configured as range_of_one, if the value OF content_transfer_content is configured as { [0, 10]:10; [11, 20]:20, then 10 may be returned if the original value falls within the range of 0 to 10, and 20 may be returned if the original value falls within the range of 11 to 20.

Content_transfer_type is configured as range_of_calc, if the value OF content_transfer_content is configured as { [0, 10]: PLUS:100, [11]: MULTIPLY:1/10, the original value is converted into a value less than or equal to 10, and then 100 is added for return, and if the value is more than or equal to 11, the original value is multiplied by 1/10 for return.

The content_transfer_type is configured as date_transfer, if the value of the content_transfer_content is configured as YYYY-MM-DDHH: mm: ss;9999-12-31 00:00:00, the analysis date value can be analyzed according to the character string type, and the analysis failure returns to 9999-12-31 00:00:00".

The content_transmission_type is configured as CONSTANT, and 2019AA may be returned if the value of the content_transmission_content is configured as 2019AA.

Content_transfer_type is configured as split_and_join, if the value of CONTENT transfer CONTENT is configured as [0,5]: [6, 18]: [19] the method comprises the steps of carrying out a first treatment on the surface of the Cutting out 1 st to 6 th bits, 7 th to 19 th bits and 20 th to last bit character strings according to original values of [ AA, BB and CC ], connecting the character strings by AA, BB, and finally adding the CC character strings; if the value of contenttransformaContent is configured to be 5: AA, the original value is divided once every 5 character strings (less than 5 calculation at last), and AA is used for connection return.

Content_transfer_type is configured as DICTIONARY, if the value of CONTENT TransformamContent is configured as { [ AAA, BBB, CCC ]: ABC; [ XXX, YYY, ZZZ ]: EFG? XYZ may be the case where the original value is AAA or BBB or CCC returns ABC, and the remaining values return XYZ if the original value is XXX or YYY or ZZZ returns EFG.

The content_transfer_type is configured as SKIP, then this field is not duplicated, i.e. no key-value pair is generated.

Content_transfer_type is configured as DEFAULT, DEFAULT (return to original field value)

According to embodiments of the present disclosure, the calculated values may be stored as key-value pairs, e.g., the storage format may be: colName (i.e., destination field name): the value of the target field. (not stored if contenttransformattype is SKIP).

As described above, the man skilled in the art can configure the topic name, the primary key field, the trigger condition type, the trigger condition content, the trigger sequence, the rule ID, the HASH calculation field name and the fragment calculation field of the kafu card, respectively, thereby generating an parsing condition. In other words, a plurality of parsing rules may be included in each parsing condition, for example, the plurality of parsing rules may be a topic name of a card, a primary key field in a data record, a trigger condition type, trigger condition content, a trigger sequence, a rule ID, a HASH calculation field name, and a shard calculation field, respectively. According to the embodiment of the disclosure, whether the data records sequentially meet the plurality of analysis rules can be verified one by one, and in the case that the data records meet the plurality of analysis rules, the data records meet the analysis conditions.

In accordance with embodiments of the present disclosure, a number of resolution conditions may be set by those skilled in the art. And under the condition that the data record does not meet the current analysis condition, continuing to verify whether the data record meets the next analysis condition. If the data record cannot satisfy any one of the plurality of analysis conditions, the data record may be recorded into an abnormal data table.

In operation S204, data information is stored in a target database.

According to embodiments of the present disclosure, for example, content parsing of a data record to obtain a key value pair, SQL may then be generated from the key value pair and a record type of the data record recorded in the Kaff card system, and the SQL statement is executed by the target database, thereby storing data information in the target database. Record types of data records may include, for example, update (update) data types and insert (insert) data types.

The following table three schematically illustrates an example of composition rules that use key-value pairs to compose an SQL statement, according to an embodiment of the disclosure.

As shown in table three, the SQL statement composed using key-value pairs includes a main SQL and a standby SQL. The target database is firstly executed according to the main SQL, and the standby SQL is executed under the condition that the execution according to the main SQL fails.

For example, if the record type is update, the main SQL generated according to the key value pair is update type SQL, and the standby SQL is Insert type SQL.

Watch III

Card record type	Main SQL	Preparation of SQL
			update	Update type SQL	Inert type SQL
insert	Inert type SQL	Update type SQL
			delete	Delete type SQL	Without any means for
key_update_before	Delete type SQL	Without any means for
			key_update_after	Update type SQL	Inert type SQL

Fig. 3 schematically illustrates a flow chart of a method of storing data information to a target database according to an embodiment of the disclosure.

As shown in fig. 3, the method may include operations S214 to S234.

In operation S214, in the case where there are a plurality of target databases, address information and a fragmentation rule of the target database are acquired.

According to embodiments of the present disclosure, the address of the target database may be read from a configuration file, for example, where the target database address may be preset by one skilled in the art. The configuration file may include, for example, a primary database address and a backup database address. When the vendor code is 0 or readOnly is abnormal, the backup server address is used for database falling. Wherein, the vendor code returns an error code for the database service defined by the database vendor, such as: mySQL defines 1062 the duplicate unique key return code that the database encounters when executing SQL.

According to an embodiment of the present disclosure, the address of the target database may also be obtained from a database centralized manager for managing a plurality of databases, for example. For example, a connection may be established with the database centralized manager address to access the database centralized manager address and obtain current service information for all databases from the database centralized manager. When the current service information of the databases is used for carrying out the exception that the database is dropped and the vendor code is 0 or the readOnly is abnormal, the current service information is re-acquired every 30 seconds, the current service information is checked to be correct after the database is taken (the database service readOnly judgment is carried out, and if and only if the readOnly state is normally acquired and is not ON, the current service information is normal), and the new normal information is used for continuing to drop the database.

The slicing rules may also be read from a configuration file, in which a person skilled in the art may set the slicing rules, according to an embodiment of the present disclosure. The sharding rule may indicate the target database to which the data information should be written, i.e. the sharding rule may indicate a correspondence between the data information and the target database. For example, the fragmentation rule may be to store the value of a particular field in the data record in the first database.

In operation S224, a respective target database to which each data sub-information in the data information belongs is determined based on the slicing rule.

The data information may be, for example, various scores of each person in a class, and the data sub-information may be, for example, a chinese score of each person in a class.

In operation S234, each data sub-information in the data information is stored in a respective target database.

According to the embodiment of the disclosure, the data storage method can analyze the data record under the condition that the data record meets the analysis condition, so that the data information of the target field can be extracted from the data record, the fields in the database can be increased or decreased according to the requirement, and the accuracy of the stored data can be ensured.

According to the embodiment of the disclosure, the method can ensure that the data record in the Kaff card system is stored in the target database without missing data, and can split one source data table into a plurality of different data tables to perform database falling in real time.

Fig. 4 schematically illustrates a method flowchart of determining whether the data record satisfies a resolution condition according to an embodiment of the present disclosure.

As shown in fig. 4, the method may include operations S401 to S404.

In operation S401, a preconfigured parsing condition is acquired, where the parsing condition indicates reference information triggering parsing of the data record.

The reference information may include, for example, at least one of a topic name, a primary key field, a trigger condition type, trigger condition content, a trigger sequence, a rule ID, a HASH calculation field name, and a fragment calculation field of the card.

The parsing condition may be, for example, a topic name of the card is "aaa", a main key field is "identification card", a trigger condition type is "start_with", and a trigger condition content is "ci_id:0001", etc.

In operation S402, the related information of the data record is compared with the reference information.

For example, it may be determined whether the topic name of the card to which the data record belongs is "aaa", whether the primary key field is "identification card", and whether the ci_id starts with "0001".

In operation S403, if the related information matches the reference information, it is determined that the data record satisfies the analysis condition.

If the topic name of the card to which the data record belongs is "aaa", the primary key field is "identification card", and the ci_id starts with "0001", it is determined that the data record satisfies the analysis condition.

Fig. 5 schematically illustrates a flow chart of a method of content parsing a data record to obtain data information including a target field according to an embodiment of the present disclosure.

As shown in fig. 5, the method may include operations S213 to S243.

In operation S213, a content rule is acquired.

In accordance with embodiments of the present disclosure, one skilled in the art may define content rules in a configuration file according to the content fields and meanings of the content fields shown in table two above, and may further obtain the content rules from the configuration file, for example.

According to an embodiment of the present disclosure, a person skilled in the art may set a plurality of content rule tables and assign a number to each content rule table, and when an analysis condition is configured, set a number of the content rule table corresponding to the analysis condition in the analysis condition, so that a content rule for analyzing a data record may be determined according to the number of the content rule table when it is determined whether the data record satisfies the analysis condition in operation S202.

As shown in Table two above, the content rules may include content fields SOURCE_COLUMN_NAME and TARGET_COLUMN_NAME, SOURCE_COLUMN_NAME representing the SOURCE field NAME and TARGET_COLUMN_NAME representing the destination field NAME. One skilled in the art can, for example, define source_column_name in the SOURCE database as "vehicle" and target_column_name in the TARGET database as "vehicle" after data in the TARGET database is stored in the TARGET database.

In operation S223, a correspondence relationship between a source field and a target field in the data record is determined based on the content rule.

For example, in the above-described embodiment, the source field "vehicle" in the source database corresponds to the target field "vehicle" in the target database.

In operation S233, an initial value of a source field is calculated to determine a target value of a target field corresponding to the source field based on a content rule.

The initial value of the source field may be calculated, for example, according to the field parsing TYPE content_transfer_type and the field detailed parsing CONTENT content_transfer_content described above in operation S203.

In operation S243, a record type of the source field is acquired from the card system, and an SQL statement is generated based on the record type and a key value pair formed by the target field and the target value, wherein the record type includes an update data type and an insert data type.

The update data type is the update type in the third table, and the insert data type is the insert type in the third table.

According to embodiments of the present disclosure, an SQL statement may be generated, for example, from key-value pairs formed from a primary key field, a record type, a target field, and a target value in a parsing condition. Wherein the primary key field may be used to identify which data sub-record in the data record to insert or update. For example, the data record includes data sub-records of Zhang three and Li four, and the data record may include fields of an identity card, an age, and the like, where the primary key field may be an identity card field. In this example, an SQL statement may be generated based on the identification card field (e.g., 371522 … …), key-value pair, and record type (e.g., insert), with the update operation being determined to be performed on the data sub-record "Zhang Sanj" based on the identification card field "371522 … …".

According to the embodiment of the disclosure, the method can isomerically parse the data record data into data with different types from the source data according to the content rule, for example, the data type data can be isomerically parsed into character type data.

According to an embodiment of the disclosure, the data processing method may further include obtaining a preset hash field, performing hash computation on a plurality of SQL statements executed by the same target database according to the hash field to determine queues respectively storing the plurality of SQL statements, and storing the plurality of SQL statements in the respective queues respectively, so that the plurality of SQL statements may be executed concurrently by the target database.

According to an embodiment of the present disclosure, the hash field may be, for example, a primary key field, and the plurality of SQL statements are divided into a plurality of different queues by performing a hash operation on the primary key field of the SQL statement, so that the target database may concurrently execute the SQL statements in the plurality of different queues.

According to an embodiment of the disclosure, to increase the usage rate of database connection, several SQLs executed by the same target database may perform hash computation according to a hash field (for example, may be a primary key value) configured by the SQL to partition the several SQLs into different queues to be executed, where the different queues to be executed will be executed concurrently.

According to the embodiment of the disclosure, the hash field is configured so that a plurality of source tables fall into one target table without causing data coverage.

According to an embodiment of the present disclosure, the method may further include obtaining a configuration parameter value; and under the condition that the number of SQL sentences in the queue reaches the configuration parameter value, controlling the target database to sequentially execute a plurality of SQL sentences according to the sequence of the SQL sentences in the queue. According to an embodiment of the present disclosure, in order to improve the database execution efficiency, the SQL scraping is performed before the SQL is executed, and the scraping stop condition may be that the number of SQL reaches the configuration parameter value and no SQL is already in the queue to be executed. When one of the accumulation stop conditions is met, the execution of the accumulation is stopped, and the execution process uses a native database connection mode.

According to an embodiment of the present disclosure, the method may further include receiving feedback information from the target database executing the SQL statement; under the condition that the feedback information indicates SQL execution abnormality, determining an abnormality reason according to the feedback information; and determining to process the abnormal operation according to the abnormal reason.

According to embodiments of the present disclosure, the feedback information may include, for example, a result of executing SQL and a vendor code of feedback. The vendor code that performs the results and feedback of SQL can be categorized, for example, into the following cases:

(1) SQL execution was successful, but the number of updates was 0.

(2) The SQL execution is abnormal, and the returned vendor code means the database connection problem.

(3) The SQL execution is abnormal, and the returned vendor code means the database read-write problem.

(4) SQL execution is abnormal, and the returned vendor code means that the unique key value problem is repeated.

(5) SQL execution is abnormal, and returned vendor code means data length problem.

(6) The SQL execution is abnormal, and the returned vendor code is not of the vendor code type contained in the above (2), (3), (4) and (5).

According to embodiments of the present disclosure, different data processing is required for different anomaly reasons.

For (1), (4), for example, standby SQL may be directly executed.

For (2), for example, the SQL can be continuously executed after the database connection is reacquired, whether the upper limit of the acquisition times is exceeded or not is judged according to the configuration, and the abnormal data table is counted after the acquisition times are exceeded.

For example, the present SQL may be retried, and whether or not the execution count upper limit is exceeded may be determined based on the configuration, and the abnormal data table may be counted after the execution count is exceeded.

For example, the exception data table may be directly written for (5).

For example, the present SQL may be retried, and whether the execution count upper limit is exceeded or not may be determined according to the configuration, and after the execution count is exceeded, the standby SQL is executed, and if the standby SQL itself is the standby SQL, the abnormal data table is counted.

According to an embodiment of the disclosure, the method may further include storing the abnormal SQL of the execution abnormality in an abnormal data table in case the feedback information still indicates that the SQL is executed abnormally after the execution of the processing abnormality operation; acquiring abnormal SQL from the abnormal data table at intervals of a preset time period; and executing abnormal SQL.

According to an embodiment of the present disclosure, after the abnormal SQL in the abnormal data table is scanned, the abnormal SQL is subjected to condition addition or preprocessing, such as: the update statement may add a time stamp condition, the insert statement may make a select query, etc. And performing next abnormal SQL execution according to the preprocessing result, feeding back the abnormal SQL execution success to an abnormal data table, and notifying the abnormal processing failure due to possible missing number risk when feeding back the abnormal processing failure.

Fig. 6 schematically illustrates a flow chart of a data storage method according to another embodiment of the present disclosure.

As shown in fig. 6, the method may include operations S601 to S613. The method may be applied, for example, to a data processing apparatus that may include a parsing module 610, a shard computation module 620, and an SQL execution module 630. Operations S601 to S605 may be performed by the parsing module 610, operations S606 to S609 may be performed by the shard calculation module 620, and operations S610 to S613 may be performed by the SQL execution module 630.

In operation S601, a card data record is acquired. For example, data records from a card system may be received in real time.

In operation S602, the data record is put into different threads for processing according to the card TOPIC to which the data record belongs. Operations S603 to S605 are performed for each thread.

In operation S603, a content rule is acquired from the data record. For example, it may be determined whether the data record satisfies the analysis condition, and the content rule of the data record is determined according to the analysis condition satisfied by the data record. The content rule may be obtained, for example, from the number of the content rule table in the analysis condition satisfied by the data record.

In operation S604, the data record is parsed into key-value pairs according to content rules. Operation S223 described above with reference to fig. 5 may be performed, for example. And calculating an initial value of the source field based on the content rule to determine a target value of a target field corresponding to the source field, wherein the target field and the target value form a key value pair.

In operation S605, an SQL statement is composed using the key-value pairs and update/insert. Operation S233 described above with reference to fig. 5 may be performed, for example.

In operation S606, the database sharding number and sharding rules are acquired. Operation S214 described above with reference to fig. 3 may be performed, for example.

In operation S607, a database slice/address to which each SQL statement belongs is calculated according to the slicing rule. Operation S224 described above with reference to fig. 3 may be performed, for example.

In operation S608, hash computation is performed according to the configured hash field, and a queue in which the SQL statement is stored is determined. For example, the above-described operations may be performed to obtain a preset hash field, and hash calculation is performed on multiple SQL statements executed by the same target database according to the hash field, so as to determine queues respectively storing the multiple SQL statements, and the multiple SQL statements are respectively stored in the respective queues, so that the multiple SQL statements may be executed concurrently by the target database.

In operation S609, the SQL statement is stored in the SQL queue to be executed.

In operation S610, according to the configuration parameter value, an SQL statement in the SQL queue to be executed is obtained, and the SQL statement is sent to the target database, so that the target database executes the SQL statement. For example, when the number of SQL statements in the queue reaches the configuration parameter value, the target database may be controlled to sequentially execute a plurality of SQL statements in the order of the SQL statements in the queue.

In operation S611, for example, feedback information from the target database to execute the SQL statement may be received, and whether the target database successfully executes the SQL statement is determined according to the feedback information. If it is determined that the target database successfully executes the SQL statement, operation S613 may be performed, and if it is determined that the target database does not successfully execute the SQL statement, operation S612 may be performed.

In operation S612, a next abnormal operation is determined according to the vendor code. For example, the reason of the abnormality may be determined according to the vendor code in the feedback information; and determining to process the abnormal operation according to the abnormal reason.

In operation S613, for example, the user may be prompted to successfully store the data, and the present storage is ended.

Fig. 7 schematically illustrates a block diagram of a data storage device 700 according to an embodiment of the disclosure.

As shown in fig. 7, the data storage 700 may include an acquisition module 710, a determination module 720, a parsing module 730, and a storage module 740.

The acquiring module 710 may, for example, perform operation S201 described above with reference to fig. 2 for acquiring a data record stored by the card system.

The determining module 720 may, for example, perform operation S202 described above with reference to fig. 2 for determining whether the data record satisfies the parsing condition.

The parsing module 730 may, for example, perform operation S203 described above with reference to fig. 2, for performing content parsing on the data record to obtain data information including a target field in case it is determined that the data record satisfies a parsing condition.

The storage module 740 may, for example, perform operation S204 described above with reference to fig. 2 for storing the data information into a target database.

According to an embodiment of the present disclosure, the storage module 740 may include: the first acquisition sub-module is used for acquiring address information and a fragmentation rule of the target database under the condition that a plurality of target databases exist; the first determining submodule is used for determining a target database to which each piece of data sub-information in the data information belongs respectively based on a slicing rule; and the storage sub-module is used for storing each piece of data sub-information in the data information into a respective target database.

According to an embodiment of the present disclosure, the determining module 720 may include: the second acquisition sub-module is used for acquiring a pre-configured analysis condition, wherein the analysis condition indicates reference information for triggering analysis of the data record; a second determining sub-module for comparing the related information of the data record with the reference information; and the third determining submodule is used for determining that the data record meets the analysis condition if the related information accords with the reference information.

According to an embodiment of the present disclosure, the parsing module 730 may include: a third obtaining sub-module, configured to obtain a content rule; a fourth determining submodule, configured to determine a correspondence between a source field and a target field in the data record based on the content rule; a calculation sub-module for calculating an initial value of the source field based on the content rule to determine a target value of a target field corresponding to the source field; and the generation sub-module is used for acquiring the record type of the source field from the Kaff card system and generating an SQL sentence based on the record type and a key value pair formed by the target field and the target value, wherein the record type comprises an update data type and an insertion data type.

According to an embodiment of the present disclosure, the parsing module 730 may include: acquiring content rules; based on the content rule, determining a corresponding relation between a source field and a target field in the data record; calculating an initial value of the source field based on the content rule to determine a target value of a target field corresponding to the source field; and acquiring a record type of the source field from the Kaff card system, and generating an SQL sentence based on the record type and a key value pair formed by the target field and the target value, wherein the record type comprises an update data type and an insertion data type.

According to an embodiment of the present disclosure, the apparatus 700 may further include an acquisition field module configured to acquire a preset hash field; the calculation module is used for carrying out hash calculation on a plurality of SQL sentences executed by the same target database according to the hash field so as to determine queues for respectively storing the SQL sentences; and the storing module is used for storing the SQL sentences into the respective queues respectively so that the SQL sentences can be executed by the target database simultaneously.

According to an embodiment of the present disclosure, the apparatus 700 may further include an acquisition configuration module for acquiring configuration parameter values; and the execution module is used for controlling the target database to execute the SQL sentences in sequence according to the sequence of the SQL sentences in the queue under the condition that the number of the SQL sentences in the queue reaches the configuration parameter value.

According to an embodiment of the present disclosure, the apparatus 700 may further include a receiving module for receiving feedback information from the target database executing the SQL statement; the reason determining module is used for determining the reason of the abnormality according to the feedback information under the condition that the feedback information indicates SQL execution abnormality; and the determining operation module is used for determining and processing abnormal operation according to the abnormal reason.

According to an embodiment of the present disclosure, the apparatus 700 may further include a storage exception module, configured to store, in an exception data table, an exception SQL for executing an exception if the feedback information still indicates that the SQL is executing the exception after executing the processing exception operation; the sentence acquisition module is used for acquiring the abnormal SQL from the abnormal data table at intervals of a preset time period; and an abnormal SQL executing module for executing the abnormal SQL.

Any number of modules, sub-modules, units, sub-units, or at least some of the functionality of any number of the sub-units according to embodiments of the present disclosure may be implemented in one module. Any one or more of the modules, sub-modules, units, sub-units according to embodiments of the present disclosure may be implemented as split into multiple modules. Any one or more of the modules, sub-modules, units, sub-units according to embodiments of the present disclosure may be implemented at least in part as a hardware circuit, such as a Field Programmable Gate Array (FPGA), a Programmable Logic Array (PLA), a system-on-chip, a system-on-substrate, a system-on-package, an Application Specific Integrated Circuit (ASIC), or in any other reasonable manner of hardware or firmware that integrates or encapsulates the circuit, or in any one of or a suitable combination of three of software, hardware, and firmware. Alternatively, one or more of the modules, sub-modules, units, sub-units according to embodiments of the present disclosure may be at least partially implemented as computer program modules, which when executed, may perform the corresponding functions.

For example, any of the acquisition module 710, the determination module 720, the parsing module 730, and the storage module 740 may be combined in one module to be implemented, or any of the modules may be split into a plurality of modules. Alternatively, at least some of the functionality of one or more of the modules may be combined with at least some of the functionality of other modules and implemented in one module. At least one of the acquisition module 710, the determination module 720, the parsing module 730, and the storage module 740 may be implemented, at least in part, as hardware circuitry, such as a Field Programmable Gate Array (FPGA), a Programmable Logic Array (PLA), a system on a chip, a system on a substrate, a system on a package, an Application Specific Integrated Circuit (ASIC), or in hardware or firmware, such as any other reasonable way of integrating or packaging the circuitry, or in any one of or a suitable combination of three of software, hardware, and firmware, according to embodiments of the present disclosure. Alternatively, at least one of the acquisition module 710, the determination module 720, the parsing module 730, and the storage module 740 may be at least partially implemented as a computer program module, which when executed, may perform the corresponding functions.

Fig. 8 schematically illustrates a block diagram of an electronic device according to an embodiment of the disclosure. The electronic device shown in fig. 8 is merely an example and should not be construed to limit the functionality and scope of use of the disclosed embodiments.

As shown in fig. 8, a computer electronic device 800 according to an embodiment of the present disclosure includes a processor 801 that can perform various appropriate actions and processes according to a program stored in a Read Only Memory (ROM) 802 or a program loaded from a storage section 808 into a Random Access Memory (RAM) 803. The processor 801 may include, for example, a general purpose microprocessor (e.g., a CPU), an instruction set processor and/or an associated chipset and/or special purpose microprocessor (e.g., an Application Specific Integrated Circuit (ASIC)), or the like. The processor 801 may also include on-board memory for caching purposes. The processor 801 may include a single processing unit or multiple processing units for performing the different actions of the method flows according to embodiments of the disclosure.

In the RAM 803, various programs and data required for the operation of the electronic device 800 are stored. The processor 801, the ROM 802, and the RAM 803 are connected to each other by a bus 804. The processor 801 performs various operations of the method flow according to the embodiments of the present disclosure by executing programs in the ROM 802 and/or the RAM 803. Note that the program may be stored in one or more memories other than the ROM 802 and the RAM 803. The processor 801 may also perform various operations of the method flows according to embodiments of the present disclosure by executing programs stored in the one or more memories.

According to an embodiment of the present disclosure, the electronic device 800 may also include an input/output (I/O) interface 805, the input/output (I/O) interface 805 also being connected to the bus 804. The electronic device 800 may also include one or more of the following components connected to the I/O interface 805: an input section 807 including a keyboard, a mouse, and the like; an output portion 807 including a display such as a Cathode Ray Tube (CRT), a Liquid Crystal Display (LCD), and a speaker; a storage section 808 including a hard disk or the like; and a communication section 809 including a network interface card such as a LAN card, a modem, or the like. The communication section 809 performs communication processing via a network such as the internet. The drive 810 is also connected to the I/O interface 805 as needed. A removable medium 811 such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, or the like is mounted on the drive 810 as needed so that a computer program read out therefrom is mounted into the storage section 808 as needed.

According to embodiments of the present disclosure, the method flow according to embodiments of the present disclosure may be implemented as a computer software program. For example, embodiments of the present disclosure include a computer program product comprising a computer program embodied on a computer readable storage medium, the computer program comprising program code for performing the method shown in the flowcharts. In such an embodiment, the computer program may be downloaded and installed from a network via the communication section 809, and/or installed from the removable media 811. The above-described functions defined in the system of the embodiments of the present disclosure are performed when the computer program is executed by the processor 801. The systems, devices, apparatus, modules, units, etc. described above may be implemented by computer program modules according to embodiments of the disclosure.

The present disclosure also provides a computer-readable storage medium that may be embodied in the apparatus/device/system described in the above embodiments; or may exist alone without being assembled into the apparatus/device/system. The computer-readable storage medium carries one or more programs which, when executed, implement methods in accordance with embodiments of the present disclosure.

According to embodiments of the present disclosure, the computer-readable storage medium may be a non-volatile computer-readable storage medium, which may include, for example, but is not limited to: a portable computer diskette, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing. In the context of this disclosure, a computer-readable storage medium may be any tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device. For example, according to embodiments of the present disclosure, the computer-readable storage medium may include ROM 802 and/or RAM 803 and/or one or more memories other than ROM 802 and RAM 803 described above.

The flowcharts and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods and computer program products according to various embodiments of the present disclosure. In this regard, each block in the flowchart or block diagrams may represent a module, segment, or portion of code, which comprises one or more executable instructions for implementing the specified logical function(s). It should also be noted that, in some alternative implementations, the functions noted in the block may occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved. It will also be noted that each block of the block diagrams or flowchart illustration, and combinations of blocks in the block diagrams or flowchart illustration, can be implemented by special purpose hardware-based systems which perform the specified functions or acts, or combinations of special purpose hardware and computer instructions.

Those skilled in the art will appreciate that the features recited in the various embodiments of the disclosure and/or in the claims may be combined in various combinations and/or combinations, even if such combinations or combinations are not explicitly recited in the disclosure. In particular, the features recited in the various embodiments of the present disclosure and/or the claims may be variously combined and/or combined without departing from the spirit and teachings of the present disclosure. All such combinations and/or combinations fall within the scope of the present disclosure.

The embodiments of the present disclosure are described above. However, these examples are for illustrative purposes only and are not intended to limit the scope of the present disclosure. Although the embodiments are described above separately, this does not mean that the measures in the embodiments cannot be used advantageously in combination. The scope of the disclosure is defined by the appended claims and equivalents thereof. Various alternatives and modifications can be made by those skilled in the art without departing from the scope of the disclosure, and such alternatives and modifications are intended to fall within the scope of the disclosure.

Claims

1. A data storage method, comprising:

acquiring a data record stored by a Kaff card system;

determining whether the data record meets analysis conditions or not, wherein the analysis conditions comprise a plurality of analysis rules, each analysis rule corresponds to one analysis field, and the analysis fields comprise a topic name of a card, a main key field in the data record, a trigger condition type, trigger condition content, a trigger sequence, a rule ID, a HASH calculation field name and a fragmentation calculation field;

performing content analysis on the data record to obtain data information comprising a target field under the condition that the data record meets analysis conditions, wherein the data record meets analysis conditions and comprises the data record meeting a plurality of analysis rules; and

Storing the data information into a target database;

wherein the obtaining the data information including the target field by performing content parsing on the data record includes:

determining a content rule for analyzing the data record according to the number of a content rule table, wherein fields in the content rule table comprise a source field name, a target field name, a field analysis type, field detail analysis content and a rule ID;

based on the content rule, determining a corresponding relation between a source field and a target field in the data record;

calculating an initial value of the source field based on the content rule to determine a target value of a target field corresponding to the source field;

acquiring a record type of the source field from the Kaff card system, and generating an SQL sentence based on the record type and a key value pair formed by the target field and the target value, wherein the record type comprises an update data type and an insertion data type;

the target database is firstly executed according to the main SQL, and the standby SQL is executed under the condition that the execution according to the main SQL fails; the type of the main SQL is the same as the record type of the data record, and the type of the standby SQL is an inserted data type under the condition that the record type is an updated data type; and in the case that the record type is the insert data type, the SQL backup type is the update data type.

2. The method of claim 1, wherein the storing the data information to a target database comprises:

under the condition that a plurality of target databases exist, acquiring address information and fragmentation rules of the target databases;

determining a target database to which each piece of data sub-information in the data information belongs respectively based on a slicing rule; and

and storing each data sub-information in the data information into a respective target database.

3. The method of claim 1, wherein the determining whether the data record satisfies a resolution condition comprises:

acquiring a pre-configured analysis condition, wherein the analysis condition indicates reference information triggering analysis of the data record;

comparing the related information of the data record with the reference information; and

and if the related information accords with the reference information, determining that the data record meets the analysis condition.

4. The method of claim 1, further comprising:

acquiring a preset hash field;

performing hash calculation on a plurality of SQL sentences executed by the same target database according to the hash field to determine queues for respectively storing the SQL sentences; and

And respectively storing the SQL sentences into respective queues so that the SQL sentences can be executed by the target database simultaneously.

5. The method of claim 4, further comprising:

acquiring configuration parameter values;

and under the condition that the number of SQL sentences in the queue reaches the configuration parameter value, controlling the target database to sequentially execute the SQL sentences according to the sequence of the SQL sentences in the queue.

6. The method of claim 1, further comprising:

receiving feedback information from the target database for executing the SQL statement;

under the condition that the feedback information indicates SQL execution abnormality, determining an abnormality reason according to the feedback information; and

and determining to process abnormal operation according to the abnormal reason.

7. The method of claim 6, further comprising:

after the abnormal handling operation is executed, storing the abnormal SQL for executing the abnormality in an abnormal data table under the condition that the feedback information still indicates that the SQL is abnormal to execute;

acquiring the abnormal SQL from the abnormal data table at intervals of a preset time period; and

and executing the abnormal SQL.

8. A data storage device, comprising:

The acquisition module is used for acquiring the data record stored by the Kaff card system;

the determining module is used for determining whether the data record meets analysis conditions or not, wherein the analysis conditions comprise a plurality of analysis rules, each analysis rule corresponds to one analysis field, and the analysis fields comprise a topic name of a Kaff card, a main key field in the data record, a trigger condition type, trigger condition content, a trigger sequence, a rule ID, a HASH calculation field name and a fragment calculation field;

the analysis module is used for carrying out content analysis on the data record to obtain data information comprising a target field under the condition that the data record meets analysis conditions, wherein the data record meets the analysis conditions and comprises the data record meeting a plurality of analysis rules; and

the storage module is used for storing the data information into a target database;

wherein, the parsing module includes:

a third obtaining sub-module, configured to determine a content rule for parsing the data record according to a number of a content rule table, where a field in the content rule table includes a source field name, a target field name, a field parsing type, a field detail parsing content, and a rule ID;

A fourth determining submodule, configured to determine a correspondence between a source field and a target field in the data record based on the content rule;

a calculation sub-module for calculating an initial value of the source field based on the content rule to determine a target value of a target field corresponding to the source field;

a generating sub-module, configured to obtain a record type of the source field from the kaff card system, and generate an SQL statement based on the record type and a key value pair formed by the target field and the target value, where the record type includes an update data type and an insert data type;

9. The apparatus of claim 8, wherein the storage module comprises:

The first acquisition sub-module is used for acquiring address information and a fragmentation rule of the target database under the condition that a plurality of target databases exist;

the first determining submodule is used for determining a target database to which each piece of data sub-information in the data information belongs respectively based on a slicing rule; and

and the storage sub-module is used for storing each piece of data sub-information in the data information into a respective target database.

10. The apparatus of claim 8, wherein the means for determining comprises:

the second acquisition sub-module is used for acquiring a pre-configured analysis condition, wherein the analysis condition indicates reference information for triggering analysis of the data record;

a second determining sub-module for comparing the related information of the data record with the reference information;

and the third determining submodule is used for determining that the data record meets the analysis condition if the related information accords with the reference information.

11. An electronic device, comprising:

one or more processors;

storage means for storing one or more programs,

wherein the one or more programs, when executed by the one or more processors, cause the one or more processors to perform the method of any of claims 1-7.

12. A computer readable storage medium having stored thereon executable instructions which when executed by a processor cause the processor to perform the method of any of claims 1 to 7.