CN111767340A

CN111767340A - Data processing method, device, electronic equipment and medium

Info

Publication number: CN111767340A
Application number: CN202010482031.0A
Authority: CN
Inventors: 唐阳光; 杨诗平; 黄景超; 毛超丹
Original assignee: Industrial and Commercial Bank of China Ltd ICBC
Current assignee: Industrial and Commercial Bank of China Ltd ICBC
Priority date: 2020-05-29
Filing date: 2020-05-29
Publication date: 2020-10-13
Anticipated expiration: 2040-05-29
Also published as: CN111767340B

Abstract

The present disclosure provides a data storage method, including: acquiring a data record stored by a card system; determining whether the data record satisfies a parsing condition; under the condition that the data record is determined to meet the analysis condition, analyzing the content of the data record to obtain data information comprising a target field; and storing the data information in a target database. The present disclosure also provides a data storage device, an electronic apparatus, and a medium.

Description

Data processing method, device, electronic equipment and medium

Technical Field

The present disclosure relates to the field of computer technologies, and in particular, to a data processing method, an apparatus, an electronic device, and a medium.

Background

There are many problems in storing data in a centralized database, such as: the database table structure can not be changed according to the application requirement, and the field is difficult to increase and decrease.

Disclosure of Invention

In view of the above, the present disclosure provides a data processing method, apparatus, electronic device and medium.

One aspect of the present disclosure provides a data storage method, including: acquiring a data record stored by a card system; determining whether the data record satisfies a parsing condition; under the condition that the data record is determined to meet the analysis condition, analyzing the content of the data record to obtain data information comprising a target field; and storing the data information in a target database.

According to an embodiment of the present disclosure, storing the data information to a target database includes: under the condition that a plurality of target databases exist, acquiring address information and fragmentation rules of the target databases; determining a target database to which each data sub-information in the data information belongs based on a fragmentation rule; and storing each data sub-information in the data information into a respective target database.

According to an embodiment of the present disclosure, determining whether the data record satisfies a parsing condition includes: acquiring a preset analysis condition, wherein the analysis condition indicates reference information for triggering analysis of the data record; comparing the related information of the data record with the reference information; and if the related information is consistent with the reference information, determining that the data record meets the analysis condition.

According to an embodiment of the present disclosure, the content parsing the data record to obtain data information including a target field includes: acquiring a content rule; determining a corresponding relation between a source field and a target field in the data record based on the content rule; based on the content rule, calculating an initial value of the source field to determine a target value of a target field corresponding to the source field; and acquiring the record type of the source field from the card system, and generating an SQL statement based on the record type and the key value pair formed by the target field and the target value, wherein the record type comprises an update data type and an insert data type.

According to an embodiment of the disclosure, the method further comprises: acquiring a preset hash field; performing hash calculation on a plurality of SQL sentences executed by the same target database according to the hash field to determine queues for respectively storing the plurality of SQL sentences; and storing the plurality of SQL sentences into respective queues respectively so that the plurality of SQL sentences can be executed by the target database concurrently.

According to an embodiment of the disclosure, the method further comprises: acquiring a configuration parameter value; and under the condition that the number of the SQL sentences in the queue reaches the configuration parameter value, controlling the target database to sequentially execute the plurality of SQL sentences according to the sequence of the SQL sentences in the queue.

According to an embodiment of the disclosure, the method further comprises: receiving feedback information from the target database for executing the SQL statement; under the condition that the feedback information indicates that SQL is executed abnormally, determining an abnormal reason according to the feedback information; and determining to process abnormal operation according to the abnormal reason.

According to an embodiment of the disclosure, the method further comprises: after the abnormal operation is processed, the feedback information still indicates that the SQL is executed abnormally, and the abnormal SQL which is executed abnormally is stored in an abnormal data table; acquiring abnormal SQL from the abnormal data table every other preset time period; and executing the abnormal SQL.

Another aspect of the present disclosure provides a data storage device comprising: the acquisition module is used for acquiring data records stored by the card system; a determining module, configured to determine whether the data record satisfies an analysis condition; the analysis module is used for analyzing the content of the data record to obtain data information comprising a target field under the condition that the data record meets the analysis condition; and the storage module is used for storing the data information into a target database.

According to an embodiment of the present disclosure, the storage module includes: the first obtaining submodule is used for obtaining the address information and the fragmentation rule of the target database under the condition that a plurality of target databases exist; the first determining submodule is used for determining a target database to which each piece of data sub-information in the data information belongs based on a fragmentation rule; and the storage sub-module is used for storing each piece of data sub-information in the data information into a respective target database.

According to an embodiment of the disclosure, the determining module includes: the second acquisition submodule is used for acquiring a preset analysis condition, and the analysis condition indicates reference information for triggering analysis of the data record; a second determining submodule for comparing information related to the data record with the reference information; and the third determining submodule is used for determining that the data record meets the analysis condition if the relevant information is consistent with the reference information.

According to an embodiment of the present disclosure, a parsing module includes: a third obtaining submodule, configured to obtain a content rule; a fourth determining submodule, configured to determine, based on the content rule, a correspondence between a source field and a target field in the data record; the calculation submodule is used for calculating the initial value of the source field based on the content rule so as to determine the target value of the target field corresponding to the source field; and the generation sub-module is used for acquiring the record type of the source field from the card system and generating an SQL statement based on the record type and the key value pair formed by the target field and the target value, wherein the record type comprises an update data type and an insert data type.

Another aspect of the present disclosure provides an electronic device including: one or more processors; a storage device to store one or more programs, wherein the one or more programs, when executed by the one or more processors, cause the one or more processors to perform the above-described method.

Another aspect of the present disclosure provides a computer-readable storage medium storing computer-executable instructions for implementing the method as described above when executed.

Another aspect of the disclosure provides a computer program comprising computer executable instructions for implementing the method as described above when executed.

Drawings

The above and other objects, features and advantages of the present disclosure will become more apparent from the following description of embodiments of the present disclosure with reference to the accompanying drawings, in which:

fig. 1 schematically illustrates an application scenario of a data storage method according to an embodiment of the present disclosure;

FIG. 2 schematically illustrates a flow diagram of a data storage method according to an embodiment of the disclosure;

FIG. 3 schematically illustrates a flow chart of a method of storing data information to a target database according to an embodiment of the present disclosure;

FIG. 4 schematically illustrates a flow chart of a method of determining whether the data record satisfies a resolution condition according to an embodiment of the present disclosure;

FIG. 5 schematically illustrates a flow chart of a method of content parsing a data record to obtain data information including a target field according to an embodiment of the disclosure;

FIG. 6 schematically illustrates a flow diagram of a data storage method according to another embodiment of the present disclosure;

FIG. 7 schematically illustrates a block diagram of a data storage device according to an embodiment of the present disclosure; and

FIG. 8 schematically shows a block diagram of an electronic device according to an embodiment of the disclosure.

Detailed Description

Hereinafter, embodiments of the present disclosure will be described with reference to the accompanying drawings. It should be understood that the description is illustrative only and is not intended to limit the scope of the present disclosure. In the following detailed description, for purposes of explanation, numerous specific details are set forth in order to provide a thorough understanding of the embodiments of the disclosure. It may be evident, however, that one or more embodiments may be practiced without these specific details. Moreover, in the following description, descriptions of well-known structures and techniques are omitted so as to not unnecessarily obscure the concepts of the present disclosure.

The terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the disclosure. The terms "comprises," "comprising," and the like, as used herein, specify the presence of stated features, steps, operations, and/or components, but do not preclude the presence or addition of one or more other features, steps, operations, or components.

All terms (including technical and scientific terms) used herein have the same meaning as commonly understood by one of ordinary skill in the art unless otherwise defined. It is noted that the terms used herein should be interpreted as having a meaning that is consistent with the context of this specification and should not be interpreted in an idealized or overly formal sense.

Where a convention analogous to "at least one of A, B and C, etc." is used, in general such a construction is intended in the sense one having skill in the art would understand the convention (e.g., "a system having at least one of A, B and C" would include but not be limited to systems that have a alone, B alone, C alone, a and B together, a and C together, B and C together, and/or A, B, C together, etc.). Where a convention analogous to "A, B or at least one of C, etc." is used, in general such a construction is intended in the sense one having skill in the art would understand the convention (e.g., "a system having at least one of A, B or C" would include but not be limited to systems that have a alone, B alone, C alone, a and B together, a and C together, B and C together, and/or A, B, C together, etc.).

The embodiment of the disclosure provides a data storage method. The data storage method can be used for storing data in the Kaffer card system into a target database. The method may include obtaining a data record stored by a card system; determining whether the data record satisfies a parsing condition; under the condition that the data record is determined to meet the analysis condition, analyzing the content of the data record to obtain data information comprising a target field; and storing the data information in a target database.

Fig. 1 schematically illustrates an application scenario 100 of a data storage method according to an embodiment of the present disclosure. It should be noted that fig. 1 is only an example of an application scenario in which the embodiments of the present disclosure may be applied to help those skilled in the art understand the technical content of the present disclosure, but does not mean that the embodiments of the present disclosure may not be applied to other devices, systems, environments or scenarios.

As shown in fig. 1, the application scenario 100 may include a source database 101, a kava system 102, a data processing device 103, and a target database 104.

The source database 101 may be, for example, a DB2, ORACLE, MyAQL, or other database.

In order to at least partly avoid the problems with using a centralized database for storing data, it is for instance possible to build an own traffic database for an application to store data using the own traffic database. According to the embodiment of the present disclosure, for example, data in the source database 101 may be synchronized into the card system 102 in real time, and then the data in the card system 102 from the source database 101 may be transferred to the service database, i.e., the target database 104. The data processing device 103 is configured to process the data record in the card, so that the data record in the card can be correctly transferred to the target database 104.

It should be noted that the data storage method provided by the embodiment of the present disclosure may be generally executed by the data processing apparatus 103.

It should be understood that the number of source and target databases in fig. 1 is merely illustrative. There may be any number of source and target databases, as desired for implementation.

FIG. 2 schematically shows a flow chart of a data storage method according to an embodiment of the present disclosure.

As shown in fig. 2, the method may include operations S201 to S204.

In operation S201, a data record stored by the card system is acquired.

For example, a request to obtain a data record may be sent to the card system and a data record sent from the card system to the data processing apparatus 103 in response to the request may be received, or the card system may send a data record to the data processing apparatus 103 in real time and the data processing apparatus 103 may receive a data record from the card system in real time.

According to embodiments of the present disclosure, the data records stored in the card system may be generated from data from DB2, ORACLE, MyAQL, etc. databases.

In operation S202, it is determined whether the data record satisfies a parsing condition.

The parsing conditions may be preconfigured by those skilled in the art according to the embodiments of the present disclosure. The parsing condition may be, for example, that a value of a certain parsing field in the data record is equal to a reference value, or that a type of a certain parsing field in the data record is a specific type, etc.

The following table one schematically shows an example of a resolution field in which a resolution condition may be configured according to an embodiment of the present disclosure. As shown in table one below, the parsing conditions that can be configured by those skilled in the art may include the snapshot name, primary key field, trigger condition type, trigger condition content, trigger order, rule ID, HASH calculation field name, and fragment calculation field of the kaffman. According to the embodiment of the disclosure, a person skilled in the art may configure the topic name, the primary key field, the trigger condition type, the trigger condition content, the trigger sequence, the rule ID, the HASH calculation field name and the fragment calculation field of the kaffman respectively, so as to generate a parsing condition.

Watch 1

topIC_NAME	Topic name of Kafka
		KEY_COLUMN	Primary key field
TRIGGER_RULE_TYPE	Type of trigger condition
		TRIGGER_RULE_CONTENT	Triggering condition content
TRIGGER_ORDER	Trigger sequence
		RULE_ID	Rule ID
HASH_COLUMN	HASH calculation field name
		SHARDING_COLUMN	Sharded computation field

The TRIGGER condition TYPE of the parsing field TRIGGER _ run _ TYPE may include, for example:

START _ WITH begins WITH something

END _ WITH ENDs WITH a certain content

CONTAIN containing a content

EQUAL to a certain value

INDEX _ OF _ CHAR _ EQUAL has a value EQUAL to a certain segment

REG _ EXP satisfies regular expression

ALL records are parsed as such

SKIP the record starting WITH a certain content

SKIP the record if SKIP _ END _ width ENDs WITH a certain content

SKIP _ CONTAIN CONTAINs a content and SKIPs the record

SKIP this record if the SKIP _ EQUAL value EQUALs a certain value

SKIP _ INDEX _ OF _ CHAR _ EQUAL SKIPs over this record if a certain value EQUALs

SKIP _ REG _ EXP satisfies the regular expression and SKIPs the record

……

According to an embodiment of the present disclosure, the TRIGGER _ RULE _ CONTENT may be stored in a character string in json format, for example, with a key as a source table field name and a value as a matching CONTENT. For example, TRIGGER _ run _ TYPE is configured to: "START _ WITH", TRIGGER _ run _ CONTENT is configured to: { "CI _ ID": and "0001", judging whether the value corresponding to the field CI _ ID in the data record starts with 0001. If the value corresponding to the field CI _ ID in the data record is beginning at 0001, the data record is determined to satisfy the analysis condition.

In operation S203, in case that it is determined that the data record satisfies the parsing condition, the data record is subjected to content parsing to obtain data information including the target field.

According to embodiments of the present disclosure, the data records may be parsed, for example, according to content rules. Table two below schematically shows an example of a content rule field according to an embodiment of the present disclosure.

Watch two

SOURCE_COLUMN_NAME	Source field name
		TARGET_COLUMN_NAME	Target field name
CONTENT_TRANSFORM_TYPE	Type of field resolution
		CONTENT_TRANSFORM_CONTENT	Detailed parsing of content for fields
RULE_ID	Rule ID (equivalent to outer bond)

CONTENT _ transfer _ TYPE may comprise, for example, the following TYPEs:

SUB _ STRING SEGMENTATION STRING

CALC _ PLUS original value PLUS a certain value

CALC _ MULTIPLY original value multiplied by a certain value

RANGE _ OF _ ONE RANGE returns a value

Range _ OF _ CALC RANGE value calculation

DATE _ transfer DATE format conversion

CONSTANT return CONSTANT

SPLIT AND splice fields for SPLIT _ AND _ JOIN

DICTIONARY data word exemplary return

SKIP this field is not duplicated

DEFAULT Default (Return to original field value)

The field CONTENT _ transfer _ CONTENT allows the CONTENT and format of the filling to be regulated differently according to the value of the field CONTENT _ transfer _ TYPE. Examples of the value of the field CONTENT _ transfer _ TYPE and the CONTENT, format and meaning of the field CONTENT _ transfer _ CONTENT allowed to be filled in are schematically shown below.

CONTENT _ transfer _ TYPE is configured as SUB _ STRING, and if the value of CONTENT _ transfer _ CONTENT is configured as [0, 10], a total of 11 th characters from 1 st to 11 th can be intercepted and returned.

The CONTENT _ transfer _ TYPE is configured as CALC _ PLUS, and if the value of CONTENT _ transfer _ CONTENT is configured as-100, the original value may be converted into a numerical value and added to (-100) for return,

the CONTENT _ transfer _ TYPE is configured as CALC _ multi, and if the value of CONTENT _ transfer _ CONTENT is configured as 1/10, the original value may be multiplied by (1/10) to return.

CONTENT _ transfer _ TYPE is configured as RANGE _ OF _ ONE, if the value OF CONTENT _ transfer _ CONTENT is configured as { [0, 10 ]: 10; [11, 20]: 20, then 10 may be returned if the original value falls within the range of 0 to 10, and 20 may be returned if the original value falls within the range of 11 to 20.

CONTENT _ transfer _ TYPE is configured as RANGE _ OF _ call, if the value OF CONTENT _ transfer _ CONTENT is configured as { [0, 10 ]: PLUS: 100, [11]: multi ply: 1/10, the original value is converted into a value less than or equal to 10, then 100 is added, and the value is returned, if the value is greater than or equal to 11, then 1/10 is multiplied.

CONTENT _ transfer _ TYPE is configured as DATE _ transfer if the value of CONTENT _ transfer _ CONTENT is configured as YYYY-MM-DD HH: mm: ss; 9999-12-3100: 00: 00, the date value can be analyzed according to the character string type, and the analysis failure returns to' 9999-12-3100: 00: 00".

CONTENT _ transfer _ TYPE is configured to status, and 2019AA may be returned if the value of CONTENT _ transfer _ CONTENT is configured to 2019 AA.

Content _ TRANSFORM _ TYPE is configured as SPLIT _ AND _ JOIN, if the value of contentTransformContents is configured as [0, 5 ]: [6, 18]: [19] (ii) a Cutting out the 1 st to 6 th, 7 th to 19 th and 20 th to last character strings from the original value, connecting by AA and BB, and finally adding CC character strings; if the value of contentTransformMContent is configured to be 5: and AA, dividing the original value once every 5 character strings (finally calculating once less than 5), and returning by using AA for connection.

CONTENT _ transfer _ TYPE is configured as dicionary, if the value of contentTransformContent is configured as { [ AAA, BBB, CCC ]: ABC; [ XXX, YYY, ZZZ ]: EFG }? XYZ, it can be that if the original value is AAA or BBB or CCC, ABC is returned, if the original value is XXX or YYY or ZZZ, EFG is returned, and the rest value is XYZ.

CONTENT _ transfer _ TYPE is configured as SKIP, this field is not duplicated, i.e. no key-value pairs are generated.

CONTENT _ TRANSFORM _ TYPE is configured to DEFAULT, then DEFAULT (return original field value)

According to an embodiment of the present disclosure, the calculated value may be stored as a key-value pair, for example, the storage format may be: colName (i.e., target field name): the value of the target field. (not stored if ContentTransformType is SKIP).

As described above, those skilled in the art can configure the snapshot name, the primary key field, the type of trigger condition, the content of trigger condition, the trigger sequence, the rule ID, the HASH calculation field name, and the fragment calculation field of the card, respectively, to generate a parsing condition. In other words, each parsing condition may include a plurality of parsing rules, for example, the parsing rules may be a topic name of the card, a primary key field in the data record, a trigger condition type, a trigger condition content, a trigger order, a rule ID, a HASH calculation field name, and a shard calculation field, respectively. According to the embodiment of the disclosure, whether the data records sequentially satisfy the plurality of analysis rules can be verified one by one, and the data records satisfy the analysis conditions under the condition that the data records satisfy the plurality of analysis rules.

According to an embodiment of the present disclosure, a plurality of resolving conditions may be set by a person skilled in the art. And in the case that the data record does not meet the current analysis condition, continuously verifying whether the data record meets the next analysis condition. If the data record fails to satisfy any of the plurality of analysis conditions, the data record may be recorded in an exception table.

In operation S204, the data information is stored in the target database.

According to an embodiment of the present disclosure, for example, a data record is subjected to content parsing to obtain a key-value pair, and then SQL may be generated according to the key-value pair and a record type of the data record recorded in the kava system, and the SQL statement is executed by a target database, so as to store data information in the target database. The record types of the data records may include, for example, an update (update) data type and an insert (insert) data type.

The following table three schematically illustrates an example of composing rules using key-value pairs to compose SQL statements in accordance with an embodiment of the disclosure.

As shown in Table three, the SQL statements composed using key-value pairs include main SQL and standby SQL. The target database is executed according to the main SQL firstly, and the backup SQL is executed under the condition that the execution according to the main SQL fails.

For example, if the record type is update, the primary SQL generated according to the key-value pair is update type SQL, and the backup SQL is Insert type SQL.

Watch III

Type of card record	Main SQL	Backup SQL
			update	Update type SQL	Insert type SQL
insert	Insert type SQL	Update type SQL
			delete	Delete type SQL	Is free of
key_update_before	Delete type SQL	Is free of
			key_update_after	Update type SQL	Insert type SQL

FIG. 3 schematically illustrates a flow chart of a method of storing data information to a target database according to an embodiment of the present disclosure.

As shown in fig. 3, the method may include operations S214 to S234.

In operation S214, in the case that a plurality of target databases exist, address information and fragmentation rules of the target databases are acquired.

According to an embodiment of the present disclosure, for example, the address of the target database may be read from a configuration file, and the target database address in the configuration file may be preset by a person skilled in the art. The configuration file may include, for example, a primary database address and a backup database address. And when the vector code is 0 or the readOnly is abnormal, subsequently using the address of the standby server to perform database dropping. Wherein the vendor code returns an error code for the database service defined by the database vendor, such as: MySQL defines 1062 as a duplicate unique key return code that a database encounters a conflict when executing SQL.

According to an embodiment of the present disclosure, the address of the target database may also be acquired from a database centralized manager for managing a plurality of databases, for example. For example, a connection can be established with the address of the database centralized manager to access the address of the database centralized manager, and the current service information of all databases can be obtained from the database centralized manager. When the current service information of the databases is used for performing database dropping and the abnormality that the vendor code is 0 or the readOnly is abnormal, the current service information is obtained again every 30s, and then whether the current service information is correct or not is verified (the judgment of the database service readOnly is performed, and if and only if the readOnly is obtained normally and is not ON, the current service information is normal), the new normal information is used for continuing the database dropping.

According to the embodiment of the present disclosure, the fragmentation rule may also be read from the configuration file, and a person skilled in the art may set the fragmentation rule in the configuration file. The fragmentation rule may indicate a target database to which the data information should be written, i.e. the fragmentation rule may indicate a correspondence between the data information and the target database. For example, a sharding rule may be to store a value for a particular field in a data record in a first database.

In operation S224, based on the slicing rule, a target database to which each piece of data sub-information in the data information belongs is determined.

The data information may be, for example, the respective achievements of each person in a class, and the data sub-information may be, for example, the Chinese achievements of each person in a class.

In operation S234, each data sub-information of the data information is stored in the respective target database.

According to the embodiment of the disclosure, the data storage method can analyze the data record under the condition that the data record meets the analysis condition, so that the data information of the target field can be extracted from the data record, the increase and decrease of the fields in the database according to the requirement are realized, and the correctness of the stored data can be ensured.

According to the embodiment of the disclosure, the method can ensure that the data records in the Kaffeta system fall into the target database for storage under the condition of not missing data, and can perform database falling on a source data table divided into a plurality of different data tables in real time.

FIG. 4 schematically illustrates a flow chart of a method of determining whether the data record satisfies a resolution condition according to an embodiment of the present disclosure.

As shown in fig. 4, the method may include operations S401 to S404.

In operation S401, a preconfigured parsing condition is obtained, where the parsing condition indicates reference information that triggers parsing of the data record.

The reference information may include, for example, at least one of a topic name, a primary key field, a trigger condition type, trigger condition content, a trigger order, a rule ID, a HASH calculation field name, and a shard calculation field of the card.

The parsing condition may be, for example, the topic name of the card is "aaa", the primary key field is "ID card", the trigger condition type is "START _ width", and the trigger condition content is "CI _ ID: 0001 ", and the like.

In operation S402, the related information of the data record is compared with the reference information.

For example, it may be determined whether the topoic name of the card to which the data record belongs is "aaa", whether the primary key field is "ID card", and whether the CI _ ID starts with "0001".

In operation S403, if the related information matches the reference information, it is determined that the data record satisfies an analysis condition.

If the topic name of the card to which the data record belongs is "aaa", the primary key field is "ID card", and the CI _ ID starts with "0001", then it is determined that the data record satisfies the resolution condition.

Fig. 5 schematically shows a flowchart of a method for content parsing of a data record to obtain data information including a target field according to an embodiment of the present disclosure.

As shown in fig. 5, the method may include operations S213 to S243.

In operation S213, a content rule is acquired.

According to the embodiment of the present disclosure, a person skilled in the art may define the content rule in the configuration file according to the content field and the meaning of the content field shown in table two above, and further, may obtain the content rule from the configuration file, for example.

According to the embodiment of the present disclosure, a person skilled in the art may set a plurality of content rule tables and assign a number to each content rule table, and set the number of the content rule table corresponding to the parsing condition in the parsing condition when configuring the parsing condition, so that the content rule for parsing the data record may be determined according to the number of the content rule table when determining whether the data record satisfies the parsing condition in operation S202.

As shown in table two above, the content rule may include the content fields SOURCE _ COLUMN _ NAME and TARGET _ COLUMN _ NAME, where SOURCE _ COLUMN _ NAME represents the SOURCE field NAME and TARGET _ COLUMN _ NAME represents the TARGET field NAME. Those skilled in the art can define the TARGET _ COLUMN _ NAME in the TARGET database as "vehicle" after the data of SOURCE _ COLUMN _ NAME in the SOURCE database is stored in the TARGET database, for example.

In operation S223, a correspondence between the source field and the target field in the data record is determined based on the content rule.

For example, in the above-described embodiment, the source field "vehicle" in the source database corresponds to the target field "vehicle" in the target database.

In operation S233, an initial value of a source field is calculated to determine a target value of a target field corresponding to the source field based on a content rule.

The initial value of the source field may be calculated, for example, according to the field parsing TYPE CONTENT _ transfer _ TYPE and the field detailed parsing CONTENT _ transfer _ CONTENT described above in operation S203.

In operation S243, a record type of the source field is obtained from the kava system, and an SQL statement is generated based on the record type and the key-value pair formed by the target field and the target value, wherein the record type includes an update data type and an insert data type.

The update data type is the update type in table three above, and the insert data type is the insert type in table three above.

According to the embodiment of the present disclosure, for example, the SQL statement may be generated according to a key-value pair formed by a primary key field, a record type, a target field, and a target value in the parsing condition. Wherein the primary key field may be used to identify which data sub-record in the data record to insert or update. For example, the data record includes data sub-records of zhang san and lie san, and the data record may include fields of an identity card, an age, and the like, wherein the primary key field may be an identity card field. In this example, the SQL statement may be generated based on the ID field (e.g., 371522 … …), the key-value pair, and the record type (e.g., insert), and the update operation to the data sub-record "Zhang three" is determined based on the ID field "371522 … …".

According to the embodiment of the disclosure, the method can perform heterogeneous parsing on the data record data into data with a different type from the source data according to the content rule, for example, data type data can be resolved into character type data.

According to the embodiment of the disclosure, the data processing method may further include obtaining a preset hash field, performing hash calculation on a plurality of SQL statements executed by the same target database according to the hash field to determine queues for respectively storing the plurality of SQL statements, and respectively storing the plurality of SQL statements in respective queues, so that the plurality of SQL statements may be concurrently executed by the target database.

According to the embodiment of the disclosure, the hash field may be, for example, a primary key field, and the primary key field of the SQL statement is hashed to divide the plurality of SQL statements into a plurality of different queues, so that the target database may concurrently execute the SQL statements in the plurality of different queues.

According to the embodiment of the disclosure, in order to increase the utilization rate of database connections, several SQL executed by the same target database may perform hash calculation according to the configured hash fields (for example, may be primary key values) to divide the several SQL into different queues to be executed, and the different queues to be executed will be executed concurrently.

According to the embodiment of the disclosure, the hash field is configured, so that data coverage is not caused when a plurality of source tables fall into one target table.

According to an embodiment of the present disclosure, the method may further include obtaining a configuration parameter value; and under the condition that the number of the SQL sentences in the queue reaches the configuration parameter value, controlling the target database to sequentially execute a plurality of SQL sentences according to the sequence of the SQL sentences in the queue. According to the embodiment of the disclosure, in order to improve the execution efficiency of the database, SQL scraping is performed before SQL is executed, and the scraping stopping condition can be that the number of SQL reaches the configuration parameter value and no SQL exists in the queue to be executed. When the two conditions meet one of the scraping stop conditions, the scraping is quitted for execution, and the execution process uses a native database connection mode.

According to an embodiment of the present disclosure, the method may further include receiving feedback information from the target database executing the SQL statement; under the condition that the feedback information indicates that the SQL is abnormally executed, determining an abnormal reason according to the feedback information; and determining to process the abnormal operation according to the abnormal reason.

According to an embodiment of the present disclosure, the feedback information may include, for example, a result of executing SQL and a vendorcode of the feedback. The vendor code for the result and feedback of the executed SQL can be classified into the following cases, for example:

(1) SQL executed successfully, but the number of updates was 0.

(2) SQL execution is abnormal, and the returned vector code means the database connection problem.

(3) SQL execution is abnormal, and the returned vector code means database reading and writing problems.

(4) SQL execution is abnormal, and the returned vector code means a problem of repeating unique key values.

(5) SQL execution is abnormal, and the returned vector code means the problem of data length.

(6) And (4) the SQL execution is abnormal, and the returned vendor code is not the vendor code type contained in the (2), (3), (4) and (5).

According to the embodiment of the disclosure, different data processing is required for different abnormal reasons.

For (1), (4), for example, the backup SQL may be executed directly.

For (2), for example, SQL may be executed after reacquiring the database connection, and it may be determined whether the acquisition number exceeds the upper limit of the acquisition number according to the configuration, and the abnormal data table may be included after the acquisition number exceeds the acquisition number.

For (3), for example, the present SQL may be retried, and it may be determined whether the execution count exceeds the upper limit of the execution count according to the configuration, and the above execution count is counted in the abnormal data table.

For (5), for example, an exception table may be directly entered.

For (6), for example, the local SQL may be retried, and it may be determined whether the execution count upper limit is exceeded or not according to the configuration, and after the execution count is exceeded, the backup SQL is executed, and if the backup SQL is used, the backup SQL is included in the abnormal data table.

According to the embodiment of the disclosure, the method may further include storing the abnormal SQL, which is executed abnormally, in an abnormal data table under the condition that the feedback information still indicates that the SQL is executed abnormally after the abnormal operation is executed and processed; acquiring abnormal SQL from the abnormal data table every other preset time period; and executing the abnormal SQL.

According to the embodiment of the disclosure, after the abnormal SQL in the abnormal data table is scanned, the abnormal SQL is conditionally added or preprocessed, such as: the update statement may add a timestamp condition, the insert statement may make a select query, etc. And performing next abnormal SQL execution according to the preprocessing result, feeding back the abnormal SQL execution to the abnormal data table after the abnormal SQL execution is successful, and notifying that manual processing is needed due to possible number leakage risk while feeding back the abnormal SQL which is failed to be processed.

FIG. 6 schematically shows a flow diagram of a data storage method according to another embodiment of the present disclosure.

As shown in fig. 6, the method may include operations S601 to S613. The method may be applied, for example, to a data processing apparatus, which may include a parsing module 610, a sharding computation module 620, and an SQL execution module 630. Operations S601 to S605 may be executed by the parsing module 610, operations S606 to S609 may be executed by the slicing calculation module 620, and operations S610 to S613 may be executed by the SQL execution module 630.

In operation S601, a card data record is acquired. For example, the data records from the kava system may be received in real time.

In operation S602, the data record is put into a different thread for processing according to the card top ic to which the data record belongs. Operations S603 to S605 are performed for each thread.

In operation S603, a content rule is acquired according to the data record. For example, it may be determined whether the data record satisfies the parsing condition, and the content rule of the data record is determined according to the parsing condition satisfied by the data record. The content rules may be obtained, for example, from the number of the content rule table in the parsing condition satisfied by the data record.

In operation S604, the data records are parsed into key-value pairs according to the content rules. For example, operation S223 described above with reference to fig. 5 may be performed. Based on the content rules, an initial value of the source field is calculated to determine a target value of a target field corresponding to the source field, wherein the target field and the target value constitute a key-value pair.

In operation S605, an SQL statement is composed using the key-value pairs and update/insert. For example, operation S233 described above with reference to fig. 5 may be performed.

In operation S606, the number of database shards and the shard rule are obtained. For example, operation S214 described above with reference to fig. 3 may be performed.

In operation S607, the database tile/address to which each SQL statement belongs is calculated according to the tile rule. For example, operation S224 described above with reference to fig. 3 may be performed.

In operation S608, a hash calculation is performed according to the configured hash field, and a queue into which the SQL statement is stored is determined. For example, the operations described above may be performed to obtain a preset hash field, perform hash calculation on a plurality of SQL statements executed by the same target database according to the hash field to determine queues for storing the plurality of SQL statements, and store the plurality of SQL statements in respective queues, so that the plurality of SQL statements may be concurrently executed by the target database.

In operation S609, the SQL statement is stored in the SQL queue to be executed.

In operation S610, according to the configuration parameter value, an SQL statement in the SQL queue to be executed is obtained, and the SQL statement is sent to the target database, so that the target database executes the SQL statement. For example, when the number of SQL statements in the queue reaches the configuration parameter value, the target database may be controlled to sequentially execute a plurality of SQL statements in the order of the SQL statements in the queue.

In operation S611, for example, feedback information from the target database executing the SQL statement may be received, and it is determined whether the target database successfully executes the SQL statement according to the feedback information. If it is determined that the target database successfully executes the SQL statement, operation S613 may be performed, and if it is determined that the target database does not successfully execute the SQL statement, operation S612 may be performed.

In operation S612, a next abnormal operation is determined according to the vendor code. For example, the cause of the abnormality may be determined according to the vendor code in the feedback information; and determining to process the abnormal operation according to the abnormal reason.

In operation S613, for example, the user may be prompted that the data is successfully stored, and the storage may be terminated.

FIG. 7 schematically shows a block diagram of a data storage device 700 according to an embodiment of the disclosure.

As shown in fig. 7, the data storage device 700 may include an acquisition module 710, a determination module 720, a parsing module 730, and a storage module 740.

The obtaining module 710 may, for example, perform operation S201 described above with reference to fig. 2, for obtaining the data record stored by the card system.

The determining module 720, for example, may perform operation S202 described above with reference to fig. 2 for determining whether the data record satisfies the parsing condition.

The parsing module 730, for example, may perform operation S203 described above with reference to fig. 2, for performing content parsing on the data record to obtain data information including the target field if it is determined that the data record satisfies the parsing condition.

The storage module 740, for example, may perform the operation S204 described above with reference to fig. 2, for storing the data information into the target database.

According to an embodiment of the present disclosure, the storage module 740 may include: the first obtaining submodule is used for obtaining the address information and the fragmentation rule of the target database under the condition that a plurality of target databases exist; the first determining submodule is used for determining a target database to which each piece of data sub-information in the data information belongs based on a fragmentation rule; and the storage sub-module is used for storing each piece of data sub-information in the data information into a respective target database.

According to an embodiment of the present disclosure, the determining module 720 may include: the second acquisition submodule is used for acquiring a preset analysis condition, and the analysis condition indicates reference information for triggering analysis of the data record; a second determining submodule for comparing information related to the data record with the reference information; and the third determining submodule is used for determining that the data record meets the analysis condition if the relevant information is consistent with the reference information.

According to an embodiment of the present disclosure, the parsing module 730 may include: a third obtaining submodule, configured to obtain a content rule; a fourth determining submodule, configured to determine, based on the content rule, a correspondence between a source field and a target field in the data record; the calculation submodule is used for calculating the initial value of the source field based on the content rule so as to determine the target value of the target field corresponding to the source field; and the generation sub-module is used for acquiring the record type of the source field from the card system and generating an SQL statement based on the record type and the key value pair formed by the target field and the target value, wherein the record type comprises an update data type and an insert data type.

According to an embodiment of the present disclosure, the parsing module 730 may include: acquiring a content rule; determining a corresponding relation between a source field and a target field in the data record based on the content rule; based on the content rule, calculating an initial value of the source field to determine a target value of a target field corresponding to the source field; and acquiring the record type of the source field from the card system, and generating an SQL statement based on the record type and the key value pair formed by the target field and the target value, wherein the record type comprises an update data type and an insert data type.

According to an embodiment of the present disclosure, the apparatus 700 may further include a field obtaining module, configured to obtain a preset hash field; the calculation module is used for performing hash calculation on a plurality of SQL sentences executed by the same target database according to the hash fields so as to determine queues for respectively storing the plurality of SQL sentences; and the storage module is used for respectively storing the SQL sentences into respective queues so that the SQL sentences can be concurrently executed by the target database.

According to an embodiment of the present disclosure, the apparatus 700 may further include an obtaining configuration module, configured to obtain a configuration parameter value; and the execution module is used for controlling the target database to sequentially execute the plurality of SQL sentences according to the sequence of the SQL sentences in the queue under the condition that the number of the SQL sentences in the queue reaches the configuration parameter value.

According to an embodiment of the present disclosure, the apparatus 700 may further include a receiving module, configured to receive feedback information from the target database to execute the SQL statement; the reason determining module is used for determining an abnormal reason according to the feedback information under the condition that the feedback information indicates that the SQL is abnormally executed; and the determining operation module is used for determining and processing abnormal operation according to the abnormal reason.

According to an embodiment of the present disclosure, the apparatus 700 may further include a storage exception module, configured to store, in an exception data table, the exception SQL that is executed abnormally under a condition that the feedback information still indicates that the SQL is executed abnormally after the exception handling operation is executed; the statement acquisition module is used for acquiring the abnormal SQL from the abnormal data table every other preset time period; and the execution abnormal SQL module is used for executing the abnormal SQL.

Any number of modules, sub-modules, units, sub-units, or at least part of the functionality of any number thereof according to embodiments of the present disclosure may be implemented in one module. Any one or more of the modules, sub-modules, units, and sub-units according to the embodiments of the present disclosure may be implemented by being split into a plurality of modules. Any one or more of the modules, sub-modules, units, sub-units according to embodiments of the present disclosure may be implemented at least in part as a hardware circuit, such as a Field Programmable Gate Array (FPGA), a Programmable Logic Array (PLA), a system on a chip, a system on a substrate, a system on a package, an Application Specific Integrated Circuit (ASIC), or may be implemented in any other reasonable manner of hardware or firmware by integrating or packaging a circuit, or in any one of or a suitable combination of software, hardware, and firmware implementations. Alternatively, one or more of the modules, sub-modules, units, sub-units according to embodiments of the disclosure may be at least partially implemented as a computer program module, which when executed may perform the corresponding functions.

For example, any of the obtaining module 710, the determining module 720, the parsing module 730, and the storing module 740 may be combined and implemented in one module, or any one of them may be split into multiple modules. Alternatively, at least part of the functionality of one or more of these modules may be combined with at least part of the functionality of the other modules and implemented in one module. According to an embodiment of the present disclosure, at least one of the obtaining module 710, the determining module 720, the parsing module 730, and the storing module 740 may be at least partially implemented as a hardware circuit, such as a Field Programmable Gate Array (FPGA), a Programmable Logic Array (PLA), a system on a chip, a system on a substrate, a system on a package, an Application Specific Integrated Circuit (ASIC), or may be implemented by hardware or firmware in any other reasonable manner of integrating or packaging a circuit, or implemented by any one of three implementations of software, hardware, and firmware, or any suitable combination of any of them. Alternatively, at least one of the obtaining module 710, the determining module 720, the parsing module 730, and the storing module 740 may be at least partially implemented as a computer program module, which when executed, may perform a corresponding function.

FIG. 8 schematically shows a block diagram of an electronic device according to an embodiment of the disclosure. The electronic device shown in fig. 8 is only an example, and should not bring any limitation to the functions and the scope of use of the embodiments of the present disclosure.

As shown in fig. 8, a computer electronic device 800 according to an embodiment of the present disclosure includes a processor 801 which can perform various appropriate actions and processes according to a program stored in a Read Only Memory (ROM)802 or a program loaded from a storage section 808 into a Random Access Memory (RAM) 803. The processor 801 may include, for example, a general purpose microprocessor (e.g., a CPU), an instruction set processor and/or associated chipset, and/or a special purpose microprocessor (e.g., an Application Specific Integrated Circuit (ASIC)), among others. The processor 801 may also include onboard memory for caching purposes. The processor 801 may include a single processing unit or multiple processing units for performing different actions of the method flows according to embodiments of the present disclosure.

In the RAM 803, various programs and data necessary for the operation of the electronic apparatus 800 are stored. The processor 801, the ROM802, and the RAM 803 are connected to each other by a bus 804. The processor 801 performs various operations of the method flows according to the embodiments of the present disclosure by executing programs in the ROM802 and/or RAM 803. Note that the programs may also be stored in one or more memories other than the ROM802 and RAM 803. The processor 801 may also perform various operations of method flows according to embodiments of the present disclosure by executing programs stored in the one or more memories.

Electronic device 800 may also include input/output (I/O) interface 805, input/output (I/O) interface 805 also connected to bus 804, according to an embodiment of the present disclosure. Electronic device 800 may also include one or more of the following components connected to I/O interface 805: an input portion 807 including a keyboard, a mouse, and the like; an output section 807 including a signal such as a Cathode Ray Tube (CRT), a Liquid Crystal Display (LCD), and the like, and a speaker; a storage portion 808 including a hard disk and the like; and a communication section 809 including a network interface card such as a LAN card, a modem, or the like. The communication section 809 performs communication processing via a network such as the internet. A drive 810 is also connected to the I/O interface 805 as necessary. A removable medium 811 such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, or the like is mounted on the drive 810 as necessary, so that a computer program read out therefrom is mounted on the storage section 808 as necessary.

According to embodiments of the present disclosure, method flows according to embodiments of the present disclosure may be implemented as computer software programs. For example, embodiments of the present disclosure include a computer program product comprising a computer program embodied on a computer readable storage medium, the computer program containing program code for performing the method illustrated by the flow chart. In such an embodiment, the computer program can be downloaded and installed from a network through the communication section 809 and/or installed from the removable medium 811. The computer program, when executed by the processor 801, performs the above-described functions defined in the system of the embodiments of the present disclosure. The systems, devices, apparatuses, modules, units, etc. described above may be implemented by computer program modules according to embodiments of the present disclosure.

The present disclosure also provides a computer-readable storage medium, which may be contained in the apparatus/device/system described in the above embodiments; or may exist separately and not be assembled into the device/apparatus/system. The computer-readable storage medium carries one or more programs which, when executed, implement the method according to an embodiment of the disclosure.

According to embodiments of the present disclosure, the computer-readable storage medium may be a non-volatile computer-readable storage medium, which may include, for example but is not limited to: a portable computer diskette, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing. In the present disclosure, a computer readable storage medium may be any tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device. For example, according to embodiments of the present disclosure, a computer-readable storage medium may include the ROM802 and/or RAM 803 described above and/or one or more memories other than the ROM802 and RAM 803.

The flowchart and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods and computer program products according to various embodiments of the present disclosure. In this regard, each block in the flowchart or block diagrams may represent a module, segment, or portion of code, which comprises one or more executable instructions for implementing the specified logical function(s). It should also be noted that, in some alternative implementations, the functions noted in the block may occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved. It will also be noted that each block of the block diagrams or flowchart illustration, and combinations of blocks in the block diagrams or flowchart illustration, can be implemented by special purpose hardware-based systems which perform the specified functions or acts, or combinations of special purpose hardware and computer instructions.

Those skilled in the art will appreciate that various combinations and/or combinations of features recited in the various embodiments and/or claims of the present disclosure can be made, even if such combinations or combinations are not expressly recited in the present disclosure. In particular, various combinations and/or combinations of the features recited in the various embodiments and/or claims of the present disclosure may be made without departing from the spirit or teaching of the present disclosure. All such combinations and/or associations are within the scope of the present disclosure.

The embodiments of the present disclosure have been described above. However, these examples are for illustrative purposes only and are not intended to limit the scope of the present disclosure. Although the embodiments are described separately above, this does not mean that the measures in the embodiments cannot be used in advantageous combination. The scope of the disclosure is defined by the appended claims and equivalents thereof. Various alternatives and modifications can be devised by those skilled in the art without departing from the scope of the present disclosure, and such alternatives and modifications are intended to be within the scope of the present disclosure.

Claims

1. A method of data storage, comprising:

acquiring a data record stored by a card system;

determining whether the data record satisfies a parsing condition;

under the condition that the data record is determined to meet the analysis condition, analyzing the content of the data record to obtain data information comprising a target field; and

and storing the data information into a target database.

2. The method of claim 1, wherein said storing the data information to a target database comprises:

under the condition that a plurality of target databases exist, acquiring address information and fragmentation rules of the target databases;

determining a target database to which each data sub-information in the data information belongs based on a fragmentation rule; and

and storing each data sub-information in the data information into a respective target database.

3. The method of claim 1, wherein the determining whether the data record satisfies a parsing condition comprises:

acquiring a preset analysis condition, wherein the analysis condition indicates reference information for triggering analysis of the data record;

comparing the related information of the data record with the reference information; and

and if the related information is consistent with the reference information, determining that the data record meets the analysis condition.

4. The method of claim 1, wherein the content parsing the data record to obtain data information including a target field comprises:

acquiring a content rule;

determining a corresponding relation between a source field and a target field in the data record based on the content rule;

based on the content rule, calculating an initial value of the source field to determine a target value of a target field corresponding to the source field;

and acquiring the record type of the source field from the card system, and generating an SQL statement based on the record type and the key value pair formed by the target field and the target value, wherein the record type comprises an update data type and an insert data type.

5. The method of claim 4, further comprising:

acquiring a preset hash field;

performing hash calculation on a plurality of SQL sentences executed by the same target database according to the hash field to determine queues for respectively storing the plurality of SQL sentences; and

and respectively storing the plurality of SQL sentences into respective queues so that the plurality of SQL sentences can be concurrently executed by the target database.

6. The method of claim 5, further comprising:

acquiring a configuration parameter value;

and under the condition that the number of the SQL sentences in the queue reaches the configuration parameter value, controlling the target database to sequentially execute the plurality of SQL sentences according to the sequence of the SQL sentences in the queue.

7. The method of claim 1, further comprising:

receiving feedback information from the target database for executing the SQL statement;

under the condition that the feedback information indicates that SQL is executed abnormally, determining an abnormal reason according to the feedback information; and

and determining to process abnormal operation according to the abnormal reason.

8. The method of claim 7, further comprising:

after the abnormal operation is processed, the feedback information still indicates that the SQL is executed abnormally, and the abnormal SQL which is executed abnormally is stored in an abnormal data table;

acquiring the abnormal SQL from the abnormal data table every other preset time period; and

and executing the abnormal SQL.

9. A data storage device comprising:

the acquisition module is used for acquiring data records stored by the card system;

a determining module, configured to determine whether the data record satisfies an analysis condition;

the analysis module is used for analyzing the content of the data record to obtain data information comprising a target field under the condition that the data record meets the analysis condition; and

and the storage module is used for storing the data information into a target database.

10. The apparatus of claim 9, wherein the storage module comprises:

the first obtaining submodule is used for obtaining the address information and the fragmentation rule of the target database under the condition that a plurality of target databases exist;

the first determining submodule is used for determining a target database to which each piece of data sub-information in the data information belongs based on a fragmentation rule; and

and the storage sub-module is used for storing each piece of data sub-information in the data information into a respective target database.

11. The apparatus of claim 9, wherein the means for determining comprises:

the second acquisition submodule is used for acquiring a preset analysis condition, and the analysis condition indicates reference information for triggering analysis of the data record;

a second determining submodule for comparing information related to the data record with the reference information;

and the third determining submodule is used for determining that the data record meets the analysis condition if the relevant information is consistent with the reference information.

12. The apparatus of claim 9, wherein the parsing module comprises:

a third obtaining submodule, configured to obtain a content rule;

a fourth determining submodule, configured to determine, based on the content rule, a correspondence between a source field and a target field in the data record;

the calculation submodule is used for calculating the initial value of the source field based on the content rule so as to determine the target value of the target field corresponding to the source field;

and the generation sub-module is used for acquiring the record type of the source field from the card system and generating an SQL statement based on the record type and the key value pair formed by the target field and the target value, wherein the record type comprises an update data type and an insert data type.

13. An electronic device, comprising:

one or more processors;

a storage device for storing one or more programs,

wherein the one or more programs, when executed by the one or more processors, cause the one or more processors to perform the method of any of claims 1-8.

14. A computer readable storage medium having stored thereon executable instructions which, when executed by a processor, cause the processor to perform the method of any one of claims 1 to 8.