WO2023125038A1 - 数据表的预处理方法、装置、电子设备及存储介质 - Google Patents

数据表的预处理方法、装置、电子设备及存储介质 Download PDF

Info

Publication number
WO2023125038A1
WO2023125038A1 PCT/CN2022/139269 CN2022139269W WO2023125038A1 WO 2023125038 A1 WO2023125038 A1 WO 2023125038A1 CN 2022139269 W CN2022139269 W CN 2022139269W WO 2023125038 A1 WO2023125038 A1 WO 2023125038A1
Authority
WO
WIPO (PCT)
Prior art keywords
data
parameter
data table
execution command
statement
Prior art date
Application number
PCT/CN2022/139269
Other languages
English (en)
French (fr)
Inventor
吕相杰
Original Assignee
中兴通讯股份有限公司
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by 中兴通讯股份有限公司 filed Critical 中兴通讯股份有限公司
Publication of WO2023125038A1 publication Critical patent/WO2023125038A1/zh

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
    • G06F16/22—Indexing; Data structures therefor; Storage structures
    • G06F16/2282—Tablespace storage structures; Management thereof
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
    • G06F16/22—Indexing; Data structures therefor; Storage structures
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
    • G06F16/23—Updating
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F21/00—Security arrangements for protecting computers, components thereof, programs or data against unauthorised activity
    • G06F21/60—Protecting data
    • G06F21/62—Protecting access to data via a platform, e.g. using keys or access control rules
    • Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
    • Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
    • Y02D—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE
    • Y02D30/00—Reducing energy consumption in communication networks
    • Y02D30/50—Reducing energy consumption in communication networks in wire-line communication networks, e.g. low power modes or reduced link rate

Definitions

  • the embodiments of the present application relate to the field of databases, and in particular to a data table preprocessing method, device, electronic equipment, and storage medium.
  • the preprocessing process of the data table in the database is mainly divided into three steps, which are creation, execution and deletion.
  • the creation stage mainly creates prepared statements
  • the execution stage mainly sets the specific value of the inserted data
  • the deletion stage deletes the previously defined prepared statements.
  • each set of data can only be inserted into the data table separately. For the insertion of a large amount of data, it will consume a lot of time and reduce the efficiency of the entire preprocessing process.
  • the main purpose of the embodiments of the present application is to provide a data table preprocessing method, device, electronic equipment, and storage medium to implement batch insertion of a large amount of data into a data table and improve preprocessing efficiency.
  • an embodiment of the present application provides a data table preprocessing method, including: creating a preprocessing statement according to the received creation command, the preprocessing statement carries a parameter package, and the parameter package includes at least one A placeholder for representing data to be inserted; receiving an execution command for executing the prepared statement; wherein, the execution command carries a data packet including multiple sets of data; the parameter packet carried in the prepared statement Packing with the data packets carried in the execution command, inserting each group of data in the data packets into the data table in batches; deleting the created prepared statement.
  • a data table preprocessing device including:
  • a creation module configured to create a prepared statement according to the received creation command, the prepared statement carries a parameter package, and the parameter package includes at least one placeholder for representing data to be inserted;
  • An execution module configured to receive an execution command for executing the prepared statement, wherein the execution command carries a data packet including multiple sets of data; and combines the parameter package carried in the prepared statement with the execution command The data packets carried in the data packet are assembled, and each group of data in the data packet is inserted into the data table in batches;
  • the delete module is used to delete the created prepared statement.
  • an embodiment of the present application also provides an electronic device, including: at least one processor; and a memory connected to the at least one processor in communication; wherein, the memory stores information that can be used by the at least one processor An instruction executed by a processor, the instruction is executed by the at least one processor, so that the at least one processor can execute the above-mentioned data table preprocessing method.
  • the embodiment of the present application also proposes a computer-readable storage medium storing a computer program, and implementing the above-mentioned data table preprocessing method when the computer program is executed by a processor.
  • a preprocessing method, device, electronic device, and storage medium for a data table proposed by the present application create a preprocessing statement containing a parameter package according to a creation command, and the parameter package contains a placeholder for representing data to be inserted, according to the received
  • the execution command to the data package containing multiple sets of data, the parameter package and the data package are combined, and the multiple sets of data in the data package are inserted into the data table in batches, so that multiple sets of data can be processed in batches instead of one set
  • the data is grouped and inserted, and then the next group of data is grouped and inserted. It takes multiple operations to complete the preprocessing of the data table, which greatly reduces the time of grouping and insertion, and improves the preprocessing efficiency of the data table.
  • Fig. 1 is the flowchart of the preprocessing method of the data table that the embodiment of the application provides;
  • Fig. 2 is the structural representation of the preprocessing device of the data table that the embodiment of the present application provides;
  • FIG. 3 is a schematic structural diagram of an electronic device provided in an embodiment of the present application.
  • the data table preprocessing method of this application is applicable to any network.
  • the embodiment of the present application relates to a preprocessing method of a data table, as shown in Figure 1, comprising:
  • Step 101 create a prepared statement according to the received creation command, the prepared statement carries a parameter package, and the parameter package includes at least one placeholder for representing the data to be inserted.
  • the parameter package includes placeholders, which represent data to be inserted, and the number of placeholders is consistent with the amount of data to be inserted.
  • the database receives a creation command sent by the client, and the database creates a prepared statement according to the creation command.
  • Step 102 receiving an execution command for executing the prepared statement, wherein the execution command carries a data packet including multiple sets of data.
  • Step 103 combining the parameter package carried in the preprocessing statement and the data package carried in the execution command, and inserting each group of data in the data package into the data table in batches.
  • the execution command includes: a statement identification field used to indicate the prepared statement currently to be executed, a batch size field and a plurality of unit fields used to indicate the number of sets of the multiple sets of data, Each of the unit fields is respectively used to carry a group of data; wherein, the number of the unit fields is the same as the number of the groups.
  • step 103 specifically includes: determining the currently to-be-executed prepared statement according to the statement identification field; obtaining the parameter package of the currently-to-be-executed prepared statement, and determining the parameter package according to the batch size field The number of executions of the currently to-be-executed prepared statement; according to the location of the placeholder in the parameter package and the number of executions, insert each group of data carried by the plurality of unit fields into the data table respectively .
  • the execution command includes: data packets of multiple sets of data, statement identification field, batch size field, and unit field.
  • the execution command may receive creation commands and execution commands sent by multiple different clients, so the current prepared statement to be executed is determined according to the statement identification field in the execution command, and the pending statement created by the creation command and the execution The pending statement that the command needs to execute corresponds to. Then determine the number of executions of the prepared statement to be executed according to the batch size field in the execution command, that is, the number of data groups that need to be inserted into the data table is the number of executions, and then according to the placeholder in the parameter pack in the prepared statement The location and execution times of each group of data are inserted into the data table respectively.
  • this application can group and insert multiple groups of data that need to be inserted into the data table in batches, and the preprocessing of the data table can be performed at one time, instead of grouping and inserting one group of data, and then performing the next group of data Data consolidation, insertion, and multiple operations are required to complete the preprocessing of the data table.
  • the preprocessing time of the data table is greatly saved.
  • the data table in order to ensure the atomicity of the data in the database, the data table needs to be locked, and the data table is unlocked after the insertion is completed. If the processing time of the data table is too long, the locking time of the data table will be too long, which will affect other processing of the data table. Therefore, the present application greatly saves the preprocessing time of the data table through the batch operation, further reduces the locking time of the data table, and ensures the processing efficiency of the data table.
  • the execution command further includes: a command field used to indicate the current protocol type; in the process of combining the parameter packet carried in the prepared statement with the data packet carried in the execution command Before, the method further includes: determining that the protocol type of the execution command is a batch type according to the command field.
  • the command field is obtained from the execution command, and the protocol type of the execution command is determined to be a batch processing type according to the command field, and then batch operations are performed on multiple sets of data to be inserted.
  • the execution command further includes: a flag bit used to indicate whether the execution command transmits and uses a new data type; when the flag bit indicates transmission and use of a new data type, the The execution command also includes: a parameter type field; the parameter type field is used to indicate the parameter type of each data carried in the unit field; according to the position of the placeholder in the parameter packet and the Execution times, before inserting each group of data carried by the plurality of unit fields into the data table respectively, it also includes: analyzing and obtaining the data carried by the unit field according to the parameter type of each data indicated by the parameter type field .
  • the parameter type field When it is determined that a new data type is used through the flag bit in the execution command, it is necessary to further obtain the parameter type field from the execution command, determine the parameter type of each data according to the parameter type field, and parse the data according to the parameter type of each data. The parsed data is then inserted into a data table. If the flag bit in the execution command indicates that no new data type is used this time, the parameter type of each data is the parameter type of the placeholder in the prepared statement, that is, the parameter type of each data is the same as the placeholder in the prepared statement. The parameter types remain the same.
  • the unit field includes: an indication field for marking whether data in the unit field is currently empty, and a parameter field for carrying a set of data.
  • the parameter field is used to carry the specific content of the parameters of each data, the length of each parameter is determined by the corresponding parameter type, and the parameter type is determined by the parameter type field.
  • the indication field is a bitmap bitmap that marks whether the data in the current unit field is empty, and each data corresponds to one bit. For example: the unit field has 8 data, and that byte is 8 bits, and the information of whether these 8 data are empty can be stored. And if it is 9 data, one byte cannot be stored, and two bytes are needed to store.
  • the parameter package further includes: the data to be inserted represented by a constant; said combining the parameter package carried in the prepared statement with the data package carried in the execution command, and combining the Insert each group of data in the data packet into the data table in batches, including: matching each group of data in the data packet carried in the execution command with the position of the placeholder in the parameter packet respectively, and matching The data is inserted into the data table.
  • the parameter package also includes the data to be inserted represented by constants, indicating that the content of the data to be inserted has been determined, and there is no need to perform matching determination.
  • the data table has an auto-increment sequence; the parameter packet carried in the prepared statement is combined with the data packet carried in the execution command, and the data packet in the data packet is Before inserting each group of data into the data table in batches, it also includes: applying to the global transaction controller for an auto-increment sequence; said combining the parameter packets carried in the preprocessing statement with the data packets carried in the execution command, Inserting each group of data in the data packet into the data table in batches includes: when the parameter packet includes a placeholder in the self-increment sequence, according to the placeholder in the non-auto-increment sequence in the parameter packet The location of the bit symbol, insert the data in the non-auto-increment sequence in the various sets of data into the data table in batches, and automatically generate the data inserted into the self-increment sequence of the data table; if the parameter package does not include the data in the In the case of the placeholder of the auto-increment sequence, insert the groups of data into the data table in batches according to the position of the placeholder in
  • the data in the non-auto-increment sequence in each group of data is inserted into the data table in batches, and at the same time
  • the auto-increment sequence in the data table is processed according to the placeholder in the auto-increment sequence and the auto-increment sequence applied to the global transaction controller when the data is inserted.
  • each group of data is directly inserted into the data table in batches according to the location of the placeholders, and after the data is inserted, the auto-increment sequence applied to the global transaction controller
  • the sequence processes the data of the auto-increment sequence in the data table separately, and automatically generates the data of the auto-increment sequence inserted into the data table.
  • the auto-increment sequence in the data table can be processed while data is inserted through the placeholder containing the auto-increment sequence and the auto-increment sequence applied to the global transaction controller.
  • Step 104 delete the created prepared statement.
  • a preprocessing method for a data table proposed by the present application According to the creation command, a preprocessing statement containing a parameter package is created.
  • the parameter package contains a placeholder for representing the data to be inserted.
  • receiving a data package containing multiple sets of data Execute the command to package the parameter package and the data package, and insert multiple sets of data in the data package into the data table in batches, so that multiple sets of data can be processed in batches, without the need to package and insert a set of data, and then For the next group of data grouping and insertion, the preprocessing of the data table can be completed only after multiple operations, which greatly reduces the time of grouping and insertion, and improves the preprocessing efficiency of the data table.
  • the embodiment of the present application relates to a data table preprocessing device, as shown in Figure 2, comprising:
  • the creation module 201 is configured to create a preprocessing statement according to the received creation command, the preprocessing statement carries a parameter package, and the parameter package includes at least one placeholder for representing data to be inserted;
  • the execution module 202 is configured to receive an execution command for executing the prepared statement, wherein the execution command carries a data packet including multiple sets of data; and combines the parameter packet carried in the prepared statement with the execution command The data packets carried in the command are assembled, and each group of data in the data packets is inserted into the data table in batches;
  • the deletion module 203 is configured to delete the created prepared statement.
  • This embodiment of the device is mainly aimed at the description of the preprocessing method of the data table provided by the method embodiment at the level of software implementation, and its implementation also needs to rely on the support of hardware, such as the functions of related modules can be deployed on the processor, so that The processor runs to implement corresponding functions, especially, the data generated by the running can be stored in the memory for subsequent inspection and use.
  • modules involved in this embodiment are logical modules, and a logical unit may be a physical unit, or a part of a physical unit, or may be realized by a combination of multiple physical units.
  • a logical unit may be a physical unit, or a part of a physical unit, or may be realized by a combination of multiple physical units.
  • units that are not closely related to solving the technical problem proposed in the present application are not introduced in this embodiment, but this does not mean that there are no other units in this embodiment.
  • This embodiment is an apparatus embodiment corresponding to the embodiment of the data table preprocessing method, and this embodiment can be implemented in cooperation with the foregoing embodiments.
  • the relevant technical details mentioned in the foregoing embodiments are still valid in this embodiment, and will not be repeated here in order to reduce repetition.
  • the relevant technical details mentioned in this embodiment can also be applied to the above method embodiments.
  • the embodiment of the present application relates to an electronic device, as shown in FIG. 3 , including: at least one processor 301; Instructions executed by the at least one processor 301, the instructions are executed by the at least one processor 301, so that the at least one processor 301 can execute the data table preprocessing method in the above-mentioned embodiment.
  • the memory and the processor are connected by a bus
  • the bus may include any number of interconnected buses and bridges, and the bus connects one or more processors and various circuits of the memory together.
  • the bus may also connect together various other circuits such as peripherals, voltage regulators, and power management circuits, all of which are well known in the art and therefore will not be further described herein.
  • the bus interface provides an interface between the bus and the transceivers.
  • a transceiver may be a single element or multiple elements, such as multiple receivers and transmitters, providing means for communicating with various other devices over a transmission medium.
  • the data processed by the processor is transmitted on the wireless medium through the antenna, further, the antenna also receives the data and transmits the data to the processor.
  • the processor is responsible for managing the bus and general processing, and can also provide various functions, including timing, peripheral interface, voltage regulation, power management, and other control functions. Instead, memory can be used to store data that the processor uses when performing operations.
  • Embodiments of the present application relate to a computer-readable storage medium storing a computer program.
  • the computer program is executed by the processor, the above data table preprocessing method is realized.
  • a storage medium includes several instructions to make a device ( It may be a single-chip microcomputer, a chip, etc.) or a processor (processor) to execute all or part of the steps of the methods described in the various embodiments of the present application.
  • the aforementioned storage media include: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), magnetic disk or optical disc, etc., which can store program codes. .

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • General Engineering & Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Physics & Mathematics (AREA)
  • Databases & Information Systems (AREA)
  • Data Mining & Analysis (AREA)
  • Software Systems (AREA)
  • Health & Medical Sciences (AREA)
  • Bioethics (AREA)
  • General Health & Medical Sciences (AREA)
  • Computer Hardware Design (AREA)
  • Computer Security & Cryptography (AREA)
  • Information Retrieval, Db Structures And Fs Structures Therefor (AREA)

Abstract

本申请提出一种数据表的预处理方法、装置、电子设备及存储介质，涉及数据库领域。数据表的预处理方法，包括：根据接收到的创建命令创建预处理语句，所述预处理语句中携带参数包，所述参数包至少包括一个用于表示待插入数据的占位符；接收用于执行所述预处理语句的执行命令，其中，所述执行命令携带包括多组数据的数据包；将所述预处理语句中携带的参数包与所述执行命令中携带的数据包进行拼包，将所述数据包中的各组数据批量插入数据表；删除所述创建的预处理语句。

Description

数据表的预处理方法、装置、电子设备及存储介质
相关申请
本申请要求于2021年12月29号申请的、申请号为202111643066.9的中国专利申请的优先权。
技术领域
本申请实施例涉及数据库领域，特别涉及一种数据表的预处理方法、装置、电子设备及存储介质。
背景技术
目前数据库中对数据表的预处理流程主要分为三个步骤，分别是创建、执行和删除。创建阶段主要创建预处理语句，执行阶段主要是设置插入数据的具体值，删除阶段是将之前定义的预处理语句删除。但是目前对数据表的预处理流程中执行阶段只能对每一组数据单独插入数据表，对于大量数据的插入，会消耗大量时间，降低了整个预处理流程的效率。
发明内容
本申请实施例的主要目的在于提出一种数据表的预处理方法、装置、电子设备及存储介质，实现对大量数据批量插入数据表，提高预处理效率。
为实现上述目的，本申请实施例提供了一种数据表的预处理方法，包括：根据接收到的创建命令创建预处理语句，所述预处理语句中携带参数包，所述参数包至少包括一个用于表示待插入数据的占位符；接收用于执行所述预处理语句的执行命令；其中，所述执行命令携带包括多组数据的数据包；将所述预处理语句中携带的参数包与所述执行命令中携带的数据包进行拼包，将所述数据包中的各组数据批量插入数据表；删除所述创建的预处理语句。
为实现上述目的，本申请实施例还提出了一种数据表的预处理装置，包括：
创建模块，用于根据接收到的创建命令创建预处理语句，所述预处理语句中携带参数包，所述参数包至少包括一个用于表示待插入数据的占位符；
执行模块，用于接收用于执行所述预处理语句的执行命令，其中，所述执行命令携带包括多组数据的数据包；并将所述预处理语句中携带的参数包与所述执行命令中携带的数据包进行拼包，将所述数据包中的各组数据批量插入数据表；
删除模块，用于删除所述创建的预处理语句。
为实现上述目的，本申请实施例还提出了一种电子设备，包括：至少一个处理器；以及，与所述至少一个处理器通信连接的存储器；其中，所述存储器存储有可被所述至少一个处理器执行的指令，所述指令被所述至少一个处理器执行，以使所述至少一个处理器能够执行以上所述的数据表的预处理方法。
为实现上述目的，本申请实施例还提出了一种计算机可读存储介质，存储有计算机程序，所述计算机程序被处理器执行时实现以上所述的数据表的预处理方法。
本申请提出的一种数据表的预处理方法、装置、电子设备及存储介质，根据创建命令创 建包含参数包的预处理语句，参数包中包含用于表示待插入数据的占位符，根据接收到包含多组数据的数据包的执行命令，将参数包和数据包进行拼包，将数据包中的多组数据批量插入数据表中，如此可以对多组数据批量处理，而无需对一组数据拼包、插入，然后再对下一组数据拼包、插入，多次操作才可完成对数据表的预处理，大大减少了拼包时间和插入时间，提高了数据表的预处理效率。
附图说明
一个或多个实施例通过与之对应的附图中的图片进行示例性说明，这些示例性说明并不构成对实施例的限定。
图1是本申请的实施例提供的数据表的预处理方法的流程图；
图2是本申请的实施例提供的数据表的预处理装置的结构示意图；
图3是本申请的实施方式提供的电子设备的结构示意图。
具体实施方式
为使本申请实施例的目的、技术方案和优点更加清楚，下面将结合附图对本申请的各实施例进行详细的阐述。然而，本领域的普通技术人员可以理解，在本申请各实施例中，为了使读者更好地理解本申请而提出了许多技术细节。但是，即使没有这些技术细节和基于以下各实施例的种种变化和修改，也可以实现本申请所要求保护的技术方案。以下各个实施例的划分是为了描述方便，不应对本申请的具体实现方式构成任何限定，各个实施例在不矛盾的前提下可以相互结合相互引用。
本申请的数据表的预处理方法适用于任何网络中。
本申请的实施例涉及一种数据表的预处理方法，如图1所示，包括：
步骤101，根据接收到的创建命令创建预处理语句，预处理语句中携带参数包，参数包至少包括一个用于表示待插入数据的占位符。
在本实施例中，参数包中包括占位符，占位符表示待插入数据，占位符的数量与待插入数据的数量一致。具体地，比如：数据库接收客户端发送的创建命令，数据库根据创建命令创建预处理语句。
步骤102，接收用于执行预处理语句的执行命令，其中，执行命令携带包括多组数据的数据包。
步骤103，将预处理语句中携带的参数包与执行命令中携带的数据包进行拼包，将数据包中的各组数据批量插入数据表。
在一实施例中，所述执行命令，包括：用于指示当前待执行的预处理语句的语句标识字段、用于指示所述多组数据的组数的批处理大小字段和多个单元字段，各所述单元字段分别用于承载一组数据；其中，所述单元字段的个数与所述组数相同。
在一实施例中，步骤103，具体包括：根据所述语句标识字段确定当前待执行的预处理语句；获取所述当前待执行的预处理语句的参数包，并根据所述批处理大小字段确定所述当前待执行的预处理语句的执行次数；根据所述参数包中的占位符所在位置和所述执行次数，将多个所述单元字段承载的各组数据分别插入所述数据表中。
也就是说，执行命令中包括：多组数据的数据包、语句标识字段、批处理大小字段、单 元字段。对于一个数据库来说，可能接收到多个不同客户端发送的创建命令和执行命令，因此根据执行命令中的语句标识字段确定当前待执行的预处理语句，即将创建命令创建的待处理语句和执行命令需要执行的待处理语句对应。然后根据执行命令中的批处理大小字段确定对待执行的预处理语句的执行次数，即需要插入数据表中的数据组的组数就是执行次数，然后根据预处理语句中参数包里面的占位符的位置和执行次数，将各组数据分别插入到数据表中。即本申请可以将需要插入数据表中的多组数据批量拼包、批量插入，一次就可以执行完对数据表的预处理，而不用对一组数据拼包、插入，然后再对下一组数据拼包、插入，多次操作才可完成对数据表的预处理。大大节省了数据表的预处理时间。
另外，在将数据插入数据表的过程中，为了保证数据库中数据的原子性需要对数据表进行加锁，插入完成后再对数据表进行解锁。若对数据表的处理时间过长，则会导致数据表加锁时间过长，影响对数据表的其他处理。因此，本申请通过批量操作大大节省了数据表的预处理时间，进而减少了对数据表的加锁时间，保证了数据表的处理效率。
在一实施例中，所述执行命令还包括：用于指示当前协议类型的命令字段；在所述将所述预处理语句中携带的参数包与所述执行命令中携带的数据包进行拼包之前，还包括：根据所述命令字段确定所述执行命令的协议类型为批处理类型。
在接收到执行命令后，从执行命令中获取命令字段，根据命令字段确定本次执行命令的协议类型为批处理类型，则对需要插入的多组数据进行批量操作。
在一实施例中，所述执行命令还包括：用于指示本次执行命令是否传输和使用新的数据类型的标记位；在所述标记位指示传输和使用新的数据类型的情况下，所述执行命令还包括：参数类型字段；所述参数类型字段用于指示承载在所述单元字段中的各数据的参数类型；在所述根据所述参数包中的占位符所在位置和所述执行次数，将多个所述单元字段承载的各组数据分别插入所述数据表中之前，还包括：根据所述参数类型字段指示的各数据的参数类型，解析得到所述单元字段承载的数据。
当通过执行命令中的标记位确定使用了新的数据类型时，就需要进一步从执行命令中获取参数类型字段，根据参数类型字段确定各数据的参数类型，根据各数据的参数类型解析数据。然后将解析后的数据插入数据表中。若执行命令中标记位指示本次没有使用新的数据类型，则各数据的参数类型就为预处理语句中占位符的参数类型，即各数据的参数类型与预处理语句中占位符的参数类型保持一致。
在一实施例中，所述单元字段包括：用于标记当前所述单元字段中数据是否为空的指示字段和用于承载一组数据的参数字段。具体地说，参数字段用于承载各数据的参数的具体内容，每个参数的长度由对应的参数类型确定，参数类型即由参数类型字段确定。指示字段是标记当前单元字段中的数据是否为空的位图bitmap，每个数据对应一位。比如：单元字段有8个数据，那一个字节也就是8位，就可以存下这8个数据是否为空的信息。而如果是9个数据，一个字节就存不下了，就需要两个字节来存。
在一实施例中，所述参数包还包括：以常量表示的待插入数据；所述将所述预处理语句中携带的参数包与所述执行命令中携带的数据包进行拼包，将所述数据包中的各组数据批量插入数据表，包括：将所述执行命令中携带的数据包中的各组数据，分别与所述参数包中的占位符所在的位置进行匹配，将匹配的数据插入所述数据表。具体地说，参数包中还包括以常量表示的待插入数据，以常量表示的待插入数据，说明待插入数据的内容已经确定，无需 再进行匹配确定，只将各组数据分别与占位符所在的位置进行匹配，将匹配的数据插入数据表即可，即对以常量表示的待插入数据不进行替换处理。相比于目前对常量表述的待插入数据也用占位符进行替换处理，进一步减少了拼包时间，提高了数据插入效率。
在一实施例中，所述数据表带有自增序列；在所述将所述预处理语句中携带的参数包与所述执行命令中携带的数据包进行拼包，将所述数据包中的各组数据批量插入数据表之前，还包括：向全局事务控制器申请自增序列；所述将所述预处理语句中携带的参数包与所述执行命令中携带的数据包进行拼包，将所述数据包中的各组数据批量插入数据表，包括：在所述参数包包括处于所述自增序列的占位符的情况下，根据所述参数包中处于非自增序列的占位符所在位置，将所述各组数据中的处于非自增序列的数据批量插入所述数据表，并自动生成插入所述数据表的自增序列的数据；在所述参数包不包括处于所述自增序列的占位符的情况下，根据所述参数包中的占位符所在位置将所述各组数据批量插入所述数据表，并自动生成插入所述数据表的自增序列的数据。
当参数包中存在占位符为自增序列的情况时，则根据非自增序列的占位符的位置，将各组数据中的处于非自增序列的数据批量插入所述数据表，同时对于数据表中带有自增序列的数据，在数据插入的同时根据处于自增序列的占位符和向全局事务控制器申请的自增序列对数据表中的自增序列进行处理。而对于参数包中不包含自增序列的占位符时，则直接根据占位符所在位置将各组数据批量插入所述数据表，数据插入完成后再根据向全局事务控制器申请的自增序列单独处理数据表中自增序列的数据，自动生成插入数据表的自增序列的数据。
当数据表中含有自增序列时，可以通过含有自增序列的占位符和向全局事务控制器申请的自增序列在数据插入的同时对数据表中的自增序列进行处理。
步骤104，删除创建的预处理语句。
本申请提出的一种数据表的预处理方法，根据创建命令创建包含参数包的预处理语句，参数包中包含用于表示待插入数据的占位符，根据接收到包含多组数据的数据包的执行命令，将参数包和数据包进行拼包，将数据包中的多组数据批量插入数据表中，如此可以对多组数据批量处理，而无需对一组数据拼包、插入，然后再对下一组数据拼包、插入，多次操作才可完成对数据表的预处理，大大减少了拼包时间和插入时间，提高了数据表的预处理效率。
此外，应当理解的是，上面各种方法的步骤划分，只是为了描述清楚，实现时可以合并为一个步骤或者对某些步骤进行拆分，分解为多个步骤，只要包括相同的逻辑关系，都在本申请的保护范围内；对流程中添加无关紧要的修改或者引入无关紧要的设计，但不改变其流程的核心设计都在该申请的保护范围内。
本申请的实施例涉及一种数据表的预处理装置，如图2所示，包括：
创建模块201，用于根据接收到的创建命令创建预处理语句，所述预处理语句中携带参数包，所述参数包至少包括一个用于表示待插入数据的占位符；
执行模块202，用于接收用于执行所述预处理语句的执行命令，其中，所述执行命令携带包括多组数据的数据包；并将所述预处理语句中携带的参数包与所述执行命令中携带的数据包进行拼包，将所述数据包中的各组数据批量插入数据表；
删除模块203，用于删除所述创建的预处理语句。
本装置实施例主要是针对方法实施例提供的数据表的预处理方法在软件实现层面上的描述，其实现还需要依托于硬件的支持，如相关模块的功能可以被部署到处理器上，以便处理 器运行实现相应的功能，特别地，运行产生的数据可以被存储到存储器中以便后续检查和使用。
本实施例中所涉及到的各模块均为逻辑模块，一个逻辑单元可以是一个物理单元，也可以是一个物理单元的一部分，还可以以多个物理单元的组合实现。此外，为了突出本申请的创新部分，本实施例中并没有将与解决本申请所提出的技术问题关系不太密切的单元引入，但这并不表明本实施例中不存在其它的单元。
本实施例为与数据表的预处理方法实施例相对应的装置实施例，本实施例可与上述实施例互相配合实施。上述实施例中提到的相关技术细节在本实施例中依然有效，为了减少重复，这里不再赘述。相应地，本实施例中提到的相关技术细节也可应用在上述方法实施例中。
本申请的实施方式涉及一种电子设备，如图3所示，包括：至少一个处理器301；以及，与所述至少一个处理器301通信连接的存储器302；其中，所述存储器302存储有可被所述至少一个处理器301执行的指令，所述指令被所述至少一个处理器301执行，以使所述至少一个处理器301能够执行上述实施方式的数据表的预处理方法。
其中，存储器和处理器采用总线方式连接，总线可以包括任意数量的互联的总线和桥，总线将一个或多个处理器和存储器的各种电路连接在一起。总线还可以将诸如外围设备、稳压器和功率管理电路等之类的各种其他电路连接在一起，这些都是本领域所公知的，因此，本文不再对其进行进一步描述。总线接口在总线和收发机之间提供接口。收发机可以是一个元件，也可以是多个元件，比如多个接收器和发送器，提供用于在传输介质上与各种其他装置通信的单元。经处理器处理的数据通过天线在无线介质上进行传输，进一步，天线还接收数据并将数据传送给处理器。
处理器负责管理总线和通常的处理，还可以提供各种功能，包括定时，外围接口，电压调节、电源管理以及其他控制功能。而存储器可以被用于存储处理器在执行操作时所使用的数据。
本申请的实施方式涉及一种计算机可读存储介质，存储有计算机程序。计算机程序被处理器执行时实现上述数据表的预处理方法。
即，本领域技术人员可以理解，实现上述实施方式方法中的全部或部分步骤是可以通过程序来指令相关的硬件来完成，该程序存储在一个存储介质中，包括若干指令用以使得一个设备(可以是单片机，芯片等)或处理器(processor)执行本申请各个实施方式所述方法的全部或部分步骤。而前述的存储介质包括：U盘、移动硬盘、只读存储器(ROM，Read-Only Memory)、随机存取存储器(RAM，Random Access Memory)、磁碟或者光盘等各种可以存储程序代码的介质。
本领域的普通技术人员可以理解，上述各实施例是实现本申请的具体实施例，而在实际应用中，可以在形式上和细节上对其作各种改变，而不偏离本申请的精神和范围。

Claims (10)

  1. 一种数据表的预处理方法，包括：
    根据接收到的创建命令创建预处理语句，所述预处理语句中携带参数包，所述参数包至少包括一个用于表示待插入数据的占位符；
    接收用于执行所述预处理语句的执行命令，其中，所述执行命令携带包括多组数据的数据包；
    将所述预处理语句中携带的参数包与所述执行命令中携带的数据包进行拼包，将所述数据包中的各组数据批量插入数据表；
    删除所述创建的预处理语句。
  2. 根据权利要求1所述的数据表的预处理方法，其中，所述执行命令，包括：用于指示当前待执行的预处理语句的语句标识字段、用于指示所述多组数据的组数的批处理大小字段和多个单元字段，各所述单元字段分别用于承载一组数据；其中，所述单元字段的个数与所述组数相同；
    所述将所述预处理语句中携带的参数包与所述执行命令中携带的数据包进行拼包，将所述数据包中的各组数据批量插入数据表，包括：
    根据所述语句标识字段确定当前待执行的预处理语句；
    获取所述当前待执行的预处理语句的参数包，并根据所述批处理大小字段确定所述当前待执行的预处理语句的执行次数；
    根据所述参数包中的占位符所在位置和所述执行次数，将多个所述单元字段承载的各组数据分别插入所述数据表中。
  3. 根据权利要求2所述的数据表的预处理方法，其中，所述执行命令还包括：用于指示本次执行命令是否传输和使用新的数据类型的标记位；
    在所述标记位指示传输和使用新的数据类型的情况下，所述执行命令还包括：参数类型字段；所述参数类型字段用于指示承载在所述单元字段中的各数据的参数类型；在所述根据所述参数包中的占位符所在位置和所述执行次数，将多个所述单元字段承载的各组数据分别插入所述数据表中之前，还包括：
    根据所述参数类型字段指示的各数据的参数类型，解析得到所述单元字段承载的数据。
  4. 根据权利要求2所述的数据表的预处理方法，其中，所述执行命令还包括：用于指示当前协议类型的命令字段；
    在所述将所述预处理语句中携带的参数包与所述执行命令中携带的数据包进行拼包之前，还包括：
    根据所述命令字段确定所述执行命令的协议类型为批处理类型。
  5. 根据权利要求2所述的数据表的预处理方法，其中，所述单元字段包括：用于标记当前所述单元字段中数据是否为空的指示字段和用于承载一组数据的参数字段。
  6. 根据权利要求1至5中任一项所述的数据表的预处理方法，其中，所述参数包还包括：以常量表示的待插入数据；
    所述将所述预处理语句中携带的参数包与所述执行命令中携带的数据包进行拼包，将所述数据包中的各组数据批量插入数据表，包括：
    将所述执行命令中携带的数据包中的各组数据，分别与所述参数包中的占位符所在的位置进行匹配，将匹配的数据插入所述数据表。
  7. 根据权利要求1至5中任一项所述的数据表的预处理方法，其中，所述数据表带有自增序列；
    在所述将所述预处理语句中携带的参数包与所述执行命令中携带的数据包进行拼包，将所述数据包中的各组数据批量插入数据表之前，还包括：向全局事务控制器申请自增序列；
    所述将所述预处理语句中携带的参数包与所述执行命令中携带的数据包进行拼包，将所述数据包中的各组数据批量插入数据表，包括：
    在所述参数包包括处于所述自增序列的占位符的情况下，根据所述参数包中处于非自增序列的占位符所在位置，将所述各组数据中的处于非自增序列的数据批量插入所述数据表，并自动生成插入所述数据表的自增序列的数据；
    在所述参数包不包括处于所述自增序列的占位符的情况下，根据所述参数包中的占位符所在位置将所述各组数据批量插入所述数据表，并自动生成插入所述数据表的自增序列的数据。
  8. 一种数据表的预处理装置，包括：
    创建模块，设置为根据接收到的创建命令创建预处理语句，所述预处理语句中携带参数包，所述参数包至少包括一个用于表示待插入数据的占位符；
    执行模块，设置为接收用于执行所述预处理语句的执行命令，其中，所述执行命令携带包括多组数据的数据包；并将所述预处理语句中携带的参数包与所述执行命令中携带的数据包进行拼包，将所述数据包中的各组数据批量插入数据表；
    删除模块，设置为删除所述创建的预处理语句。
  9. 一种电子设备，包括：
    至少一个处理器；以及，
    与所述至少一个处理器通信连接的存储器；其中，
    所述存储器存储有可被所述至少一个处理器执行的指令，所述指令被所述至少一个处理器执行，以使所述至少一个处理器能够执行如权利要求1至7中任一项所述的数据表的预处理方法。
  10. 一种计算机可读存储介质，存储有计算机程序，其中，所述计算机程序被处理器执行时实现权利要求1至7中任一项所述的数据表的预处理方法。
PCT/CN2022/139269 2021-12-29 2022-12-15 数据表的预处理方法、装置、电子设备及存储介质 WO2023125038A1 (zh)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
CN202111643066.9 2021-12-29
CN202111643066.9A CN116414820A (zh) 2021-12-29 2021-12-29 数据表的预处理方法、装置、电子设备及存储介质

Publications (1)

Publication Number Publication Date
WO2023125038A1 true WO2023125038A1 (zh) 2023-07-06

Family

ID=86997657

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2022/139269 WO2023125038A1 (zh) 2021-12-29 2022-12-15 数据表的预处理方法、装置、电子设备及存储介质

Country Status (2)

Country Link
CN (1) CN116414820A (zh)
WO (1) WO2023125038A1 (zh)

Families Citing this family (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN116975032B (zh) * 2023-07-14 2024-04-12 南京领行科技股份有限公司 数据对齐方法、系统和电子设备及存储介质

Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20090216809A1 (en) * 2008-02-26 2009-08-27 International Business Machines Corporation Method for updating databases
CN102271057A (zh) * 2011-08-02 2011-12-07 大唐移动通信设备有限公司 参数更新方法和设备
CN102521409A (zh) * 2011-12-28 2012-06-27 北京人大金仓信息技术股份有限公司 一种基于通信协议的数据库数据批量更新方法
CN102521408A (zh) * 2011-12-27 2012-06-27 北京人大金仓信息技术股份有限公司 一种通过jdbc接口访问平面文件的方法
CN109829327A (zh) * 2018-12-15 2019-05-31 中国平安人寿保险股份有限公司 敏感信息处理方法、装置、电子设备及存储介质

Patent Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20090216809A1 (en) * 2008-02-26 2009-08-27 International Business Machines Corporation Method for updating databases
CN102271057A (zh) * 2011-08-02 2011-12-07 大唐移动通信设备有限公司 参数更新方法和设备
CN102521408A (zh) * 2011-12-27 2012-06-27 北京人大金仓信息技术股份有限公司 一种通过jdbc接口访问平面文件的方法
CN102521409A (zh) * 2011-12-28 2012-06-27 北京人大金仓信息技术股份有限公司 一种基于通信协议的数据库数据批量更新方法
CN109829327A (zh) * 2018-12-15 2019-05-31 中国平安人寿保险股份有限公司 敏感信息处理方法、装置、电子设备及存储介质

Also Published As

Publication number Publication date
CN116414820A (zh) 2023-07-11

Similar Documents

Publication Publication Date Title
US7577900B2 (en) Mechanism for maintaining data format synchronization between different entities
CN109829287A (zh) Api接口权限访问方法、设备、存储介质及装置
CN109388417B (zh) 一种通信协议的更新方法、系统及终端设备
WO2023125038A1 (zh) 数据表的预处理方法、装置、电子设备及存储介质
CN109165266A (zh) 表单生成方法、装置、计算机设备和存储介质
US20090024640A1 (en) Apparatus and method for improving efficiency of content rule checking in a content management system
CN110308944B (zh) 配置文件处理方法、系统、计算机设备和存储介质
CN111177113B (zh) 数据迁移方法、装置、计算机设备和存储介质
CN107193917A (zh) 基于iid的全站系统配置文件无损数据更新方法
CN106998260A (zh) 一种基于以太网链路的fpga设备升级方法及系统
CN108304251B (zh) 线程同步方法及服务器
EP2563061B1 (en) Data cut-over method and apparatus
CN112163025A (zh) 数据库数据导出方法、装置、计算机设备及存储介质
CN116502273A (zh) 基于数据血缘的动态数据脱敏方法、装置和设备
CN106293862B (zh) 一种可扩展标记语言xml数据的解析方法和装置
CN116560629A (zh) 一种数据传输对象的序列化方法、装置、介质和设备
US11941032B2 (en) Key name generation method and apparatus and non-transitory computer-readable storage medium
US20110055279A1 (en) Application server, object management method, and object management program
CN112306466A (zh) Aar包生成方法、电子设备及存储介质
CN110647546A (zh) 第三方规则引擎生成方法及装置
WO2023097903A1 (zh) 报文传输方法、装置、电子设备和存储介质
CN112039729B (zh) 设备识别方法、服务器、电子设备和存储介质
CN110309133B (zh) 批量数据的处理方法和装置
US20100325151A1 (en) Method and apparatus for searching in a memory-efficient manner for at least one query data element
CN115529302B (zh) 一种应用程序网络通信数据交互流程的建模与仿真方法

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 22914293

Country of ref document: EP

Kind code of ref document: A1