CN111209304A - Data processing method, device and system - Google Patents

Data processing method, device and system Download PDF

Info

Publication number
CN111209304A
CN111209304A CN201911394329.XA CN201911394329A CN111209304A CN 111209304 A CN111209304 A CN 111209304A CN 201911394329 A CN201911394329 A CN 201911394329A CN 111209304 A CN111209304 A CN 111209304A
Authority
CN
China
Prior art keywords
data
null value
bit
content
data information
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Granted
Application number
CN201911394329.XA
Other languages
Chinese (zh)
Other versions
CN111209304B (en
Inventor
何鹏秋
任阳
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Huawei Cloud Computing Technologies Co Ltd
Original Assignee
Huawei Technologies Co Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Huawei Technologies Co Ltd filed Critical Huawei Technologies Co Ltd
Priority to CN201911394329.XA priority Critical patent/CN111209304B/en
Publication of CN111209304A publication Critical patent/CN111209304A/en
Application granted granted Critical
Publication of CN111209304B publication Critical patent/CN111209304B/en
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
    • G06F16/24—Querying
    • G06F16/245—Query processing
    • G06F16/2455—Query execution
    • G06F16/24552—Database cache management
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
    • G06F16/27—Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
    • G06F16/28—Databases characterised by their database models, e.g. relational or object models
    • G06F16/284—Relational databases
    • Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
    • Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
    • Y02D—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE
    • Y02D10/00—Energy efficient computing, e.g. low power processors, power management or thermal management

Landscapes

  • Engineering & Computer Science (AREA)
  • Databases & Information Systems (AREA)
  • Theoretical Computer Science (AREA)
  • Data Mining & Analysis (AREA)
  • Physics & Mathematics (AREA)
  • General Engineering & Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Computational Linguistics (AREA)
  • Computing Systems (AREA)
  • Information Retrieval, Db Structures And Fs Structures Therefor (AREA)

Abstract

The application discloses a data processing method, a device and a system, and belongs to the field of data storage. The method comprises the following steps: receiving a write-in request carrying first data information for writing into a database, wherein the first data information comprises a data bit and a null value marking bit, and the data information stored in the database is the data information without the null value marking bit; when the content of the null value marking bit indicates that the content of the data bit is a null value, updating the content of the data bit in the first data information into an effective null value mark, and storing the first data information after deleting the null value marking bit in the first data information, wherein the effective null value mark is different from the content of the data bit of the data information stored in the database; and when the content of the null value marking bit indicates that the content of the data bit is not a null value and the content of the data bit is different from the valid null value mark, storing the first data information after deleting the null value marking bit in the first data information. The data reading efficiency of the database can be improved.

Description

Data processing method, device and system
Technical Field
The present application relates to the field of data storage, and in particular, to a data processing method, apparatus and system.
Background
In a database system, stored data information generally includes data bits and null value (null) flag bits, where the data bits carry actual data in the data information, and the null value flag bits are used to identify whether the content of the data bits is a null value.
In currently open-source databases, null flag bits occupy one or more units of data, such as a byte (byte) or a bit (bit). For example, the greenplus database uses a boolean variable array to store whether the content of each field data (i.e. the data carried in the data bits) of a tuple is null, and the MySQL database and the impala database indicate whether the content of the data bits is null by a bit.
However, in such a manner of identifying whether the content of the data bit is null, two data (i.e., the data bit and the null flag bit) need to be read each time one piece of data information is acquired, which easily affects the data access efficiency of the database.
Disclosure of Invention
The embodiment of the application provides a data processing method, device and system, which can improve the data access efficiency of a database.
The technical scheme is as follows:
in a first aspect, a data processing method is provided, which can implement conversion from a composite format to a fusion format, and the conversion process includes:
receiving a write-in request carrying first data information for writing to a database, wherein the first data information comprises a data bit and a null value marking bit, the content of the null value marking bit is used for identifying whether the content of the data bit is a null value, and the data information stored in the database is the data information without the null value marking bit; when the content of the null value marking bit indicates that the content of the data bit is a null value, updating the content of the data bit in the first data information to be the valid null value mark, and storing the first data information after deleting the null value marking bit in the first data information, wherein the content of the valid null value mark is different from the content of the data bit of the data information stored in the database; and when the content of the null value marking bit indicates that the content of the data bit is not a null value and the content of the data bit is different from the valid null value mark, storing the first data information after deleting the null value marking bit in the first data information.
In the embodiment of the present application, when the content of the null flag bit indicates that the content of the data bit is null, the content of the data bit in the first data information is updated to the valid null flag, and the first data information is stored after the null flag bit in the first data information is deleted, and when the content of the null flag bit indicates that the content of the data bit is not null, and the content of the data bit is different from the valid null flag, the first data information is stored after the null flag bit in the first data information is deleted. Therefore, the conversion from the composite format to the fusion format is realized, two data (namely a data bit and a null value marking bit) do not need to be read when one data information is acquired each time, and the data access efficiency of the database is improved.
Optionally, the method further comprises:
when the content of the null value marking bit indicates that the content of the data bit is not a null value and the content of the data bit is the same as the valid null value mark, storing the first data information; and adding the null value marking bit outside the data bit in other stored data information of the database.
Optionally, the method further comprises:
when the content of the null value marking bit indicates that the content of the data bit is not a null value and the content of the data bit is the same as the effective null value mark, updating the effective null value mark by adopting an alternative null value mark to obtain a new effective null value mark, wherein the alternative null value mark is different from the content of the data bit of the data information stored in the database;
updating the content of a data bit in first target data information in the database to a new valid null value mark, wherein the content of the data bit is the data information of an original valid null value mark;
deleting the null value mark bit in the first data information and storing the first data information.
The setting of the valid null value flag affects whether the fusion format is valid or not, and further affects the performance of the database. The embodiments of the present application take the following various setting modes as examples:
in a first arrangement, the length of the data bits is a fixed length of x bytes, the length of the valid null flag is y bytes, y is greater than x, and x and y are each integer multiples of 2.
By setting the length of the valid null value flag to be greater than the length of the data bit, it can be ensured that the content (i.e. the carried data) in the data bit is different from the valid null value flag, thereby avoiding the occurrence of the same condition between the valid null value flag and the data bit, and ensuring the validity of the valid null value flag for the data information written at will.
Illustratively, y is n times x, and n is an integer multiple of 2. Since the longer the length of the valid null flag, the more storage space it occupies, y is usually 2 times x, i.e. n is 2. Therefore, the content in the data bit is ensured to be different from the valid null value mark, and simultaneously, the small storage space is ensured to be occupied as much as possible.
In the embodiment of the present application, in the first setting mode, although the memory occupied for storing the first data information is the same as the memory occupied for storing the composite format in the database, since the valid data flag is data that cannot be used in the data bit of the first data information, the validity of the data in the fusion format in the whole data processing process can be ensured all the time. And because when data information is read in the cache, the data information is not required to be read twice, and only the data bits are required to be read, the probability of cache miss and the data access time can be effectively reduced, and the data access efficiency is improved.
In a second arrangement, the database supports a 64-bit operating system, the length of the first data message is variable, that is, the data bits of the first data message carry variable-byte-length data, and the variable-byte-length data usually uses the format of a data address to store the actual storage location of the data in the database. And 0 xffffffffffffffffffffff is a data address that is not currently used, so the valid null flag may be 0 xffffffffffffffffff.
Corresponding to the second setting mode, after deleting the null value flag bit in the first data information, the process of storing the first data information by the management node includes: and after deleting the null value mark bit in the first data information, storing the content in the data bit in the first data information in a data address format.
In the second setting mode, since the valid data flag is data that cannot be used when storing the data bit of the first data information, the data in the fusion format can be guaranteed to be valid all the time in the whole data processing process.
Optionally, the embodiment of the present application may also implement conversion from the fusion format to the composite format, where the conversion process is equivalent to the inverse process of the conversion from the composite format to the fusion format. The conversion process further comprises:
receiving a read request for reading second data information from the database;
when the stored data information in the database is the data information without the null value marking bit, comparing the content of the data bit of the second data information stored in the database with the effective null value mark;
when the content of the data bit is the same as the valid null value flag, outputting the second data information after adding the null value marking bit outside the data bit of the second data information, wherein the content of the added null value marking bit indicates that the content of the data bit of the second data information is a null value;
when the content of the data bit is different from the valid null value flag, the second data information is output after the null value marking bit is added outside the data bit of the second data information, and the content of the added null value marking bit indicates that the content of the data bit of the second data information is not a null value.
In a second aspect, the present application provides a data processing apparatus, which may include at least one module, where the at least one module may be configured to implement the data processing method provided in the first aspect or various possible implementations of the first aspect.
In a third aspect, the present application provides a computing device comprising a processor and a memory. The memory stores computer instructions; the processor executes the computer instructions stored by the memory to cause the computing device to perform the methods provided by the first aspect or the various possible implementations of the first aspect, to cause the computing device to deploy the data processing apparatus provided by the second aspect or the various possible implementations of the second aspect.
In a fourth aspect, the present application provides a computer-readable storage medium having stored therein computer instructions that instruct a computing device to execute the method provided by the above first aspect or various possible implementations of the first aspect, or instruct the computing device to deploy the data processing apparatus provided by the above second aspect or various possible implementations of the second aspect.
In a fifth aspect, the present application provides a computer program product comprising computer instructions stored in a computer readable storage medium. A processor of the computing device may read the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions to cause the computing device to perform the method provided by the above-mentioned first aspect or various possible implementations of the first aspect, to cause the computing device to deploy the data processing apparatus provided by the above-mentioned second aspect or various possible implementations of the second aspect.
In a sixth aspect, there is provided a data processing system comprising: a management node and a data node, the management node comprising the second aspect or various possible implementations of the second aspect of the data processing apparatus or the computing device of the third aspect.
In a seventh aspect, a chip is provided, which may include programmable logic circuits and/or program instructions, and when the chip is run, is configured to implement the data processing method according to any one of the first aspect.
The technical scheme provided by the embodiment of the application has the following beneficial effects:
in the embodiment of the present application, when the content of the null flag bit indicates that the content of the data bit is null, the content of the data bit in the first data information is updated to the valid null flag, and the first data information is stored after the null flag bit in the first data information is deleted, and when the content of the null flag bit indicates that the content of the data bit is not null, and the content of the data bit is different from the valid null flag, the first data information is stored after the null flag bit in the first data information is deleted. Therefore, the conversion from the composite format to the fusion format is realized, two data (namely a data bit and a null value marking bit) do not need to be read when one data information is acquired each time, and the data access efficiency of the database is improved.
Conventional database systems only support one data format, and accordingly, both written data and read data must be data in the format supported by the database system. In the embodiment of the application, the management node can realize the mutual conversion of the composite format and the fusion format, so that the database system is compatible with the two data formats. Therefore, data among the databases supporting different data formats can be mutually transmitted, and the flexibility of data processing in the databases is improved. In addition, in the embodiment of the present application, the data format currently supported by the database in the database system may be set according to a specific scenario (such as a use of a corresponding application program), for example, set by a management node or manually set, so as to further improve flexibility of data processing in the database.
Drawings
Fig. 1 is a schematic diagram of a database system according to a data processing method provided in an embodiment of the present application;
fig. 2 is a schematic diagram of a database system according to another data processing method provided in the embodiment of the present application;
fig. 3 is a schematic flowchart of a data processing method according to an embodiment of the present application;
FIG. 4 is a schematic diagram of valid null flags corresponding to data bits of different lengths according to an embodiment of the present disclosure;
FIG. 5 is a schematic flow chart diagram of another data processing method provided in the embodiments of the present application;
fig. 6 is a schematic structural diagram of a data processing apparatus according to an embodiment of the present application;
FIG. 7 is a schematic structural diagram of another data processing apparatus according to an embodiment of the present application;
FIG. 8 is a schematic structural diagram of another data processing apparatus provided in an embodiment of the present application;
fig. 9 is a schematic structural diagram of another data processing apparatus according to an embodiment of the present application;
fig. 10 is a possible basic hardware architecture of a computer device, which is schematically provided by the embodiment of the present application.
Detailed Description
To make the objects, technical solutions and advantages of the present application more clear, embodiments of the present application will be described in further detail below with reference to the accompanying drawings.
For the convenience of the reader, the following explains the terms referred to in the embodiments of the present application.
Cache miss: cache (Cache) is a concept relative to an internal memory in a database system, and is generally subdivided into three layers, including an L1 Cache, an L2 Cache and an L3 Cache, and the read-write delay of the Cache is sequentially increased, so that the implementation cost is sequentially reduced. When data access is performed on a database, if the accessed data is stored in a cache, the data is called a "Hit" (Hit), and otherwise, the data is called a "Miss" (Miss). The speed of a Central Processing Unit (CPU) accessing the cache is between that of a register and a memory (difference of order of magnitude), and the cache miss is reduced, so that the data access efficiency can be remarkably improved.
Data information: the data information includes at least data bits for carrying data.
Null (null): in database systems, null values are used to represent missing element values. That is, the data bit where the null value is located is null.
The compound format is as follows: in database systems, the format in which data bits and null (null) flag bits are used to describe data information is referred to as a composite format. Where the null flag bits occupy one or more data units, such as one byte (byte) or one bit (bit). For example, the greenplus database, MySQL database, and impala database support only composite format data storage. When the content in the null flag bit indicates true, the value representing the data bit is null, also called unknown (unknown), i.e., data is missing. For data information with the content in the null value marking bit being indicated as true, the specific data (namely the data value) in the data bit does not need to be searched, so that the searching and querying efficiency is improved. For example, the content of the null flag bit is 1, indicating true.
Fusion format: the storage format in which null flags are used in data bits of the data information to describe that the data is null is referred to as a fusion format. In a database that supports the fusion format, for data bits whose data value is not null, the data of the data bit itself is used. For example, only fused-format data storage is supported in an On-Line Analytical Processing (OLAP) system.
Null value flag: the contents of the data bits are described as null using unusual or even non-existent special data, which is called null flag.
Relational database (Relational database): a relational Database is one of Distributed Databases (DDB), and a relational Database refers to a Database that uses a relational model to organize data and stores data in rows and columns, where a row of data is a minimum unit of data reading and writing, and is also called a record. In a relational database, a series of rows and columns are called data tables, and a data table can be regarded as a two-dimensional table. The relational model can be simply understood as a two-dimensional table model. In the relational database, data are stored in a structured manner, each field of each data table is defined according to a preset rule (that is, the structure of the table is predefined), and then the data are stored according to the structure of the data table. The data in each field in the data table is also referred to as field data. In the relational database, the data information is field data.
Referring to fig. 1, fig. 1 is a schematic diagram of an application environment of a Database System, such as a Distributed Database System (DDBS), according to an embodiment of the present application, related to a data processing method. The Database System 100 may be a server or a server cluster composed of a plurality of servers, and the Database System 100 includes a Database Management System (DMS) 110 and a Database 120. In the database system 100, the database management system 110 is mainly used for managing data, and the database 120 is mainly used for storing data. The application (or client) 200 can use the data in the database system 100, i.e. perform data access, such as data reading and writing. For example, in a distributed database system, the application 200 may transparently operate on the database 120 through the database management system 110, and the data in the database 120 is stored in different local databases, managed by one or more database management systems, run on different machines, supported by different operating systems, and connected together by different communication networks.
As shown in fig. 2, the database system 100 includes: a management node (also called database engine, coordinator)111 and a data node 121. The database management system may be deployed on a management node 111 and the database may be deployed on one or more data nodes (data nodes) 121. When the database system is a distributed database system, the database is deployed over a plurality of data nodes. Each data node may be distributed across one or more storage media, such as hard disks or diskettes.
The management node 111 is configured to manage the corresponding data node 121, and implement operations of the application 200 on the data node 121, such as data adding operations, data deleting operations, data modifying operations, or data querying operations.
In this embodiment, the management node 111 may be a single node, or a designated data node or an elected data node in the one or more data nodes 121, which may be one or more operators, one or more application instances, one or more processes, one or more processing engines, a server, or a server cluster composed of multiple servers. Each data node represents a set minimum processing unit of the database system. For example, each data node may perform processes for one application instance or one database that manages and/or stores data. The database system may be deployed in a server or a server cluster composed of a plurality of servers, and one database system may be referred to as one data storage. It is noted that, in some application scenarios, the database system may also include the application 200.
It should be noted that the data processing method provided in the embodiment of the present application may also be applied to other data systems as long as the system supports data storage, reading, and analysis, and the embodiment of the present application is only schematically illustrated by taking a database system as an example.
The embodiment of the present application provides a data processing method, which may be applied to a management node, where the management node may be an existing management node in a database system as shown in fig. 2, or may be a newly added management node, or may be a management node of another data system. As shown in fig. 3, it is assumed that when the management node receives a write request, the data information stored in the database is the data information with null flag bits removed. That is, the database supports data storage in the converged format at this time. Therefore, in order to ensure that the received data in the composite format is stored in the database supporting the data storage in the converged format, the management node needs to perform format conversion of the data, see subsequent steps 201 to 206.
Step 201, a management node receives a write request carrying first data information for writing to a database.
In the embodiment of the present application, the first data information includes data bits and null flag bits. As previously described, the null flag bit is used to identify whether the contents of the data bit are null. Therefore, the first data information is known to be data in a composite format. The write request can be generated by an application program or other nodes, for example, the write request can be generated by one data node in the database system, and is used for writing the data needing to be redistributed into another data node during data processing of the database; for another example, the write request is generated for a management node in another database system, for writing data in the another database system into the database system; for another example, when the management node is one operator in the database, the write request is generated for another operator in the database, for example, the another operator is the next operator of the operator corresponding to the management node, and the first data information in the write request is the calculation result of the another operator; the write request generation device in the embodiment of the present application is not limited, as long as the first data information carried by the write request generation device is data in a composite format.
In actual implementation, after receiving the write request, the management node needs to detect the data format supported by the current database, and then performs the subsequent step when the data format supported by the management node is the fusion format.
Step 202, the management node detects whether the content of the null value flag bit indicates that the content of the data bit is null value. When the content of the null value flag bit indicates that the content of the data bit is null value, execute step 203; when the contents of the null flag bits indicate that the contents of the data bits are not null, step 204 is performed.
Optionally, when the content of the null flag bit indicates true, determining the content of the data bit to be a null; when the contents of the null flag bit indicate false, it is determined that the contents of the data bit are not null. Illustratively, the content of the null flag bit is 1, indicating true; the contents of the null flag bit are 0 indicating false.
Step 203, the management node updates the content of the data bit in the first data information to a valid null flag, and stores the first data information after deleting the null flag bit in the first data information.
As mentioned above, the null flag is a special data that is not common or even present. The valid null flag is different from the contents of the data bits of the data information already stored in the database. The setting of the valid null value flag affects whether the fusion format is valid or not, and further affects the performance of the database. The embodiments of the present application take the following various setting modes as examples:
in a first arrangement, the length of the data bits is a fixed length of x bytes, the length of the valid null flag is y bytes, y is greater than x, and x and y are each integer multiples of 2.
By setting the length of the valid null value flag to be greater than the length of the data bit, it can be ensured that the content (i.e. the carried data) in the data bit is different from the valid null value flag, thereby avoiding the occurrence of the same condition between the valid null value flag and the data bit, and ensuring the validity of the valid null value flag for the data information written at will.
Illustratively, y is n times x, and n is an integer multiple of 2. Since the longer the length of the valid null flag, the more storage space it occupies, y is usually 2 times x, i.e. n is 2. Therefore, the content in the data bit is ensured to be different from the valid null value mark, and simultaneously, the small storage space is ensured to be occupied as much as possible.
Common fixed lengths of the data bits are 1 byte, 2 bytes, 4 bytes and 8 bytes, and the lengths of the valid null flags are 2 bytes, 4 bytes, 8 bytes and 16 bytes, respectively, in a one-to-one correspondence. As shown in fig. 4, fig. 4 schematically shows the valid null flags corresponding to different lengths of data bits. Wherein, when the length of the data bit is 1 byte, the valid null flag may be 0X 81; when the length of the data bit is 2 bytes, the valid null flag may be 0X 8100; when the length of the data bit is 4 bytes, the valid null flag may be 0X 81000000; the valid null flag may be 0X 81000000000000000000 when the data bits are 8 bytes in length. The valid null flag bit is schematically illustrated in fig. 4, and is not limited thereto, as long as the length of the valid null flag bit is twice the length of the data bit.
Corresponding to the first setting mode, after deleting the null value flag bit in the first data information, the process of storing the first data information by the management node includes: and after deleting the null value mark bit in the first data information, adopting y bytes to store the content in the data bit in the first data information.
In the conventional technology, if the data is stored in a composite format, in the cache, data reading needs to be performed twice each time one data information is accessed, that is, a null value mark bit and a data bit are respectively read, so that cache miss probability and data access time are increased, and data access efficiency is further reduced.
In the embodiment of the present application, in the first setting mode, although the memory occupied for storing the first data information is the same as the memory occupied for storing the composite format in the database, since the valid data flag is data that cannot be used in the data bit of the first data information, the validity of the data in the fusion format in the whole data processing process can be ensured all the time. And because when data information is read in the cache, the data information is not required to be read twice, and only the data bits are required to be read, the probability of cache miss and the data access time can be effectively reduced, and the data access efficiency is improved.
In practical implementation, the length of the valid null value flag may also be the same as the length of the data bit, so that the content of the data bit may be directly stored according to the actual length of the data bit, thereby reducing the occupation of the data information on the memory during storage.
In a second arrangement, the database supports a 64-bit operating system, the length of the first data message is variable, that is, the data bits of the first data message carry variable-byte-length data, and the variable-byte-length data usually uses the format of a data address to store the actual storage location of the data in the database. And 0 xffffffffffffffffffffff is a data address that is not currently used, so the valid null flag may be 0 xffffffffffffffffff.
Corresponding to the second setting mode, after deleting the null value flag bit in the first data information, the process of storing the first data information by the management node includes: and after deleting the null value mark bit in the first data information, storing the content in the data bit in the first data information in a data address format.
In the second setting mode, since the valid data flag is data that cannot be used when storing the data bit of the first data information, the data in the fusion format can be guaranteed to be valid all the time in the whole data processing process.
Step 204, the management node detects whether the content of the data bit is the same as the valid null flag. When the content of the data bit is different from the valid null flag, go to step 205; when the contents of the data bits are the same as the valid null flag, step 206 is performed.
Since the detection of the content of the null flag bit in the foregoing step 202 indicates that the content of the data bit is not a null value, it indicates that the data bit carries actual data, but the data has the same possibility as the valid null flag, which affects the reliability of the valid null flag. Therefore, the management node needs to detect whether the content of the data bit is the same as the valid null flag. For example, the management node may directly compare the contents of the data bits with the valid null flag, i.e., may determine whether the two are the same.
Step 205, after deleting the null value flag bit in the first data information, the management node stores the first data information.
When the management node detects that the content of the data bit is different from the valid null value flag, it indicates that the reliability of the valid null value flag is not affected by the actual data carried in the data bit, so that the null value flag bit in the first data information is deleted, the obtained new first data information is data information in a fusion format, and the first data information is stored in the database in a format supported by the database.
Step 206, the management node executes the processing flow for the flag conflict situation.
When the management node detects that the content of the data bit is the same as the valid null value flag, it indicates that the reliability of the valid null value flag is affected by the fact that the data bit carries actual data, and if the first data information is directly stored in the database, data failure of the database in the fusion format may be caused. In the embodiment of the present application, the condition that the content of the data bit is the same as the valid null flag is referred to as a flag collision condition. The management node needs to execute a processing procedure of the flag conflict condition to avoid the influence caused by the flag conflict condition. The processing flow can be realized in various optional ways, and the following optional ways are taken as examples in the embodiment of the present application to explain:
in an alternative, flag collisions are avoided by performing format recovery.
As shown in step 201, when the management node receives the aforementioned write request, the database supports data storage in the converged format. When the content of the null flag bit indicates that the content of the data bit is not null and the content of the data bit is the same as the valid null flag, the management node may store the first data information and other data information already stored in the database in a composite format. Since the first data information itself is data in a composite format, it can be directly stored in the database, and other stored data information (i.e. data information other than the first data information) is data in a fusion format, so that format conversion is required. Null value marking bits can be added outside data bits in other stored data information of the database, so that the other data information is converted into data in a composite format. It should be noted that the actions of storing the first data information and adding the null flag bit to the data bit in the other stored data information of the database may be executed simultaneously or sequentially, and the execution order of the first data information and the null flag bit is not limited in this embodiment of the present application.
In a second alternative, a valid null flag update is performed to avoid flag collisions.
Optionally, the management node may be preset with a plurality of null value flags, and each null value flag may be manually set or may be determined by the management node based on a preset rule. Among the null value flags, the null value flag currently in effect is a valid null value flag, which is different from the content of the data bits of the data information stored in the database. The processing flow for flag conflict condition includes:
step A1, when the content of the null value marking bit indicates that the content of the data bit is not null value and the content of the data bit is the same as the valid null value flag, updating the valid null value flag with the alternative null value flag to obtain a new valid null value flag, where the alternative null value flag is different from the content of the data bit of the data information stored in the database.
Alternatively, the management node may select, as the candidate null value flag, a null value flag that is different from the current valid null value flag and different from the content of the data bit of the data information already stored in the database, among the plurality of null value flags.
In a first alternative example, the management node may store a plurality of null value flags (the null value flags are different from each other) in a certain order, for example, store the null value flags in an array manner. When selecting the candidate null value mark, the management node may sequentially obtain null value marks from a null value mark next to the current valid null value mark, compare the obtained null value mark with the contents of the data bits of each piece of data information stored in the database after obtaining one null value mark each time, determine the null value mark as the candidate null value mark, and stop obtaining the null value mark until the obtained null value mark is different from the contents of the data bits of all pieces of data information stored in the database.
As shown in table 1, table 1 is an exemplary null value flag table, which records a plurality of null value flags stored by the management node, and table 1 assumes that the management node stores 3 null value flags, which are 0X8100, 0X8101, and 0X8102, respectively. Assuming that the content of a data bit of the first data information is 0X8100 and the current valid null value flag is 0X8100, the content of the data bit is the same as the valid null value flag, obtaining the next null value flag 0X8101 of the valid null value flag 0X8100, comparing the null value flag with the content of the data bit of each data information stored in the database, and determining the data bit of the first data information as an alternative null value flag if the content of the data bit of the null value flag 0X8101 is different from the content of the data bit of all the stored data information in the database; if the content of the null value mark 0X8101 is the same as the content of the data bit of any stored data information in the database, obtaining the next null value mark 0X8102 of the null value mark 0X8101, comparing the null value mark with the content of the data bit of each data information stored in the database again, and if the content of the data bit of the null value mark 0X8102 is different from the content of the data bit of all stored data information in the database, determining the data bit as an alternative null value mark.
TABLE 1
Null sign Whether or not it is effective
0X8100 Is effective
0X8101
0X8102
In a second alternative example, after receiving one write request each time, when the content of the null value flag bit indicates that the content of the data bit is not a null value, the management node may compare w null value flags except for the current valid null value flag in the plurality of null value flags with the content of the data bit, respectively, and determine the null value flag identical to the content of the data bit as an invalid null value flag, when detecting that the content of the data bit is different from the valid null value flag; then, when the candidate null value flag needs to be selected, the management node may determine the candidate null value flag from the null value flags other than the invalid null value flag in the w null value flags, for example, randomly select one null value flag other than the invalid null value flag as the candidate null value flag. w is a positive integer, w is less than or equal to r-1, and r is the total number of the null value marks.
Alternatively, the management node may store a plurality of null value flags in a certain order, for example, store the null value flags in an array manner. The w null flags may be adjacent w null flags after the current valid null flag. For example, in this case, the management node may select, from among the w null flags other than the invalid null flag, a null flag closest to the current valid null flag as the candidate null flag.
Still taking the foregoing table 1 as an example, assuming that w is equal to 1 and the null flags are stored in a certain order, after each write request is received, when the content of the null flag bit indicates that the content of the data bit is not a null value, the management node may divide the current valid null flag from the null flags when detecting that the content of the data bit is different from the valid null flag: 1 null value mark 0X8101 except 0X8100 is compared with the content of the data bit respectively, and the null value mark which is the same as the content of the data bit is determined as an invalid null value mark; the management node may determine the alternative null value flag from null value flags other than the invalid null value flag among the 1 null value flags when the alternative null value flag needs to be selected. For example, at the current time, if the content of the null value flag 0X8101 is different from that of the data bit in the previously received write request, that is, if there is no invalid null value flag, the null value flag 0X8101 is determined as the candidate null value flag.
It should be noted that, after the management node updates the valid null value flag with the alternative null value flag to obtain a new valid null value flag, the management node may identify the original valid null value flag as invalid and identify the new valid null value flag as valid in an array storing a plurality of null value flags. Null flags identified as invalid may be disabled in subsequent procedures or deleted directly.
Alternatively, if the candidate null value flags cannot be obtained in step a1, for example, the content of the data bits of the data information stored in the database is the same for all the null value flags except the valid null value flag, or for example, all the null value flags except the valid null value flag are invalid null value flags. In an alternative example, the management node may issue an alarm message to prompt the user to reset the null flag, and in another alternative, the management node may perform the first alternative, and perform format recovery to avoid flag collision.
Step a2, updating the content of the data bit in the first target data information in the database to a new valid null value flag, where the content of the data bit is the data information of the original valid null value flag.
Since the valid null value flag is updated in step a1, and the content of the data bit in the database is the original valid null value flag and needs to be updated, the management node screens out the data information whose content is the original valid null value flag from the database, which is referred to as the first target data information in the embodiment of the present application, and then updates the content of the data bit in the first target data information in the database to the new valid null value flag.
Still taking table 1 as an example, assuming that the content of the data bit of the first target data information is 0X8100 and the new valid null flag is 0X8101, the content of the data bit is updated to 0X 8101.
And step A3, deleting the null value flag bit in the first data information, and storing the first data information.
Since the valid null flag is updated and the null flags in the management node are different from each other, the content of the data bit of the first data information is different from the valid null flag. In addition, since the first data information is data in a composite format, the first data information needs to be converted into data in a fusion format and then stored. The management node deletes the null value flag bit in the first data information, and the process of storing the first data information may refer to step 203, which is not described in this embodiment.
It should be noted that, in the foregoing step 203, the foregoing two setting manners can effectively reduce or even avoid the failure of the fusion format. In some more reliable database systems, the foregoing steps 204 and 206 may not be executed, and the step 205 is executed directly, that is, the content of the default data bit is different from the valid null flag, so that the detection process may be simplified, and the data access efficiency may be improved.
However, the data in some database systems may change continuously according to different application scenarios, and there is a probability that the content of the data bit is the same as the valid null flag, or alternatively, in the foregoing step 203, the valid data flag is not set by the foregoing two setting manners, and therefore there is a probability that the content of the data bit is the same as the probability of the valid null flag, and thus, the foregoing steps 204 and 206 still need to be performed.
To sum up, in the embodiment of the present application, when the content of the null value flag bit indicates that the content of the data bit is the null value, will in the first data information the content of the data bit is updated to be the valid null value flag, and is deleting after the null value flag bit in the first data information, store the first data information, work as the content of the null value flag bit indicates that the content of the data bit is not the null value, just the content of the data bit with the valid null value flag is different, deleting after the null value flag bit in the first data information, store the first data information. Therefore, the conversion from the composite format to the fusion format is realized, two data (namely a data bit and a null value marking bit) do not need to be read when one data information is acquired each time, and the data access efficiency of the database is improved.
In the foregoing steps 201 to 206, it is assumed that the database supports data storage in the fusion format, and in actual implementation of the embodiment of the present application, if the database supports data storage in the composite format, the received first data information may be directly stored after step 201.
In the embodiment of the present application, the conversion from the composite format to the fusion format is realized through the foregoing steps 201 to 206. Further, the embodiment of the present application may also implement conversion from the fusion format to the composite format, where the conversion process is equivalent to the inverse process of the conversion from the composite format to the fusion format. As shown in fig. 5, the conversion process may refer to steps 207 to 210 below.
Step 207, the management node receives a read request for reading the second data information from the database.
In the embodiment of the present application, it is assumed that the stored data information in the database is the data information from which the null value flag bit is removed, and at this time, the database supports data storage in the fusion format.
Step 208, the management node compares the content of the data bit of the second data information stored in the database with the valid null flag.
Step 209, when the content of the data bit is the same as the valid null flag, the management node outputs the second data information after adding a null flag bit outside the data bit of the second data information, and the content of the added null flag bit indicates that the content of the data bit of the second data information is a null value.
When the content of the data bit is the same as the valid null value mark, the content of the data bit is actually null value, the management node outputs the second data information after adding a null value marking bit outside the data bit of the second data information, and the content of the added null value marking bit indicates that the content of the data bit of the second data information is null value, so that the data of the second data information in the fusion format can be converted into the data in the composite format. Referring to step 202 above, assume that when the content of the null flag bit indicates true, the content of the data bit is determined to be null; when the contents of the null flag bit indicate false, it is determined that the contents of the data bit are not null. The content of the added data bit indicates true.
Illustratively, the content of the null flag bit is 1, indicating true; the contents of the null flag bit are 0 indicating false. The content of the added data bit is 1.
And step 210, when the content of the data bit is different from the valid null flag, the management node outputs the second data information after adding a null flag bit outside the data bit of the second data information, wherein the content of the added null flag bit indicates that the content of the data bit of the second data information is not a null value.
When the content of the data bit is the same as the valid null value mark, the content of the data bit is not actually null value, the content of the data bit is the content of the management node, the management node outputs the second data information after adding a null value marking bit outside the data bit of the second data information, and the content of the added null value marking bit indicates that the content of the data bit of the second data information is not null value, so that the data of the second data information in the fusion format can be converted into the data in the composite format. The content of the added data bit is indicated as false, as in step 209. Illustratively, the content of the added data bit is 0.
It should be noted that, referring to step 203, since the management node stores the data bits of the data information in different manners of setting the two different valid null flags of step 203 when storing the data information, the data reading manner is also different according to the two different manners of setting the valid null flags when reading the data information.
When the valid null flag is set in the first setting manner in step 203, in steps 209 and 210, after adding a null flag bit to the data bit of the second data information, the process of outputting the second data information includes:
and extracting the content of x bytes from the content of the data bits of the stored y bytes of second data information to be used as the data bits, adding null value marking bits outside the data bits to obtain the second data information, and outputting the second data information.
When the valid null flag is set in the second setting manner in step 203, in steps 209 and 210, after adding a null flag bit to the data bit of the second data information, the process of outputting the second data information includes:
and converting the content of the data bit of the second data information in the data address format into the original content when the data bit is written, adding a null value marking bit outside the data bit subjected to the data format conversion to obtain the second data information, and outputting the second data information.
In the foregoing steps 207 to 210, it is assumed that the database supports data storage in the fusion format, and in practical implementation of the embodiment of the present application, if the database supports data storage in the composite format, the stored second data information may be directly read after step 207.
In practical implementation of the embodiment of the present application, the management node may include a plurality of processing engines, for example, a composite format data processing engine and a fusion format data processing engine, where the composite format data processing engine is configured to process composite format data, and the fusion format data processing engine is configured to process fusion format data. By arranging two processing engines supporting different data formats, the data formats supported by the database system can be quickly switched, and the data processing efficiency is improved.
It should be noted that, the order of the steps of the data processing method provided in the embodiment of the present application may be appropriately adjusted, and the steps may also be increased or decreased according to the circumstances, and any method that can be easily conceived by a person skilled in the art within the technical scope disclosed in the present application should be included in the protection scope of the present application, and therefore, the details are not described again.
To sum up, in the embodiment of the present application, when the content of the null value flag bit indicates that the content of the data bit is the null value, will in the first data information the content of the data bit is updated to be the valid null value flag, and is deleting after the null value flag bit in the first data information, store the first data information, work as the content of the null value flag bit indicates that the content of the data bit is not the null value, just the content of the data bit with the valid null value flag is different, deleting after the null value flag bit in the first data information, store the first data information. Therefore, the conversion from the composite format to the fusion format is realized, two data (namely a data bit and a null value marking bit) do not need to be read when one data information is acquired each time, and the data access efficiency of the database is improved.
Conventional database systems only support one data format, and accordingly, both written data and read data must be data in the format supported by the database system. In the embodiment of the application, the management node can realize the mutual conversion of the composite format and the fusion format, so that the database system is compatible with the two data formats. Therefore, data among the databases supporting different data formats can be mutually transmitted, and the flexibility of data processing in the databases is improved. In addition, in the embodiment of the present application, the data format currently supported by the database in the database system may be set according to a specific scenario (such as a use of a corresponding application program), for example, set by a management node or manually set, so as to further improve flexibility of data processing in the database.
Furthermore, when the management node is a module (such as one or more operators, one or more application instances, or one or more processes) in the database system, data between modules supporting different data formats can also be mutually transmitted, so that flexibility of data processing between modules in the database system is improved.
Fig. 6 is a schematic structural diagram of a data processing apparatus 300 according to an embodiment of the present application, where as shown in fig. 6, the apparatus 300 includes:
a first receiving module 301, configured to receive a write request carrying first data information for writing to a database, where the first data information includes a data bit and a null value flag bit, the content of the null value flag bit is used to identify whether the content of the data bit is a null value, and the data information stored in the database is data information from which the null value flag bit is removed;
a first processing module 302, configured to update the content of the data bit in the first data information to the valid null flag when the content of the null flag bit indicates that the content of the data bit is a null value, and store the first data information after deleting the null flag bit in the first data information, where the valid null flag is different from the content of the data bit of the data information already stored in the database;
a second processing module 303, configured to, when the content of the null flag bit indicates that the content of the data bit is not a null value and the content of the data bit is different from the valid null flag, delete the null flag bit in the first data information, and then store the first data information.
Fig. 7 is a schematic structural diagram of another data processing apparatus 300 according to an embodiment of the present application, and as shown in fig. 7, the apparatus 300 may further include:
a third processing module 304, configured to store the first data information when the content of the null flag bit indicates that the content of the data bit is not a null value, the content of the data bit is the same as the valid null flag, and the content of the null flag bit indicates that the content of the data bit is not a null value;
a flag adding module 305, configured to add the null flag bit in addition to the data bit in the other stored data information of the database.
Fig. 8 is a schematic structural diagram of another data processing apparatus 300 according to an embodiment of the present application, where as shown in fig. 8, the apparatus 300 further includes:
a flag updating module 306, configured to update the valid null value flag with an alternative null value flag to obtain a new valid null value flag when the content of the null value flag bit indicates that the content of the data bit is not a null value and the content of the data bit is the same as the valid null value flag, where the alternative null value flag is different from the content of the data bit of the data information already stored in the database;
the flag updating module 306 is further configured to update the content of the data bit in the first target data information in the database to a new valid null value flag, where the content of the data bit in the first target data information is the data information of the original valid null value flag;
a new flag deleting module 307, configured to delete the null value flag bit in the first data information, and store the first data information.
Optionally, the length of the data bit is x bytes, the length of the valid null flag is y bytes, y is greater than x, and x and y are both integer multiples of 2;
the flag deletion new module 307 is configured to:
and after deleting the null value mark bit in the first data information, adopting y bytes to store the content in the data bit in the first data information.
Optionally, y is 2 times the x.
Optionally, the database supports a 64-bit operating system, the length of the first data information is variable, and the valid null flag is 0 xffffffffffffff.
Fig. 9 is a schematic structural diagram of another data processing apparatus 300 according to an embodiment of the present application, where as shown in fig. 9, the apparatus 300 further includes:
a second receiving module 308, configured to receive a read request for reading second data information from the database;
a comparing module 309, configured to compare, when the stored data information in the database is the data information from which the null flag bit is removed, the content of the data bit of the second data information stored in the database with the valid null flag;
an output module 310, configured to output the second data information after adding the null flag bit outside the data bit of the second data information when the content of the data bit is the same as the valid null flag, where the content of the added null flag bit indicates that the content of the data bit of the second data information is a null value;
the output module 310 is further configured to output the second data information after adding the null flag bit outside the data bit of the second data information when the content of the data bit is different from the valid null flag, where the content of the added null flag bit indicates that the content of the data bit of the second data information is not a null value.
To sum up, in the embodiment of the present application, when the content of the null value flag bit indicates that the content of the data bit is the null value, will in the first data information the content of the data bit is updated to be the valid null value flag, and is deleting after the null value flag bit in the first data information, store the first data information, work as the content of the null value flag bit indicates that the content of the data bit is not the null value, just the content of the data bit with the valid null value flag is different, deleting after the null value flag bit in the first data information, store the first data information. Therefore, the conversion from the composite format to the fusion format is realized, two data (namely a data bit and a null value marking bit) do not need to be read when one data information is acquired each time, and the data access efficiency of the database is improved.
Optionally, fig. 10 is a possible basic hardware architecture of a computer device, which is schematically provided by the embodiment of the present application.
Referring to fig. 10, the computer device 400 includes a processor 401, a memory 402, a communication interface 403, and a bus 404.
In the computer device 400, the number of the processors 401 may be one or more, and fig. 10 only illustrates one of the processors 401. Alternatively, the processor 401 may be a Central Processing Unit (CPU). If the computer device 400 has multiple processors 401, the types of the multiple processors 401 may be different, or may be the same. Optionally, the plurality of processors 401 of the computer device 400 may also be integrated as a multi-core processor.
Memory 402 stores computer instructions and data; the memory 402 may store computer instructions and data necessary to implement the data processing methods provided herein, e.g., the memory 402 stores instructions for implementing the steps of the data processing methods. The memory 402 may be any one or any combination of the following storage media: nonvolatile memory (e.g., Read Only Memory (ROM), Solid State Disk (SSD), hard disk (HDD), optical disk), volatile memory.
The communication interface 403 may be any one or any combination of the following devices: a network interface (e.g., an ethernet interface), a wireless network card, etc. having a network access function.
The communication interface 403 is used for the computer apparatus 400 to perform data communication with other computer apparatuses or terminals.
The bus 404 may connect the processor 401 with the memory 402 and the communication interface 403. Thus, via bus 404, processor 401 may access memory 402 and may also interact with other computer devices or terminals via communication interface 403.
In the present application, the computer device 400 executes computer instructions in the memory 402, so that the computer device 400 implements the data processing method provided in the present application, or so that the computer device 400 deploys a data processing apparatus.
In an exemplary embodiment, a non-transitory computer-readable storage medium including instructions, for example, a memory including instructions, which are executable by a processor of a server to perform the emoticon recommendation method according to the embodiments of the present application is also provided. For example, the non-transitory computer readable storage medium may be a ROM, a Random Access Memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, an optical data storage device, and the like.
An embodiment of the present application provides a data processing system, including: a management node and a data node, the management node comprising the data processing apparatus 300 of any of the preceding claims or the computer device 400 of any of the preceding claims.
In the above embodiments, the implementation may be wholly or partially realized by software, hardware, firmware, or any combination thereof. When implemented in software, may be implemented in whole or in part in the form of a computer program product comprising one or more computer instructions. When loaded and executed on a computer, cause the processes or functions described in accordance with the embodiments of the application to occur, in whole or in part. The computer may be a general purpose computer, a network of computers, or other programmable device. The computer instructions may be stored in a computer readable storage medium or transmitted from one computer readable storage medium to another, for example, the computer instructions may be transmitted from one website, computer, server, or data center to another website, computer, server, or data center by wire (e.g., coaxial cable, fiber optic, digital subscriber line) or wirelessly (e.g., infrared, wireless, microwave, etc.). The computer-readable storage medium can be any available medium that can be accessed by a computer or a data storage device including one or more available media integrated servers, data centers, and the like. The usable medium may be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium, or a semiconductor medium (e.g., solid state disk), among others.
It can be clearly understood by those skilled in the art that, for convenience and brevity of description, the specific working processes of the system, the apparatus and the module described above may refer to the corresponding processes in the foregoing method embodiments, and are not described herein again.
The above description is only exemplary of the present application and should not be taken as limiting, as any modification, equivalent replacement, or improvement made within the spirit and principle of the present application should be included in the protection scope of the present application.
In this application, the terms "first" and "second" are used for descriptive purposes only and are not to be construed as indicating or implying relative importance. The term "at least one" means one or more, and the term "plurality" means two or more, unless expressly defined otherwise. A refers to B and refers to the simple variation where A is the same as B or A is B.
It will be understood by those skilled in the art that all or part of the steps for implementing the above embodiments may be implemented by hardware, or may be implemented by a program instructing relevant hardware, where the program may be stored in a computer-readable storage medium, and the above-mentioned storage medium may be a read-only memory, a magnetic disk or an optical disk, etc.
The above description is only exemplary of the present application and should not be taken as limiting, as any modification, equivalent replacement, or improvement made within the spirit and principle of the present application should be included in the protection scope of the present application.

Claims (17)

1. A method of data processing, the method comprising:
receiving a write-in request carrying first data information for writing to a database, wherein the first data information comprises a data bit and a null value marking bit, the content of the null value marking bit is used for identifying whether the content of the data bit is a null value, and the data information stored in the database is the data information without the null value marking bit;
when the content of the null value marking bit indicates that the content of the data bit is a null value, updating the content of the data bit in the first data information to be the valid null value mark, and storing the first data information after deleting the null value marking bit in the first data information, wherein the content of the valid null value mark is different from the content of the data bit of the data information stored in the database;
and when the content of the null value marking bit indicates that the content of the data bit is not a null value and the content of the data bit is different from the valid null value mark, storing the first data information after deleting the null value marking bit in the first data information.
2. The method of claim 1, further comprising:
when the content of the null value marking bit indicates that the content of the data bit is not a null value and the content of the data bit is the same as the valid null value mark, storing the first data information;
and adding the null value marking bit outside the data bit in other stored data information of the database.
3. The method of claim 1, further comprising:
when the content of the null value marking bit indicates that the content of the data bit is not a null value and the content of the data bit is the same as the effective null value mark, updating the effective null value mark by adopting an alternative null value mark to obtain a new effective null value mark, wherein the alternative null value mark is different from the content of the data bit of the data information stored in the database;
updating the content of a data bit in first target data information in the database to a new valid null value mark, wherein the content of the data bit is the data information of an original valid null value mark;
deleting the null value mark bit in the first data information and storing the first data information.
4. The method according to any one of claims 1 to 3, wherein the data bits have a length of x bytes, the valid null flag has a length of y bytes, y is greater than x, and x and y are each integer multiples of 2;
after deleting the null value flag bit in the first data information, the storing the first data information includes: and after deleting the null value mark bit in the first data information, adopting y bytes to store the content in the data bit in the first data information.
5. The method of claim 4, wherein y is 2 times the x.
6. The method of any of claims 1 to 3, wherein the database supports a 64-bit operating system, the length of the first data message is variable, and the valid null flag is 0 XFFFFFFFFFFFFFFFFFFFFF.
7. The method of any of claims 1 to 6, further comprising:
receiving a read request for reading second data information from the database;
when the stored data information in the database is the data information without the null value marking bit, comparing the content of the data bit of the second data information stored in the database with the effective null value mark;
when the content of the data bit is the same as the valid null value flag, outputting the second data information after adding the null value marking bit outside the data bit of the second data information, wherein the content of the added null value marking bit indicates that the content of the data bit of the second data information is a null value;
when the content of the data bit is different from the valid null value flag, the second data information is output after the null value marking bit is added outside the data bit of the second data information, and the content of the added null value marking bit indicates that the content of the data bit of the second data information is not a null value.
8. A data processing apparatus, characterized in that the apparatus comprises:
receiving a write-in request carrying first data information for writing to a database, wherein the first data information comprises a data bit and a null value marking bit, the content of the null value marking bit is used for identifying whether the content of the data bit is a null value, and the data information stored in the database is the data information without the null value marking bit;
when the content of the null value marking bit indicates that the content of the data bit is a null value, updating the content of the data bit in the first data information to be the valid null value mark, and storing the first data information after deleting the null value marking bit in the first data information, wherein the content of the valid null value mark is different from the content of the data bit of the data information stored in the database;
and when the content of the null value marking bit indicates that the content of the data bit is not a null value and the content of the data bit is different from the valid null value mark, storing the first data information after deleting the null value marking bit in the first data information.
9. The apparatus of claim 8, further comprising:
when the content of the null value marking bit indicates that the content of the data bit is not a null value and the content of the data bit is the same as the valid null value mark, storing the first data information;
and adding the null value marking bit outside the data bit in other stored data information of the database.
10. The apparatus of claim 8, further comprising:
when the content of the null value marking bit indicates that the content of the data bit is not a null value and the content of the data bit is the same as the effective null value mark, updating the effective null value mark by adopting an alternative null value mark to obtain a new effective null value mark, wherein the alternative null value mark is different from the content of the data bit of the data information stored in the database;
updating the content of a data bit in first target data information in the database to a new valid null value mark, wherein the content of the data bit is the data information of an original valid null value mark;
deleting the null value mark bit in the first data information and storing the first data information.
11. The apparatus according to any one of claims 8 to 10, wherein the data bits have a length of x bytes, the valid null flag has a length of y bytes, y is greater than x, and x and y are integer multiples of 2;
after deleting the null value flag bit in the first data information, the storing the first data information includes: and after deleting the null value mark bit in the first data information, adopting y bytes to store the content in the data bit in the first data information.
12. The apparatus of claim 11, wherein y is 2 times the x.
13. The apparatus of any of claims 8 to 10, wherein the database supports a 64-bit operating system, the length of the first data message is variable, and the valid null flag is 0 xffffffffffffffffff.
14. The apparatus of any one of claims 8 to 13, further comprising:
receiving a read request for reading second data information from the database;
when the stored data information in the database is the data information without the null value marking bit, comparing the content of the data bit of the second data information stored in the database with the effective null value mark;
when the content of the data bit is the same as the valid null value flag, outputting the second data information after adding the null value marking bit outside the data bit of the second data information, wherein the content of the added null value marking bit indicates that the content of the data bit of the second data information is a null value;
when the content of the data bit is different from the valid null value flag, the second data information is output after the null value marking bit is added outside the data bit of the second data information, and the content of the added null value marking bit indicates that the content of the data bit of the second data information is not a null value.
15. A computer device comprising a processor and a memory;
the computer device performs the data processing method of any of claims 1 to 7 when the processor executes the computer instructions stored by the memory.
16. A computer-readable storage medium, characterized in that the computer-readable storage medium comprises computer instructions that instruct a computer device to perform the data processing method of any of claims 1 to 7.
17. A data processing system, comprising:
a data management node and a data storage node, the data management node comprising a data processing apparatus as claimed in any of claims 8 to 14.
CN201911394329.XA 2019-12-30 2019-12-30 Data processing method, device and system Active CN111209304B (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN201911394329.XA CN111209304B (en) 2019-12-30 2019-12-30 Data processing method, device and system

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN201911394329.XA CN111209304B (en) 2019-12-30 2019-12-30 Data processing method, device and system

Publications (2)

Publication Number Publication Date
CN111209304A true CN111209304A (en) 2020-05-29
CN111209304B CN111209304B (en) 2023-04-07

Family

ID=70786529

Family Applications (1)

Application Number Title Priority Date Filing Date
CN201911394329.XA Active CN111209304B (en) 2019-12-30 2019-12-30 Data processing method, device and system

Country Status (1)

Country Link
CN (1) CN111209304B (en)

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN112395294A (en) * 2020-11-27 2021-02-23 浪潮云信息技术股份公司 Database data management method and system and database

Citations (8)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2002061569A2 (en) * 2001-01-29 2002-08-08 Thinprint Gmbh Method for processing printing orders in server-based networks, and corresponding server-based network
US20020111955A1 (en) * 2000-07-28 2002-08-15 Colby Latha S. Maintaining pre-computed aggregate views incrementally in the presence of non-minimal changes
US20120166400A1 (en) * 2010-12-28 2012-06-28 Teradata Us, Inc. Techniques for processing operations on column partitions in a database
TW201250469A (en) * 2011-06-14 2012-12-16 Phison Electronics Corp Memory management table proceesing method, memory controller and memory storage apparatus
US20150242481A1 (en) * 2013-04-16 2015-08-27 Hitachi, Ltd. Computer system, computer system management method, and program
CN106681995A (en) * 2015-11-05 2017-05-17 阿里巴巴集团控股有限公司 Data caching method and data query method and device
CN107544869A (en) * 2017-05-27 2018-01-05 新华三技术有限公司 A kind of data reconstruction method and device
CN109240948A (en) * 2018-08-21 2019-01-18 北京京东尚科信息技术有限公司 The data cached storage of one kind and method for cleaning and device

Patent Citations (8)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20020111955A1 (en) * 2000-07-28 2002-08-15 Colby Latha S. Maintaining pre-computed aggregate views incrementally in the presence of non-minimal changes
WO2002061569A2 (en) * 2001-01-29 2002-08-08 Thinprint Gmbh Method for processing printing orders in server-based networks, and corresponding server-based network
US20120166400A1 (en) * 2010-12-28 2012-06-28 Teradata Us, Inc. Techniques for processing operations on column partitions in a database
TW201250469A (en) * 2011-06-14 2012-12-16 Phison Electronics Corp Memory management table proceesing method, memory controller and memory storage apparatus
US20150242481A1 (en) * 2013-04-16 2015-08-27 Hitachi, Ltd. Computer system, computer system management method, and program
CN106681995A (en) * 2015-11-05 2017-05-17 阿里巴巴集团控股有限公司 Data caching method and data query method and device
CN107544869A (en) * 2017-05-27 2018-01-05 新华三技术有限公司 A kind of data reconstruction method and device
CN109240948A (en) * 2018-08-21 2019-01-18 北京京东尚科信息技术有限公司 The data cached storage of one kind and method for cleaning and device

Cited By (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN112395294A (en) * 2020-11-27 2021-02-23 浪潮云信息技术股份公司 Database data management method and system and database
CN112395294B (en) * 2020-11-27 2023-07-18 浪潮云信息技术股份公司 Database data management method and system and database

Also Published As

Publication number Publication date
CN111209304B (en) 2023-04-07

Similar Documents

Publication Publication Date Title
CN104935654B (en) Caching method, write-in point client in a kind of server cluster system and read client
US11176110B2 (en) Data updating method and device for a distributed database system
CN106844676B (en) Data storage method and device
CN107391033B (en) Data migration method and device, computing equipment and computer storage medium
US11455117B2 (en) Data reading method, apparatus, and system, avoiding version rollback issues in distributed system
CN110851474A (en) Data query method, database middleware, data query device and storage medium
CN113704790A (en) Abnormal log information summarizing method and computer equipment
CN112650692A (en) Heap memory allocation method, device and storage medium
CN116521956A (en) Graph database query method and device, electronic equipment and storage medium
CN111209304B (en) Data processing method, device and system
CN114443598A (en) Data writing method and device, computer equipment and storage medium
CN112115521B (en) Data access method and device
CN111639087A (en) Data updating method and device in database and electronic equipment
CN115599295A (en) Node capacity expansion method and device of storage system
CN110968267B (en) Data management method, device, server and system
CN110083509B (en) Method and device for arranging log data
CN113849482A (en) Data migration method and device and electronic equipment
CN114297196A (en) Metadata storage method and device, electronic equipment and storage medium
CN113806389A (en) Data processing method and device, computing equipment and storage medium
CN114020525A (en) Fault isolation method, device, equipment and storage medium
CN113220237A (en) Distributed storage method, device, equipment and storage medium
CN114722261A (en) Resource processing method and device, electronic equipment and storage medium
US11947822B2 (en) Maintaining a record data structure using page metadata of a bookkeeping page
CN115934583B (en) Hierarchical caching method, device and system
CN111797062B (en) Data processing method, device and distributed database system

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
TA01 Transfer of patent application right
TA01 Transfer of patent application right

Effective date of registration: 20220221

Address after: 550025 Huawei cloud data center, jiaoxinggong Road, Qianzhong Avenue, Gui'an New District, Guiyang City, Guizhou Province

Applicant after: Huawei Cloud Computing Technologies Co.,Ltd.

Address before: 518129 Bantian HUAWEI headquarters office building, Longgang District, Guangdong, Shenzhen

Applicant before: HUAWEI TECHNOLOGIES Co.,Ltd.

GR01 Patent grant
GR01 Patent grant