CN110162672B - Data processing method and device, electronic equipment and readable storage medium - Google Patents
Data processing method and device, electronic equipment and readable storage medium Download PDFInfo
- Publication number
- CN110162672B CN110162672B CN201910390733.3A CN201910390733A CN110162672B CN 110162672 B CN110162672 B CN 110162672B CN 201910390733 A CN201910390733 A CN 201910390733A CN 110162672 B CN110162672 B CN 110162672B
- Authority
- CN
- China
- Prior art keywords
- data
- value
- special state
- special
- state
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/90—Details of database functions independent of the retrieved data types
- G06F16/903—Querying
Landscapes
- Engineering & Computer Science (AREA)
- Databases & Information Systems (AREA)
- Theoretical Computer Science (AREA)
- Computational Linguistics (AREA)
- Data Mining & Analysis (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The embodiment of the invention realizes that the special data state in a data table is replaced by the value which does not appear in the data table by setting the mapping relation between the special data state and the value which is not contained in the data table, wherein the data table is a data set with saturated data volume or a data fragment of the data set. Therefore, only the corresponding relation between the special state of the data and the value which does not appear in the selected data table needs to be recorded, and the storage cost is low.
Description
Technical Field
The present invention relates to the field of computer technologies, and in particular, to a data processing method and apparatus, an electronic device, and a readable storage medium.
Background
A particular state of data means that the data does not represent a particular value. Special states of data are very common in databases. For example, in a database, if a constraint of a column allows a null value, there is a high probability that data in the null state exists in the column. For another example, in some databases, when a delete operation is performed on data, the data is not actually deleted from the database, but rather is marked as deleted. Data in a special state usually adopts a different representation method from that of ordinary data due to its special data meaning.
The prior art generally employs the following method to represent the special values: any value is selected to represent data in a particular state and a bitmap is maintained for each particular state to record whether the data is in that particular state. However, this representation method needs to record whether and what kind of special state the data is in for each data, and the storage overhead is large. Therefore, how to propose a lower-cost representation method for the special state of the data is the problem to be solved by the invention.
Disclosure of Invention
In view of this, embodiments of the present invention provide a data processing method and apparatus, an electronic device, and a readable storage medium, which aim to select an absent value in a data table to represent a special state of data, and only record a corresponding relationship between the special state of data and the selected absent value, so that storage overhead is relatively low.
In a first aspect, a data processing method is provided, including:
determining a first data set, wherein the first data set is formed by values not contained in a data table, and the data table is a data set with saturated data volume or a data fragment of the data set;
determining a kind set of data special states in a data table;
setting the mapping relation between the first data set and the category set in response to the number of elements contained in the first data set not being less than the number of elements contained in the category set, so that each data special state corresponds to at least one value;
and setting the data special state in the data table to be a value corresponding to the data special state.
Further, the determining the first set of data comprises:
determining a second data set consisting of values contained in the data table and a data type of the data table;
determining the value range of the data type;
and subtracting the value contained in the second data set from the value range to determine a first data set.
Further, the method further comprises:
in response to the number of elements included in the first set of data being less than the number of elements included in the set of categories, dividing the set of categories into a first set of categories and a second set of categories, the number of elements included in the first set of categories being equal to the number of elements included in the first set of data;
setting a mapping relation between the first data set and the first category set, so that each data special state in the first category set corresponds to one value;
and establishing a bitmap index for each data special state in the second category set.
Further, the data special state comprises a null value, a deleted state and an invalid value.
Further, the method further comprises:
in response to receiving an instruction for deleting data in the data table, determining a value corresponding to a deleted state of the data;
and replacing the deleted data in the data table by the value corresponding to the deleted state.
Further, the method further comprises:
receiving an instruction for modifying the data table, wherein the modification instruction comprises a modified value;
removing the modified value from the first data set in response to the modified value corresponding to the special state of the data;
in response to the first data set containing a value that does not correspond to a data special state, resetting a corresponding value for the data special state corresponding to the modified value, and replacing the modified value in the data table with the reset value;
and modifying data according to the modification instruction.
Further, the method further comprises:
receiving an instruction for inquiring data in the data table;
and responding to the data value inquired according to the inquiry instruction and having a corresponding data special state, and returning a value according to the data special state corresponding to the data value.
In a second aspect, there is provided an electronic device comprising a memory for storing one or more computer program instructions and a processor, wherein the one or more computer program instructions are executed by the processor to implement the method as described above.
In a third aspect, a data processing apparatus is provided, including:
the device comprises a first statistical module and a second statistical module, wherein the first statistical module is used for determining a first data set, the first data set is formed by values which are not contained in a data table, and the data table is a data set with saturated data volume or a data fragment of the data set.
And the second statistical module is used for determining the category set of the special states of the data in the data table.
And the data processing module is used for setting the mapping relation between the first data set and the category set when the number of elements contained in the first data set is not less than the number of elements contained in the category set, so that each data special state corresponds to at least one value.
And the control module is used for setting the special data state in the data table to be a value corresponding to the special data state.
In a fourth aspect, there is provided a computer readable storage medium for storing a data set, the computer program instructions, when executed by a processor, implementing the method as described above.
According to the embodiment of the invention, the special data state in the data table is replaced by the value which does not appear in the data table by setting the mapping relation between the special data state and the value which is not contained in the data table, and the data table is a data set with saturated data volume or data fragments of the data set. Therefore, only the corresponding relation between the special state of the data and the value which does not appear in the selected data table needs to be recorded, and the storage cost is low.
Drawings
The above and other objects, features and advantages of the present invention will become more apparent from the following description of the embodiments of the present invention with reference to the accompanying drawings, in which:
FIG. 1 is a diagram of a prior art bitmap index;
FIG. 2 is a flow chart of a data processing method according to an embodiment of the present invention;
FIG. 3 is a flow chart of determining a first set of data according to an embodiment of the present invention;
FIG. 4 is a flow chart of another data processing method according to an embodiment of the present invention;
FIG. 5 is a flow chart of another data processing method according to the embodiment of the present invention;
FIG. 6 is a diagram of a data processing apparatus according to an embodiment of the present invention;
fig. 7 is a schematic diagram of an electronic device according to an embodiment of the invention.
Detailed Description
The present invention will be described below based on examples, but the present invention is not limited to only these examples. In the following detailed description of the present invention, certain specific details are set forth. It will be apparent to one skilled in the art that the present invention may be practiced without these specific details. Well-known methods, procedures, and procedures have not been described in detail so as not to obscure the present invention.
Further, those of ordinary skill in the art will appreciate that the drawings provided herein are for illustrative purposes and are not necessarily drawn to scale.
Unless the context clearly requires otherwise, throughout the description and the claims, the words "comprise", "comprising", and the like are to be construed in an inclusive sense as opposed to an exclusive or exhaustive sense; that is, what is meant is "including, but not limited to".
In the description of the present invention, it is to be understood that the terms "first," "second," and the like are used for descriptive purposes only and are not to be construed as indicating or implying relative importance. In addition, in the description of the present invention, "a plurality" means two or more unless otherwise specified.
Fig. 1 is a diagram illustrating a bitmap index of the prior art, and as shown in fig. 1, the bitmap index is a table including only two data values, which may be, for example, 0 and 1. Each column of data in a data table may have different numbers of special states, for example, part of the columns in the data table contains 3 special states, part of the columns contains 1 special state, and part of the columns does not contain special states. The bitmap indices shown in fig. 1 as 11, 12 and 13 can only be established for each column.
When a bitmap index is established for a special state of data in a data table, the data bitmap index corresponds to a column in the data table. Further, when each column of the data table contains four rows of data, the bitmap index is a 4 × 1 table. And determining the position coordinates of the special state of the data needing to establish the bitmap index in the data table, and marking in the bitmap index according to the position coordinates. For example: and when the second row in the same column in the data table is null, the third row is in a deleted state, and the fourth row is an invalid value, setting the data in the bitmap with the value of 1 to represent the special state of the data. As shown, the bitmap index 11 is a null bitmap index, the value of the second row is 1, and the remaining values are 0. The bitmap index 12 is an invalid value bitmap index, the fourth row has a value of 1, and the remaining values are 0. The bitmap index 13 is a deleted bitmap index, the value of the third row is 1, and the remaining values are 0. Further, when the special state of the data in the data table is represented by a bitmap index, the data in the data table in the special state may be arbitrarily replaced.
The method for establishing the bitmap index needs to record whether the data is in a special state and in which special state for each data, so at least one bitmap index needs to be established for each special state of the data, and the storage cost is high.
Fig. 2 is a flowchart of a data processing method according to an embodiment of the present invention, and as shown in fig. 2, the data processing method according to the embodiment of the present invention includes:
step S100: a first set of data is determined.
Specifically, the first data set is composed of values not included in a data table, which is a data set or a data slice of a data set with a saturated data amount. Further, the data table is data of a data setIn the slicing case, the data slices of the data set in the system are divided into encapsulated data slices and unencapsulated data slices, and the difference between the encapsulated data slices and the unencapsulated data slices is whether the data volume is saturated or not. And if the data volume of the data fragment reaches saturation, the data fragment is considered to be the encapsulated data fragment, otherwise, the data fragment is considered to be the non-encapsulated data fragment. The data processing method is used for processing the data of the packaged data fragments, and the special state of the non-packaged data fragments is still recorded by using a traditional bitmap index-based method. Therefore, any value in the data type value range of the data values in the unpackaged data slice may be normal data. For example, corresponding to the unsigned int32 type, anywhere from 0-2 before packaging32The value of-1 is considered to be the normal value for the normal data state. And when data is added into the unpackaged data fragments to enable the data volume of the data fragments to reach saturation, triggering the process of carrying out data processing on the data fragments with the saturated data volume. That is, values that do not appear in the data set are selected to represent the special state of the data, and bitmap indexes originally used for representing the special state of the data are discarded.
For example, if the first data set is U, the set including all data values in the data table is K, and the data type range in the set is N, then U is N-K, that is, all values included in the first data set are the data type range in the data table minus the values included in the data set. The first data set may be, for example, U ═ U0,u1,u2,…,up-1Denotes, p is a positive integer. The first data set includes P elements, each element being a value.
Step S200: a set of categories of special states of data in the data table is determined.
Specifically, the data special state may include, for example, a null value, a deleted state, and an invalid value. The set of categories may be, for example, by S ═ S0,s1,s2,…,sq-1Q is a positive integer. The collection of categories includes q elements, each element being a special state of the data.
Step S300: and setting the mapping relation between the first data set and the category set.
Specifically, the step S300 is in response to the number of elements included in the data set not being less than the number of elements included in the category set, i.e., p not being less than q. After setting the mapping relationship between the first data set and the category set, each data special state corresponds to at least one value, such as s0And u0Corresponds to, s1And u9And the like. Further, the mapping relationship is stored by a table record.
Step S400: the data special state in the data table is set to a value corresponding to the data special state.
Specifically, the data special state in the data table is set to a value corresponding to the data special state according to the mapping relation between the first data set and the category set. For example at s0And u0Corresponds to, s1And u9In the corresponding case, the state s in the data table is0Is set as u0The state is s1Is set as u9。
The data processing method can realize that the special state of the data in the data table is replaced by the value which does not appear in the data table by setting the mapping relation between the special state of the data and the value. The method only needs to record the corresponding relation between the special state of the data and the value which does not appear in the selected data table, and reduces the storage expense of the data.
Fig. 3 is a flowchart of determining a first data set according to an embodiment of the present invention, and as shown in fig. 3, the method for determining the first data set includes:
step S110: determining a second data set contained in the data table and a data type of the data table.
Specifically, a second data set K is created and all data in the data table is inserted into K. Judging whether the data to be inserted exist in K or not in the inserting process, if so, discarding the data; otherwise, the data is inserted into K. After the insert operation is completed, the second data set K contains n data, i.e. K ═ K0,k1,…,kn-1N is a positive integer. And simultaneously acquiring the data types of the values in the data table, wherein the data types can be, for example: int, float, double, etc.
Step S120: and determining the value range of the data type.
Specifically, the value range of the data type is further determined according to the data type determined in step S110, for example, when the data type is int, the value range with a sign is-231～231And (4) the unsigned value range of the integer data of-1 is 0-4294967295. In this embodiment, the range of values of the data type may be represented by N.
Step S130: and subtracting the value contained in the second data set from the value range to determine a first data set.
In particular, the first set of data, i.e. U-N-K, is determined by subtracting the values contained in the second set of data within the range of values of said data type. For example, subtracting the second data set K ═ K { K } from the range of unsigned values 0 ~ 4294967295 when the data type is int0,k1,…,kn-1All values contained in (c): k is a radical of0,k1,…,kn-1Obtaining a first data set U ═ U0,u1,u2,…,up-1}。
Fig. 4 is a flowchart of another data processing method according to an embodiment of the present invention, and as shown in fig. 4, the data processing method includes:
step S100: a first set of data is determined.
Specifically, the first data set is composed of values not contained in a data table, which is a data set or a data slice of a data set with saturated data amount.
Step S200: a set of categories of special states of data in the data table is determined.
Specifically, the data special state may include, for example, a null value, a deleted state, and an invalid value.
Step S300: and setting the mapping relation between the first data set and the category set.
Specifically, the step S300 is in response to the number of elements included in the data set not being less than the number of elements included in the category set, i.e., the p is not less than q. After setting the mapping relationship between the first data set and the category set, each data special state corresponds to at least one value, such as s0And u0Corresponds to, s1And u9And the like. Further, the mapping relationship is stored by a table record.
Step S500: the set of categories is divided into a first set of categories and a second set of categories.
In particular, said step S500 is responsive to the number of elements contained in said first set of data being less than the number of elements contained in said set of categories, i.e. said p is less than q. Wherein the number of elements of the first set of categories is equal to the number of elements of the first set of data. Further, when the number of elements in the first data set is 0, the number of elements in the first category set is 0, and all elements in the category set are all elements in the second category set.
For example, when the number p of elements included in the first data set U is 4 and the number q of elements included in the category set S is 7, the data set S is divided into a first category set S 'and a second category set S ", where the number q' of elements included in the first category set is 4 and the number q" of elements included in the second category set is 3.
Step S600: and setting the mapping relation between the first data set and the first kind set.
Specifically, the mapping relationship between the first data set and the first class set is set so that each data special state included in the first class set corresponds to a value included in the first data set. For example, in step S500, a one-to-one correspondence relationship is established between 4 values of the data set and 4 data special states of the first type set, and further, a table is established to store a mapping relationship between the first data set and the first type set.
Step S700: a bitmap index is established for each data special state in the second category set.
Specifically, a bitmap corresponding to a column in the data table is established for each data special state. When a data special state appears in the data table, marking is carried out at the same coordinate position in the bitmap, for example, when the data special state appears at the 3 rd row and the 7 th row of the second column in the data table, marking 1 at the position of the 2 nd row and the 7 th row in the bitmap, and marking 0 at other positions. And recording and representing the special states of the data in the second type set by establishing a bitmap index for each special state of the data.
Step S400: the data special state in the data table is set to a value corresponding to the data special state.
Specifically, after step S300 or S600, the first data set and the category set or the first category set form a mapping relationship and are stored. And setting all data special states in the data table or the data special states included in the first data set to be values corresponding to the data special states according to the mapping relation between the first data set and the category set or the first category set.
Fig. 5 is a flowchart of another data processing method according to an embodiment of the present invention, as shown in fig. 5, when data processing is performed, each type of data operation generally requires special processing on data in a special state, and therefore, when the operation is performed, it is necessary to identify whether the data is in the special state, and then perform corresponding processing. In the present invention, if the data special state has a corresponding value, it is necessary to determine whether the data is in the special state according to the data value.
The data processing method in this embodiment is based on step S400, that is, the first data set and the category set have a mapping relationship, and all or part of the data special states in the data table are replaced by corresponding values. The steps of deleting, modifying and querying by the user include:
step S800: and determining the value corresponding to the deleted data state.
Specifically, in response to receiving an instruction to delete data in the data table, the step S800 determines a value in the data set corresponding to the deleted data state according to the mapping relationship between the stored category set and the data set.
Step S900: and replacing the deleted data in the data table by the value corresponding to the deleted state.
Specifically, for example, when the value corresponding to the deleted state is 1, the value of the deleted data in the data table is replaced with 1.
Further, when the deleted state of the data in the data table is represented by a method of establishing a bitmap index, the deleted state of the data has no corresponding value, and the deleted data in the bitmap index for representing the deleted state of the data may be set to be a deleted state.
Step S1000: in response to the modified value corresponding to the data special state, removing the modified value from the first data set.
Specifically, the step S1000 receives an instruction for modifying data in the data table, where the modification instruction includes a modified value. In response to the modified value corresponding to the data special state, removing the modified value from the first data set. For example: when the modified value is 7, removing 7 from the first data set.
Step S1100: and in response to the first data set containing a value which has no corresponding relation with the special data state, resetting a corresponding value for the special data state corresponding to the modified value, and replacing the modified value in the data table with the reset value.
Specifically, it is queried whether the first data set includes a value having no correspondence with the data special state, if yes, a corresponding value is reset for the data special state corresponding to the modified value in step S1000, and the modified value in the data table is replaced with the reset value. For example, the modified value is 7, and the data special state corresponding to the modified value is null. And if a value 9 which has no corresponding relation with a special state of the data is determined in the first data set, resetting a corresponding value 9 for a null value, and replacing the modified value 7 in the data table with the reset value 9. Further, the mapping relation between the first data set and the category set is updated and recorded in a table. And if the first data set does not contain a value which has no corresponding relation with the special data state, establishing a bitmap index for the special data state.
Step S1200: and modifying the data according to the modification instruction.
Specifically, the data table is modified according to the modification instruction, that is, the modified data in the data table is replaced by the modified value.
Step S1300: and responding to the data value inquired according to the inquiry instruction and having a corresponding data special state, and returning a value according to the data special state corresponding to the data value.
Specifically, step 1300 is to respond to receiving an instruction to query the data table, and further to respond to that the data value queried according to the query instruction has a corresponding data special state, and return a value according to the data special state corresponding to the data value. For example, skipping when the data special state is a deleted state and returning a null value when the data special state is null.
Further, when the special state of part of the data in the data table is represented by a method of establishing a bitmap index, a value is returned according to the special state of the data corresponding to the data value.
The data processing method only needs to delete, modify and query the data according to the mapping relation between the first data set and the category set, and does not need to repeatedly select and read the bitmap index, so that the data quantity needing to be queried in the data processing process can be reduced, and the data processing efficiency is improved.
Fig. 6 is a schematic diagram of a data processing apparatus according to an embodiment of the present invention, as shown in fig. 6, the data processing apparatus includes a first statistical module 61, a second statistical module 62, a data processing module 63, and a control module 64. In particular, the first statistical module 61 is configured to determine a first data set, where the first data set is composed of values not included in a data table, and the data table is a data set with saturated data volume or a data slice of the data set. The second statistical module 62 is used to determine the set of categories of the special states of the data in the data table. The data processing module 63 is configured to set a mapping relationship between the first data set and the category set when the number of elements included in the first data set is not less than the number of elements included in the category set, so that each data special state corresponds to at least one value. The control module 64 is configured to set a data special state in the data table to a value corresponding to the data special state.
The data processing device can replace the special data state in the data table with the value which does not appear in the data table by setting the mapping relation between the special data state and the value. The method only needs to record the corresponding relation between the special state of the data and the value which does not appear in the selected data table, and reduces the storage expense of the data.
Fig. 7 is a schematic view of an electronic device according to an embodiment of the present invention, where in this embodiment, the electronic device includes a server, a terminal, and the like. As shown, the electronic device includes: at least one processor 72; a memory 71 communicatively coupled to the at least one processor; and a communication component 73 communicatively coupled to the storage medium, the communication component receiving and transmitting data under control of the processor; the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to implement the data processing method in the above embodiments.
In particular, the memory 71, as a non-volatile computer-readable storage medium, may be used to store non-volatile software programs, non-volatile computer-executable programs, and modules. The processor 72 executes various functional applications of the device and data processing, i.e., implements the above-described data processing method, by executing nonvolatile software programs, instructions, and modules stored in the memory.
The memory 71 may include a storage program area and a storage data area, wherein the storage program area may store an operating system, an application program required for at least one function; the storage data area may store a list of options, etc. Further, the memory 71 may include high speed random access memory, and may also include non-volatile memory, such as at least one magnetic disk storage device, flash memory device, or other non-volatile solid state storage device. In some embodiments, the memory 71 optionally includes memory 71 remotely located from the processor 72, and these remote memories may be connected to external devices via a network. Examples of such networks include, but are not limited to, the internet, intranets, local area networks, mobile communication networks, and combinations thereof.
One or more modules are stored in the memory 71 and, when executed by the one or more processors 72, perform the data processing method in any of the method embodiments described above.
The product can execute the method provided by the embodiment of the application, has corresponding functional modules and beneficial effects of the execution method, and can refer to the method provided by the embodiment of the application without detailed technical details in the embodiment.
The present invention also relates to a computer-readable storage medium for storing a computer-readable program for causing a computer to perform some or all of the above-described method embodiments.
That is, as can be understood by those skilled in the art, all or part of the steps in the method for implementing the embodiments described above may be implemented by a program instructing related hardware, where the program is stored in a storage medium and includes several instructions to enable a device (which may be a single chip, a chip, or the like) or a processor (processor) to execute all or part of the steps of the method described in the embodiments of the present application. And the aforementioned storage medium includes: a U-disk, a removable hard disk, a Read-Only Memory (ROM), a Random Access Memory (RAM), a magnetic disk or an optical disk, and other various media capable of storing program codes.
The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention, and various modifications and changes may be made by those skilled in the art. Any modification, equivalent replacement, or improvement made within the spirit and principle of the present invention should be included in the protection scope of the present invention.
Claims (9)
1. A data processing method, comprising:
determining a first data set, wherein the first data set is formed by values not contained in a data table, and the data table is a data set with saturated data volume or a data fragment of the data set;
determining a kind set of data special states in a data table;
setting the mapping relation between the first data set and the category set in response to the number of elements contained in the first data set not being less than the number of elements contained in the category set, so that each data special state corresponds to at least one value;
setting a data special state in the data table to a value corresponding to the data special state;
in response to the number of elements included in the first set of data being less than the number of elements included in the set of categories, dividing the set of categories into a first set of categories and a second set of categories, the number of elements included in the first set of categories being equal to the number of elements included in the first set of data;
setting a mapping relation between the first data set and the first category set, so that each data special state in the first category set corresponds to one value;
and establishing a bitmap index for each data special state in the second category set.
2. The method of claim 1, wherein the determining the first set of data comprises:
determining a second data set consisting of values contained in the data table and a data type of the data table;
determining the value range of the data type;
and subtracting the value contained in the second data set from the value range to determine a first data set.
3. The method of claim 1, wherein the data special states include null values, deleted states, invalid values.
4. The method of any one of claims 1-3, further comprising:
in response to receiving an instruction for deleting data in the data table, determining a value corresponding to a deleted state of the data;
and replacing the deleted data in the data table by the value corresponding to the deleted state.
5. The method of any one of claims 1-3, further comprising:
receiving an instruction for modifying the data table, wherein the modification instruction comprises a modified value;
removing the modified value from the first data set in response to the modified value corresponding to the special state of the data;
in response to the first data set containing a value that does not correspond to a data special state, resetting a corresponding value for the data special state corresponding to the modified value, and replacing the modified value in the data table with the reset value;
and modifying data according to the modification instruction.
6. The method of any one of claims 1-3, further comprising:
receiving an instruction for inquiring data in the data table;
and responding to the data value inquired according to the inquiry instruction and having a corresponding data special state, and returning a value according to the data special state corresponding to the data value.
7. An electronic device comprising a memory and a processor, wherein the memory is configured to store one or more computer program instructions, wherein the one or more computer program instructions are executed by the processor to implement the method of any of claims 1-6.
8. A data processing apparatus, comprising:
the data processing device comprises a first statistic module, a second statistic module and a third statistic module, wherein the first statistic module is used for determining a first data set, the first data set is formed by values which are not contained in a data table, and the data table is a data set with saturated data volume or data fragments of the data set;
the second statistical module is used for determining a kind set of the special data states in the data table;
a data processing module, configured to set a mapping relationship between the first data set and the category set when the number of elements included in the first data set is not less than the number of elements included in the category set, so that each data special state corresponds to at least one value;
the control module is used for setting the special data state in the data table to a value corresponding to the special data state;
the data processing module is further configured to:
in response to the number of elements included in the first set of data being less than the number of elements included in the set of categories, dividing the set of categories into a first set of categories and a second set of categories, the number of elements included in the first set of categories being equal to the number of elements included in the first set of data;
setting a mapping relation between the first data set and the first category set, so that each data special state in the first category set corresponds to one value;
and establishing a bitmap index for each data special state in the second category set.
9. A computer readable storage medium storing computer program instructions, which when executed by a processor implement the method of any one of claims 1-6.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910390733.3A CN110162672B (en) | 2019-05-10 | 2019-05-10 | Data processing method and device, electronic equipment and readable storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910390733.3A CN110162672B (en) | 2019-05-10 | 2019-05-10 | Data processing method and device, electronic equipment and readable storage medium |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN110162672A CN110162672A (en) | 2019-08-23 |
| CN110162672B true CN110162672B (en) | 2021-07-27 |
Family
ID=67634361
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201910390733.3A Active CN110162672B (en) | 2019-05-10 | 2019-05-10 | Data processing method and device, electronic equipment and readable storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN110162672B (en) |
Families Citing this family (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111581032B (en) * | 2020-05-21 | 2023-06-27 | 抖音视界有限公司 | Method, device, equipment and storage medium for operating data and rolling back data |
| CN113626444B (en) * | 2021-08-26 | 2023-11-28 | 平安国际智慧城市科技股份有限公司 | Table query method, device, equipment and medium based on bitmap algorithm |
| CN114443703A (en) * | 2021-12-15 | 2022-05-06 | 北京达佳互联信息技术有限公司 | Data processing method and device, electronic equipment and storage medium |
Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN104054073A (en) * | 2011-11-15 | 2014-09-17 | 起元科技有限公司 | Data clustering, segmentation, and parallelization |
| CN105930926A (en) * | 2016-04-19 | 2016-09-07 | 杨展业 | SAAS-based IOT data analyzing, monitoring and predicting system and method |
| CN107679129A (en) * | 2017-09-21 | 2018-02-09 | 无线生活（杭州）信息科技有限公司 | A kind of big data processing method and processing device |
| CN108052394A (en) * | 2017-12-27 | 2018-05-18 | 福建星瑞格软件有限公司 | The method and computer equipment of resource allocation based on SQL statement run time |
| CN108366259A (en) * | 2018-01-15 | 2018-08-03 | 华南理工大学 | A kind of reversible steganography method of three coefficient of H.264/AVC video of high-fidelity |
| CN109344142A (en) * | 2018-08-22 | 2019-02-15 | 中国平安人寿保险股份有限公司 | Data processing method, device, electronic equipment and storage medium |
-
2019
- 2019-05-10 CN CN201910390733.3A patent/CN110162672B/en active Active
Patent Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN104054073A (en) * | 2011-11-15 | 2014-09-17 | 起元科技有限公司 | Data clustering, segmentation, and parallelization |
| CN105930926A (en) * | 2016-04-19 | 2016-09-07 | 杨展业 | SAAS-based IOT data analyzing, monitoring and predicting system and method |
| CN107679129A (en) * | 2017-09-21 | 2018-02-09 | 无线生活（杭州）信息科技有限公司 | A kind of big data processing method and processing device |
| CN108052394A (en) * | 2017-12-27 | 2018-05-18 | 福建星瑞格软件有限公司 | The method and computer equipment of resource allocation based on SQL statement run time |
| CN108366259A (en) * | 2018-01-15 | 2018-08-03 | 华南理工大学 | A kind of reversible steganography method of three coefficient of H.264/AVC video of high-fidelity |
| CN109344142A (en) * | 2018-08-22 | 2019-02-15 | 中国平安人寿保险股份有限公司 | Data processing method, device, electronic equipment and storage medium |
Non-Patent Citations (4)
| Title |
|---|
| Adrian.数据库表中的空值(NULL).《https://www.cnblogs.com/stephenwang/archive/2006/03/30/362710.html》.2006,1. * |
| exhuma.How should I store "unknown" and "missing" values in a variable, while still retraining the difference between "unknown" and "missing".《https://softwareengineering.stackexchange.com/questions/376845/how-should-i-store-unknown-and-missing-values-in-a-variable-while-still-ret》.2018,1-2. * |
| Jake VanderPlas.处理缺失数据.《http://www.oreilly.com.cn/radar/?p=151》.2015,1. * |
| lwjaiyjk3.数据库中表中状态字段status的设计问题.《https://blog.csdn.net/yujiak/article/details/79002310?utm_source=blogxgwz1》.2018,1. * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN110162672A (en) | 2019-08-23 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN110162672B (en) | Data processing method and device, electronic equipment and readable storage medium | |
| US8738593B2 (en) | Method and apparatus for reducing index sizes and increasing performance of non-relational databases | |
| CN110781372B (en) | Method and device for optimizing website, computer equipment and storage medium | |
| CN111061758B (en) | Data storage method, device and storage medium | |
| CN104036187B (en) | Method and system for determining computer virus types | |
| CN112473140B (en) | Task processing method and device, electronic equipment and storage medium | |
| CN107577787B (en) | Method and system for storing associated data information | |
| CN112734982A (en) | Storage method and system for unmanned vehicle driving behavior data | |
| CN109885577A (en) | Data processing method, device, terminal and storage medium | |
| CN107402753B (en) | Method and device for refreshing hard disk firmware | |
| CN114548059A (en) | Method and device for managing structured data, storage medium and electronic equipment | |
| CN110535943A (en) | Data processing method, device, electronic equipment and storage medium | |
| US6763358B2 (en) | Method and system for activating column triggers in a database management system | |
| CN116955341A (en) | Database integrity evaluation method, system and application thereof | |
| CN108376054B (en) | Processing method and device for indexing identification data | |
| CN110727895A (en) | Sensitive word sending method and device, electronic equipment and storage medium | |
| US20240020303A1 (en) | Systems and methods for optimized cloud database query execution | |
| CN111125158A (en) | Data table processing method, device, medium and electronic equipment | |
| CN109815213A (en) | It is deleted on a kind of Append-Only database and the method and system of modification data | |
| CN103778218A (en) | Cloud computation-based standard information consistency early warning system and method | |
| CN104091127A (en) | Method and device for clearing privacy records | |
| CN112419058A (en) | Data management method and device, storage medium and electronic equipment | |
| CN109446162B (en) | Method and system for determining data state of target mobile terminal in mobile internet | |
| CN110751204A (en) | Data fusion method and device, storage medium and electronic device | |
| CN114615533B (en) | Method, device, equipment and storage medium for managing and controlling burnt key code |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |