CN112506651B - Method and equipment for data operation in large-data-volume environment - Google Patents
Method and equipment for data operation in large-data-volume environment Download PDFInfo
- Publication number
- CN112506651B CN112506651B CN202011363989.4A CN202011363989A CN112506651B CN 112506651 B CN112506651 B CN 112506651B CN 202011363989 A CN202011363989 A CN 202011363989A CN 112506651 B CN112506651 B CN 112506651B
- Authority
- CN
- China
- Prior art keywords
- data
- integer
- char
- bit
- memory space
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 238000000034 method Methods 0.000 title claims abstract description 41
- 238000013500 data storage Methods 0.000 claims abstract description 39
- 230000004044 response Effects 0.000 claims description 37
- 230000004048 modification Effects 0.000 claims description 5
- 238000012986 modification Methods 0.000 claims description 5
- 238000004590 computer program Methods 0.000 description 5
- 238000010586 diagram Methods 0.000 description 3
- 230000008569 process Effects 0.000 description 3
- 230000008901 benefit Effects 0.000 description 2
- 230000000694 effects Effects 0.000 description 2
- 238000003672 processing method Methods 0.000 description 2
- 230000009286 beneficial effect Effects 0.000 description 1
- 238000004364 calculation method Methods 0.000 description 1
- 238000013499 data model Methods 0.000 description 1
- 230000007547 defect Effects 0.000 description 1
- 230000008030 elimination Effects 0.000 description 1
- 238000003379 elimination reaction Methods 0.000 description 1
- 230000006870 function Effects 0.000 description 1
- 230000005055 memory storage Effects 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
- 238000012163 sequencing technique Methods 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/50—Allocation of resources, e.g. of the central processing unit [CPU]
- G06F9/5005—Allocation of resources, e.g. of the central processing unit [CPU] to service a request
- G06F9/5011—Allocation of resources, e.g. of the central processing unit [CPU] to service a request the resources being hardware resources other than CPUs, Servers and Terminals
- G06F9/5016—Allocation of resources, e.g. of the central processing unit [CPU] to service a request the resources being hardware resources other than CPUs, Servers and Terminals the resource being the memory
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0602—Interfaces specially adapted for storage systems specifically adapted to achieve a particular effect
- G06F3/0608—Saving storage space on storage systems
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Software Systems (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Human Computer Interaction (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention provides a method and equipment for data operation in a large-data-volume environment, wherein the method comprises the following steps: setting a data storage mode of a char type memory space; storing a corresponding number of integers in data to be operated by using each char in a char type memory space based on a data storage mode; each data is operated on based on the way the data is stored. By using the scheme of the invention, the space for storing the big data in the memory can be saved, and the efficiency of data operation is greatly improved.
Description
Technical Field
The field relates to the field of computers, and more particularly to a method and apparatus for data manipulation in a large data volume environment.
Background
With the development of networks, at present, more and more data need to be processed by a server, and the situation of processing a large amount of data is more and more common. An increasing number of methods for processing these data are emerging on the corresponding market. The existing treatment methods include the following methods: 1. storing all processing methods in the memory, putting all data into the memory, and processing the data in a sequencing or duplicate checking mode; 2. a local elimination method, wherein a part of data is found out from the data for data operation, and then the rest data is compared with the data found in advance; 3. the dividing and treating method divides the whole data into n parts of data with the same volume according to a certain proportion and respectively operates the n parts of data or operates recursion for many times. Finally, the required data is obtained.
The whole memory storage processing method is the most widely applied method at present. The defects are obvious, the large data quantity is huge, and if the large data quantity is completely stored in a memory, a large amount of memory space is occupied for operation. If the most frequent 10 of 10 hundred million queries (each of 8B) need to be searched, the memory required for 10 hundred million queries is about 10^9 × 8B ^ 8GB memory, considering that each query word takes 8B. A typical computer cannot support such a large memory usage. The method is similar to the sorting method, and the time for traversing one hundred million numbers is very huge. The time complexity of the method is O (n + m 2), wherein m is the size of the container and n is the size of the residual data volume. The divide and conquer method is between the first and the second methods, divides and processes big data, reduces the use of memory, and allocates the memory to the blind computers respectively, and combines a plurality of machines for operation. The computing power of a plurality of computers is reduced, the used memory space is reduced, and the computing efficiency is improved. However, the cooperative operation of a plurality of computers involves a problem of cooperative operation between the computers. The architecture is complicated due to the problems of computing backup and cooperative operation.
Disclosure of Invention
In view of this, an embodiment of the present invention provides a method and a device for data operation in a large data volume environment, and by using the technical solution of the present invention, a space for storing large data in a memory can be eliminated, so as to greatly improve the efficiency of data operation.
In view of the above object, an aspect of an embodiment of the present invention provides a method for data operation in a large data volume environment, including the steps of:
setting a data storage mode of a char (for defining character type variables in C or C + +) type memory space;
storing a corresponding number of integers in the data to be operated by using each char in the char type memory space based on the data storage mode;
and calculating each data based on the stored data mode.
According to an embodiment of the present invention, the manner of setting the char type memory space to store data includes:
the manner in which the char-type memory space stores data is set to one of 1 integer bit 128 state, 2 integer bits 8 state, or 4 integer bits 2 state.
According to an embodiment of the present invention, storing a corresponding number of integers in data to be operated using each char in a char type memory space based on the manner of storing data includes:
judging the data storage mode of the char type memory space;
in response to the char-type memory space storing data in the form of 1 integer Bit 128 states, the first Bit of each char is used to store information on whether an integer is stored in memory, and the second through eighth bits are used to store the state bits of the integer.
According to an embodiment of the present invention, storing a corresponding number of integers in data to be operated using each char in a char type memory space based on the manner of storing data includes:
judging the data storage mode of the char type memory space;
in response to the manner in which the char-type memory space stores data is 2 integer bits with 8 states, the first Bit of each char is used to store information whether a first integer is stored in memory, the second through fourth bits store status bits of the first integer, the fifth Bit stores information whether a second integer is stored in memory, and the sixth through eighth bits store status bits of the second integer.
According to an embodiment of the present invention, storing a corresponding number of integers in data to be operated using each char in a char type memory space based on the manner of storing data includes:
judging the data storage mode of the char type memory space;
in response to the fact that the manner of storing data in the char-type memory space is 4 integer bits and 2 states, the first Bit of each char is used to store information on whether a first integer is stored in the memory, the second Bit stores the state Bit of the first integer, the third Bit stores information on whether a second integer is stored in the memory, the fourth Bit stores the state Bit of the second integer, the fifth Bit stores information on whether a third integer is stored in the memory, the sixth Bit stores the state Bit of the third integer, the seventh Bit stores information on whether a fourth integer is stored in the memory, and the eighth Bit stores the state Bit of the fourth integer.
According to an embodiment of the present invention, the performing the operation on each data based on the manner of storing the data includes:
judging the data storage mode of the char type memory space;
judging which kind of operation is performed on the data in response to the fact that the mode of storing the data in the char type memory space is 2 integer bits and 8 states;
in response to performing an increment operation on data, dividing the data by the number of integer bits in the manner in which the data is stored to obtain a quotient and a remainder;
and finding the Bit pointed by the quotient and the remainder, and positioning the pointed Bit at 1.
According to an embodiment of the present invention, the performing the operation on each data based on the manner of storing the data includes:
judging the data storage mode of the char type memory space;
judging which kind of operation is performed on the data in response to the fact that the mode of storing the data in the char type memory space is 2 integer bits and 8 states;
in response to performing a state modification operation on data, dividing the data by the number of integer bits in the manner in which the data is stored to obtain a quotient and a remainder;
and finding out the quotient and the remainder pointing to the Bit position of the representative state, and carrying out OR operation on the Bit position of the representative state after zero setting and the new state Bit.
According to an embodiment of the present invention, the performing the operation on each data based on the manner of storing the data includes:
judging the data storage mode of the char type memory space;
judging which operation is performed on the data in response to the fact that the mode of storing the data in the char type memory space is 2 integer bits and 8 states;
in response to performing a delete operation on data, dividing the data by the number of integer bits in the manner in which the data is stored to obtain a quotient and a remainder;
the Bit to which the quotient and remainder point and the Bit representing the state are zeroed.
According to an embodiment of the present invention, the performing the operation on each data based on the manner of storing the data includes:
judging the data storage mode of the char type memory space;
judging which kind of operation is performed on the data in response to the fact that the mode of storing the data in the char type memory space is 2 integer bits and 8 states;
in response to performing a query operation on data, dividing the data by the number of integer bits in the manner in which the data is stored to obtain a quotient and a remainder;
judging whether the Bit pointed by the quotient and the remainder is 0 or not;
in response to the pointed to Bit being 0, return that the looked up data is not present.
In another aspect of the embodiments of the present invention, there is also provided an apparatus for data operation in a large data volume environment, the apparatus including:
the device comprises a setting module, a storage module and a processing module, wherein the setting module is configured to set the data storage mode of a char type memory space;
the storage module is configured to store a corresponding number of integers in the data to be operated by using each char in the char type memory space based on the data storage mode;
an operation module configured to operate on each data based on the manner in which the data is stored.
The invention has the following beneficial technical effects: the method for data operation in the large data volume environment provided by the embodiment of the invention is characterized in that a data storage mode of a char type memory space is set; based on the data storage mode, each char in the char type memory space is used for storing a corresponding number of integers in the data to be operated; according to the technical scheme of operating each data based on the data storage mode, the space for storing big data in the memory can be saved, and the data operation efficiency is greatly improved.
Drawings
In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the embodiments or the prior art will be briefly described below, and it is obvious that the drawings in the following description are only some embodiments of the present invention, and it is obvious for those skilled in the art that other embodiments can be obtained by using the drawings without creative efforts.
FIG. 1 is a schematic flow chart diagram of a method of data manipulation in a large data volume environment, according to one embodiment of the present invention;
FIG. 2 is a schematic diagram of an apparatus for data manipulation in a large data volume environment, according to one embodiment of the present invention.
Detailed Description
In order to make the objects, technical solutions and advantages of the present invention more apparent, the following embodiments of the present invention are described in further detail with reference to the accompanying drawings.
In view of the above objects, a first aspect of embodiments of the present invention proposes an embodiment of a method for data operation in a large data volume environment. Fig. 1 shows a schematic flow diagram of the method.
As shown in fig. 1, the method may include the steps of:
s1, setting the data storage mode of the char type memory space, and setting three types of storage modes according to the requirement;
s2 stores a corresponding number of integers in the data to be computed by using each char in the char type memory space based on the data storage manner, and according to the set storage manner, each char may store 1 data, 2 data, or 4 data, which is not a real storage but stores a corresponding state, thereby saving a large amount of storage space;
s3 performs operations on each data based on the manner in which the data is stored, including operations of adding, modifying, deleting, and looking up data.
Take a 64-bit compiler computer as an example: one char type of data occupies 1 byte (8-Bit) of space. One integer occupies 4 bytes (32-Bit). Suppose we want to compute a data model that is an add-drop look-and-check of 100 integers and their states. The conventional operation method applies an array space of 100 integers to store the 100 integers, and requires 100 × 4 to 400 bytes. If the states of the 100 integers are expressed as integers, for example, an integer has 3 states, where 1 represents 1 occurrence of the integer, 2 represents 2 occurrences of the integer, and 3 represents 3 occurrences of the integer. Describing the state of 100 integers requires that the memory space be opened up to 100 × 4-400 bytes (3200 bits). The total space required to perform this data scene operation is 800 bytes.
Taking a storage mode in the technical scheme of the invention as an example, the space to be opened up is as follows: the method is characterized in that a char type is used as an application unit to apply for 1 char space with 8 bits, and each Bit has two states of 0 and 1. We can describe two integers and their states in these 8 consecutive Bit spaces. Bit [0] represents whether a value exists in the integer array, Bit [1-3] is used to describe the state of the integer, Bit [4] is used to describe whether the next integer exists in the integer array, and Bit [5-8] is used to describe the state of the integer. Such a char space may store 2 integers and their states. The 100 integers and states require only 50 char spaces, i.e., 50 bytes (400 bits). If 100 states exist in each integer within 100 billion, 10000000000 × 8 (double-type integer value storage) 1 (in the case of a char-type storage state) needs to be applied in the conventional mode, namely 800 megabyte space (74G memory space), and a new computing mode 1 char space can describe an integer and the state of the integer needs to be applied in 10000000000 × 1 (in the case of a char-type integer storage and state) 9.3G memory space. Therefore, the advantage of the memory space calculation method is larger when the number of integer states is smaller. The 7-Bit space in the state can be saved by 8 times, the 3-Bit space can be saved by 16 times, and the 1-Bit space can be saved by 32 times.
By the technical scheme, the space for storing the big data in the memory can be saved, and the efficiency of data operation is greatly improved.
In a preferred embodiment of the present invention, the setting of the manner of storing data in the char type memory space includes:
the manner in which the char-type memory space stores data is set to one of 1 integer bit 128 state, 2 integer bits 8 state, or 4 integer bits 2 state. The integer Bit herein refers to the number of bits of an 8-Bit stored integer in one char, rather than the number of bits of the stored integer.
In a preferred embodiment of the present invention, the storing, by using each char in the char-type memory space, a corresponding number of integers in the data to be operated on based on the manner of storing the data includes:
judging the data storage mode of the char type memory space;
in response to the char-type memory space storing data in the form of 1 integer Bit 128 states, the first Bit of each char is used to store information on whether an integer is stored in memory, and the second through eighth bits are used to store the state bits of the integer.
In a preferred embodiment of the present invention, the storing, by using each char in the char-type memory space, a corresponding number of integers in the data to be operated on based on the manner of storing the data includes:
judging the data storage mode of the char type memory space;
in response to the manner in which the char-type memory space stores data is 2 integer bits with 8 states, the first Bit of each char is used to store information on whether a first integer is stored in memory, the second through fourth bits store status bits of the first integer, the fifth Bit stores information on whether a second integer is stored in memory, and the sixth through eighth bits store status bits of the second integer.
In a preferred embodiment of the present invention, the storing, by using each char in the char-type memory space, a corresponding number of integers in the data to be operated on based on the manner of storing the data includes:
judging the data storage mode of the char type memory space;
in response to the fact that the manner of storing data in the char-type memory space is 4 integer bits and 2 states, the first Bit of each char is used to store information on whether a first integer is stored in the memory, the second Bit stores the state Bit of the first integer, the third Bit stores information on whether a second integer is stored in the memory, the fourth Bit stores the state Bit of the second integer, the fifth Bit stores information on whether a third integer is stored in the memory, the sixth Bit stores the state Bit of the third integer, the seventh Bit stores information on whether a fourth integer is stored in the memory, and the eighth Bit stores the state Bit of the fourth integer.
In a preferred embodiment of the present invention, the performing the operation on each data based on the manner of storing the data includes:
judging the data storage mode of the char type memory space;
judging which kind of operation is performed on the data in response to the fact that the mode of storing the data in the char type memory space is 2 integer bits and 8 states;
in response to performing an increment operation on data, dividing the data by the number of integer bits in the manner in which the data is stored to obtain a quotient and a remainder;
and finding the Bit pointed by the quotient and the remainder, and positioning the pointed Bit at 1. For example, to add 61 to the data, determine the position of 61 in the requested space, and say 61/2 ═ 30 or 1, the identity position of 61 is at the 4 th Bit of the 30 th position of the subscript of the char array. The run bit operation ORs the 4 th bit of pointer p [30] with 0x 1. Thus a value of 1 for this Bit indicates 61 that it has entered the array.
In a preferred embodiment of the present invention, the performing the operation on each data based on the manner of storing the data includes:
judging the data storage mode of the char type memory space;
judging which kind of operation is performed on the data in response to the fact that the mode of storing the data in the char type memory space is 2 integer bits and 8 states;
in response to performing a state modification operation on data, dividing the data by the number of integer bits in the manner in which the data is stored to obtain a quotient and a remainder;
and finding out the quotient and the remainder pointing to the Bit position of the representative state, and carrying out OR operation on the Bit position of the representative state after zero setting and the new state Bit. For example, modifying the state of integer 61, with 61/2 ═ 30 or 1, indicates that the identity position of 61 is at the 4 th Bit of the index 30 position of the char array, i.e., modifying the value of 5-7 bits of pointer p [30], and a 3-Bit value can store 8 states. If the actual condition is, for example, the modified state is 3, the state is set to zero and then the bit operation is performed by using 0x11 and changing three bits.
In a preferred embodiment of the present invention, the performing the operation on each data based on the manner of storing the data includes:
judging the data storage mode of the char type memory space;
judging which kind of operation is performed on the data in response to the fact that the mode of storing the data in the char type memory space is 2 integer bits and 8 states;
in response to performing a delete operation on data, dividing the data by the number of integer bits in the manner in which the data is stored to obtain a quotient and a remainder;
the Bit to which the quotient and remainder point and the Bit representing the state are zeroed. For example, if the integer 61 is deleted, 61/2 is equal to 30 or more than 1, and the identity position of 61 is indicated as the 4 th Bit of the index 30 position of char array, the state of the fourth index Bit of the pointer p [30] is set to zero, and then the state space is set to zero by performing a Bit operation with 0x0 and the subsequent 5-7 bits. Indicating that the number 61 has been deleted from the array.
In a preferred embodiment of the present invention, the performing the operation on each data based on the manner of storing the data includes:
judging the data storage mode of the char type memory space;
judging which kind of operation is performed on the data in response to the fact that the mode of storing the data in the char type memory space is 2 integer bits and 8 states;
in response to performing a query operation on data, dividing the data by the number of integer bits in the manner in which the data is stored to obtain a quotient and a remainder;
judging whether the Bit pointed by the quotient and the remainder is 0 or not;
in response to the pointed to Bit being 0, return that the looked up data is not present. And inquiring the state mode of a certain integer, firstly calculating the identity position subscript of the integer in an array memory, wherein num/2 is used for obtaining that the position residue 0 is the No. 0Bit of the char space, and the residue 1 is the No. 4 Bit of the char space. And then using the subscript and 0x0 to carry out Bit operation to detect whether the subscript is 0 or 1, if the subscript is 0, the result indicates that the integer does not exist in the memory array, and if the subscript is 1, carrying out Bit operation on a subsequent 3-Bit space to obtain the state of the integer.
By the technical scheme, the space for storing the big data in the memory can be saved, and the efficiency of data operation is greatly improved.
It should be noted that, as will be understood by those skilled in the art, all or part of the processes in the methods of the above embodiments may be implemented by instructing relevant hardware through a computer program, and the above programs may be stored in a computer-readable storage medium, and when executed, the programs may include the processes of the embodiments of the methods as described above. The storage medium may be a magnetic disk, an optical disk, a Read-Only Memory (ROM), a Random Access Memory (RAM), or the like. The embodiments of the computer program may achieve the same or similar effects as any of the above-described method embodiments corresponding thereto.
Furthermore, the method disclosed according to an embodiment of the present invention may also be implemented as a computer program executed by a CPU, and the computer program may be stored in a computer-readable storage medium. The computer program, when executed by the CPU, performs the above-described functions defined in the method disclosed in the embodiments of the present invention.
In view of the above object, a second aspect of the embodiments of the present invention proposes an apparatus for data operation in a large data volume environment, as shown in fig. 2, the apparatus 200 includes:
the device comprises a setting module, a storage module and a processing module, wherein the setting module is configured to set the data storage mode of a char type memory space;
the storage module is configured to store a corresponding number of integers in the data to be operated by using each char in the char type memory space based on the data storage mode;
an operation module configured to operate on each data based on the manner in which the data is stored.
The above-described embodiments, particularly any "preferred" embodiments, are possible examples of implementations, and are set forth only for a clear understanding of the principles of the invention. Many variations and modifications may be made to the above-described embodiments without departing from the spirit and principles of the technology described herein. All such modifications are intended to be included within the scope of this disclosure and protected by the following claims.
Claims (8)
1. A method for data operation in a large data volume environment, comprising the steps of:
setting a data storage mode of a char type memory space;
based on the data storage mode, each char in the char type memory space is used for storing a corresponding number of integers in the data to be operated;
each data is operated on based on the way the data is stored,
the method for setting the memory space of the char type memory to store data comprises the following steps: the manner of storing data in the char-type memory space is set to one of a 1 integer bit 128 state, a 2 integer bit 8 state, or a 4 integer bit 2 state,
storing a corresponding number of integers in the data to be operated using each char in the char type memory space based on the manner of storing data comprises:
judging the data storage mode of the char type memory space;
in response to the manner in which the char-type memory space stores data is 2 integer bits with 8 states, the first Bit of each char is used to store information on whether a first integer is stored in memory, the second through fourth bits store status bits of the first integer, the fifth Bit stores information on whether a second integer is stored in memory, and the sixth through eighth bits store status bits of the second integer.
2. The method as claimed in claim 1, wherein storing a corresponding number of integers in the data to be operated on using each char in the char-type memory space based on the manner of storing the data comprises:
judging the data storage mode of the char type memory space;
in response to the char-type memory space storing data in the form of 1 integer Bit 128 states, the first Bit of each char is used to store information on whether an integer is stored in memory, and the second through eighth bits are used to store the state bits of the integer.
3. The method as claimed in claim 1, wherein storing a corresponding number of integers in the data to be operated on using each char in the char-type memory space based on the manner of storing the data comprises:
judging the data storage mode of the char type memory space;
in response to the fact that the manner of storing data in the char-type memory space is 4 integer bits and 2 states, the first Bit of each char is used to store information on whether a first integer is stored in the memory, the second Bit stores the state Bit of the first integer, the third Bit stores information on whether a second integer is stored in the memory, the fourth Bit stores the state Bit of the second integer, the fifth Bit stores information on whether a third integer is stored in the memory, the sixth Bit stores the state Bit of the third integer, the seventh Bit stores information on whether a fourth integer is stored in the memory, and the eighth Bit stores the state Bit of the fourth integer.
4. The method of claim 1, wherein operating on each data based on the manner in which the data is stored comprises:
judging the data storage mode of the char type memory space;
judging which kind of operation is performed on the data in response to the fact that the mode of storing the data in the char type memory space is 2 integer bits and 8 states;
in response to performing an increment operation on data, dividing the data by the number of integer bits in the manner in which the data is stored to obtain a quotient and a remainder;
and finding the Bit pointed by the quotient and the remainder, and positioning the pointed Bit at 1.
5. The method of claim 1, wherein operating on each data based on the manner in which the data is stored comprises:
judging the data storage mode of the char type memory space;
judging which operation is performed on the data in response to the fact that the mode of storing the data in the char type memory space is 2 integer bits and 8 states;
in response to performing a state modification operation on data, dividing the data by the number of integer bits in the manner in which the data is stored to obtain a quotient and a remainder;
and finding out the quotient and the remainder pointing to the Bit position of the representative state, and carrying out OR operation on the Bit position of the representative state after zero setting and the new state Bit.
6. The method of claim 1, wherein operating on each data based on the manner in which the data is stored comprises:
judging the data storage mode of the char type memory space;
judging which kind of operation is performed on the data in response to the fact that the mode of storing the data in the char type memory space is 2 integer bits and 8 states;
in response to performing a delete operation on data, dividing the data by the number of integer bits in the manner in which the data is stored to obtain a quotient and a remainder;
the Bit to which the quotient and remainder point and the Bit representing the state are zeroed.
7. The method of claim 1, wherein operating on each data based on the manner in which the data is stored comprises:
judging the data storage mode of the char type memory space;
judging which kind of operation is performed on the data in response to the fact that the mode of storing the data in the char type memory space is 2 integer bits and 8 states;
in response to performing a query operation on data, dividing the data by the number of integer bits in the manner in which the data is stored to obtain a quotient and a remainder;
judging whether the Bit pointed by the quotient and the remainder is 0 or not;
in response to the pointed to Bit being 0, return that the looked up data is not present.
8. An apparatus for data manipulation in a large data volume environment, the apparatus comprising:
the setting module is configured to set a data storage mode of a char type memory space;
the storage module is configured to store a corresponding number of integers in the data to be operated by using each char in the char type memory space based on the data storage mode;
an operation module configured to operate on each data based on the manner in which the data is stored,
wherein the setting module is further configured to: the manner of storing data in the char-type memory space is set to one of a 1 integer bit 128 state, a 2 integer bit 8 state, or a 4 integer bit 2 state,
the storage module is further configured to:
judging the data storage mode of the char type memory space;
in response to the manner in which the char-type memory space stores data is 2 integer bits with 8 states, the first Bit of each char is used to store information on whether a first integer is stored in memory, the second through fourth bits store status bits of the first integer, the fifth Bit stores information on whether a second integer is stored in memory, and the sixth through eighth bits store status bits of the second integer.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202011363989.4A CN112506651B (en) | 2020-11-27 | 2020-11-27 | Method and equipment for data operation in large-data-volume environment |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202011363989.4A CN112506651B (en) | 2020-11-27 | 2020-11-27 | Method and equipment for data operation in large-data-volume environment |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN112506651A CN112506651A (en) | 2021-03-16 |
| CN112506651B true CN112506651B (en) | 2022-06-17 |
Family
ID=74967231
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202011363989.4A Active CN112506651B (en) | 2020-11-27 | 2020-11-27 | Method and equipment for data operation in large-data-volume environment |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN112506651B (en) |
Families Citing this family (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN115185981B (en) * | 2022-09-14 | 2022-11-25 | 吉奥时空信息技术股份有限公司 | Data duplication checking method and device considering super-large table |
Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101826054A (en) * | 2009-03-04 | 2010-09-08 | 安凯（广州）软件技术有限公司 | Memory management method of micromemory system |
| US20180101365A1 (en) * | 2016-10-07 | 2018-04-12 | International Business Machines Corporation | Reducing waste areas inside strings |
-
2020
- 2020-11-27 CN CN202011363989.4A patent/CN112506651B/en active Active
Patent Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101826054A (en) * | 2009-03-04 | 2010-09-08 | 安凯（广州）软件技术有限公司 | Memory management method of micromemory system |
| US20180101365A1 (en) * | 2016-10-07 | 2018-04-12 | International Business Machines Corporation | Reducing waste areas inside strings |
Also Published As
| Publication number | Publication date |
|---|---|
| CN112506651A (en) | 2021-03-16 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US5659737A (en) | Methods and apparatus for data compression that preserves order by using failure greater than and failure less than tokens | |
| US7587401B2 (en) | Methods and apparatus to compress datasets using proxies | |
| CN111258966A (en) | Data deduplication method, device, equipment and storage medium | |
| CN111949710B (en) | Data storage method, device, server and storage medium | |
| US6396420B1 (en) | Delta compression of floating point coordinate data | |
| CN111881012A (en) | Log recording method, device, equipment and storage medium | |
| CN109582231B (en) | Data storage method and device, electronic equipment and storage medium | |
| EP4154406A1 (en) | Compression/decompression using index correlating uncompressed/compressed content | |
| CN111274245B (en) | Method and device for optimizing data storage | |
| CN112506651B (en) | Method and equipment for data operation in large-data-volume environment | |
| CN109634983B (en) | Method, apparatus, device and medium for determining recall point of interest information | |
| CN112748866A (en) | Method and device for processing incremental index data | |
| CN113760894A (en) | Data calling method and device, electronic equipment and storage medium | |
| US20130204839A1 (en) | Validating Files Using a Sliding Window to Access and Correlate Records in an Arbitrarily Large Dataset | |
| CN115080684B (en) | Network disk document indexing method and device, network disk and storage medium | |
| US6510499B1 (en) | Method, apparatus, and article of manufacture for providing access to data stored in compressed files | |
| CN114238250A (en) | Program file compression method and device and electronic equipment | |
| CN112487111A (en) | Data table association method and device based on KV database | |
| CN114065727A (en) | Information duplication eliminating method, apparatus and computer readable medium | |
| CN113419792A (en) | Event processing method and device, terminal equipment and storage medium | |
| CN113805787A (en) | Data writing method, device, equipment and storage medium | |
| CN113535710B (en) | Searching method, searching device, terminal equipment and storage medium | |
| CN116664129B (en) | Block chain account book data iteration method, electronic equipment and readable storage medium | |
| CN109165220B (en) | Data matching calculation method | |
| CN113656406A (en) | Index page merging method, device, equipment and medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |