CN104572810A

CN104572810A - Method for carrying out operation processing on massive files by using bitmap

Info

Publication number: CN104572810A
Application number: CN201410652811.XA
Authority: CN
Inventors: 国睿
Original assignee: Guang Xigu Development In Science And Technology Co Ltd Of Shenzhen
Current assignee: Guang Xigu Development In Science And Technology Co Ltd Of Shenzhen
Priority date: 2014-11-17
Filing date: 2014-11-17
Publication date: 2015-04-29

Abstract

The invention discloses a method for carrying out operation processing on massive files by using a bitmap. The method comprises the following steps: inserting one datum into the bitmap; judging whether a certain number exists in the bitmap or not; printing all data sets in the bitmap; carrying out duplication removal; taking a union set or a crossed set or a difference set. The method can be used for carrying out operation functions of taking the crossed set and the union set, carrying out the duplication removal, and taking the difference set and the like on massive data sets, so that the data operation processing speed is extremely accelerated.

Description

Bitmap is utilized to carry out the method for mass file calculation process

Technical field

The present invention relates to the method utilizing bitmap to carry out mass file calculation process.

Background technology

In a lot of software systems, all there is similar following application scenarios:

From two lot number codes, get both and occur simultaneously.Such as QQ has 3,000 ten thousand members, and Elopichthys bambusa has 2,000 ten thousand, and needing to extract is the list that member is also Elopichthys bambusa user.

Carry out duplicate removal process to a collection of number, 10.1 short greatly activities have been carried out in such as certain electric business website, have 7,000 ten thousand person-times to be browsed by the login of QQ number and bought.After deriving these purchaser records, extract QQ list of numbers, each QQ only extracts once.

Union is got to two batch datas, the user such as playing A game has 3,000,000, the user playing B game has 4,500,000, a new game of the same type is had to go on the market, the player prepared A, B two plays throws in advertisement, but each user can only throw in once, the Players list extracting A, B two game is needed to get union, and duplicate removal.

Be extracted in a data set, but not in the part of another data set.Such as there are 3,000 ten thousand QQ member users, 2,000 ten thousand QQ Elopichthys bambusa users, need extraction to be QQ member, but be not the user list of Elopichthys bambusa.

This type of application can abstractly be, data set A has M element, and data set B has N number of element, needs to provide a kind of method, gets common factor, gets union, to certain data set duplicate removal, gets difference set to A and B;

In the application of general software systems, get the functions such as common factor duplicate removal for data set, general usage data storehouse, or the sort sequence that operating system provides, uniq duplicate removal, comm extracts and occurs simultaneously or union.

When data volume is less time, when such as the scale of A, B data set is within 10W, said method can solve problem.But when data set scale increases, said method computing is consuming time sharply to be increased.

Summary of the invention

The object of this invention is to provide a kind of method utilizing bitmap to carry out mass file calculation process.

To achieve these goals, technical scheme provided by the invention is: provide a kind of method utilizing bitmap to carry out mass file calculation process, comprises the steps: that insertion data put in place figure; Judge certain number whether in bitmap; Print all data sets in bitmap; Duplicate removal process; Get union or common factor or difference set.

Need 512M space storage bitmap data, all data need initialization to reset.

Insert data to put in place figure, needing the position of correspondence is 1.

Judge whether data centralization has existed a number, need to judge whether corresponding bit position is 1.

Compared with prior art, the present invention utilizes bitmap to carry out in the method for mass file calculation process, inserts data and to put in place figure; Judge certain number whether in bitmap; Print all data sets in bitmap; Duplicate removal process; Get union or common factor or difference set.The present invention is based on the bit arithmetic of C/C++ language, provide and the calculation functions such as common factor, union, duplicate removal, difference set are got to massive data sets.Very big raising data operation processing speed.

By following description also by reference to the accompanying drawings, the present invention will become more clear, and these accompanying drawings are for explaining embodiments of the invention.

Accompanying drawing explanation

Fig. 1 is that the present invention utilizes bitmap to carry out the schematic diagram of first embodiment of the method for mass file calculation process.

Fig. 2 is that the present invention utilizes bitmap to carry out the schematic diagram of second embodiment of the method for mass file calculation process.

Fig. 3 is that the present invention utilizes bitmap to carry out the schematic diagram of the 3rd embodiment of the method for mass file calculation process.

Fig. 4 is that the present invention utilizes bitmap to carry out the schematic diagram of the 4th embodiment of the method for mass file calculation process.

Fig. 5 is the schematic diagram that in computer system, a byte is made up of 8 bit positions.

Embodiment

With reference now to accompanying drawing, describe embodiments of the invention, element numbers similar in accompanying drawing represents similar element.

Whether core of the present invention utilizes bitmap to record certain data, have and occurred in data centralization.The time complexity that bitmap is searched is constant, greatly improves treatment effeciency.

As shown in Figure 5, in computer systems, which, a byte is made up of 8 bit positions, and each position can be 0 or 1 two states.A byte can represent at most whether 8 numbers exist, and such as, from 0-7bit, represents whether these 8 numerals of 0-7 have in data centralization respectively.If there is existence, the bit position 1 of correspondence.

32 conventional at present signless integers, span is 0 to 4294967295.If represented by a byte, need 4G internal memory.In 32-bit operating system, application program free memory is generally within 2G.Represent with bit position, then only need 4G/8=512M space, improve space efficiency.

Please refer to Fig. 1, in the embodiment shown in this figure, ask the common factor of two manifolds, is first read manifold A; Each data call insert in A is inserted bitmap; Read manifold B; Judge data in manifold B whether in bitmap; If so, then common factor is belonged to; If not, then common factor is not belonged to.

Please refer to Fig. 2, in the embodiment shown in this figure, ask the difference set of two manifolds, is first read manifold A; Each data call insert in A is inserted bitmap; Read manifold B; Judge data in manifold B whether in bitmap; If so, then difference set is not belonged to; If not, then difference set is belonged to.Therefore visible, ask the difference set of two data sets and the common factor asking two data sets, its flow process is substantially identical.

Please refer to Fig. 3, in the embodiment shown in this figure, ask the union of two data, is first read manifold A; Each data call insert in A is inserted bitmap; Read manifold B; Each data call insert in B is inserted bitmap; Print the data comprised in whole bitmap.

Please refer to Fig. 4, in the embodiment shown in this figure, for carrying out duplicate removal process to a manifold, is first read manifold A; Each data call insert in A is inserted bitmap; Print the data comprised in whole bitmap.

Test comparison result is as follows:

Remarks: test environment is CPU i51.8G, internal memory 2G, operating system is RedHat enterprise version Linux 6.032BitX86.

As can be seen from test comparison data, when data scale is less, without advantage, even may there is the decline of efficiency in the present invention.When data scale is about 10W bar, there is more than 60% improved efficiency; When data scale reach 1,000,000, ten million, even more than one hundred million after, the present invention can improve ten times of time efficiencies to decades of times.

Exemplify the example in the present invention's practical application below:

Such as, in light breath paddy registered user 1,000,000 members, micro-letter bean vermicelli 2,000,000, needs to extract:

Not only 1, be website members but also be the user profile of micro-letter bean vermicelli

2, be website members but be not the user profile of micro-letter bean vermicelli

3, be micro-letter bean vermicelli but be not the user profile of website members

Can use above method, thus carry out follow-up marketing work, the user of such as " be micro-letter bean vermicelli but be not website members " sends micro-letter encouragement and adds member.

Above disclosedly be only the preferred embodiments of the present invention, certainly can not limit the interest field of the present invention with this, therefore according to the equivalent variations that the present patent application the scope of the claims is done, still belong to the scope that the present invention is contained.

Claims

1. utilize bitmap to carry out a method for mass file calculation process, it is characterized in that, comprise the steps: that insertion data put in place figure; Judge certain number whether in bitmap; Print all data sets in bitmap; Duplicate removal process; Get union or common factor or difference set.

2. utilize bitmap to carry out the method for mass file calculation process as claimed in claim 1, it is characterized in that, need 512M space storage bitmap data, all data need initialization to reset.

3. utilize bitmap to carry out the method for mass file calculation process as claimed in claim 1, it is characterized in that, insert data and to put in place figure, needing the position of correspondence is 1.

4. utilize bitmap to carry out the method for mass file calculation process as claimed in claim 1, it is characterized in that, judge whether data centralization has existed a number, need to judge whether corresponding bit position is 1.