CN117331946A

CN117331946A - Method, device, equipment and computer readable storage medium for constructing mail index

Info

Publication number: CN117331946A
Application number: CN202311598878.5A
Authority: CN
Inventors: 杨良志; 白琳; 汪志新; 卢业波; 白小刚; 瞿勇金; 王向军
Original assignee: Richinfo Technology Co ltd
Current assignee: Richinfo Technology Co ltd
Priority date: 2023-11-28
Filing date: 2023-11-28
Publication date: 2024-01-02

Abstract

Embodiments of the present application provide a method, apparatus, device, and computer-readable storage medium for constructing mail indexes. The method comprises the steps of obtaining a target database of constructed mails; sorting the shaping data in the target database according to the order from small to large; performing logical OR operation on the first integer data after sequencing and preset characters to obtain target coding data; performing difference operation on the two adjacent integer data after sequencing to obtain a difference set of the two adjacent integer data; performing logical OR operation on the data in the difference set and the preset characters respectively to obtain a coded data set; and constructing a mail index based on the target coded data and the coded data set. In this way, the number of disk reads can be greatly reduced, and the retrieval speed and user experience can be improved.

Description

Method, device, equipment and computer readable storage medium for constructing mail index

Technical Field

Embodiments of the present application relate to the field of data processing, and in particular, to a method, an apparatus, a device, and a computer readable storage medium for constructing a mail index.

Background

In the inverted index technique, a btree is generally used as a basic data structure, and an entry is used as a key value of the btree, where each key may be allocated with a plurality of data blocks. In the above storage structure, there are a large number of integer numbers to be stored, such as mail id, total number of entries contained in one mail, number of times one entry appears in a certain mail, total number of times one entry appears in the inverted index database. Typically, there are no more than 42 hundred million mail items in the reverse index library of the mail system, and when this number is exceeded, a new reverse index library needs to be created.

In the current inverted index library, the shaping number is usually saved in a fixed-length byte number, such as 4 bytes of integer type, 2 bytes of short type and 8 bytes of long type; and storing ids of all documents affiliated to the entry in the data block affiliated to the entry in an integer of fixed length. Therefore, through the existing index storage method, more btree data blocks are occupied, more delay is caused, and user experience is seriously affected.

Disclosure of Invention

According to the embodiment of the application, the mail index construction scheme is provided, the block number occupied by each term can be reduced, the disk reading times are greatly reduced, and the retrieval speed and the user experience are improved.

In a first aspect of the present application, a method of constructing a mail index is provided. The method comprises the following steps:

acquiring a target database of the constructed mail;

sorting the shaping data in the target database according to the order from small to large;

performing logical OR operation on the first integer data after sequencing and preset characters to obtain target coding data;

performing difference operation on the two adjacent integer data after sequencing to obtain a difference set of the two adjacent integer data;

performing logical OR operation on the data in the difference set and the preset characters respectively to obtain a coded data set;

and constructing a mail index based on the target coded data and the coded data set.

Further, the target database includes:

the method comprises the steps of obtaining shaping data in a mail to be processed;

and performing inverted index on the shaping data to construct a target database.

Further, the method further comprises the following steps:

the shaped data in the set of differences is arranged in a bit-to-bit manner.

Further, the logic or operation of the arranged first integer data and the preset character is performed, and the obtaining the target coding data includes:

extracting the first n bits of the first shaping data after arrangement, and carrying out logical OR operation on the first n bits and a preset character to obtain target coding data; n is a positive integer.

In a second aspect of the present application, a mail index construction apparatus is provided. The device comprises:

the acquisition module is used for acquiring a target database of the constructed mail;

the arrangement module is used for ordering the shaping data in the target database according to the order from small to large;

the first operation module is used for carrying out logical OR operation on the ordered first integer data and preset characters to obtain target coding data;

the second operation module is used for carrying out difference operation on the two adjacent integer data after sequencing to obtain a difference set of the two adjacent integer data;

the third operation module is used for carrying out logical OR operation on the data in the difference set and the preset characters respectively to obtain a coded data set;

and the construction module is used for constructing a mail index based on the target coded data and the coded data set.

Further, the target database includes:

Further, the method further comprises the following steps:

the shaped data in the set of differences is arranged in a bit-to-bit manner.

extracting the first n bits of the first shaping data after arrangement, and carrying out logical OR operation on the first n bits and a preset character to obtain target coding data; n-bit positive integer.

In a third aspect of the present application, an electronic device is provided. The electronic device includes: a memory and a processor, the memory having stored thereon a computer program, the processor implementing the method as described above when executing the program.

In a fourth aspect of the present application, there is provided a computer readable storage medium having stored thereon a computer program which when executed by a processor implements a method as according to the first aspect of the present application.

According to the method for constructing the mail index, the target database of the constructed mail is obtained; based on the bit of the shaping data in the target database, arranging the shaping data in the target database; performing logical OR operation on the arranged first integer data and preset characters to obtain target coding data; performing difference operation on the two adjacent integer data after arrangement to obtain a difference set of the two adjacent integer data; performing logical OR operation on the data in the difference set and the preset characters respectively to obtain a coded data set; based on the target coded data and the coded data set, a mail index is constructed, the number of blocks occupied by each term is greatly reduced, the retrieval efficiency is improved, and the customer experience is improved.

It should be understood that the description in this summary is not intended to limit key or critical features of embodiments of the present application, nor is it intended to be used to limit the scope of the present application. Other features of the present application will become apparent from the description that follows.

Drawings

The above and other features, advantages and aspects of embodiments of the present application will become more apparent by reference to the following detailed description when taken in conjunction with the accompanying drawings. In the drawings, wherein like or similar reference numerals denote like or similar elements, in which:

FIG. 1 is a flow chart of a method of constructing a mail index according to an embodiment of the present application;

FIG. 2 is a schematic diagram of search terms according to an embodiment of the present application;

FIG. 3 is a block diagram of a mail index building apparatus according to an embodiment of the present application;

fig. 4 is a schematic structural diagram of a terminal device or a server suitable for implementing an embodiment of the present application.

Detailed Description

For the purposes of making the objects, technical solutions and advantages of the embodiments of the present disclosure more apparent, the technical solutions of the embodiments of the present disclosure will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present disclosure, and it is apparent that the described embodiments are some embodiments of the present disclosure, but not all embodiments. All other embodiments, which can be made by one of ordinary skill in the art based on the embodiments in this disclosure without inventive faculty, are intended to be within the scope of this disclosure.

In addition, the term "and/or" herein is merely an association relationship describing an association object, and means that three relationships may exist, for example, a and/or B may mean: a exists alone, A and B exist together, and B exists alone. In addition, the character "/" herein generally indicates that the front and rear associated objects are an "or" relationship.

Fig. 1 shows a flowchart of a method of constructing a mail index according to an embodiment of the present disclosure. The method comprises the following steps:

s110, acquiring a target database of the constructed mail.

Wherein the target database comprises a database which can be constructed by the following steps:

and obtaining the shaping data in the mail to be processed in a wired or wireless mode, performing inverted index on the shaping data, and constructing a target database (an inverted index database of a mail system). Wherein in the target database there is a large amount of shaping data. For example, the number of times a term appears in the entire target database, which may be in the order of hundred million, between 0-42 hundred million, such data requiring 4-byte unsigned plastic accommodation; mail ids may be distributed between 0-42 billions, such data requiring 4 byte unsigned plastic accommodation; the number of times a term appears in a mail may be between 0 and 65535, such data needs to be accommodated in a 2 byte unsigned plastic.

And S120, sorting the shaping data in the target database according to the order from small to large.

In some embodiments, the shaping data in the target database is sorted in order of magnitude to facilitate subsequent encoding of the shaping data. For example, the integer data ids are ordered, id1, id2 … ….

S130, performing logical OR operation on the first integer data after sequencing and the preset characters to obtain target coding data.

In practical applications, the term in the target database rarely appears more than 1 million times in the whole library, but in extreme cases it may reach 10 hundred million levels, in which case a large amount of space is wasted using a fixed 4 byte store. Therefore, in order to improve the retrieval efficiency and reduce the occupation of disk blocks, the arranged integer data is encoded in the following manner:

extracting the first n bits of the first shaping data after sequencing, and carrying out logical OR operation on the first n bits and a preset character to obtain target coding data; n is a positive integer, preferably 7; the preset character string can be set according to the actual application scene, for example, 0x80. That is, when performing the operation, 7 bits are taken from the lower order to the upper order each time and 0x80 is logically or-operated, 1 is complemented at the highest bit, and bits which are less than 7 bits at last are not complemented at the highest bit. When decoding, the byte with the highest order 0 must be the first byte after encoding a certain shaping number, and the byte between the byte and the next byte with the highest order 0 can be decoded into a shaping number, and the byte with the highest order not 0 must not be the first byte after encoding.

The following is a description of encoding/decoding by taking 257 numbers as an example:

257 bits:0001 0000 0001;

storing it as an int type requires 4 bytes, 0000 0000 0000 0000 0000 0001 0000 0001.

The coding process comprises the following steps:

when the method is used for encoding, the first 7 bits corresponding to 257, namely the numbers 0001 0000 0001 and 0x80, namely 0x1000 0000, are logically or operated to obtain 1000 0001, an 8 th bit is added and is 1, and the 8 th bit, namely the highest bit, is a flag bit, which indicates that the byte is not the highest byte of a number;

257, i.e., 0001 0000 0001, has less than 7 bits, and because of less than 7 bits, it is no longer necessary to perform a logical OR operation with 0x80, but it is only necessary to leave 0001 0 as it is, which constitutes an independent byte and the highest bit must be 0, and the highest bit 0 indicates that it must be the highest byte of a certain number.

The resulting 257, 0001 0000 0001 encoded bits sequence 0000 0010 1000 0001 stored on disk takes only 2 bytes.

The decoding process comprises the following steps:

process of decoding 0000 0010 1000 0001 on disk into 257:

encountering byte 0000 0010, the most significant bit of which is not 0 being the most significant byte of an integer; when encountering byte 1000 0001, the most significant bit is 1, which is certainly not the most significant byte of a certain integer, and the most significant bit is a flag bit introduced during encoding, so that the most significant bit must be ignored, and only the remaining 7 bits are certainly from the number before encoding, the lower 7 bits 000 0001 of the most significant bit are taken to be spliced with the preceding byte 0000 0010, and 0000 0010 000 0010 is 257.

Further, after the integer is coded, a large amount of data originally needing 4 bytes in the target database can be coded into less than 4 bytes for storage, and the purpose of compact and efficient storage and thus the searching efficiency is improved is achieved.

And S140, performing difference operation on the two adjacent integer data after sequencing to obtain a difference set of the two adjacent integer data.

In some embodiments, the difference operation is performed on two adjacent integer data after arrangement to obtain a set of differences between the two adjacent integer data.

Further, integer data in the set of differences is arranged in a bit-low to high manner.

S150, carrying out logical OR operation on the data in the difference set and the preset characters respectively to obtain a coded data set.

In some embodiments, referring to the encoding step in step S130, the data in the arranged difference set is respectively logically or-operated with the preset character to obtain an encoded data set.

In some embodiments, take the integer data as mail ID as an example: in the target database, the term is used as the key of the btree, the mail id of the term is stored in a data block managed by a term, the mail id must be kept unique in the whole inverted index database and must be defined as a 4-byte shaping number type, one term may be subordinate to hundreds of thousands of mails or even more mails, and at this time, the id of the hundreds of thousands of mails or even millions of mails need to be stored in a large amount, so that a large number of blocks need to be read during retrieval to cause delay increase. In this disclosure, mail ids within a term's data block are encoded. the blocks corresponding to term are the main data of the inverted index, so that the number of the blocks occupied by each term is greatly reduced, the number of disk reading times can be greatly reduced, and the retrieval speed can be greatly improved.

Specifically, the 1 st mail id to which term belongs is saved by variable length coding, then the difference of the 2 nd mail id to which term belongs minus the 1 st mail id is saved, further the difference of the 3 rd mail id to which term belongs minus the 2 nd mail id is saved, and so on … …, the difference is mostly smaller but may be larger in extreme cases, at this time, the coding operation is performed on the difference, and the step S130 is referred to. After the above difference is encoded by the method of the present disclosure, only one byte is needed, and if the encoding method of the present disclosure is not adopted, 4 bytes are needed.

As shown in fig. 2, term, i.e., an entry in the index, represents the word "south china" in the drawing. Each term corresponds to one or more data blocks, and the incremental mail id stored in the data blocks is the encoded shaping number.

Referring to fig. 2, id1=200, id2=268, id3=290 are taken as an example. At this time, 1 byte is occupied by id1, 2 bytes are occupied by id2, and 2 bytes are occupied by id3 according to the foregoing encoding method. The first shaping number 200 stored in the data block takes 2 bytes, then the stored id2-id1 = 268-200 = 68, 68 is the difference of the second id minus the first id, only one byte is needed for encoding the difference 68, and 2 bytes are needed if id2 = 268 is stored; similarly, the 3 rd position holds id 3-id2=290-268=22 with only one byte instead of two bytes.

S160, constructing a mail index based on the target coding data and the coding data set.

In some embodiments, a mail index is constructed based on the target encoded data and the set of encoded data.

Taking the search id as an example, when searching the mail id corresponding to a term, the corresponding data block is read to the memory, the 1 st mail id is read first, then each difference value is read according to the decoding algorithm in the step S130, the 2 nd mail id is obtained by adding the 1 st difference value to the first mail id, the 3 rd mail id … … is obtained by adding the 2 nd difference value to the 2 nd mail id, and so on, all the mail ids can be obtained.

The encoding method adopted by the present disclosure can be used for saving the mail id of term membership, and not saving the original mail id, but saving the difference value of two adjacent ids, thereby greatly reducing the disk block occupation.

According to the embodiment of the disclosure, the following technical effects are achieved:

the search speed and the user experience are improved while the occupation of the disk blocks is greatly reduced.

It should be noted that, for simplicity of description, the foregoing method embodiments are all expressed as a series of action combinations, but it should be understood by those skilled in the art that the present application is not limited by the order of actions described, as some steps may be performed in other order or simultaneously in accordance with the present application. Further, those skilled in the art will also appreciate that the embodiments described in the specification are all alternative embodiments, and that the acts and modules referred to are not necessarily required in the present application.

The foregoing is a description of embodiments of the method, and the following further describes embodiments of the device.

Fig. 3 shows a block diagram of a mail index constructing apparatus according to an embodiment of the present application, including, as shown in fig. 3:

an obtaining module 310, configured to obtain a target database of the constructed mail;

an arrangement module 320, configured to order the shaping data in the target database in order from small to large;

the first operation module 330 is configured to perform a logical or operation on the ordered first integer data and a preset character, so as to obtain target encoded data;

the second operation module 340 is configured to perform a difference operation on the two adjacent integer data after the sorting, so as to obtain a set of differences between the two adjacent integer data;

a third operation module 350, configured to perform logical or operation on the data in the difference set and the preset characters, respectively, to obtain a coded data set;

a construction module 360 is configured to construct a mail index based on the target encoded data and the set of encoded data.

It will be clear to those skilled in the art that, for convenience and brevity of description, specific working procedures of the described modules may refer to corresponding procedures in the foregoing method embodiments, which are not described herein again.

Fig. 4 shows a schematic diagram of a structure of a terminal device or a server suitable for implementing an embodiment of the present application.

As shown in fig. 4, the terminal device or the server includes a Central Processing Unit (CPU) 401, which can perform various appropriate actions and processes according to a program stored in a Read Only Memory (ROM) 402 or a program loaded from a storage section 408 into a Random Access Memory (RAM) 403. In the RAM403, various programs and data required for the operation of the terminal device or the server are also stored. The CPU 401, ROM402, and RAM403 are connected to each other by a bus 404. An input/output (I/O) interface 405 is also connected to bus 404.

The following components are connected to the I/O interface 405: an input section 406 including a keyboard, a mouse, and the like; an output portion 407 including a Cathode Ray Tube (CRT), a Liquid Crystal Display (LCD), and the like, and a speaker, and the like; a storage section 408 including a hard disk or the like; and a communication section 409 including a network interface card such as a LAN card, a modem, or the like. The communication section 409 performs communication processing via a network such as the internet. The drive 410 is also connected to the I/O interface 405 as needed. A removable medium 411 such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, or the like is installed on the drive 410 as needed, so that a computer program read therefrom is installed into the storage section 408 as needed.

In particular, the above method flow steps may be implemented as a computer software program according to embodiments of the present application. For example, embodiments of the present application include a computer program product comprising a computer program embodied on a machine-readable medium, the computer program comprising program code for performing the method shown in the flowcharts. In such an embodiment, the computer program may be downloaded and installed from a network via the communication portion 409 and/or installed from the removable medium 411. The above-described functions defined in the system of the present application are performed when the computer program is executed by a Central Processing Unit (CPU) 401.

It should be noted that the computer readable medium shown in the present application may be a computer readable signal medium or a computer readable storage medium, or any combination of the two. The computer readable storage medium can be, for example, but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or a combination of any of the foregoing. More specific examples of the computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer diskette, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing. In the context of this document, a computer readable storage medium may be any tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device. In the present application, however, a computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, with computer-readable program code embodied therein. Such a propagated data signal may take any of a variety of forms, including, but not limited to, electro-magnetic, optical, or any suitable combination of the foregoing. A computer readable signal medium may also be any computer readable medium that is not a computer readable storage medium and that can communicate, propagate, or transport a program for use by or in connection with an instruction execution system, apparatus, or device. Program code embodied on a computer readable medium may be transmitted using any appropriate medium, including but not limited to: wireless, wire, fiber optic cable, RF, etc., or any suitable combination of the foregoing.

The flowcharts and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods and computer program products according to various embodiments of the present application. In this regard, each block in the flowchart or block diagrams may represent a module, segment, or portion of code, which comprises one or more executable instructions for implementing the specified logical function(s). It should also be noted that, in some alternative implementations, the functions noted in the block may occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved. It will also be noted that each block of the block diagrams and/or flowchart illustration, and combinations of blocks in the block diagrams and/or flowchart illustration, can be implemented by special purpose hardware-based systems which perform the specified functions or acts, or combinations of special purpose hardware and computer instructions.

The units or modules described in the embodiments of the present application may be implemented by software, or may be implemented by hardware. The described units or modules may also be provided in a processor. Wherein the names of the units or modules do not in some cases constitute a limitation of the units or modules themselves.

As another aspect, the present application also provides a computer-readable storage medium that may be included in the electronic device described in the above embodiments; or may be present alone without being incorporated into the electronic device. The computer-readable storage medium stores one or more programs that when executed by one or more processors perform the methods described herein.

The foregoing description is only of the preferred embodiments of the present application and is presented as a description of the principles of the technology being utilized. It will be appreciated by persons skilled in the art that the scope of the application referred to in this application is not limited to the specific combinations of features described above, but it is intended to cover other embodiments in which any combination of features described above or their equivalents is possible without departing from the spirit of the application. Such as the above-mentioned features and the technical features having similar functions (but not limited to) applied for in this application are replaced with each other.

Claims

1. A method of constructing a mail index, comprising:

acquiring a target database of the constructed mail;

2. The method of claim 1, wherein the target database comprises:

3. The method as recited in claim 2, further comprising:

the shaped data in the set of differences is arranged in a bit-to-bit manner.

4. The method of claim 3, wherein performing a logical or operation on the aligned first integer data and the predetermined character to obtain the target encoded data comprises:

5. A mail index constructing apparatus, comprising:

6. The apparatus of claim 5, wherein the target database comprises:

7. The apparatus as recited in claim 6, further comprising:

the shaped data in the set of differences is arranged in a bit-to-bit manner.

8. The apparatus of claim 7, wherein the performing a logical or operation on the aligned first integer data and the predetermined character to obtain the target encoded data comprises:

9. An electronic device comprising a memory and a processor, the memory having stored thereon a computer program, characterized in that the processor, when executing the computer program, implements the method according to any of claims 1-4.

10. A computer readable storage medium, on which a computer program is stored, which computer program, when being executed by a processor, implements the method according to any of claims 1-4.