CN111737264A

CN111737264A - Information processing method and system

Info

Publication number: CN111737264A
Application number: CN202010695547.3A
Authority: CN
Inventors: 郭子聪; 王电轻; 黄魏楠
Original assignee: Zhizhe Sihai Beijing Technology Co ltd
Current assignee: Zhizhe Sihai Beijing Technology Co ltd
Priority date: 2020-07-20
Filing date: 2020-07-20
Publication date: 2020-10-02

Abstract

The present disclosure provides an information processing method for querying an element having a numeric tag with a numeric value having at least one digit, each digit having a bit value, the method comprising: creating, for each digit, a plurality of sets associated with a size of a bit value on the digit; creating a bitmap index from each of the plurality of sets; receiving a query request for the numeric tag; and executing the query request by using the bitmap index to generate a query result. The present disclosure also provides an information processing system.

Description

Information processing method and system

Technical Field

The present disclosure relates to the field of information technologies, and in particular, to an information processing method, system, electronic device, and computer readable medium.

Background

At present, with the popularization and high-speed development of internet technology, people figure systems can not be left in internet shopping, internet recruitment, internet social contact, internet question answering and the like, and the systems are widely applied to the business fields of commodity recommendation, news, service and the like and are important technical bases for high-degree data fine operation of internet companies.

In the crowd portrayal system, the most important data base is a user label, the accuracy of the user label highly influences the final usability of the crowd portrayal system, and partial information of a user, such as age, behavior data and the like, has a very fine-grained query requirement under a specific condition.

The crowd portrayal system analyzes and stores user IDs, user labels and label values, and can deeply depict and abstract all characteristics of a user by depending on label information of various dimensions. In order to extract a group of users with the same tag information to find the needs of the same group of users, the crowd portrayal system needs to have the capability of supporting instant query, which needs to balance the technical cost and the use experience.

Some parts of the value of the user tag are numerically characterized, such as user age, income, etc. Some of them, though not numerical values, can be converted into numerical values for preservation, such as sex, marital, etc. Therefore, the storage and query of the numerical labels are the most basic requirements, and especially in a crowd portrayal system, the related storage and query requirements of the precise numerical labels are more rigid requirements.

Therefore, an information processing method is urgently needed, and under the existing data architecture, the query result of the numerical label is returned to the terminal as soon as possible at the lowest technical cost, and finally the query requirement of the terminal can be responded in second level is met.

Disclosure of Invention

In view of this, an object of the embodiments of the present disclosure is to provide an information processing method and system, which generate a corresponding bitmap index through a specially designed set, so that the space occupied by data storage is reduced, the query speed is increased, and the time returned to a user is reduced to millisecond level.

According to a first aspect of the present disclosure, there is provided an information processing method for querying an element having a numeric tag whose numeric value has at least one digit, each digit having a bit value, the method comprising:

creating, for each digit, a plurality of sets associated with a size of a bit value on the digit;

creating a bitmap index from each of the plurality of sets;

receiving a query request for the numeric tag;

and executing the query request by using the bitmap index to generate a query result.

In a possible embodiment, the creating sets associated with the size of the bit value on the digit includes:

such that the element in each set satisfies the condition that the value of the value associated with the element at that digit is greater than or equal to a set value.

In one possible embodiment, the set value for each set is incremented in sequence among the plurality of sets.

In one possible embodiment, the numeric label is n-ary, having m digits, where n is an integer greater than or equal to 2 and m is an integer, and creating, for each digit, a plurality of sets associated with the size of the bit value on that digit, includes:

screening out elements with the bit value of the digit being greater than or equal to 0 to obtain a first set;

screening out elements with the bit value of the digit being greater than or equal to 1 to obtain a second set;

repeating the steps until the elements with the bit value of the digit larger than or equal to n-1 are screened out to obtain an nth set; and

repeating the above steps for each digit of the numerical label to obtain m × n sets.

In one possible embodiment, the method further comprises:

traversing each digit of the numeric label to generate the plurality of sets and a bitmap index for each set, respectively.

In one possible embodiment, the element is a user identifier.

In a possible embodiment, the executing the query request by using the bitmap index to generate a query result specifically includes:

and performing logical operation by using the bitmap index according to the relational expression to generate the query result.

According to a second aspect of the present disclosure, there is provided an information processing system for querying an element having a numeric tag with a numeric value having at least one digit, each digit having a bit value, comprising:

a set creation unit configured to create, for each digit, a plurality of sets associated with a bit value size on the digit;

a bitmap index unit configured to create a bitmap index from each of the plurality of sets;

a request receiving unit configured to receive a query request regarding the numeric tag;

and the request query unit is configured to execute the query request by using the bitmap index to generate a query result.

According to a third aspect of the present disclosure, there is provided an electronic device comprising: a memory, a processor and a computer program stored on the memory and executable on the processor, the processor implementing the method according to the second aspect when executing the program.

According to a fourth aspect of the present disclosure, there is provided a computer readable storage medium having stored thereon executable instructions which, when executed by a processor, cause the processor to perform the method of the second aspect.

According to the information processing system and method provided by the embodiment of the disclosure, the user ID corresponding to the matched tag value is screened out through the preset condition, the corresponding set is established, and the set is converted into the bitmap index by using the bitmap index generation method. And after receiving a query request, performing a logical operation by using the bitmap index through a logical expression to generate the query result. The query speed is high, and the storage efficiency is high.

Additional features and advantages of the disclosure will be set forth in the description which follows, and in part will be obvious from the description, or may be learned by the practice of the embodiments of the disclosure. The objectives and other advantages of the disclosure may be realized and attained by the structure particularly pointed out in the written description and drawings.

Drawings

In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings needed to be used in the embodiments will be briefly described below, and it is obvious that the drawings in the following description are only some embodiments of the present application, and it is obvious for those skilled in the art to obtain other drawings without creative efforts. The foregoing and other objects, features and advantages of the application will be apparent from the accompanying drawings. Like reference numerals refer to like parts throughout the drawings. The drawings are not intended to be to scale as practical, emphasis instead being placed upon illustrating the subject matter of the present application.

FIG. 1 shows a schematic diagram of typical Internet crowd imagery data, according to an embodiment of the disclosure.

FIG. 2 illustrates a schematic diagram of an exemplary numeric tag in accordance with an embodiment of the present disclosure.

FIG. 3 shows a schematic diagram of an exemplary information handling system in accordance with an embodiment of the present disclosure.

FIG. 4 illustrates a schematic diagram of an exemplary collection in accordance with an embodiment of the present disclosure.

FIG. 5 illustrates a schematic diagram of typical bit value splitting according to an embodiment of the present disclosure.

Fig. 6 shows a schematic diagram of a typical vector transposition according to an embodiment of the present disclosure.

FIG. 7 illustrates a diagram of an exemplary expanded set, in accordance with embodiments of the present disclosure.

FIG. 8 illustrates a diagram of an exemplary bitmap index, in accordance with an embodiment of the present disclosure.

Fig. 9 illustrates a schematic diagram of an exemplary information processing method according to an embodiment of the present disclosure.

Fig. 10 shows a schematic structural diagram of an electronic device for implementing an embodiment of the present disclosure.

Detailed Description

Hereinafter, embodiments of the present disclosure will be described with reference to the accompanying drawings. It should be understood that the description is illustrative only and is not intended to limit the scope of the present disclosure. Moreover, in the following description, descriptions of well-known structures and techniques are omitted so as to not unnecessarily obscure the concepts of the present disclosure.

The terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the disclosure. The words "a", "an" and "the" and the like as used herein are also intended to include the meanings of "a plurality" and "the" unless the context clearly dictates otherwise. Furthermore, the terms "comprises," "comprising," and the like, as used herein, specify the presence of stated features, steps, operations, and/or components, but do not preclude the presence or addition of one or more other features, steps, operations, or components.

All terms (including technical and scientific terms) used herein have the same meaning as commonly understood by one of ordinary skill in the art unless otherwise defined. It is noted that the terms used herein should be interpreted as having a meaning that is consistent with the context of this specification and should not be interpreted in an idealized or overly formal sense.

As people portrayal systems have evolved, numeric tags have become increasingly granular, such as those that have been classified generally before into 20-29, 30-39, and 40-49 years of age, which are coarser. It is now possible to store directly at the actual age, to a finer granularity. In the prior art, two solutions are mostly adopted when retrieving numerical tags: according to the scheme I, query is not carried out through a data architecture of a label system, and offline aggregation is carried out through hive, presto and other modes; and in the second scheme, each value is taken as a separate tag and is compatible into the data architecture of the members (tag- > members).

In the first scheme, the engines such as hive and presto are used for query, and the biggest problems are response time and calculation cost. Even if presto query is used, waiting for about 30 seconds is needed, user experience is damaged, high-cost solid state disk equipment is needed, operation cost is high, and when the user uses the device at high frequency, great query pressure is caused on other services for normally using data query.

In the second scheme, each numerical value is stored as a tag, which has two problems. One is that the amount is uncontrollable, if the maximum value of the value is in the hundred million level, then hundreds of millions of label enumeration values are needed, and for any storage engine, the uncontrollable amount of data is dangerous, and the risk of performance degradation and even breakdown exists at any time. And secondly, the interval query pressure is large, if the required data interval is large, the label with the order of magnitude is required to be calculated if the required data interval is across the order of magnitude, and both the io consumption and the calculation consumption are very large.

In view of this, the present disclosure divides values and optimizes storage and query performance by a new organization form of a data plane and using a set idea, so that the index calculation speed is fast, the data storage efficiency is high, and the query response speed is fast. Specifically, the present disclosure provides an information processing method for querying an element having a numeric tag with a numeric value having at least one digit, each digit having a bit value, the method comprising: creating, for each digit, a plurality of sets associated with a size of a bit value on the digit; creating a bitmap index from each of the plurality of sets; receiving a query request for the numeric tag; and executing the query request by using the bitmap index to generate a query result. The present disclosure is described in detail below with reference to the attached drawings.

FIG. 1 shows a schematic diagram of typical Internet crowd imagery data, according to an embodiment of the disclosure. Fig. 1 has a table form in which user identifiers, i.e., user IDs, mainly serve to distinguish different users whose data volumes are likely to be on the order of billions in practice.

The user labels are gender, age and marital, reflect certain attributes, states, behavior characteristics and the like of the user, and are the most concerned parts in the crowd images. As technology develops, the granularity of user tags becomes finer, and finer tags can retain more user features, reflect more detailed portions of the user's attributes, but have greater and greater challenges for the server.

Like the age label, is numerical, and the content of the label can be expressed in numerical values. The same is the income, the daily average login times, the number of the consumed commodities of each order, the number of the orders in a period of time and the like. Although the contents of the user tags are not numerical, the user tags are simple and generally efficient to search because they can enumerate different situations that may occur and the number of the different situations is small, for example, the gender is male and female, the marriage is not married, married and divorced, and more subdivided may be a spouse, a marriage and a reweld. There are also some numerical labels, whose contents are numerical values, but have strong personal attributes, which are not repeated, such as identification numbers, and the query usually does not involve operations. In addition, the numerical value may be an integer or a decimal, and may be stored in a database in a form of a shaping, a floating point, or the like.

Fig. 2 shows a schematic diagram of a numeric tag. Age and number of items purchased each year are typical numerical labels. Typical numerical tags have a one-to-many relationship of user identifiers (user IDs), such as a user aged 30 years, a user who purchases 300 products each year, a user who logs in an account 10 times per month, and the like, and each user has a very large scale, which brings a great challenge to a query, especially when the value requested to be queried is an interval or an intersection of several numerical tags, and the computation is very expensive. For convenience of explanation, the disclosure will be described in terms of age and number of items purchased per year.

FIG. 3 shows a schematic diagram of an exemplary information handling system 300 in accordance with an embodiment of the present disclosure. The system 300 includes a database 301, the database 301 including a set creation unit 302 and a bitmap index unit 303, in addition to the stored data itself. The system 300 further comprises a request receiving unit 304, a request querying unit 305. The set creation unit 302, the bitmap indexing unit 303 are typically part of the database 301, for example, as part of a database management system (DBMS). The request receiving unit 304 and the request inquiring unit 305 may be a part of the database 301, may be an external interface of the database, or may be an external processing unit.

The set creating unit 302 obtains tag information of each dimension of the user, where the tag information includes an identifier of the user, i.e., a user ID, a plurality of tags, and a numerical value corresponding to each tag. Each value consists of a digit and a value on the digit. The set creation unit 302 creates, for each digit, a plurality of sets associated with the size of the bit value on that digit. The bitmap index unit 303 can generate bitmap indexes from these sets.

The tag information of 10 users is stored in the user information list, and the tag information is the age of each of the users U1-U10 and the number of commodities purchased each year (see fig. 2).

A set of age tags is first established. For example, the elements of set 401 should satisfy: the bit value of the single digit is greater than or equal to 0, then the set 401 contains 10 elements, U1, U2, U3 through U10.

The set then built up in turn meets the following condition, where the elements of set 402 should satisfy: the bit value of the single digit is greater than or equal to 1, then the set 402 contains 4 elements, U2, U4, U7, U9. Since there is no case where the bit value of the ones digit of the age label value is 2 or more, the other sets 403 to 410 (i.e., the elements in the set satisfy any value of 3-9 or more of the bit values of the ones digit) are empty sets.

By analogy, multiple sets for other tags may be established for tens, hundreds, etc. digits.

There is also a method of creating a set, which uses a method of expanding a set. Such as when the number is 322 for each number of units purchased annually.

The Value 322 is first split, as shown in fig. 5, where Value _ list represents the array obtained after splitting, and is stored in the form of row vector.

The row vectors are then converted to column vectors, as shown in fig. 6, which illustrates a schematic diagram of a typical vector transpose in accordance with an embodiment of the present disclosure.

Finally, the columns are expanded, for example, the split table can group by the fields of digit and bit values, and the columns of user id can be put into a set. As shown in fig. 7, a schematic diagram of an exemplary extended set is shown, where digit 0 represents a unit bit, digit 1 represents a ten bit, and digit 2 represents a hundred bit, according to an embodiment of the disclosure. The elements represent elements in the set that satisfy the expansion condition.

More generally, the method of expansion may be:

sets associated with the magnitude of the bit value on the digit are created such that the element in each set satisfies the condition that the bit value on the digit of the value associated with the element is greater than or equal to a set value. Wherein the set value for each set is sequentially incremented in the plurality of sets.

For example, the numeric tag is n-ary, and has m digits, where n is an integer greater than or equal to 2, and m is an integer, creating multiple sets associated with the size of a bit value on each digit, and screening out elements whose bit value is greater than or equal to 0 of the digit to obtain a first set; screening out elements with the bit value of the digit being greater than or equal to 1 to obtain a second set; repeating the steps until the elements with the bit value of the digit larger than or equal to n-1 are screened out to obtain an nth set; and repeating the steps for each digit of the numerical label to obtain m × n sets.

The bitmap index unit 303 generates a bitmap index by using a bitmap index coding technique from the plurality of sets created by the set creating unit 302. The Bitmap Index coding technique may be any one of the prior art techniques, such as Bitmap Index, FastBit, and the like, and the disclosure is not limited thereto.

Age tag set 801, representing that the element of the set satisfies the condition that the value of the place value of the single digit is greater than or equal to 0, the bitmap index generated is 1111111111, as shown by vector 802.

Age tag set 803, representing that the element of the set satisfies the condition that the place value of the ones digit is greater than or equal to 1, generates a bitmap index of 0101001010, as shown by vector 804.

Age label set 805, representing that the element of the set satisfies the condition that the value of the bit of the decadic is greater than or equal to 3, generates a bitmap index of 0011000110, as shown by vector 806.

The request receiving unit 804 receives a query request, which may be a query for one or more values, for example, a query for users in 31 years old, or a query for one or more value intervals, for example, a query for users in 22-28 years old.

The request query unit 305 generates a logical expression according to the query condition.

For a query request of a 31-year-old user, the logical expression generated by the request query unit 305 is: intersection of the set 801, the set 803, and the set 805, that is, AND operation is performed on the

bitmap index vectors

802, 804, and 806.

The logical expression of the query value request may also be generated using the following steps:

step a: extracting the digit and bit value of the label and the numerical value to be inquired;

step b: calculating a difference set of a set of values to be queried and a set smaller than the values to be queried by 1 in a plurality of sets of lowest digits;

if there are other digits, go through the method in step b to get multiple difference sets of all digits.

And calculating the intersection of the difference sets of each digit to obtain a set meeting the query condition as a query result.

Another possible query request is that the query request is a value interval.

For example, a query request is an element that wants to fetch a tag having a bit value in one digit of 2 or more and 8 or less, and bit values in ten and hundred digits of 0:

assume that a set with a bitvalue greater than or equal to 2 is satisfied for an element, denoted as set S1.

Assume that a set with a bitvalue greater than or equal to 8 is satisfied for an element, denoted as set S2.

Assume that a set whose tens value is greater than or equal to 1 is satisfied for an element, denoted as set S3.

Assume that a set for an element satisfying a percentile value greater than or equal to 1 is denoted as set S4.

The logic of the operation of evaluating the query result is: subtracting set S2 from set S1 results in the element satisfying all sets having a place value greater than or equal to 2 and less than or equal to 8, subtracting set S3 from set S4 results in the element satisfying all sets having a place and tens value equal to 0 and a place value greater than or equal to 2 and less than or equal to 8. I.e., S5= S1-S2-S3-S4, this result S5 is the query result we want.

The logic expression of the query value interval request can also be generated by the following steps:

step c: extracting the digit and bit value of the label and the numerical value to be inquired;

step d: calculating a difference set of sets of two endpoints of the value to be queried in a plurality of sets of the lowest digit, namely calculating a difference set between a maximum value of the interval and a set of minimum values of the interval;

if there are other digits, go through the method in step d to get multiple difference sets of all digits.

Since the sets are in a one-to-one correspondence with the corresponding bitmap indexes, the logical expression for the bitmap indexes can also be generated by the same method. The present disclosure describes a method of logical operations on sets for the sake of easier understanding of the description, and is an exemplary description.

Compared with the prior art, the scheme disclosed by the invention can respond to the request at the second level and return the result. In the case that the data is 5-bit data, a storage scheme that one bitmap index is stored for each value is not needed, namely 100000 bitmap indexes are stored, only 5 (digits) × 10 (carry) =50 bitmap indexes are needed, and storage efficiency is greatly optimized. The bitmap index generation speed is high, spark is used for calculation, and the calculation can be completed in 10 minutes.

An information processing method for querying an element having a numeric tag with a numeric value having at least one digit, each digit having a bit value, the method comprising:

step 901 creates, for each digit, a plurality of sets associated with the size of the bit value on that digit; wherein the creating of the plurality of sets associated with the size of the bit value on the digit comprises: such that the element in each set satisfies the condition that the value of the value associated with the element at that digit is greater than or equal to a set value. Wherein the element may be a user identifier.

Step 902 creates a bitmap index from each of the plurality of sets; wherein the set value for each set is sequentially incremented in the plurality of sets.

The numeric label in

step

901 or 902 may be n-ary, having m digits, where n is an integer greater than or equal to 2 and m is an integer, said creating for each digit a plurality of sets associated with the size of the bit value on that digit, including in particular: screening out elements with the bit value of the digit being greater than or equal to 0 to obtain a first set; screening out elements with the bit value of the digit being greater than or equal to 1 to obtain a second set; repeating the steps until the elements with the bit value of the digit larger than or equal to n-1 are screened out to obtain an nth set; and repeating the steps for each digit of the numerical label to obtain m × n sets.

As in

steps

901 and 902, each digit of the numeric label may be traversed to generate the sets and the bitmap index for each set, respectively.

Step 903 receives a query request for the numeric label;

step 904 executes the query request using the bitmap index to generate a query result, wherein the query result may be generated by performing a logical operation using the bitmap index according to a relational expression to generate the query result.

Fig. 10 shows a schematic structural diagram of an electronic device for implementing an embodiment of the present disclosure. As shown in fig. 10, the electronic apparatus 1000 includes a Central Processing Unit (CPU) 1001 that can perform various appropriate actions and processes in accordance with a program stored in a Read Only Memory (ROM) 1002 or a program loaded from a storage section 1008 into a Random Access Memory (RAM) 1003. In the RAM1003, various programs and data necessary for the operation of the electronic apparatus 1000 are also stored. The CPU 1001, ROM1002, and RAM1003 are connected to each other via a bus 1004. An input/output (I/O) interface 1005 is also connected to bus 1004.

The following components are connected to the I/O interface 1005: an input section 1006 including a keyboard, a mouse, and the like; an output section 1007 including a display such as a Cathode Ray Tube (CRT), a Liquid Crystal Display (LCD), and the like, and a speaker; a storage portion 1008 including a hard disk and the like; and a communication section 1009 including a network interface card such as a LAN card, a modem, or the like. The communication section 1009 performs communication processing via a network such as the internet. The driver 1010 is also connected to the I/O interface 1005 as necessary. A removable medium 1011 such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, or the like is mounted on the drive 1010 as necessary, so that a computer program read out therefrom is mounted into the storage section 1008 as necessary.

In particular, according to an embodiment of the present disclosure, the processes described above with reference to the flowcharts may be implemented as computer software programs. For example, embodiments of the present disclosure include a computer program product comprising a computer-readable medium carrying instructions that, in such embodiments, may be downloaded and installed from a network via communications portion 1009 and/or installed from removable media 1011. The instructions, when executed by the Central Processing Unit (CPU) 1001, perform the various method steps described in this disclosure.

Although example embodiments have been described, it will be apparent to those skilled in the art that various changes and modifications may be made without departing from the spirit and scope of the disclosed concept. Accordingly, it should be understood that the above-described exemplary embodiments are not limiting, but illustrative.

Claims

1. An information processing method for querying an element having a numeric tag with a numeric value having at least one digit, each digit having a bit value, the method comprising:

creating a bitmap index from each of the plurality of sets;

receiving a query request for the numeric tag;

2. The information processing method of claim 1, wherein said creating sets associated with bit value sizes on the digit comprises:

3. The information processing method according to claim 2, wherein the set value with respect to each set is sequentially incremented in the plurality of sets.

4. The information processing method according to claim 1 or 2, wherein the numeric label is n-ary, having m digits, where n is an integer greater than or equal to 2 and m is an integer, and creating, for each digit, a plurality of sets associated with a size of a bit value on that digit, specifically comprises:

5. The information processing method of claim 1, the method further comprising:

6. The information processing method according to claim 1, wherein the element is a user identifier.

7. The information processing method according to claim 1, wherein the executing the query request using the bitmap index to generate a query result specifically comprises:

8. An information handling system for querying an element having a numeric tag with a numeric value having at least one digit, each digit having a bit value, comprising:

9. An electronic device, comprising:

one or more processors;

a memory for storing one or more programs,

wherein the one or more programs, when executed by the one or more processors, cause the one or more processors to perform the method of any of claims 1-7.

10. A computer readable medium having stored thereon executable instructions which, when executed by a processor, cause the processor to perform the method of any one of claims 1 to 7.